Introduction

Renewable energy in Australia

1.1 In 1992 the Australian Government ratified the United Nations Framework Convention on Climate Change (UNFCCC). The objective of the UNFCCC framework is the ‘stabilisation of greenhouse gas concentrations in the atmosphere at a level that would prevent dangerous anthropogenic interference with the climate system’.⁶ Under the framework, Australia set emissions reduction targets for the two Kyoto Protocol periods (2008–12 and 2013–20) and for 2030 under the Paris Agreement.⁷

1.2 Successive governments have taken a range of actions to support the development of Australia’s renewable energy industry⁸ and meet emissions reduction targets. These actions include the establishment of the:

• Renewable Energy Target in 2001 to encourage additional generation of electricity from sustainable and renewable resources and reduce greenhouse gas emissions in the electricity sector⁹;
• Clean Energy Finance Corporation (CEFC)¹⁰ and the Australian Renewable Energy Agency (ARENA) in 2012 to support investment in renewable technologies;
• Emissions Reduction Fund in November 2014, a voluntary scheme that provides incentives for new practices and technologies that reduce emissions¹¹; and
• Safeguard Mechanism in July 2016 to provide a framework for large emitters to measure, report and manage their greenhouse gas emissions.

Australian Renewable Energy Agency

1.3 ARENA is a corporate Commonwealth entity within the Industry, Science, Energy and Resources Portfolio.¹² It is established under the Australian Renewable Energy Agency Act 2011 (ARENA Act). The ARENA Act’s objectives are to:

• improve the competitiveness of renewable energy technologies; and
• increase the supply of renewable energy in Australia.¹³

1.4 The ARENA Act provides ARENA with over $2.2 billion of Australian Government funding from 2012–13 to 2021–22.¹⁴ ARENA’s functions under the Act include¹⁵;

• providing financial assistance¹⁶ for research, development, demonstration, commercialisation or deployment of renewable energy technologies, or the storage and sharing of information and knowledge about renewable energy technologies; and
• collecting, analysing, interpreting and disseminating information and knowledge relating to renewable energy technologies and projects.

1.5 Australian businesses, research institutions, Commonwealth entities, state or territory owned corporations or their subsidiaries and local government or councils are eligible for ARENA funding. ARENA classifies its funding responsibilities on an ‘innovation chain’ that is measured using technological and commercial readiness scales. ARENA’s role in the renewable energy sector with respect to other funding bodies and delivery partners is depicted in Figure 1.1.

Figure 1.1: ARENA’s role in the renewable energy ‘innovation’ chain

 

 

Note a: These figures are ARENA funding for approved applications up to January 2020. ARENA has also approved applications of $121 million for grants it classifies as studies. These studies cut across the innovation chain categories shown in this figure.

Source: ANAO analysis from ARENA documents.

Funding co-contributions

1.6 ARENA considers the amount of co-contribution from project proponents and partners as part of the merit assessment process. ARENA’s reported breakdown of the contributions from other organisations involved in ARENA funded projects is presented in Table 1.1.

Table 1.1: Committed co-contributions as at 31 December 2019

Source: ARENA, ARENA at a glance — 30 December 2019.

Types of renewable energy technology funded

1.7 ARENA’s reported breakdown of funding commmitments by renewable energy technology type or focus at 30 June 2019 is presented in Figure 1.2. Solar photovoltaic (PV) related projects have received the largest funding commitments.

Figure 1.2: ARENA funding commmitments by technology type 2012–19, at 30 June 2019

Source: ARENA, Annual Report 2018–19, Table 2.

Grant program and project funding

1.8 A breakdown of ARENA’s approved grant applications by year as at January 2020 is provided in Table 1.2.

1.9 ARENA utilises two grant funding processes: continually open application processes and targeted, competitive funding rounds. Applications to the continually open application processes are individually assessed against the merit criteria for the program. Targeted competitive rounds fund the highest merit applications up to the available funding allocated by ARENA to the round. The majority of funding has been awarded through continually open rounds (Figure 1.3).

Figure 1.3: Funding awarded through open and competitive processes at January 2020

Source: ANAO analysis of ARENA data.

Recent grant programs

1.10 Since 2016–17 ARENA has awarded the majority of grant funding through the Advancing Renewables Program (Table 1.3). This program provides grants through both open processes and competitive rounds. Funded activities include the development, demonstration, deployment or commercialisation of a renewable energy technology, including feasibility studies, pilot projects and demonstration projects.

Governance and staffing arrangements

1.11 The ARENA Board is the accountable authority under the Public Governance, Performance and Accountability Act 2013 (PGPA Act).¹⁷ It is responsible for ensuring that ARENA complies with the ARENA Act and setting ARENA’s strategies and policies.¹⁸

1.12 Under the ARENA Act, ARENA can only employ two staff members — the Chief Executive Officer (CEO) and the Chief Financial Officer (CFO).¹⁹ Other staff are required to be provided by the Department of Industry, Science, Energy and Resources (the department).²⁰ ARENA may also directly engage consultants to provide technical and specialist advisory services²¹, and use labour hire firms. ARENA’s total full time equivalent (FTE) at 30 June 2019 was 98.3 —the CEO, CFO, 28 staff provided by the Department of the Environment and Energy, and the remainder engaged as contractors and consultants.

1.13 Restrictions under the ARENA Act that limit ARENA’s ability to directly employ workers, combined with reducing departmental staff allocations over time, has led to what ARENA considers to be a heavy reliance on contractor (labour hire) and consultancy arrangements. ARENA has noted that the staffing model is of greater cost to the Commonwealth than could otherwise be the case.

Rationale for undertaking the audit

1.14 Substantial funding has been committed to renewable energy activities through ARENA. As ARENA approaches the end of its legislated funding, it is important to provide the Parliament with assurance over the effectiveness of ARENA’s grants management in improving the competitiveness of renewable energy technologies and increasing the supply of renewable energy in Australia.

Audit approach

Audit objective, criteria and scope

1.15 The objective of the audit was to assess the effectiveness of grant program management by ARENA. To form a conclusion against this objective, the ANAO developed the following high-level criteria:

• Does grant selection support the achievement of ARENA’s objectives?
• Are grant funding agreements managed effectively?
• Does evaluation of grant programs indicate that ARENA is achieving its objectives?

1.16 The audit scope focused on ARENA’s:

• strategic planning;
• design of grant program guidelines;
• application assessment and selection processes;
• grant funding agreement management; and
• evaluation and reporting of grants programs and organisational performance.

1.17 The scope of the audit did not include:

• the $200 million Clean Energy Innovation Fund which is financed and primarily managed by the CEFC, with subject matter expertise and contract management assistance provided by ARENA²²; and
• ARENA’s legislated functions that do not directly relate to grants management, such as providing advice to the Minister and liaising with state and territory governments and other authorities.

Audit methodology

1.18 Audit procedures included:

• examination of ARENA documentation;
• analysis of grant management system data;
• sample testing of application assessments and funding agreement management documentation;
• observation of grants assessment processes;
• field visits to projects funded by ARENA;
• consideration of public contributions to the ANAO; and
• interviews with ARENA staff.

1.19 The audit was conducted in accordance with ANAO Auditing Standards at a cost to the ANAO of approximately $489,600.

1.20 The team members for this audit were Joshua Francis, Kate Wilson, Isaac Gravolin, Taela Edwards, Sara Casey, Mark Rodrigues, and Michael White.

2.1 To effectively achieve its legislated objectives of increasing the supply and competiveness of renewable energy, ARENA’s grant selection needs to fund activities that support those objectives. The following criteria were used to assess if ARENA’s grant selection supports the achievement of its objectives:

• Are strategic plans clear and consistent with ARENA’s objectives?
• Are grant program guidelines appropriate?
• Are grant application assessments and selections consistent with grant guidelines and frameworks?

Are strategic plans clear and consistent with ARENA’s objectives?

2.2 In 2018–19 ARENA had four key strategic planning documents, three of which are required by legislation (Table 2.1).

Table 2.1: ARENA’s 2018–19 strategic planning documents

Note a: Department of Finance, Corporate plans for Commonwealth entities — Resource Management Guide No. 132, January 2017.

Note b: Paragraph 4; ARENA Act subsections 19(3) and 27(3).

Note c: Innovating Energy: ARENA’s Investment Plan 2017, p. 5. This investment plan remained in force until 2019.

Source: ANAO analysis.

The corporate plan

2.3 The Commonwealth performance framework sets out that the corporate plan is designed to be an entity’s primary planning document.²³ Corporate plans are developed at the beginning of the reporting cycle to set out an entity’s strategies for achieving its purposes and how success will be measured.²⁴

2.4 The Public Governance, Performance and Accountability Rule 2014 (PGPA Rule) section 16E sets out the minimum content requirements for all Commonwealth entities’ corporate plans. The ARENA Corporate Plan 2018/19 – 2021/22 includes all required elements as shown in Table 2.2. However, as discussed at paragraphs 2.9 to 2.25 below, issues have been identified with the presentation of the 2018–19 performance framework.

Table 2.2: ARENA 2018–19 corporate plan content requirements

Legend: ◆ Meets minimum requirements ▲ Information presented but improvements required

Source: ANAO analysis.

ARENA’s purpose statement

2.5 Under the PGPA Act, the purposes of a Commonwealth entity are to include the objectives, functions or role of the entity. Entities should also consider existing authoritative documents when defining their purposes, such as enabling legislation as passed by the parliament.²⁵

2.6 A concise and measurable statement of the purposes of an entity underpins a robust performance reporting framework.²⁶ A clear and concise purpose statement clearly states:

• what the entity aims to achieve;
• how the entity will do this; and
• to what effect, that is, the impact achieved when an entity successfully delivers its purposes.²⁷

2.7 The ARENA Corporate Plan 2018/19 – 2021/22 states that ARENA’s purpose is ‘to accelerate Australia’s shift to affordable and reliable renewable energy’.²⁸ This purpose statement is readily identifiable, concise and easily understood.

2.8 However, the purpose does not explicitly address all of the objectives and functions of the ARENA Act.²⁹ In addition, while the purpose statement clearly states what ARENA aims to achieve, the purpose would benefit from including how it will achieve this aim and the expected effect on Australians once this is achieved.³⁰

Do the activities adequately address the purpose?

2.9 Under the Commonwealth performance framework³¹, the description of purposes and activities in the corporate plan is the foundation for developing meaningful performance information. The corporate plan should focus on the high-level activities that will provide a reader some insight and understanding of how the purpose will be achieved. Each activity should be explicitly linked to a purpose and specify the contribution it makes to achieving an entity’s purposes.³²

2.10 ARENA’s 2018–19 performance framework is made up of its purpose, activities, outcomes, performance measures and impacts. A summary is presented in Table 2.3. ARENA has taken a similar approach to presenting the information in 2019–20.

2.11 The activities described in the 2018–19 corporate plan’s performance framework mostly support appropriate performance reporting. The activities could be more clearly aligned to ARENA’s purpose statement, as there is insufficient context for the reader to align the activities to the ‘affordable’ and ‘reliable’ elements of the purpose. This should be considered in the context of the suggested improvements to the purpose statement (see paragraph 2.8).

Are the performance measures appropriate?

2.12 In order to allow entities to assess whether performance measures are appropriate, the Department of Finance has provided guidance to entities on the characteristics of ‘good’ performance information — relevant, reliable and complete.³³ The ANAO has drawn on the Department of Finance’s guidance and other sources to develop audit criteria to assess the appropriateness of performance information.³⁴ These criteria can be found at Appendix 2 and have been used in each of the ANAO’s audits of entities’ performance statements to date.

2.13 This assessment examined the 23 performance measures defined in the ARENA Corporate Plan 2018–19 and ARENA’s two performance measures in the Environment and Energy Portfolio Budget Statements for the same year.³⁵

2.14 In addition to these performance measures, ARENA’s 2018–19 performance framework identified six impacts and four outcomes.³⁶ Under the Commonwealth performance framework, these impacts and outcomes meet the definition of performance measures.³⁷ In the annual report, ARENA also reported directly against two of six of these impacts and all four of the outcomes using case studies. Therefore, these six impact and four outcome statements are also included in this assessment.

Relevance

2.15 A relevant performance measure allows readers to assess an entity’s progress in fulfilling its purpose. Relevant performance measures:

• clearly indicate who benefits and how they benefit from the entity’s activities;
• are focused on a significant aspect of the entity’s purpose, via the activities, and makes the attribution of the entity’s activities clear; and
• are understandable, providing sufficient information in a clear and concise manner.³⁸

2.16 ARENA’s performance measures are either relevant or mostly relevant. The relevance assessment of ARENA’s 2018–19 performance measures is presented in Table 2.4. The most consistent strengths of the performance measures is that they are understandable and use concise, plain English. The relevance of these performance measures would be improved by clearly and consistently:

• indicating who will benefit and how they will benefit from the achievement of the performance measures;
• articulating how each individual measure will contribute to the achievement of ARENA’s purpose; and
• clarifying the attribution of results, including acknowledging when a result would be significantly impacted by factors beyond the direct effect of ARENA’s activities.

Table 2.4: Relevance assessment of ARENA’s 2018–19 performance measures

Note a: See Appendix 2 for a description of each characteristic.

Source: ANAO analysis.

Reliability

2.17 A reliable performance measure allows readers to form clear expectations and consistently assess an entity’s progress in fulfilling its purpose. Reliable performance measures are:

• measurable — they use and disclose information sources and methodologies (including a basis or baseline for measurement or assessment, for example a target or benchmark) that were fit-for-purpose; and
• free from bias — they allow for clear interpretation and an objective basis for assessment of the results.³⁹

2.18 The 2018–19 performance measures are only partly reliable, or not reliable because the reader cannot independently interpret the results reported — the reader must rely on ARENA to tell them what constitutes ‘good’ performance (Table 2.5). This was because most of the measures did not:

• specify a target or baseline; and/or
• did not provide a clear measurement method and information source.

Table 2.5: Reliability assessment of ARENA’s 2018–19 performance measures

Note a: See Appendix 2 for a description of each characteristic.

Source: ANAO analysis.

Completeness

2.19 A complete set of performance criteria allow for the overall assessment of an entity’s progress in fulfilling its purpose. A complete set of performance measures are:

• balanced — they provide a basis for assessment of both the efficiency and effectiveness of the entity in fulfilling its purpose, rely on a mixture of quantitative and qualitative measurement bases and assess a mixture of short, medium and long-term objectives; and
• collective — collectively address the entity’s purpose through the activities identified in the corporate plan.⁴⁰

2.20 In order to support the Parliament and the public in assessing how well an entity is performing, including how they are using the resources that have been entrusted to them, entities should identify a set of measures that demonstrate both efficiency and effectiveness.⁴¹ ARENA’s 2018–19 performance measures did not include any efficiency measures.

2.21 ARENA’s 2018–19 performance measures did present an appropriate balance of qualitative and quantitative information.

2.22 Accountability performance information (the highest level of the performance information hierarchy) is information that reports on outcomes and impacts to demonstrate whether public resources are delivering on government objectives.⁴² Performance reporting on accountability information is the focus of the PGPA Act.⁴³ While lower levels of performance information are also important, they should be used to support and advance accountability information.⁴⁴

2.23 Of ARENA’s 35 performance measures in 2018–19, 29 per cent are accountability information, 60 per cent are tactical/strategic information and 11 per cent are management information.

2.24 ARENA’s 2018–19 performance measures collectively address ARENA’s 2018–19 purpose to ‘accelerate Australia’s shift to affordable and reliable renewable energy.’ However, the ARENA 2018–19 Annual Report did not report against four of the six impacts and therefore did not present a complete picture of the extent to which ARENA is achieving its purpose.

2.25 Overall, improvements to ARENA’s performance measurement framework are required to provide the Parliament and the public meaningful performance information with a clear line of sight between planned and actual performance. In January 2020, following preliminary findings from the ANAO, ARENA commenced a process to improve its performance measurement framework.

Alignment of the PGPA Act and ARENA Act planning documents

2.28 To achieve a shared understanding within ARENA and among stakeholders of what ARENA will focus on in pursuit of its objectives, additional strategic planning documents should be aligned to the corporate plan.

2.29 The documents are mostly consistent in identifying ARENA’s purpose, impacts, focus areas and investment priorities. However, while the focus areas and investment priorities (key elements of the ARENA Act documents) are stated in the corporate plan, they are not integrated into or covered by the performance framework. The ‘guiding principles for ARENA investment’, one of the two key elements of the ARENA General Funding Strategy 2018/19 – 2020/21, are not aligned to any of the key elements of the other strategic planning documents.

2.30 While it is not expected that all documents will present precisely the same information, to be meaningful to the user it is expected that they would be built around and expand on the same key concepts. The strategic documents do not consistently present ARENA’s outcomes (Table 2.6) and activities (Table 2.7).

Table 2.6: Presentation of outcomes across the strategic planning documents

Legend: ◆ appropriately aligned ▲ partially aligned ■ not aligned – no comparable item

Note a: ‘Outcomes’ did not appear in any of the other strategic documents. The outcome statement in the PBS was not considered as part of this assessment, as under the Commonwealth performance framework it is intended to be a higher-level, purpose-style statement.

Source: ANAO analysis of ARENA Corporate Plan 2018/19 – 2020/21 and Innovating Energy: ARENA’s Investment Plan 2017.

Table 2.7: Presentation of activities across the strategic planning documents

Legend: ◆ appropriately aligned ▲ partially aligned ■ not aligned – no comparable item

Note a: ‘Activities’ did not appear in any of the other strategic documents.

Source: ANAO analysis of ARENA Corporate Plan 2018/19 – 2020/21 and Annual Business Plan 2018–19.

2.31 A shared understanding within ARENA and among stakeholders of what ARENA will focus on in pursuit of its purpose may be inhibited because the key planning documents are not fully aligned. There is merit in ARENA reviewing and updating these documents to improve the consistency in its messages.

Are grant guidelines appropriate and aligned with strategic plans?

Are grant program guidelines appropriate?

Commonwealth Grants Rules and Guidelines and internal grant framework and policies

2.32 The Commonwealth Grants Rules and Guidelines (CGRGs) are issued by the Finance Minister under the PGPA Act and establish the expectations for all non-corporate Commonwealth entities in relation to grants administration. While ARENA is a corporate Commonwealth entity, ARENA’s Internal Governance and Control Framework states:

Although ARENA is not subject to the Commonwealth Grant Rules and Guidelines (CGRGs) or Commonwealth Procurement Rules (CPRs), wherever possible ARENA has developed its frameworks, policies and plans utilising the core principles contained in these documents to ensure consistency with recognised Commonwealth best practice.⁴⁵

2.33 ARENA’s Financial Assistance Manual outlines the rules, procedures and good practices its officials are to apply when administering its grants.⁴⁶ The manual is structured around ARENA’s four main stages of program delivery. These are the:

• design stage — development and approval of grant program guidelines;
• selection stage — identification of funding recipients and the execution of funding agreements with successful applicants;
• delivery stage — management of funding agreements; and
• evaluation stage — assessment of the performance of its grant projects and programs.

2.34 The manual is supported by other policy and procedure documents, including grant administration specific documents such as the Standard Operating Procedure Assessment Process and the ARENA Conflict of Interest Policy. The financial assistance manual and other grants related policy and procedure documents refer ARENA officials to the grant program guidelines.

Consistency with Commonwealth Grants Rules and Guidelines

2.35 The majority of ARENA’s funding since 2016–17 has been committed though the Advancing Renewables Program (see paragraph 1.10 and Table 1.3). The Advancing Renewables Program guidelines were tested for consistency against key guideline related elements of the CGRGs.

2.36 Consistent with the CGRGs, the program’s guidelines and funding agreement template:

• contain straightforward and easily understood eligibility and merit criteria;
• specify the weighting of assessment criteria;
• specify the grant application approval processes;
• inform applicants of terms and conditions they will need to meet during the life of the grant such as performance and financial reporting;
• describe complaint handling and review mechanisms;
• are revised where significant changes have been made to a grant opportunity;
• outline conflict of interest policies; and
• provide for consideration of the risk associated with proposed projects.

Value for money — assessing additionality

2.37 A key principle of the CGRGs is that achieving value for money should be a prime consideration in all phases of grants administration.⁴⁷ The Advancing Renewables Program guidelines state that ‘once a proposal has been assessed to have satisfied the eligibility criteria, it will be assessed for overall value for money against the merit criteria.’⁴⁸ The five merit criteria are:

1. Contributes to the Program Outcomes;
2. Applicant capability and capacity;
3. Activity design, methodology, risk and compliance;
4. Financial viability and co-funding commitment; and
5. Knowledge sharing.⁴⁹

2.38 For each merit criterion, the guidelines provide examples of the types of information that applicants can provide to demonstrate merit. These information types map to most of the value for money principles set out in the CGRGs.

2.39 The CGRGs also note that guidelines should support an assessment of the additionality of applications: ‘a fundamental appraisal criterion is that a grant should add value by achieving something worthwhile that would not occur without the grant’.⁵⁰ The assessment of program additionality has been a feature in Commonwealth grant programs⁵¹ and in other programs delivered by corporate Commonwealth entities.⁵²

2.40 In the context of providing financial assistance to a growing industry, there is a risk that ARENA may support projects that might have proceeded without public funding. In November 2019, 11 per cent of ARENA funding recipients (that responded to market research commissioned by ARENA) indicated that their project would have been successful without ARENA’s involvement.⁵³

2.41 The ARENA General Funding Strategy 2018/19 – 2020/21 outlines the principles that:

• ARENA does not fund activities that are commercially viable without ARENA support; and
• ARENA seeks to provide the minimum financial assistance needed, and in the most appropriate form, to deliver the desired outcome, thus maximising value for public investment.⁵⁴

2.42 However, the Advancing Renewables Program guidelines do not describe how applications will be assessed for additionality, except in relation to the knowledge sharing activities related to projects. The guidelines do not reference the general funding strategy’s principles as a standard that will be used in the merit assessment. There are no other descriptions of how ARENA assesses if the grant will add value by achieving something worthwhile that would not occur without the grant.

Are grant programs aligned with strategic plans?

Grant program guidelines design and approval

2.45 As noted in Chapter 1, paragraph 1.9, ARENA has provided funding through two distinct grant processes: targeted competitive funding rounds and a continually open process (non-competitive funding). The majority of ARENA’s funding has been awarded through open processes (Figure 1.3).

2.46 The ARENA Corporate Plan 2018/19 – 2020/21 describes the expected benefits for each application process:

• targeted funding rounds are intended to create competitive tension, support price discovery and allow the pursuit of specific outcomes in focus areas; and
• continually open processes are intended to provide applicants with a funding mechanism that best meets their needs and the needs of the market.⁵⁵

2.47 ARENA’s continually open process has been run through the Advancing Renewables Program since 2016 and its predecessor the Emerging Renewables Program between 2012 and 2016.⁵⁶ Prior to 2016, ARENA also ran a number of other open grant processes under different sets of grants guidelines.

2.48 ARENA runs competitive processes when it identifies the circumstances to achieve value for money through a competitive process. This included the four Research and Development program competitive rounds and five competitive processes run through the Advancing Renewables Program guidelines. ARENA’s changing investment priorities over time, with reference to large scale solar projects, are described in Case study 1.

2.49 The CGRGs state that it is best practice for officials to identify the most appropriate application processes for achieving value for money. Both competitive and open grant processes are identified as options in the CGRGs. Documenting the rationale for not using competitive merit-based application processes when grant guidelines are approved is considered best practice under the CGRGs.⁶³ Consistent with the CGRGs and the ARENA Act, ARENA:

• documented the rationale for the Advancing Renewables Program as a continually open program by referencing the Emerging Renewables Program’s design in a paper to the ARENA Board; and
• gained the approval of the Minister for Industry and Science for the Advancing Renewables Program guidelines to use a continually open program in July 2015.⁶⁴

2.50 Given the high proportion of its funding that is awarded through open application processes, it is important that the ARENA Board continually assures itself that this approach is delivering value for money.

Investment priorities and alignment with grant program guidelines

2.51 ARENA considers that the updating of its investment priorities to favour new innovation over the areas it has funded in the past is an important element of ensuring value for money through its open grants processes. Since its establishment in 2012, ARENA has published five updates to its general funding strategy and four updates to its investment plan. These updates have seen projects that would have been considered of high merit under the investment priorities in 2012–13 no longer be a focus under the 2019 investment priorities.

2.52 The reliance on the investment priorities in ARENA’s strategic plans requires alignment between strategic planning documents and grant guidelines, which was demonstrated by:

• program and round objectives either mapping to or directly aligned with key elements of ARENA’s strategic planning documents; and
• the merit assessment criteria outlined in each of the grant program guidelines appropriately aligning with their respective program objectives.

Are grant application assessments and selections consistent with grant guidelines and frameworks?

2.53 ARENA’s grant program guidelines set out the application assessment processes, criteria and requirements for applicants. In assessing the applications, ARENA’s staff are guided by an internal assessment framework made up of the policies, procedures, templates, systems and controls.

Application assessment process

2.54 ARENA uses two different processes to assess all applications: a one-stage process that involves the assessment of a ‘full application’ and a two-stage process that involves an ‘expression of interest’ followed by a ‘full application’.⁶⁵ Applications to the Advancing Renewables Program seeking more than $500,000 are to be assessed through the two-stage process, unless this requirement is waived.⁶⁶ An overview of the assessment process is provided in Figure 2.1.

Figure 2.1: Overview of ARENA application assessment process

 

 

Note: Steps in green are the decision points covered by ANAO’s sample testing.

Source: ANAO analysis from ARENA documentation.

2.55 The ANAO identified four key decision points in the full application assessment process:

• eligibility assessment undertaken by ARENA staff against the grant guidelines eligibility criteria and approved by ARENA manger or executive;
• merit assessment undertaken by ARENA’s advisory panel against the grant guidelines merit criteria⁶⁷;
• funding decision by the board or the board’s delegate to enter into funding agreement negotiations, informed by the merit assessment report and supporting analysis from ARENA; and
• execution decision for ARENA to enter into the grant funding agreement by the board’s delegate. This is the final approval in the application assessment process.

2.56 As at 23 July 2019 ARENA had completed assessment of 992 potential projects⁶⁸, with 28 per cent successful in becoming funded projects with executed funding agreements (Table 2.8).

Table 2.8: Potential Project Assessment Outcomes

Source: ANAO analysis of ARENA Grants Management System data extracted 23 July 2019.

Application assessments

2.57 The ANAO reviewed decision records of a sample of applications against key requirements identified from the guidelines and ARENA’s internal assessment framework. These key requirements apply to the decisions at the eligibility assessment, merit assessment, funding and execution decision steps in the full application assessment process (green boxes in Figure 2.1).⁶⁹

2.58 ARENA’s assessment and selection processes were generally implemented in accordance with its grant guidelines and assessment framework, as:

• eligibility assessments were completed;
• the correct decision-maker⁷⁰ was engaged at each key stage;
• applications assessed as ‘high’ merit were funded;
• consideration of risk was documented; and
• consideration of value for money was generally appropriate, except for additionality.

Consideration of additionality

2.59 ARENA’s grant guidelines do not adequately describe how it will assess the additionality of project applications so that it does not fund projects that would proceed without ARENA support (see paragraphs 2.39–2.42). In assessing how additionality is presented to the ARENA decision-maker as part of the broader value for money merit assessments for the 44 sampled applications, it was identified that:

• all included evidence of consideration of additionality with respect to knowledge sharing;
• the consideration of additionality from a financial perspective was not as clearly presented to the ARENA decision-maker. Advice on the financial viability of a project without ARENA grant funding was documented in 11 out of 44 application assessments; and
• for two successful applications the merit assessment report stated the projects may be viable without ARENA funding, and it could not be identified in subsequent decision records how these additionality issues had been addressed.⁷¹

2.60 As well as outlining how it will assess additionality in its grant guidelines, there is merit in ARENA standardising how it documents its consideration of additionality. This will assist ARENA in demonstrating proper use of public funds under the PGPA Act.

Batched approvals of competitive funding rounds

2.61 ARENA batches the assessment and approval of the applications for competitive funding rounds. This differs from continuously open Advancing Renewables Program applications, where each individual application has a decision document that covers that specific application.

2.62 The ANAO’s sample of 44 applications included 18 applications that were part of batched approval processes. Decision documents across the competitive rounds also do not have a consistent approach to the presentation of information to the decision-maker.

2.63 The application-specific information provided to the decision-maker for batched approvals for competitive funding rounds was often limited to summarised merit assessment information and did not attach individual project merit assessment reports. The format of this summarised information varied across the different approval documents review. It generally included a merit ranked list of all applications with an available funding cut off, with summary project description and summary assessment comments. When compared to decisions on the open Advancing Renewables Program which attached individual project merit assessment reports, this approach provided less application-specific information to the decision-maker.

Conditions imposed on merit and funding decisions

2.64 In the merit assessment reports and the funding decision approvals, conditions or caveats can be placed on approvals to be addressed for the application to be awarded funding. In the sample of 44 applications, conditions or caveats could be identified in 15 merit assessment reports and 17 funding decisions. The review of these application assessments found:

• nine instances where a condition or caveat made in a merit assessment report could not be clearly identified as being addressed in the subsequent funding approval decision documentation; and
• four instances in execution decision documentation where there was insufficient information to determine if conditions imposed by the decision-maker at the previous funding decision approval had been met.

2.65 For transparency and clarity, records of decision should clearly and consistently document the recommendations and conditions from previous stages and how these have been addressed.

Workflow systems

2.66 ARENA’s ability to provide quality assurance over its application assessment process is limited by its lack of information system-based workflows, standardised templates and electronic record keeping. These limitations are also relevant to ARENA’s grants management processes. Further analysis is presented in paragraphs 3.17–3.19. A related recommendation, Recommendation no.3, is at paragraph 3.20.

3.1 Effective grant management is required to ensure that ARENA’s grants achieve its objectives with value for money. The following criteria were used to assess the effectiveness of ARENA’s grant management:

• Are appropriate arrangements in place to monitor the performance of funding agreements?
• Are governance arrangements for the oversight of grant funding agreements robust?
• Are risks and issues identified and dealt with appropriately?

Are appropriate arrangements in place to monitor the performance of funding agreements?

Grant funding agreements

3.2 The Commonwealth Grants Rules and Guidelines (CGRGs) and ARENA’s internal grants management framework⁷² recognise that well-structured and appropriately managed grant funding agreements underpin effective grant management. Well-structured grant funding agreements contain core project management elements such as milestones, outcomes and performance measures. Commonwealth entities that monitor project milestones and take appropriate actions to manage risks to grant outcomes are well-placed to deliver effective grants management.

Grant agreement elements

3.3 ARENA’s funding agreement template includes core project management elements such as risk management plans, milestones, deliverables, outcomes, reporting and acquittal arrangements. The funding agreement template does not contain clear guidance to support the development of performance information for inclusion in funding agreements.

3.4 The CGRGs note that ‘performance reporting requirements and other information sought from grantees are key inputs used by officials in evaluating whether outcomes have been achieved and whether a particular grant activity achieved value with relevant money’.⁷³ In a November 2018 paper to its audit committee, ARENA recognised the need to improve the definition and recording of project level performance information. The funding agreement template and guidance has not been updated to address this shortcoming. There is merit in ARENA improving its grant funding template guidance for developing performance information in funding agreements.

Milestones and payments

3.5 Grant funding agreements require clear milestone and payment structures that are proportionate to risk if they are to be effectively managed.⁷⁴ ARENA’s grant management policy notes that risk increases with the value of ARENA funding committed.

3.6 A sample of 44 funding agreements showed the structure of agreement milestones to be largely proportionate in terms of number of milestones, proportion of total funding to be paid at each milestone and spacing over the project duration.⁷⁵ Analysis of ARENA’s grant management system shows the average number of milestones recorded for each project funded increases as the value of ARENA’s committed funding increases (Figure 3.1).

Figure 3.1: Average number of milestone by value of committed ARENA funding

Source: ANAO analysis of ARENA Grants Management System data extracted 23 July 2019.

Milestones scheduled beyond ARENA’s current funding

3.7 While financial year 2021–22 is the final year that ARENA can request funding from the Consolidated Revenue Fund⁷⁶, the ARENA Act does not contain sunset provisions for ARENA’s functions post-2021–22. The 2019–20 portfolio budget statements for the Environment and Energy Portfolio forecast that ARENA will operate at a deficit in 2021–22 and 2022–23.⁷⁷ ARENA’s 2019–20 corporate plan outlines that in the absence of any funding changes, ARENA intends to:

• commit the majority of its remaining funds in 2019–20 so that these projects have at least two years to be completed and for funds to be paid out; and
• manage residual contract obligations beyond 30 June 2022.⁷⁸

3.8 Since 2015 the ARENA board has considered a series of papers on post-2022 arrangements for grant funding agreements, including legal advice on ARENA’s funding options and legislated obligations. Based on this advice and its considerations, ARENA estimates it will have access to $55.5 million at the end of 2021–22 and $9.3 million at the end of 2022–23 available to meet its grant and operating expenses when it can no longer draw down from the Consolidated Revenue Fund.

3.9 In June 2019 the ARENA board agreed to the structuring of milestone payments in 2022–23 and beyond. The board was presented with two key reasons for moving away from the previous policy of not scheduling grant payments beyond 30 June 2022. These were:

• the optimal structuring of new projects requires milestone payments to be scheduled post-30 June 2022; and
• that 35 to 40 per cent of existing projects experience schedule delay each year. This results in grant expenditure shifting to future financial years by way of contract variation.⁷⁹

3.10 At 30 June 2019 ARENA also had 31 grant funding agreements with recoupment mechanisms that will operate out to 2036. The identifiable maximum recoupable value across these mechanisms is $400.5 million.⁸⁰ These recoupment mechanisms will require ongoing monitoring to determine if the conditions of recoupment are met.

3.11 As ARENA identified in its 2019–20 corporate plan, post-2021–22 operations are now in its four-year planning timeframe. The board has considered an appropriate range of matters in relation to post-2022 operations. The ARENA Board’s most recent discussion of funding, operational and legal issues relating to post-2022 operations occurred at its December 2019 meeting.⁸¹ It tasked ARENA management five actions on these matters and made one workforce planning related decision.

Assurance of funding agreement compliance

3.12 Effective ongoing monitoring and management of grant funding agreements are elements of the CGRGs key principles.⁸² This provides assurance that grants are proceeding as planned and is an important part of achieving value for money. ARENA’s grant management framework identifies that monitoring of projects is essential to ensure compliance with the terms and conditions of the financial assistance, particularly through the delivery of milestones.

3.13 ARENA’s milestones and payments approval process involves a grant manager making a recommendation to the ARENA decision-maker to accept evidence that the milestone has been completed and release any associated payment. This approval is documented through a Milestone Approval Minute that the decision-maker approves in hard copy. The milestone payment approval processes are not system-based.⁸³

3.14 A review of the records of supporting evidence for 167 decisions to approve milestones or make funding payments across the 44 sampled funding agreements found that:

• supporting documents could be validated for 93 per cent of decisions reviewed; and
• the supporting evidence could not be validated for seven per cent of decisions reviewed.⁸⁴

3.15 ARENA’s milestone payment procedures require the record of decision to be stored on a project’s official hard copy file and supporting evidence on ARENA’s electronic shared drive. ARENA’s milestone payment procedures do not require records of decision and supporting evidence for approvals to be saved or linked in the grant management system. Storing or linking decision records and evidence against the relevant milestone in the grant management system would provide a clear and robust record of the decision and supporting information.

3.16 ARENA has largely demonstrated that milestones were substantiated with documentary evidence. However, ARENA’s use of hard copy supporting and decision documents outside of its grants management system increases risks that appropriate documentation to support decisions is not provided to the decision-maker. The current paper-based record keeping approach also limits ARENA’s ability to provide assurance that the appropriate evidence was considered by the decision-maker.

Grants management business process and information systems

3.17 At the time of the audit field work, two-thirds of the 62 steps in ARENA’s grants assessment and management processes are not performed directly on its grants management system (Table 3.2).⁸⁵ These non-system-based steps occur via email, spreadsheets, word-processor documents and hard copy files, with the grants management system updated at some steps in the processes. Manually-compiled, spreadsheet- and document-based processes have an increased risk of error and less effective controls compared to automated, system-based processes.⁸⁶

Table 3.1: Percentage of assessment and grant management process steps by system

Source: ANAO analysis of ARENA documents.

3.18 Previous reviews and internal audits have made recommendations that ARENA better aligns its grant administration processes and its information systems. ARENA has implemented a range of actions in response, however it has still not achieved fully digital processes with complete electronic information management.

3.19 Increasing the integration between ARENA’s grant administration processes and its information systems will reduce the risk of errors in the grants administration processes and allow ARENA to provide greater assurance over its management of grants.

Funding agreement variations

3.22 ARENA’s grant management framework recognises that there are circumstances where it is appropriate to vary a grant. Effective variations management ensures that varied grants continue to deliver value for money, achieve program outcomes and do not create probity issues.

Number and type of variations

3.23 ARENA has not established a benchmark or tolerance level for the number of variations it will accept per grant. During the course of audit fieldwork in June 2019, ARENA management reported to the Board that between 35 and 40 per cent of projects experience schedule delays each year, leading to project variations.

3.24 In July 2019, ARENA implemented a variation tracking module in its grant management system.⁸⁷ Prior to this, information on project variations could be determined only from the individual project files. This limited ARENA’s ability to inform itself on whether the rate of variations is increasing or decreasing over time.⁸⁸

3.25 Of the 44 projects examined as part of sample testing, 26 (59 per cent) had been varied at least once.⁸⁹ Changes to agreement timing by milestone was the most common rationale in the variations reviewed. The classification of the rationales presented to the ARENA decision-maker⁹⁰ for these variations is presented in Table 3.2.

Table 3.2: Rationales presented in the funding agreement variations review by ANAO

Note a: 81 variations were identified. Some variations have more than one rationale. Percentage is expressed as a proportion of the 81 variations.

Source: ANAO analysis of 81 variations across 44 ARENA funding agreements.

Variation policies, procedures and information systems

3.26 The variation policy and process documents use inconsistent terminology, particularly in the classification of variations (Table 3.3). ARENA had limited procedural guidance for variations prior to the introduction of standard operating procedures for variations in July 2019. While these procedures have improved its variation documentation, inconsistencies remain across the suite of procedural guidance and policy. These inconsistencies and the lack of system-based approvals limit the assurance that varied funding agreements still deliver value for money, achieve program outcomes and do not raise probity issues.⁹¹

Table 3.3: Comparison of ARENA’s variation documents

Source: ANAO analysis of ARENA documentation.

Variation consideration of value for money, program criteria and probity

3.27 ARENA’s policy and process documents identify that value for money, program criteria and probity are important matters when considering a variation. The ANAO assessed evidence of value for money, program criteria and probity considerations in the records of decision for 81 variations across the 44 sampled funding agreements.

3.28 Variation approval records containing clear evidence of the decision-maker directly considering all three elements of value for money, program criteria and probity of the variations could be identified for only four of these variations.⁹² ARENA advised ANAO that these matters are only included in information provided to a decision-maker if the matter is considered material in the context of the specific variation.

3.29 Case study 2 presents an illustrative example of the risks and issues that can arise when controls are not effective through variation policies and procedures. This case study was selected as it highlights all three issues of value for money, program criteria and probity of the variations. It is not representative of the process undertaken for all variations.

3.30 Effective management of variations is also required due to the rate of variations to milestone dates and ARENA’s timeline under its legislation. ARENA’s variations policy recognises the importance of value for money, merit against program guidelines and probity considerations when assessing variation requests. The variation policies and procedures do not provide effective controls to ensure these matters are appropriately considered for all variations.

Are governance arrangements for the oversight of grant funding agreements robust?

Funding agreement oversight arrangements

Grant funding governance and operational structures

3.33 ARENA has established an oversight structure that links the operational teams managing grant funding agreements, through operational and executive management, to the ARENA Board:

• at the operational level, the Project Delivery Branch is responsible for grant funding agreement management. The allocation of individual grant funding agreements between two teams is based on project risk⁹³;
• there are clear management roles, responsibilities and reporting lines between the Project Delivery Branch’s teams, the Executive Leadership Team and the ARENA Board; and
• this main line of reporting is supported by additional committees, including the Project Portfolio Management Committee and the Risk and Audit Committee.

3.34 Overall, ARENA has in place an appropriate range of business management and oversight committees with a suitable range of governance responsibilities.

Funding agreement oversight information

3.35 The Project Delivery Branch’s managers and the ARENA Board receive structured and regular reporting on grants management. Specific matters for noting or decision are brought to committees as agenda items with papers when required.

3.36 During the course of the audit, ARENA implemented business improvement projects that improved the information available to its staff, managers and governance committees with:

• a set of electronic reporting dashboards covering performance metrics such as milestones due and payment processing times. These dashboards can be accessed on ARENA’s intranet. Some of these metrics are incorporated into governance committee reporting, such as the CEO and Financial Reports to the ARENA Board;
• the transition of the Project Delivery Branch’s spreadsheet-based risk assessment process for grants to a system-based process undertaken in ARENA’s financial management information system⁹⁴; and
• a standardised reporting dashboard in the grant management system that replaced a manually compiled management report that captured operational information including milestones overdue; contracts with major issues, variations; and summary grant management system statistics.

3.37 The implementation of structured, electronic reporting tools will assist in addressing the data integrity and control risks that previously existed with ARENA’s reliance on manually compiled reports. It also increases the type and quality of information available to support funding agreement oversight.

Are risks and issues identified and dealt with appropriately?

Organisational risk management

3.38 Accountable authorities of all Commonwealth entities must establish and maintain appropriate systems of risk oversight, management and internal control.⁹⁵ ARENA’s board has established a risk framework that is consistent with the nine elements of the Commonwealth Risk Management Policy. This framework includes:

• policies and procedures such as a risk management policy, a governance and internal control framework;
• operational policies, procedures and systems with embedded risk management;
• a quarterly strategic risk report that includes the risk appetite statement and strategic risk management information; and
• project level risk information in the CEO report presented at each board meeting that includes summary information about extreme risk projects and in some months, statistics on the risk categories across the project portfolio (Table 3.4).

Table 3.4: Project level risk classifications presented to the ARENA Board

Source: ARENA Board Reporting September 2019.

Grant funding agreement risk management

3.39 ARENA’s risk assessment procedure for grant funding agreements was updated in February 2019. The procedure documents how ARENA assigns an overall risk classification to each project.

3.40 A project risk assessment is to be completed for each funding agreement on a quarterly basis by the relevant ARENA grant manager. Grant managers are also required to upgrade these risk ratings at any time a major matter comes to their attention. The risk assessments are used to classify each project in one of four categories from low to extreme risk, as shown in Table 3.4.

3.41 The risk classification assigned to each project is used to inform grant management decisions, including:

• which Project Delivery Branch team manages the project⁹⁶; and
• the approval of milestones and variations by ARENA staff under powers delegated from the CEO. These delegated approval limits are proportional to risk, with more senior staff able to approve higher value milestones and variations.

3.42 In February 2019, the project risk assessment was transitioned from a spreadsheet-based process to the financial management information system. Converting this process into a system-based process improves the controls over this risk information. Off-system document-based processes, such as variations and milestone approvals, still need to be manually updated with a project’s risk classification.

Presentation of risk and issue information to decision-makers

3.43 Between August and October 2017 ARENA updated its contract management policies, procedures and milestone approval templates, including the requirements for the presentation of risk and issue information to decision-makers. The updated milestone approval minute template included a section for presenting project risk rating information. These policies and procedures did not mandate the use of the milestone approval minute template.

3.44 The sample of 44 funding agreements included 97 milestone approvals made after October 2017 (Table 3.5). Of the 97 milestone approval decisions since October 2017, 80 contained appropriate communication of project risks and issues to the decision-maker.

3.45 Decision records that did not communicate risks and issues to the decision-maker included two milestone approval minutes that did not complete the project risk rating section and 15 approvals that did not use the milestone approval minute template. The presentation of risk and issues would be improved by mandating a standard approach for documenting and approving milestone approvals.

Table 3.5: Communication of project risks and issues to decision-maker

Source: ANAO analysis of ARENA documents.

Funding agreement sanctions

3.46 Grant funding agreements require appropriate sanctions that provide mechanisms for effective risk and issue management. The funding agreement template contains clauses for the: suspension of funding; reduction in funding; termination or reduction in scope with cause; termination for an insolvency event; and the repayment of ARENA funding.

3.47 At July 2019 eight per cent of ARENA’s funding agreements were classified as terminated (Table 3.6). The ARENA 2018–19 Annual Report identifies that ARENA has received $63.5 million through the return of grants since 2014–15.

Table 3.6: ARENA funding agreement status — 23 July 2019

Note a: Executed ARENA funding agreements where the project’s activities have not yet commenced.

Note b: This status only applies to projects under variation at a particular point in time. When variations are complete, that status is changed back to Active. It is not a cumulative count of all grants that have been varied.

Source: ANAO analysis of ARENA Grants Management System data extracted on 23 July 2019.

3.48 No actions were identified that were inconsistent with sanctions established by the guidelines and funding agreements in the review of 44 sampled funding agreements.

3.49 ARENA informed the ANAO that no funding recipients have lodged formal disputes under the dispute clauses of the funding agreements. ANAO’s review of ARENA’s complaints register did not identify any complaints that relate to ARENA undertaking actions that may be inconsistent with funding agreements.

Management of probity risks

3.50 The Joint Committee of Public Accounts and Audit has highlighted the importance of Commonwealth entities fully complying with conflict of interest requirements.⁹⁷ The CGRGs identify probity and accountability as key principles of grants administration.⁹⁸ This includes that accountable authorities should:

• ensure that entity policy and management processes for conflict of interest are published to support probity and transparency; and
• put in place appropriate mechanisms for identifying and managing potential conflicts of interest for grant opportunities.

3.51 ARENA has a range of probity measures in place:

• documented conflict of interest policies, supported by an ongoing training program;
• probity registers are maintained for the board, executive, staff and assessment panel;
• an annual declaration of interests process, in addition to ongoing disclosure requirements; and
• mechanisms recognising that ARENA deals with market sensitive information, such as an embargo register.

3.52 ARENA application processes and templates assist the board, executive, staff and the assessment panel to identify potential probity issues. Applicants are required to identify all entities involved in the application. This information is then provided to those involved in the assessment process to assist them in identifying any probity issues. Board and assessment panel papers clearly identify entities that are subject to deliberation.

3.53 The assessment advisory panel member independence declaration process is the only probity activity that is managed on an information system.⁹⁹ ANAO analysis of data extracted from the grant management system on 23 July 2019 found:

• this step was completed for all 3,201 panel member assessments recorded in system; and
• for the eight assessments where a panel member identified a conflict, the system did not allow those panel members to continue assessing that application.¹⁰⁰

3.54 The advisory panel member’s merit assessment of an application is only one of the 31 steps in ARENA’s application assessment process.¹⁰¹ All other probity activities and registers across the application assessment and grant management processes are managed in spreadsheets and documents. This creates the same integrity and control risks that are to be addressed by Recommendation no.3 (see paragraph 3.20).

3.55 Probity issues in relation to evaluation of service providers are further discussed in paragraphs 4.31–4.38.

4.1 Evaluating grant activities to ensure that objectives are achieved and relevant money is appropriately managed is a key part of grants management. The following criteria were used to assess if evaluation of grant programs indicate that ARENA is achieving its objectives:

• Are evaluation frameworks and reporting arrangements appropriate?
• Does evaluation demonstrate the impact of objectives?
• Are lessons from evaluations actioned appropriately?

Are evaluation frameworks and reporting arrangements appropriate?

ARENA’s policies and procedures for evaluation and performance reporting

4.2 ARENA’s financial assistance manual sets out general principles for the evaluation of its grant activities.¹⁰² The key evaluation and performance reporting policy and procedure documents consist of:

• the 2019 Evaluation Framework and supporting program and portfolio specific monitoring and evaluation plans; and
• the 2018–19 Performance Framework Guide (draft).¹⁰³

4.3 The 2019 Evaluation Framework was a major update and restructure to the 2015 version it replaced. This update made improvements that included establishing clear governance and reporting arrangements and a schedule of evaluations to be conducted from 2019 to ‘post 2022’.¹⁰⁴

4.4 For the versions of the evaluation framework and performance framework guide in place during 2018–19, there was a lack of clear references between documents. The performance framework guide did not clearly explain how ARENA’s evaluation and performance processes aligned with the Commonwealth performance framework. The poor linkage between ARENA’s evaluation framework and its performance measurement framework limited the potential contribution of evaluations to ARENA’s performance statements.¹⁰⁵

4.5 ARENA’s performance framework would be improved by clearly and publicly identifying in advance how the evaluations will be used as performance measurement methods and information sources in ARENA’s performance framework. This is related to Recommendation no.1 (see paragraph 2.26).

Reporting and assurance arrangements

4.6 Section 16F of the PGPA Rule requires an accountable authority to certify that their performance statements accurately present the entity’s performance for the reporting period. It is important that an accountable authority has confidence that the entity has adequate processes in place to provide sufficient assurance to support this certification.¹⁰⁶ Relevant processes may include:

• detailed audit committee review and endorsement;
• effective use of internal audit resources;
• documented procedures and processes supporting the performance reporting cycle;
• progressive reporting of results to internal stakeholders such as the accountable authority and audit committee throughout the year; and
• management certifications supported by suitable quality assurance of results.¹⁰⁷

Assurance by the ARENA Risk and Audit Committee and ARENA Management

4.7 Section 45 of the PGPA Act requires accountable authorities to appoint an audit committee and that audit committee must perform its functions in accordance with the PGPA Rules. ARENA’s Risk and Audit Committee’s (RAC) charter incorporates the audit committee functions specified in section 17 of the PGPA Rules.¹⁰⁸

4.8 The ARENA RAC Charter includes that the RAC is to review and provide written advice as assurance about the appropriateness of ARENA’s performance reporting. This includes reviewing the framework for developing and reporting key performance indicators and ARENA’s annual performance statements. The RAC’s 2018–19 and 2019–20 work plans include responsibilities and actions that aligned with Department of Finance guidance for audit committee’s roles and functions for reviewing the appropriateness of an organisation’s performance reporting.¹⁰⁹

4.9 The RAC considered performance framework matters at each of its 10 meetings between August 2017 and November 2019. The ARENA Board received 25 papers related to performance framework matters from the RAC or management between June 2017 and December 2019. ARENA also commissioned an internal audit that reviewed the 2017–18 and 2018–19 performance statements that was completed in November 2019.

4.10 Despite this range of activities by ARENA management and the RAC, shortcomings with ARENA’s 2018–19 Performance Measurement Framework outlined in Chapter 2, paragraphs 2.13–2.23, were not identified. The ARENA Board’s ability to identify these shortcomings when approving the 2018–19 Performance Framework in June 2018 was diminished by the lack of effective management and audit committee assurance.¹¹⁰

Reporting against the corporate plan

4.13 Regular reporting to key internal stakeholders of progress in achieving the measures and other commitments outlined in an entity’s corporate plan supports good decision-making and is an indicator of an entity’s commitment to positioning the corporate plan as the entity’s primary planning document.¹¹¹

4.14 ARENA reports on its corporate plan performance measures on an annual basis to the audit committee and the ARENA Board through presentation of the draft annual report. ARENA’s executive leadership team meetings review the performance measures in the annual work plan on a bi-monthly basis.¹¹² The annual work plan measures do not directly align to the performance measures outlined in ARENA’s corporate plan.

4.15 More frequent internal reporting using the corporate plan’s performance framework would increase ARENA’s ability to measure its performance and address issues as they arise.

Reporting of periodic evaluations to ARENA governance bodies

4.16 The outcomes and recommendations of evaluations should be considered by relevant governance bodies to ensure that appropriate action is taken and tracked. To determine whether there is appropriate governance oversight of evaluations, the ANAO examined the papers of ARENA’s key governance bodies.¹¹³ A consistent approach to the presentation of periodic evaluation to governance bodies could not be identified.

4.17 The 2015 Evaluation Framework did not establish clear governance roles. This issue has been addressed in the 2019 version. Application of the governance roles defined in the 2019 Evaluation Framework will facilitate the improved oversight of periodic evaluations by ARENA’s governing committees.

Public reporting of knowledge sharing

4.18 One of ARENA’s functions under its Act is ‘to collect, analyse, interpret and disseminate information and knowledge relating to renewable energy technologies and projects’.¹¹⁴ ARENA publishes a range of information on its website as part of this legislative knowledge sharing function. Data shows 364 reports, presentations and other knowledge sharing documents were available on the ARENA website as at 14 October 2019. These documents were downloaded 20,698 times from March 2018 to October 2019.¹¹⁵

4.19 ARENA’s knowledge sharing section of its website uses a thematic search approach that has been developed to allow users to find knowledge sharing products without needing to know the name of a specific project.¹¹⁶ However, not all knowledge sharing material is directly linked to the project that produced it.

4.20 A comparison of the knowledge sharing documents data and the number of ARENA projects with committed funding shows:

• 229 of these documents are directly linked to 139 grant projects in the data set. This was 29 per cent of all projects with committed funding as at 30 September 2019.
• 135 of these documents are not linked to an individual grant project in the data set. ARENA advised that not all documents can be linked back to individual grants as some documents are not produced by a specific project.
• 71 per cent of ARENA’s projects do not have a linked document in the knowledge sharing document data set supplied to ANAO.

4.21 As noted in Chapter 2, knowledge sharing is a key element of the additionality of ARENA’s grant funding.¹¹⁷ As well as the thematic search approach, presenting specific information on the knowledge sharing outputs that will be publicly released from each grant would provide stakeholders with a clearer understanding of what is to be produced. Stakeholder expectations for clear project level linkages to knowledge sharing outputs was highlighted in market research presented to ARENA in November 2019.

4.22 Links to knowledge sharing products on each project’s ARENA webpage, or a description of what these deliverables are if they cannot be published, would improve accountability and transparency.

Public reporting through corporate documents

4.23 The PGPA Act and ARENA Act require ARENA to publish a range of strategic and corporate documents and its grants guidelines. The required documents are available on ARENA’s website.¹¹⁸ ARENA’s annual report includes an index of compliance with legislated reporting requirements.¹¹⁹ These requirements include lists of financial assistance provided. ARENA includes its Annual Performance Statements in its annual report as required by the Commonwealth performance framework.

4.24 The 2018–19 annual report did not provide a complete or reliable assessment of ARENA’s progress against the performance measures and impacts set out in the 2018–19 corporate plan.¹²⁰ The 2018–19 performance statement included information against ARENA’s identified performance measures except for the ‘cost and/or revenue improvements for renewable energy technologies’.

4.25 ARENA also used three case studies to report against two of its six impact statements, but did not link any quantitative performance measures to them.¹²¹ Case studies should not be relied upon as a stand-alone measurement, unless: the scope is predetermined; activities considered are clearly stated; and measurement methods detailed in advance.¹²² ARENA’s 2018–19 corporate plan did not include this context for the use of its case studies. The quantitative performance measures presented in the 2018–19 annual report did not support a complete and reliable assessment of ARENA’s performance for the reasons presented in Chapter 2.¹²³

4.26 In the two years to December 2019 ARENA finalised 10 evaluations under its evaluation framework. Of these, eight were focused on ARENA’s outcomes. Two are publically available on the ARENA website.¹²⁴ In order to provide greater transparency on the effectiveness in delivering its legislated objectives, ARENA should publish all of its evaluations.

Does evaluation demonstrate the impact of objectives?

4.27 ARENA uses two main approaches to inform itself on whether it is meeting its objectives: its performance measurement framework, examined in Chapter 2, 2.11–2.24, and periodic evaluations.

4.28 In relation to evaluations ARENA’s Evaluation Framework requires that evaluations meet specified criteria to demonstrate the proper use of public resources, including for:

• Appropriateness — the extent to which the objective of the activity contributes to ARENA’s purpose, for example, what would have happened in the absence of the activity or would alternative approaches have been more appropriate?
• Effectiveness — the extent to which intended outcomes are being achieved, including any unintended positive or negative consequences; and
• Efficiency — the extent to which inputs are minimised to achieve the activity’s objectives and outcomes, including whether processes were efficient and represent value for money.

Evaluations undertaken

4.29 A breakdown of the type and focus of the 10 periodic evaluations since 2017 are presented in Table 4.1.

Table 4.1: Characteristics of ARENA evaluations finalised since 2017

Note a: Types of evaluation have been adapted from Auditor-General Report No. 47 of 2018–19 Evaluating Aboriginal and Torres Strait Islander Programs.

Note b: ARENA’s legislated objectives do not explicitly refer to economic impacts.

Note c: ARENA uses the Portfolio concept to evaluate a group of grants with a common characteristic. This can be a technology type such as solar, or another characteristic such as regional locations.

Source: ANAO analysis.

Evaluation findings

4.30 Of the eight outcome evaluations, three were considered to clearly conclude on the impact of the grant programs on ARENA’s objectives. Overviews of these conclusions are presented in Table 4.2. The other five outcome evaluation made positive findings about ARENA’s administration of its grant activities, but did not provide clear and direct conclusions about the impacts of grant activities on ARENA’s legislative objectives.

Table 4.2: Summary of evaluation conclusions that address ARENA’s impact

Source: Evaluation reports provided to ARENA.

Robustness of evaluations

4.31 All 10 evaluations were commissioned by ARENA and conducted by third parties. The ANAO reviewed the robustness of ARENA’s 10 evaluations based on the questions presented in Table 4.3.

Table 4.3: ANAO robustness assessment of evaluations

Legend: ◆ robust ▲ minor issues ■ major issues

Source: ANAO analysis.

4.32 The strengths of the evaluations are that they all clearly identify the type of evaluation and the methodologies that will be used.

4.33 The issues identified across the evaluations included:

• most of the outcome evaluations did not present clear, impact focused conclusions against the objectives, limiting the clarity and value of the evaluation to the reader; and
• most evaluations either did not clearly outline the evidence base for the evaluations, or did not balance internal or secondary information sources, such as interviews or existing ARENA records, with other external evidence sources or benchmarking.

4.34 Three of the eight parties commissioned to conduct evaluations are also recipients of ARENA grant funding. The final reports for these evaluations did not contain disclosure of these potential conflicts and any actions taken to manage them. ARENA defines a conflict of interest as:

a situation where there is, or appears to be, a divergence between the interests of a person and their professional responsibilities to ARENA, such that an independent observer might reasonably question whether the actions or decisions of that person are influenced by their interests.¹²⁵

4.35 ARENA’s Conflict of Interest Policy requires all potential service providers involved in procurement to declare potential conflict of interest. This policy then requires the relevant ARENA delegate to assess the materiality of declared conflicts and identify actions to manage material conflicts.

4.36 Table 4.4 describes the three cases where ARENA grant funding recipients were also evaluation services providers to ARENA. ARENA was unable to provide evidence that the materiality of these conflicts were considered by the delegate, or that actions to manage material conflicts were documented.

Table 4.4: ARENA consideration of conflict of interests by evaluation providers.

Note a: These actions are required under ARENA Conflict of Interest Policy.

Note b: Grants from this organisation were transferred to ARENA in 2012 and were part the scope of this evaluation.

Source: ANAO analysis of ARENA records.

4.37 Two members of the evaluator team for Case 1 in Table 4.4 sit on ARENA’s advisory panel and were members of the predecessor organisation’s assessment research advisory panel. These panels were involved in the merit assessment process for grants covered by the evaluation. A declaration of this conflict could not be identified in the material provided to ARENA by the provider.

4.38 There is no evidence that ARENA had identified this conflict, considered its materiality or documented actions relating to this procurement. The evaluation provider in Case 1 of Table 4.4 is also the organisation that received ARENA funding through processes discussed in Case Study 2, presented in Chapter 2.

ARENA’s impact on objectives

4.41 As noted in Chapter 1, paragraphs 1.3–1.4, under the ARENA Act, ARENA’s:

• objectives are to improve the competitiveness of renewable energy technologies and increase the supply of renewable energy in Australia; and
• functions include providing financial assistance for, and disseminating information and knowledge relating to, renewable energy technologies and projects.

Trends in Australian Renewable Energy since 2012

4.42 The Australian Energy Regulator has reported the generation capacity of renewable energy in the National Electricity Market has increased since 2012–13. This is primarily through the installation of wind and solar (Figure 4.1).

Figure 4.1: National Electricity Market Generation capacity change 2012–13 to 2018–19

Source: Australian Energy Regulator, State of the energy market 2018 (2019 data update), Figure 2.20.

4.43 The Australian Energy Regulator’s annual report on residential electricity price trends shows that residential electricity prices are expected to decrease between 2018–19 and 2021–22 for most Australian states and territories. The report noted:

• New renewable generation means that prices of electricity are lower during peak renewable production periods, which may lead to lower wholesale electricity purchase costs depending on the hedging profiles of retailers.
• Additional supply [provided by renewable energy generation] places downward pressure on wholesale pricing.¹²⁶

4.44 The Department of the Environment and Energy’s emissions projections for 2019 note that since 2016 falling emissions from electricity generation have been driven by large amounts of renewable generation entering the market.¹²⁷

Attribution of ARENA’s activities to these trends

4.45 These trends in Australian renewable energy are consistent with ARENA’s legislative purpose and objectives. As noted in Chapter 1, paragraph 1.2, there are a range of Australian government activities designed to support the development of renewable energy technologies in Australia. These measures and other policy, economic and social factors, may be accelerating Australia’s shift to affordable and reliable renewable energy.

4.46 It is important that ARENA’s evaluations identify the contribution of ARENA’s grant activities to the current trends in the Australian renewable energy to assess its impact on its objectives. In aggregate, ARENA’s evaluations do not demonstrate the extent to which ARENA’s outcomes are impacting on its purpose and legislated objectives as required by its evaluation framework.

4.47 ARENA’s ability to attribute the impact of its grants activities to the trends in Australian renewable energy would be strengthened by:

• addressing the shortcomings with the performance measurement framework as outlined in Recommendation no.1;
• clearly assessing additionality in the grant selection process as outlined in Recommendation no.2; and
• ensuring periodic evaluations make clear and robust conclusions as to whether the program or portfolio is effectively achieving ARENA’s objectives.

Are lessons from evaluations actioned appropriately?

4.48 To determine if lessons from evaluations were actioned appropriately, the ANAO reviewed:

• board papers supporting the approval of strategic planning documents and grant funding guidelines;
• the implementation arrangements for the Future Needs Review process evaluation; and
• ARENA’s system for tracking the implementation of evaluation recommendations.

Evaluation input to board approvals of strategic document and guidelines

4.49 The board’s approval of strategic planning documents and grant funding guidelines are supported by papers from management describing the development of these documents. The ANAO’s review of 15 of these papers that were presented to the board between May 2017 and December 2019 found limited references to periodic evaluations in these papers. These board papers were informed by lessons from a range of other information sources including stakeholder consultation and workshops; outputs from ARENA funded grants; government priorities; ARENA’s investment priorities; and legal advice.

Future Needs Review

4.50 The process evaluation identified in Table 4.1 was the Future Needs Review completed in April 2017. The aim of the review was to provide the board with recommendations to improve ARENA’s effectiveness in pursuing priorities and objectives. It covered organisational structure, processes and procedures, stakeholder experience, skills and capabilities and response to previous reviews. The review made 29 recommendations, all of which were accepted by ARENA.

4.51 To monitor the status of the implementation of recommendations, the ARENA board established a Review Implementation Oversight Committee in June 2017. The oversight committee provided regular updates to the board until November 2018 when the Board considered the recommendations to be implemented aside from two business as usual matters. Some information technology projects to address the remaining actions continued with direct reporting to the board. On advice from ARENA management, the board closed all outstanding recommendations from the review in April 2019.

4.52 ARENA reported to the Board that key outcomes arising from the review have included:

• stronger business planning processes, including earlier preparation of the corporate plan;
• greater policy alignment with the department and other energy agencies;
• increase in CEO delegations;
• benchmarking of administrative costs, with costs kept within the established thresholds; and
• approval of a Business Case for Enhanced Technology (ICT investment) which formed the basis for systems improvement including grants management system configuration and new financial management information system (FMIS).

4.53 Initiatives arising from the review were examined by the ANAO with respect to its adoption of a portfolio management to grant management, grant management system improvements and increased focus on milestone processing and payments. As noted in the report, these initiatives will support ARENA’s management of its grant programs.¹²⁸

Evaluation recommendation implementation and tracking

4.54 ARENA monitors the implementation of agreed evaluation recommendations in a spreadsheet evaluation tracker. The framework states the tracker is to document:

• ARENA’s response to each recommendation;
• ARENA’s agreed action in response to each recommendation;
• a status update on the implementations of actions every six months; and
• the Executive Leadership Team member that is responsible for implementing that action.

4.55 The framework does not outline procedures for closing recommendations, such as the committee or officer responsible for reviewing and closing recommendations and the level of documentation required to substantiate closure. The inclusion of recommendation closure procedures in the framework would strengthen the assurance provided through the tracker.

4.56 At February 2020, 75 evaluation recommendations were recorded on the ARENA evaluation tracker. The information recorded on the spreadsheet indicated that ARENA had taken appropriate action in relation to 67 of these recommendations. For six recommendations, the information recorded was not clear enough to determine whether appropriate action had been taken. Actions for two recommendations relating to the grants management system were marked as complete, stating the issues raised would be considered in another review, but no information on the outcomes of the further review was captured on the tracker. Based on the analysis presented in Chapters 2 and 3¹²⁹, these two evaluation recommendations have not been appropriately actioned.

Appendix 1 Australian Renewable Energy Agency response

 

 

 

 

 

 

Appendix 2 Criteria for the assessment of the appropriateness of performance information

Table A.1: Criteria for the assessment of the appropriateness of performance information