1. The Public Governance Performance and Accountability Act 2013 (PGPA Act) places explicit obligations on accountable authorities for the quality and reliability of performance information and requires Australian Government entities to report their performance to Parliament in a way that meaningfully reflects their organisation’s purpose and achievements. This is an important aspect of the Australian Government’s public accountability system, enabling the Parliament and the public to assess whether Australian Government entities deliver value for money and achieve the outcomes for which they are funded.

2. Recognising that public value is not only concerned with financial performance, the PGPA Act requires Australian Government entities to prepare annual performance statements. Australian Government entities have prepared annual performance statements and included those statements in their annual reports, with effect from the 2015–2016 reporting period.

3. The PGPA Act also makes provision for annual performance statements to be examined by the Auditor-General. Following a request from the Minister for Finance in August 2019, the Australian National Audit Office (ANAO) commenced a pilot program of performance statements audits. During the pilot program there was improvement in the standard of performance statements preparation and reporting for each of the audited entities¹, demonstrating that mandated audits of performance statements can drive more transparent and meaningful performance reporting to Parliament.

4. Notwithstanding the progress made by entities during the pilot program, entities’ performance reporting functions and supporting systems will need to mature if they are to play a more proactive role in strategic planning and quality assurance. In addition, the ANAO will need to increase awareness within the sector of its methodology for conducting performance statements audits and continue to refine the methodology to enable the Auditor-General to provide the auditee with clear, concise and timely findings.

5. The ANAO was provided additional funding in the 2021–22 Budget to support the staged roll-out of annual performance statements audits, from six audits in 2021–22 increasing to 19 audits in 2024–25 (comprising the material entities by income and expenditure). Implementation of mandatory auditing of entities’ annual performance statements would give the Parliament the same level of assurance over the quality and reliability of non-financial performance information that it currently receives for financial information presented in financial statements.

6. The Joint Committee of Public Accounts and Audit (JCPAA) has played an active and important role in the implementation of the PGPA Act through the Public Management Reform Agenda (PMRA). The JCPAA has recommended amending the PGPA Act to enable mandatory audits of annual performance statements by the Auditor-General to encourage the provision of high-quality performance information to support parliamentary accountability of entity performance² and amending the Auditor-General Act 1997 so that audits of annual performance statements are able to be initiated without the need for approval or direction from the committee or Finance Minister.³

7. This report reflects on the outcome of the ANAO’s annual performance statements audit pilot program and the ANAO’s preparation for the staged implementation of an annual performance statements assurance audit program.

Performance reporting in the Australian Government

Developments in the Commonwealth Performance Framework

8. The requirement for Australian Government entities to prepare annual performance statements under the PGPA Act took effect from 1 July 2015 with entities preparing and reporting annual performance statements for the first time in the 2015–16 reporting period.

9. In August 2019, the Minister for Finance wrote to the Auditor-General requesting the conduct of a program of pilot assurance audits of annual performance statements of Australian Government entities subject to the PGPA Act in consultation with the JCPAA.

10. The Auditor-General agreed to the Finance Minister’s request and in 2020 commenced a pilot performance statements audit program of 2019–20 performance statements of the Department of Social Services (DSS), the Attorney-General’s Department (AGD) and the Department of Veterans’ Affairs (DVA). The same three entities’ 2020–21 annual performance statements were audited in 2021.

11. The performance statements pilot program has demonstrated that accessible and understandable audit conclusions can be issued that clearly set out to the user the extent to which the performance statements can be relied upon to assess the performance of the entity. It has also demonstrated that the issuing of timely audit conclusions has been challenging. Ideally audit conclusions on an entity’s performance statements and financial statements would be issued in time to enable both to be included in the entity’s annual report. The framework does not currently require the audited entity to include the performance statements audit conclusion in its annual report.

12. The ANAO is conducting six performance statements audits in 2021–2022, which will increase to 19 audits by 2024–25.

The benefits of high quality performance statements for the Parliament and the sector

High quality performance information

13. High quality annual performance statements will deliver on the Parliament’s key objective in establishing the performance framework requirements in the PGPA Act to improve the quality and reliability of performance information in the Australian public sector. High quality annual performance statements that comply with the framework will support the Parliament’s accountability and scrutiny function through the legislative process and parliamentary committees.

14. High quality performance statements enable entities to show the Parliament and the public whether policies and programs are delivering the results intended with the resources provided. The information that supports high quality performance statements will also provide a valuable evidence base for entities to justify new policy proposals and evaluate existing policy and program settings.

15. The ANAO’s performance statements audits are designed to align with the Parliament’s expectations as established in the PGPA Act and the PGPA Rule (see Appendix 1). They provide assurance to the Parliament and also facilitate high quality performance reporting across the sector. Auditing contributes to the quality of preparation and presentation by entities and provides independent assurance to the Parliament that the performance statements can be relied upon.

Findings and recommendations from the Pilot Program

Audit themes

16. Five themes emerged from findings through the audit process in 2020–21, representing opportunities for improvement and areas of focus for entities.

17. For composite measures — a performance measure with several targets — the entity’s corporate plan needs to clearly set out how the results from each target will be weighted and the proportion of targets that must be met for the measure to be considered achieved.

18. Measures based on case studies and surveys need to be supported by a clear methodology that explains the basis for selecting case studies and identifies how surveys will be conducted. Entities need to pay particular attention where there is a heightened risk of bias in measuring and assessing performance.

19. A measure that is inwardly focused on what the entity does to enable an output to be produced will generally be assessed as an ‘input’ or an ‘activity’ — not an output. If the measure is assessed as an input or an activity, it would not normally meet the intent of the PGPA Rule.

20. Entities need to ensure there are appropriate disclosures in the performance statements regarding key information, known limitations with source data and the methodology for measuring results.

21. Entities need to ensure that processes are in place to keep records and provide their own assurance over the systems and sources that inform their performance results.

Audit conclusions and findings

22. On 7 April 2022, the Minister for Finance tabled the Auditor-General’s Independent assurance reports on AGD’s, DVA’s and DSS’ 2020–21 annual performance statements.⁴

23. Across the three 2020–21 performance statements audits, entities’ annual performance statements were largely compliant with the requirements of the performance framework and fairly presented the performance of the entity. There were some exceptions where specific measures did not meet those requirements. The ANAO made significant findings and reported exceptions as qualifications to our audit conclusion for 14.9 per cent of the three entities’ performance measures.

Lessons learned and future opportunities

Progress and areas to improve

24. Overall, engagement with entities in this new audit function has been positive. Entities have demonstrated their willingness to improve the quality of the annual performance statements they present to Parliament. An emerging better practice is the development of performance measure profiles and preparation manuals. These documents are designed to underpin the performance measures included in the entity’s annual performance statements and the process of preparation of the performance statements.

25. Entities’ performance reporting functions and supporting systems will need to mature and play a more proactive role in strategic planning and quality assurance. This should include:

• ensuring performance measures meet the requirements of the PGPA Rule;
• having processes to gain assurance over the reliability and verifiability of the data source and methodology, and the completeness and accuracy of results;
• keeping records to demonstrate why and how internal assurance processes are undertaken; and
• constructing efficiency measures for outputs or results, which could involve linking money spent and resources applied to the results achieved.

26. Consistent with their role for financial statements, audit committees have an important role to play in supporting entities to improve the quality of their performance statements.

27. The 2020–21 audits showed that the ANAO needs to increase awareness within the sector of its methodology for conducting performance statements audits. The current shortfall in awareness may reflect the absence of a dedicated performance information function within entities, similar to the Chief Financial Officer (CFO) function which includes preparation of annual financial statements. Likewise, the absence of a network of performance reporting professionals to build capability and confidence could be addressed as the audit program grows.

28. The Department of Finance and the ANAO have discussed the establishment of a performance statements ‘Community of Practice’ to drive improvement in the effectiveness and the efficacy of the process and improve the profile and professionalism of the performance reporting function within the sector.

29. Performance statements audits will have a timeline consistent with financial statements audits such that audit conclusions can be tabled in the entity’s annual report by the end of October each year. Achieving this timeframe will require improvements in planning, systems and processes for both the ANAO and entities.

30. This report recognises the progress that has been made during the performance statements pilot and the likely benefits for the Parliament and the sector as the audit program matures. Nonetheless, the staged roll-out of performance statements auditing will be challenging for the public sector. The disciplines applied to ensure informative and accurate financial reporting, which have been developed over many decades, are largely absent from performance reporting.

31. The ANAO will provide the Parliament with regular updates on the progress of the performance statements audit program. The Finance Minister has noted that a JCPAA inquiry to review the audit methods and outcomes each year during the roll-out of the performance statements pilot program would inform incremental improvements in the program and practice and inform the design of legislation going forward.⁵

1.1 Performance reporting is the main way that Australian Government entities demonstrate to the Parliament and the public how well they have used public resources to deliver programs and services and achieve outcomes. It is also fundamental to good management, governance, and decision-making. Effective performance reporting, therefore, plays an important role in maintaining public trust and confidence in the public sector and the government.⁶

The Commonwealth Performance Framework

1.2 The intended benefit of performance reporting is to improve the efficiency and effectiveness of program delivery, as well as to provide accountability for achieving results that matter to the Parliament and the public. It allows entities to learn from experience, improve program performance and allocate limited resources to where they have the most impact.

1.3 Historically, the emphasis has been on financial reporting and the preparation of financial statements. Parliament, through the Public Governance, Performance and Accountability Act 2013 (PGPA Act), introduced new requirements for the preparation of performance information and annual performance statements.

1.4 The Commonwealth Performance Framework (the framework) sets out requirements for performance reporting and the preparation of annual performance statements. It consists of the PGPA Act, the accompanying Public Governance, Performance and Accountability Rule 2014 (PGPA Rule) and guidance issued by the Department of Finance (Finance). (See Appendices 1 and 2)

1.5 The framework aims to:

• improve both financial and non-financial performance information by placing obligations on officials for the quality and reliability of performance information; and
• provide the Australian Parliament and the public with transparent and meaningful information through a combination of financial and non-financial reporting.⁷

1.6 The framework is designed to enable greater parliamentary and public scrutiny and accountability for improved performance. It also assists entities to prioritise policies and programs and allocate resources accordingly.⁸

1.7 The requirement for Australian Government entities to prepare annual performance statements under the PGPA Act took effect from 1 July 2015 with entities preparing and reporting annual performance statements for the first time in the 2015–16 reporting period.

The requirements of the Commonwealth Performance Framework

1.8 While the framework is principles-based, it establishes requirements that entities must meet. These relate to:

• preparing and publishing documents that plan and report against entity performance;
• establishing a basic structure to the performance information contained in these documents;
• planning, assessing and reporting entity performance against stated purposes and key activities with performance measures that meet specified requirements; and
• incorporating detailed analysis of results specifying impacts to intended outcomes.

1.9 A key objective of the framework is to establish a strong performance reporting system to demonstrate to the Parliament and the public that resources are being used efficiently and effectively by Australian Government entities.⁹ To this end, the framework ‘aims to improve the line of sight between what was intended and what was delivered’.¹⁰ A reader should be able to identify each performance measure as it is presented in the Portfolio Budget Statements (PBS), the entity’s corporate plan and the entity’s annual performance statements.

1.10 The clear line of sight between an entity’s planning documents and its key reporting document is known as ‘the clear-read’ principle. In its March 2021 guide to preparing the 2021–22 PBS, Finance states:

…there must be a clear linkage from the Appropriation Bills to the PB statements, to individual entities’ corporate plan and annual report. Entities should present performance information clearly and consistently (and ensure it is reconcilable) between publications within and across reporting cycles.¹¹

1.11 To achieve a clear line of sight between planned and actual performance, the framework, through a Finance Secretary’s Direction¹², establishes requirements for entities to clearly structure their performance information in their PBS and corporate plan.

PBS requirements for performance measures in each program

1.12 The PBS framework requires entities to have performance measures that provide coverage across the entity’s programs. The Finance Secretary’s Direction issued in December 2021 states that for each existing program, the entity must ‘report at least one high level performance measure and planned performance results, including targets where it is reasonably practicable to set a target’. For new or materially changed programs, the entity is required to report all performance measures and planned performance results, including targets where it is reasonably practicable to set a target.

1.13 The Australian National Audit Office (ANAO)’s performance statements audit methodology includes an assessment of entity performance measures in each program of the PBS.

Corporate plan requirements for performance measures

1.14 Subsection 16E(2) item 5 of the PGPA Rule requires that the entity’s corporate plan include details of how the entity’s performance will be measured and assessed through ‘specified performance measures for the entity that meet the requirements of section 16EA’ and ‘specified targets for each of those performance measures for which it is reasonably practicable to set a target’.

1.15 Section 16EA of the PGPA Rule states that the performance measures for an entity meet the requirements of this section if, in the context of the entity’s purposes or key activities, they:

1. relate directly to one or more of those purposes or key activities; and
2. use sources of information and methodologies that are reliable and verifiable; and
3. provide an unbiased basis for the measurement and assessment of the entity’s performance; and
4. where reasonably practicable, comprise a mix of qualitative and quantitative measures; and
5. include measures of the entity’s outputs, efficiency and effectiveness if those things are appropriate measures of the entity’s performance; and
6. provide a basis for an assessment of the entity’s performance over time.

1.16 An integral part of the ANAO’s performance statements audit methodology is an assessment of whether the entity’s performance measures meet each of the requirements of section 16EA of the PGPA Rule, as well as the requirement for targets in subsection 16E(2). These requirements form the test of whether an entity’s performance measures are ‘appropriate’ to measure and assess the entity’s performance in achieving its purposes.

The ANAO’s role in auditing performance information

1.17 The Auditor-General’s functions include auditing the annual performance statements of Australian Government entities in accordance with the PGPA Act as set out in section 15 of the Auditor-General Act 1997. The ANAO’s role in conducting audits of annual performance statements is currently subject to the request of the Minister for Finance or the responsible Minister for an Australian Government entity rather than initiated by the Auditor-General.¹³ The series of audits undertaken to date have been conducted at the request of the Finance Minister under section 40 of the PGPA Act.

1.18 The ANAO conducted three performance audits of selected entities’ 2015–16, 2016–17 and 2017–18 performance statements and three performance audits of selected entities’ 2015–16, 2016–17 and 2017–18 corporate plans.¹⁴ These audits were designed to assess entities’ performance against the framework and develop the ANAO’s methodology to support future implementation of annual audits of performance statements.

1.19 The findings from these performance audits were mixed, with two of the 10 entities examined having systems and assurance approaches in place to measure and report reliably and the remaining eight requiring improvement to ensure performance information could be relied upon to demonstrate performance against objectives and purposes. The common areas for improvement included:

• meeting the performance reporting requirements of the PGPA Act and PGPA Rule;
• developing measures to report on efficiency;
• revising audit committee charters to ensure they included the PGPA Rule requirement to review the appropriateness of performance reporting;
• improving results to ensure readers understand the connection between results, internal and external influences and how these informed the assessment of progress against purposes; and
• including descriptions of information sources, targets and methodologies for measurement and basis for assessment of performance.

1.20 In September 2017 at a hearing of the Joint Committee of Public Accounts and Audit (JCPAA), the Auditor-General reflected on the experience of these performance audits:

The growing maturity of Commonwealth entities’ annual financial statements can in part be attributed to the regular external audit scrutiny applied by the ANAO. Engagement with entities throughout a financial year provides the opportunity to resolve matters affecting the reliability of financial statements in real time. This approach, accompanied by the ANAO’s bi-annual reporting of significant and moderate audit issues to the parliament, has played a role in entities moving towards the more mature financial reporting processes observed today. The introduction of annual audits of performance statements could be expected to lead to similar improvements to the maturity of entities’ performance measurement and reporting. We will continue to build on our audit methodology in this area and intend to position the ANAO to be able to audit the annual performance statements of Commonwealth entities in a similar way to the audit of financial statements, if required to do so…

[P]ast experience demonstrates leaving external review to periodic performance audits is unlikely to drive the desired level of improvement. This in turn may result in the current reform agenda for performance reporting going the same way as previous ones, with modest improvement and ongoing frustration of the parliament with the quality of performance reporting by entities.¹⁵

The role of the Joint Committee of Public Accounts and Audit

1.21 The JCPAA has played an important role in overseeing the government’s implementation of the PGPA Act through the Public Management Reform Agenda (PMRA).¹⁶ It held two inquiries and made recommendations into the progress of the PMRA (JCPAA Reports 453 and 457) and conducted a third inquiry and made recommendations into the framework in response to ANAO reports (JCPAA Report 469).¹⁷

1.22 In its Report 469, tabled in December 2017, the JCPAA agreed with the Auditor-General’s evidence that the growing maturity of Australian Government entities’ annual financial statements can be attributed to the regular external audit scrutiny applied by the ANAO, and that the introduction of mandatory annual audits of performance statements could be expected to lead to similar improvements in the maturity of entity performance statements.¹⁸ The report recommended that the government amend the PGPA Act to enable mandatory audits of annual performance statements by the Auditor-General of entities selected by the Auditor-General for review.¹⁹ In addition, the JCPAA recommended in March 2022 that the Auditor-General Act 1997 be amended so that audits are able to be initiated without the need for approval or direction from the committee or Finance Minister.²⁰

The review of the PGPA Act and the Finance Minister’s request

1.23 In accordance with section 112 of the PGPA Act, an independent review of its operation (and the PGPA Rule) was required as soon as practicable after 1 July 2017. In September 2017, the Minister for Finance appointed external reviewers. In September 2018, the reviewers provided a report to the Minister for Finance that made 52 recommendations, eight of which related to improving the quality of performance reporting.²¹ The Review found that:

[B]roadly speaking, Finance, the Auditor-General and the Joint Committee of Public Accounts and Audit agree that the overall quality of published performance information is better than it was before the framework was introduced, but that progress has been uneven, and in some cases modest.²²

1.24 The Independent Review recommended that:

The Finance Minister, in consultation with the Joint Committee of Public Accounts and Audit, should request that the Auditor-General pilot assurance audits of annual performance statements to trial an appropriate methodology for these audits. The Committee should monitor the implementation of the pilot on behalf of the Parliament. (Recommendation 8)²³

1.25 In August 2019, the Minister for Finance wrote to the Auditor-General requesting the conduct of a program of pilot audits of annual performance statements of Australian Government entities subject to the PGPA Act in consultation with the JCPAA. The Minister noted that a pilot will allow the Auditor-General to further develop and refine an appropriate methodology prior to potential introduction of mandatory assurance audits of Australian Government entities’ annual performance statements.

1.26 In November 2019, the Auditor-General agreed to the Finance Minister’s request, noting that the pilot would consist of an audit of three entities’ 2019–20 performance statements. The Auditor-General notified the Finance Minister that:

While the Australian National Audit Office (ANAO) would meet the cost of conducting the pilot in 2019–20, costs could only be met through a reduction in the number of performance audits undertaken; transition to full implementation of performance statements audits will require additional budget funding; and that the scale and timing of full implementation will be informed by the pilot.

The ANAO’s annual performance statements audit pilot

1.27 In February 2020, the Auditor-General advised the JCPAA that the Department of Social Services (DSS), the Attorney-General’s Department (AGD) and the Department of Veterans’ Affairs (DVA) had been selected to participate in the pilot.

1.28 The 2020 audits were reasonable assurance engagements under Australian Standard on Assurance Engagements (ASAE) 3000 Assurance Engagements Other than Audits or Reviews of Historical Financial Information. The methodology was developed from the ANAO’s methodology for auditing entities’ financial statements, and from the ANAO’s methodology for performance audits of entities’ corporate plans and performance statements.²⁴

1.29 In December 2020, the Auditor-General provided his Independent assurance reports on DVA’s and AGD’s 2019–20 annual performance statements to the Minister for Finance. The Minister tabled these auditor’s reports in February 2021. The reports are published on Finance’s website.²⁵

1.30 While both DVA’s and AGD’s 2019–20 performance statements were found to be largely compliant with the requirements of the performance framework, the audit conclusion for each entity included qualifications related to specific performance measures. Notwithstanding these qualifications, both DVA and AGD’s performance statements provide an example for the sector of how to align purposes with key activities and results, and transparently report these results based on targets and different types of performance measure (outputs, effectiveness and proxies for efficiency).

1.31 The interim management letter issued to DSS in July 2020 identified that the department’s performance measures included in the 2019–20 Corporate Plan are not sufficiently reliable — and thereby not sufficiently appropriate — to measure the achievement of DSS’ purposes.²⁶ By agreement, in November 2020 the ANAO withdrew from the 2019–20 DSS audit engagement.

The 2021 pilot audits

1.32 To maintain the momentum from the 2020 pilot, in February 2021 the ANAO commenced audits of the same three entities’ 2020–21 performance statements.

1.33 The Auditor-General provided his Independent assurance reports on AGD’s, DVA’s and DSS’ 2020–21 annual performance statements to the Minister for Finance who tabled them on 7 April 2022. The reports are published on Finance’s website.²⁷

1.34 As with the previous year, the Auditor-General’s conclusion for all three entities’ 2020–21 performance statements included qualifications related to specific performance measures (see chapter 4). Nonetheless, AGD demonstrated improvement in its internal processes, its engagement with the ANAO, the readability of its statements and the department resolved previous year findings. DVA also addressed previous year findings and implemented additional processes to provide quality assurance over its performance results and the data and methodologies applied to calculate results.

1.35 DSS made revisions to its performance measures from the previous reporting period. The revised performance measures were mostly compliant with the Rule and were accompanied by performance measure profiles that explain the context and rationale for each measure, key activity and program (see chapter 4).

1.36 For both the 2020 and 2021 pilot audits, the ANAO demonstrated that, based on its audit methodology, an accessible and understandable audit conclusion can be issued that clearly sets out to the user of the performance statements what information in the statements can or cannot be relied upon to inform an assessment of the entity’s performance.

1.37 For both the 2020 and 2021 pilot audits, the issuing of timely audit conclusions has been challenging. Ideally audit conclusions on entity’s performance statements and financial statements would be issued in time to enable both to be included in the entity’s annual report. The framework does not currently require the audited entity to include the performance statements audit conclusion in its annual report as is the case for financial statements audit opinions.

1.38 None of the five audit conclusions issued were prepared in time for publication with the entities’ annual reports. The ANAO observes that there were several reasons for this, including entities’ ability to address issues raised during the audit and provide timely evidence to meet requirements for testing ‘completeness and accuracy’. The commencement date for the audits also impacted timeframes — the work required in the audits far exceeded the planned timeframes. These issues are discussed in chapters 3 and 4 of this report.

Ongoing funding and support from the Parliament

1.39 As part of the 2021–22 Budget, the ANAO was provided additional funding to support the staged implementation of a program of performance statements audits, from three audits in 2020–2021 increasing to six audits in 2021–2022, ten audits in 2022–2023 and 19 audits in 2024–25. Significantly, this funding builds into the core business of the sector assurance to the Parliament of non-financial reporting information, closing this assurance gap.²⁸

1.40 As Finance notes in its 2020–21 annual performance statements, the department is working with the ANAO on the design and implementation of this expanded audit assurance program of annual performance statements.²⁹

Parliament’s expectation for high quality performance information

2.1 In enacting the PGPA Act and establishing the Commonwealth Performance Framework (framework), Parliament has set the expectation that entities will develop and report high quality performance information in their annual performance statements. The framework sets standards in relation to the quality of performance information, performance monitoring, evaluation and reporting.³⁰

2.2 Achieving high quality performance reporting has been an important but elusive issue for decades, and there continues to be a strong interest in improving the quality and reliability of performance information in the Australian public sector. Despite many initiatives³¹, Australian Government entities still struggle to tell their performance story easily and clearly, which can lead to criticisms about public sector performance, transparency and accountability.

2.3 The Australian National Audit Office (ANAO) provides assurance to the Parliament that entities’ reporting meets the standards required by the framework. ANAO annual performance statements audits are designed to provide the Parliament with assurance that its expectation for high quality and meaningful information to measure and assess the performance of entities is being met, and the integrity of the framework is upheld.

2.4 There are two main benefits from public sector entities producing high quality and meaningful performance information in their annual performance statements. The first is to provide effective public accountability by informing the Parliament and the public about how well taxpayer money is being used and the outcomes being achieved. The second is to provide performance information that entities can use to measure their impact and drive improvements in the efficiency and effectiveness of policies and programs and how government allocates taxpayer funds.

How ANAO assurance should lead to higher quality and reliable performance information

2.5 Since its establishment in 1901, the ANAO has provided the Parliament with assurance that public sector entities’ financial statements have complied with the relevant accounting standards and present fairly the financial position of the entity.

2.6 On the whole, through the regular annual auditing of financial statements, the Parliament can be assured that financial reporting in the Australian public sector is robust. The number of audit findings in financial statements has declined over the past five years, indicating ongoing improvement in financial reporting and generally sound financial management within the sector.³²

2.7 Over time, entities have developed effective systems to plan and report their financial performance in their financial statements. In conducting audits and providing assurance to the Parliament, the ANAO has played an important role in increasing and maintaining the quality and reliability of entities’ financial reporting.

2.8 The 2020 and 2021 pilot performance statements audits demonstrated that processes and systems supporting the preparation of annual performance statements are not as robust as those in place for the preparation of financial statements, which have been developed over many decades. Consequently, each of the five audit conclusions related to the pilot audits contained several findings and qualifications related to specific performance measures. Chapter 3 of this report explains the type and the content of these qualifications.

2.9 Qualified audit conclusions are more likely to be issued in the early stages of implementing the new audit framework for annual performance statements and as entities continue to improve their performance reporting systems and processes. This will be an important part of the process for improving the quality of entities’ performance information.

2.10 As with financial statements audits, regular auditing of annual performance statements will assist entities to improve the quality and reliability of their annual performance statements and provide Parliament with assurance that the quality of entities’ performance information is consistent with the framework and Parliament’s expectations.

2.11 The quality of annual performance statements improved between the 2020 and 2021 pilot audits, including a gradual shift from generic measures to targeted measures that better reflect the impact of services and programs. Further progress will, however, be necessary before the audit conclusions do not include significant findings and qualifications.

A strategic roll-out

2.12 To drive higher quality performance information, the roll-out of the ANAO’s performance statements audit program will be important. In the next four years, the ANAO will audit the same entities as the previous year while adding additional entities. In 2022, for example, the same three entities will be audited as in 2021 (Department of Social Services (DSS), Attorney General’s Department (AGD) and Department of Veterans’ Affairs (DVA)) plus an additional three entities — the Department of the Treasury, the Department of Agriculture, Water and the Environment and the Department of Education, Skills and Employment. In 2023, these six entities will be audited plus an additional four.

2.13 Entity selection will initially be drawn from the major portfolio entities in the Australian public sector to align with the ANAO’s parliamentary reporting focus for financial statements where a yearly assessment of the key financial controls of major entities is published. The entities included in the report are selected on the basis of their contribution to the income, expenses, assets and liabilities of the sector. The 2020–21 Interim Report on Key Financial Controls of Major Entities examined 25 entities, including all departments of state and other major Australian Government entities.³³

2.14 As demonstrated during the pilot program, this cumulative process will enable the ‘repeat’ entities to improve the quality and reliability of their performance statements, including to address the issues that were identified the previous year. The ‘repeat’ entities will also be well placed to share their knowledge and experience with ‘new’ entities and more broadly across the sector.

How the Parliament benefits from audit assurance and improved performance reporting

2.15 A key objective of the Commonwealth Performance Framework (2015–16 to present) is to provide the Parliament with meaningful performance information.³⁴ The Explanatory Memorandum to the Public Governance, Performance and Accountability Bill described this objective as follows:

…for the Parliament to properly fulfil its oversight function, performance information is crucial to assessing whether policy goals are being achieved. Performance information also shows how effectively the public sector has performed. The quality of information is more important than the quantity.³⁵

2.16 The specific requirements of the Public Governance, Performance and Accountability Act 2013 (PGPA Act) and the Public Governance, Performance and Accountability Rule 2014 (PGPA Rule) reflect that high quality, meaningful performance information for the Parliament is information that:

• aligns the entity’s purposes, key activities and performance measures and targets;
• measures what was achieved against what the entity had planned to achieve;
• measures the impact and cost effectiveness of the entity in achieving its purposes and policy goals; and
• measures what was achieved based on reliable data sources and methodologies.

2.17 The auditing of annual performance statements should offer the following benefits for the Parliament.

• In the short-term, the Auditor-General’s audit conclusions provide Parliament with a clear view on the extent to which performance statements can be relied upon to form an assessment of an entity’s performance.
• In the medium to longer term, a program of performance statements audits will improve the quality and reliability of performance information presented in annual performance statements. It will also support informed legislative and parliamentary committee debate on the quality of outcomes and the return on investment being achieved.

How the sector will benefit from audit assurance and improved performance reporting

2.18 An audit program of entities’ externally reported performance information offers the sector two main benefits. The first is to require quality and transparency about what programs and policies have achieved and whether there has been a return on the government’s investment. The second is to promote a culture of policy and program evaluation that can inform long-term strategic planning and support evidence-based decision-making.

A return on investment

2.19 The private sector rigorously measures its performance and determines whether the return is worth the investment of shareholder funds. Evaluation is an integral part of business models and helps inform future resource allocation decisions, with analysis not only focusing on spending, but testing effectiveness against ambition.

2.20 The ANAO’s program of performance statements audits will assist the sector to demonstrate to the Parliament that policies and programs are delivering the results intended with the resources provided.

2.21 High quality performance reporting and evaluation is essential as Australian Government entities today are facing a new set of complex and longer-term challenges, including being impacted by the COVID-19 pandemic.³⁶ To support this focus, new approaches to performance and public accountability are needed, including:

• developing new ways of reporting on the management of complex long-term issues that matter to the public;
• explaining more clearly how public organisations use public money to generate value for the Australian community; and
• aligning the Australian Government entity’s funding requirements to the services and outcomes being delivered to Australian citizens rather than only to the inputs and outputs of its activities.

2.22 Given the cross-cutting nature of many complex problems, it may also require integrating and sharing results between Australian Government entities to describe performance in a way that includes outcomes, sectors, and the whole-of-government.³⁷ For example, the Consolidated Financial Statements provide a picture of the financial results of the whole of the government. There is currently no equivalent approach for consolidating annual performance statements.

From compliance to strategic planning

2.23 The framework requires entities to report their performance information annually. Auditing of this information will indicate to entities whether the information is accurate, current and able to be relied upon to support decision-making. This information is thereby a platform for entities to plan and improve future policy and programs and to better allocate their resources.

2.24 Without adequate performance information, entities can struggle to evaluate their policies and programs to identify whether current settings are working as intended. ANAO performance audits have highlighted the difficulties that entities have in evaluating their policies and programs, especially where there is inadequate baseline data.³⁸

2.25 Current government guidance highlights the importance of entities obtaining and using high quality evidence and information to develop and evaluate their policies and programs.

• In developing policy, it is an expectation of government that new policy proposals will be supported by solid evidence that is established through impact analysis. Impact analysis ensures decision-makers have confidence that proposed policies are well-designed, well-targeted and fit-for-purpose.³⁹
• In December 2021, the Finance Minister endorsed a Commonwealth Evaluation Policy and supporting toolkit.⁴⁰ The aim of the Policy is to:
  • embed an APS culture of evaluation and learning from experience;
  • improve the impact of programs through evidence-based policy and delivery; and
  • support Australian Government entities to improve the quality of performance reporting through adherence to consistent, quality standards and approaches.

2.26 The Department of Finance (Finance) has also been working with stakeholders in the sector to embed evaluation planning in new policy proposals and regulation impact statements.⁴¹

2.27 High quality performance statements can help entities to improve their policy development and evaluation processes by providing information on outcomes and return on investment instead of focussing only on costs. This in turn should assist to improve their performance and deliver greater value for the public funds they have been entrusted to manage.

2.28 In this context, the pilot program has already had an impact, resulting in a maturing of performance statements governance and preparation processes in entities that have been audited (see chapters 3 and 4). While the entities generally have less well-coordinated processes compared to the preparation of financial statements, it is consistent with the way financial statements maturity evolved following the introduction of accrual accounting.⁴²

2.29 Audit committees, Chief Financial Officers and finance teams are integral to good governance of Australian Government entities. To tell their performance story well, Australian Government entities may need to raise the profile of the planning and performance reporting function.

2.30 When the audit program is fully implemented and mature, the ANAO would expect to see that the information in entities’ annual performance statements is a key part of the entity’s coordinated approach to policy and program design and planning strategically for the future, including to help better target policies and programs and maximise the chance of achieving desired results.

2.31 Finance and the ANAO will play a key role in moving the sector beyond a compliance mindset. Finance’s work to build and embed a culture of evaluation and learning in the public sector will complement the ANAO’s performance statements audit program.

Public sector auditing of non-financial performance in Australia

2.32 The audit of performance information is not unique to the Australian public sector. Such information is also audited in several state and territory jurisdictions.

2.33 In Western Australia, state agencies report their effectiveness and efficiency Key Performance Indicators (KPIs) in their annual reports. The Auditor General then forms an audit opinion on each agency’s KPIs as a whole. This opinion is based on whether the indicators fairly represent the indicated performance for the audit period, and if they are relevant and appropriate to assist users to assess the agency’s performance.⁴³ This process occurs annually and the audit opinion is published in the audited agency’s annual report.

2.34 Local Government Victoria (LGV) collects and publishes a range of information about council performance. Councils are required to prepare a performance report in line with the Local Government Performance Reporting Framework and reports are audited annually by the Victorian Auditor-General’s Office.⁴⁴ LGV publishes councils’ performance results on a dashboard⁴⁵ annually.

2.35 In the Australian Capital Territory, territory authorities list objectives, performance criteria and other measures in their annual ‘statement of intent’. They then assess their performance in relation to this in their annual report and ‘statement of performance’. This statement is scrutinised by the ACT Auditor-General who provides a limited assurance report to be attached to the authority’s annual report.⁴⁶

2.36 Standards for the preparation and audit of non-financial performance information are not set independently in Australia.⁴⁷ Rather, each jurisdiction has established its own framework, mainly through primary and subordinate legislation. This contrasts with financial statements, where accounting and auditing standards are independently set, both nationally and internationally, which promotes consistency across jurisdictions.⁴⁸ These issues are discussed in more detail below.

Auditing of non-financial performance in the public sector internationally

2.37 Several other jurisdictions have established national legislative frameworks with comparable requirements for entity performance reporting. Canada, the United States of America (USA), New Zealand, South Africa and the United Kingdom (UK) each have a framework similar to the Commonwealth framework in terms of the characteristics of the framework, the quality of information required and requirements for publishing performance measures, targets and results.

2.38 The Office of the Auditor-General New Zealand (OAG) has been conducting audits on how entities can improve their performance and reporting for public accountability purposes. Results and observations from this process were published in a 2021 report by the OAG on ‘The problems, progress, and potential of performance reporting’.⁴⁹

2.39 South Africa has a performance statements audit program. Canada, the USA and the UK do not have the equivalent of a performance statements audit program. They do, however, have central agencies that are engaged in improving performance reporting (see Appendix 3).

Accounting standards relating to performance information

2.40 In 2009, the Australian Accounting Standards Board (AASB) initiated a project on service performance reporting to respond to feedback received about needs of not-for-profit (NFP) private entities’ financial statements users.

2.41 In August 2015, the AASB issued an Exposure Draft (ED) 270 Reporting Service Performance Information for comment. The ED proposed principles and mandatory requirements for NFP entities in both the private and public sectors to report service performance information that is useful for accountability and decision-making purposes. The proposals in the ED were not required to be included as part of the financial statements; therefore, it did not specify whether service performance information reported is required to be audited; this is a matter for each entity’s regulator.

2.42 Feedback received generally agreed with the proposed objectives and principles of ED 270. Since that time, the AASB has not significantly progressed on this topic. The AASB has noted in consulting on its agenda for 2022–2026⁵⁰ that it intends to recommence the service performance reporting project in the period of 2022–2026 because of its priority and close relationship with the sustainability reporting project (see below).

New Zealand accounting and auditing standards

2.43 In November 2017, the New Zealand Accounting Standards Board of the External Reporting Board issued Public Benefit Entity Financial Reporting Standard 48 Service Performance Reporting (PBE FRS 48). This standard establishes service performance reporting requirements with effect from 1 January 2022. PBE FRS 48 reflects the view that service performance information is an essential part of a general purpose financial report and that an entity should report its service performance information alongside its financial statements.⁵¹ Consequently, a new audit standard, New Zealand Auditing Standard 1 The Audit of Service Performance Information, was issued in February 2019 by the New Zealand Auditing and Assurance Standards Board of the External Reporting Board. This standard took effect from 1 January 2022.

Integrated reporting

2.44 The value of non-financial performance information is recognised in the private sector. A growing number of companies are using the international framework for integrated reporting to better communicate to their stakeholders how they create value over time. The framework goes well beyond the traditional financial reporting requirements of companies.

2.45 Integrated reporting is designed to clearly show how an organisation’s strategy, governance, performance, and prospects, in the context of its external environment, create value in the short, medium, and long term.⁵²

2.46 The goal of integrated reporting is to provide investors with the information they need to make more effective capital allocation decisions that facilitate better long-term investment returns.

2.47 Integrated reporting is gaining ‘traction’ from corporations and regulators. For example, it is referred to in the ASX Corporate Governance Principles and Recommendations and the Australian Securities and Investments Commission’s guidance in relation to the preparation of operating and financial reviews.

Sustainability reporting in Australia and internationally

2.48 Sustainability reporting refers to the practice of corporations and other organisations measuring and publicly reporting on their economic, social and environmental performance.⁵³ In the Australian private sector, there is not compulsory sustainability reporting as such. Companies are required to disclose any information that shareholders would reasonably need to make an informed assessment of an entity’s operations and business strategies. There are also recommendations on corporate governance practices around environmental and social risks for publicly listed companies in Australia. The current legal requirements for certain entities in terms of disclosing non-financial information are related to specific federal Acts including the Modern Slavery Act 2018, the Workplace Gender Equality Act 2012, and the National Greenhouse and Energy Reporting Act 2007.

2.49 The annual reports of Australian Government departments, Parliamentary departments, Commonwealth authorities, Commonwealth companies and other Australian Government agencies must under section 516A of the Environment Protection and Biodiversity Conservation Act 1999, all now include a report on environmental matters in their annual reports. These disclosures are not subject to audit as they do not form part of the financial report of Australian Government entities. They are also not part of entity annual performance statements.

2.50 International investors with global investment portfolios are increasingly calling for high quality, transparent, reliable and comparable reporting by companies on climate and other environmental, social and governance (ESG) matters. On 3 November 2021, the IFRS Foundation Trustees announced the creation of a new standard-setting board — the International Sustainability Standards Board (ISSB) — to help meet this demand.⁵⁴

2.51 The intention is for the ISSB to deliver a comprehensive global baseline of sustainability-related disclosure standards that provide investors and other capital market participants with information about companies’ sustainability-related risks and opportunities to help them make informed decisions.

2.52 In response to international developments, the Financial Reporting Council (FRC), AASB and the Auditing and Assurance Standards Board (AUASB) recently released a Position Statement on Extended External Reporting (EER) and Assurance.⁵⁵ This Position Statement outlines an approach by which the AASB intends to develop reporting requirements for sustainability-related information, simultaneously with the relevant assurance standards developed by the AUASB, rather than via a separately established board at this time. The Boards announced their intention to establish reporting requirements for sustainability-related information that will, as far as is practicable, align with significant international developments in the same area.

2.53 The AUASB has adopted the guidance from the IAASB, Non-Authoritative Guidance on Applying ISAE 3000 (Revised) to Extended External Reporting Assurance Engagements⁵⁶, for use in Australia.⁵⁷

3.1 Performance statements audits provide Parliament with assurance over the quality and reliability of entities’ annual performance statements and also provide findings and recommendations to entities. The aim of issuing findings and recommendations to each entity is to drive improvement in the preparation of the performance statements and help entities to disclose relevant, accurate and useful information regarding their performance. Entities can work to resolve interim findings in the final phase of the audit and to use final findings to resolve matters for the following year. In this way, performance statements audits are designed to improve transparency and drive continuous improvement in the quality of performance information.

Phases of a performance statements audit

3.2 Performance statements audits are structured similarly to financial statements audits with a planning, interim and final phase. As in financial statements, the intent of this structure is to enable the auditee to address matters identified at the interim phase of the audit and to reflect on final findings in the following reporting year. (See Appendix 4)

Planning phase

3.3 To ensure an effective audit process, the nature, timing and extent of audit procedures must be considered.⁵⁸ In the planning phase of a performance statements audit, auditors seek to gain an understanding of the entity and the entity’s processes, its risk environment, and roles and responsibilities for the production of the performance statements.

3.4 Auditors typically collect publicly available documentation and request internal entity performance documentation to support this process. For example, material may be requested to explain the context and the evidence base for each performance measure in the annual report, corporate plan and Portfolio Budget Statements (PBS).

3.5 The entities audited in 2021 each developed this type of documentation. For example, Department of Social Services (DSS) ‘Program Profiles’ establish a template for performance information preparation in accordance with the requirements of the Public Governance, Performance and Accountability Rule 2014 (PGPA Rule). For each program, DSS Program Profiles set out:

• the alignment of the relevant purpose, outcome, key activity and performance measure(s) and target(s);
• the policy or program context for the ‘key activity’;
• why the measure selected is important to report achievement of the key activity;
• why the target was chosen;
• the type (output, effectiveness, efficiency / qualitative, quantitative) and the timeframe of the measure;
• the data source and methodology to support the measure including the data provider and the source data owner;
• the calculation required to derive a result; and
• lines of responsibility and accountability for the receiving and storing source data and calculating a result in accordance with the methodology.

3.6 An engagement letter to agree entity and audit processes and set out audit criteria, key milestones and responsibilities is provided by the Australian National Audit Office (ANAO) and acknowledged by the entity.

Interim phase

3.7 The interim phase of the audit occurs prior to the entity’s preparation of its annual performance statements. During the interim phase, the audit assesses the corporate plan and the PBS and the internal control framework that supports the production of the performance statements. This provides an understanding of the likelihood or risk of errors within the performance statements and allows for detailed testing to be planned and undertaken.

3.8 At the conclusion of the interim audit phase an Interim Management Letter (IML) is provided to the entity. The IML indicates the ANAO’s assessment of the internal control environment, the ‘clear read’ of performance information, and whether each of the entity’s measures meet the specific requirements of the PGPA Rule, especially the requirements set out in section 16EA of the Rule and whether targets have been set in line with section 16E(2)(5)(b). The IML contains the ANAO’s findings and recommendations which are provided to the entity for comment and response.

3.9 The IML is designed to assist the entity to address matters relating to the performance statements and/or particular measures prior to the final phase of the audit, and to provide the entity with an opportunity to produce performance statements on which the ANAO will be able to provide an unqualified conclusion.

Final phase

3.10 During the final phase of the audit, the ANAO’s focus is on the performance information and analysis contained in the entity’s performance statements. The audit concentrates on the following questions.

• Do the performance measures meet the requirements of the PGPA Rule? This is based on the information in the PBS, the corporate plan, the draft performance statements, the entity’s internal documentation, and the ANAO’s assessment of results in the performance statements based on the entity’s methodology and the evidence obtained.
• Has the entity provided workpapers that sufficiently demonstrate how data sources, evidence and the methodology were applied to calculate the reported result in the performance statements?
• Is the ANAO able to obtain the evidence required to verify the reported result based on the entity’s methodology for measurement and assessment?
• Can the entity provide adequate assurance over the systems used to report performance information, and the evidence generated from these systems?
• Do the performance statements meet the requirements of section 16F of the PGPA Rule and the requirements for the presentation of annual reports in section 17AC of the PGPA Rule?
• Have the IML findings and recommendations been adequately addressed?

3.11 The audit process then assesses the overall results of testing and evidence obtained in order to form the audit conclusion. Importantly, the audit conclusion aims to inform the user about the quality and reliability of information presented in the performance statements (from which the user forms a view about the entity’s performance) and does not provide an opinion about the substantive performance of the entity.

3.12 The audit conclusion is normally discussed with the entity in advance of provision to the accountable authority to ensure the results of the audit are clearly understood and to allow for any last details to be finalised. Once completed the Auditor-General provides his Independent assurance report to the Finance Minister.

3.13 The ANAO also prepares a Final Management Letter (FML) which contains the ANAO’s findings and recommendations from the audit. Consistent with the approach for financial statements, the FML categorises findings as significant (A), moderate (B) and minor (C) to emphasise their relative importance and support entities to prioritise improvement efforts.

3.14 The FML is provided to the entity for comment and response. The entity’s responses to, and action on, the FML findings and recommendations are a key reference point for the ANAO in commencing an audit of the entity’s performance statements for the following reporting year. Table 3.1 summarises audit findings presented to entities that participated in the 2020–21 pilot program.

Table 3.1: Performance statements audit findings 2020–21

Source: ANAO compilation of findings. See Appendix 5

Findings by theme

3.15 This section briefly discusses key themes identified during the pilot audit program, which represent opportunities for improvement and areas of focus for entities. More detail is provided in Appendix 6.

Use of composite measures and wording of measures and targets

3.16 The first theme relates to how entities construct a performance measure and target. It is important that the following be considered when creating performance measures and targets.

• For composite measures — a performance measure with several targets:
  • the entity’s corporate plan clearly sets out how the results from each target will be weighted;
  • the corporate plan explains the proportion of targets that must be met for the measure as a whole to be considered achieved; and
  • internal documents explain why the targets were chosen and where appropriate, why they were chosen over other targets.
• The wording of the measure and the target align (for example, if the measure aims to achieve a percentage, the target should also refer to a percentage).
• The target is able to assess a matter or matters that substantially address what the measure aims to achieve; that is if the target is achieved, it should reflect that the aim of the measure has been substantially achieved.

3.17 Appendix 6 shows that there was at least one significant (A), moderate (B) and minor (C) finding relating to these matters.

Case studies and surveys — meeting the requirements of sections 16E and 16EA of the PGPA Rule

3.18 The second theme relates to entities ensuring that their performance measures meet the specific requirements of the PGPA Rule. Appendix 6 shows that significant or moderate findings were issued to entities where their performance measures did not meet the requirement for data sources and methodologies to be reliable and verifiable and provide an unbiased basis for assessment.

Unbiased basis for assessment

3.19 Entities need to pay particular attention where there is a heightened risk of bias in measuring and assessing performance. The 2020–21 performance statements audits highlighted the following:

• for measures based on case studies, the selection and scope of these case studies needs to be clearly identified in the corporate plan;
• for measures based on surveys, internal documents created at the beginning of the reporting period should identify the population, the sample size, how the survey will be conducted (including controls over the survey process and data) and an acceptable response rate;
• where survey measures are based on third parties self-reporting, there should be an assessment of the risk of positive bias and how these risks will be addressed and disclosed; and
• where measures are based on an internal measurement, the entity should ensure there are adequate controls in place to demonstrate that the result was objectively assessed.

Reliable and verifiable

3.20 A biased basis for measuring and assessing the result of a measure may also be a sign that the data source and methodology may not provide a reliable basis for assessment. Appendix 7 shows that several measures that were assessed as both biased and unreliable formed the basis for a qualification.

The requirement for measures of output, effectiveness and efficiency

3.21 The third theme relates to whether the type of performance measure meets the intent of the framework. The PGPA Rule requires measures of the entity’s outputs, efficiency and effectiveness to be included if those things are appropriate measures of the entity’s performance (subsection 16EA(e)).

3.22 A measure that is inwardly focused on what the entity does to enable an output to be produced will generally be assessed as an ‘input’ or an ‘activity’⁵⁹ — not an output. If the measure is assessed as an input or an activity, it would not normally meet the intent of the PGPA Rule⁶⁰ and the ANAO would consider them as supporting information.

3.23 In the 2020–21 performance statements audits, 7 of the 107 (6.54 per cent) performance measures across the three entities were assessed as ‘inputs’ or ‘activities’ and were therefore excluded from a full assessment against the requirements of section 16EA of the PGPA Rule. However, the results for these seven measures were still assessed for ‘completeness and accuracy’ during the final phase of the audit.

3.24 The exclusion of input and activity measures from testing against the full requirements of section 16EA of the PGPA Rule is designed to drive performance reporting that measures the impact that an entity makes. The presentation of input or activity-based measures does not automatically attract a finding, although a finding may be made in cases where:

• input and activity measures account for a sizeable proportion of the entity’s total performance measures; and
• an input or activity measure is all that the entity provides to explain its performance in relation to a key activity or an outcome.

3.25 In these cases, the reader may have inadequate information to assess the entity’s performance.

Disclosure and presentation

3.26 The fourth theme relates to the disclosure in the performance statements of key information on source data and methodology, and how the results for measures are presented.

3.27 In general, the three entities’ 2020–21 performance statements clearly disclosed key information and the known limitations with source data and the methodology. One finding was issued where the entity did not disclose survey response rates. The following matters resulted in significant and minor findings (see Appendix 6):

• failure to identify criteria for the selection of case studies in the corporate plan;
• non-disclosure of survey response rates and methodologies in the performance statements; and
• failure to identify population and sampling approaches for survey-based measures.

3.28 Section 17AC of the PGPA Rule requires information in annual reports that is ‘relevant, reliable, concise, understandable and balanced’.⁶¹ For performance measures where the movement of several variables is being assessed, the use of a graph or a chart could supplement the narrative in the annual report.

Providing assurance over completeness and accuracy of results

3.29 The fifth theme relates to how entities gain assurance over the completeness and accuracy of reported results. Appendix 6 shows that there was at least one significant (A), moderate (B), and minor (C) finding relating to these matters. These findings reflect the importance of entities’ processes to keep records and provide their own assurance over the systems and sources that inform their performance results. These processes ensure that the reader can have confidence that the reported results are accurate. Chapter 4 discusses the role of audit committees to contribute to this process.

Bases for qualification

3.30 As noted previously, the Auditor-General issued audit conclusions that included qualifications related to specific performance measures for all three audits in 2020–21.⁶² These are referred to as ‘except for’ audit conclusions on each of the three 2020–21 performance statements audits, as the qualifications did not apply to the annual performance statements as a whole, but to specific measures reported in the statements. This section explains an ‘except for’ qualification and the specific context for the Attorney General’s Department (AGD), DSS and Department of Veterans’ Affairs (DVA) conclusions.

Modified audit conclusions

3.31 An auditor’s conclusion may be ‘modified’ in one of three ways.

• A ‘qualified conclusion’ may be expressed when the auditor, having obtained sufficient appropriate audit evidence, concludes that the performance statements as a whole are largely compliant with the PGPA Act or the PGPA Rule, but there are matters of non-compliance that, individually or in aggregate, are material but not pervasive to the performance statements. This could include where the auditor is unable to obtain sufficient appropriate audit evidence for specific performance measures, which, while material, is not pervasive to the performance statements. A qualified conclusion may also be referred to as an ‘except for’ qualification.
• A ‘disclaimer of conclusion’ is expressed when the auditor, having been unable to obtain sufficient appropriate audit evidence on which to base the conclusion, determines that the possible effects on the [performance] statements of undetected non-compliance could be both material and pervasive.
• An ‘adverse conclusion’ is expressed when the auditor, having obtained sufficient appropriate audit evidence, determines that matters of non-compliance, individually or in aggregate, are both material and pervasive to the [performance] statements.⁶³

3.32 As noted above the Auditor-General’s conclusion on each of the three 2020–21 performance statements audits was an ‘except for’ qualification, which is described as follows in the auditor’s reports:

In my opinion, except for the effects and possible effects of the matters described in the Bases for Qualified Conclusion section of my report, the attached 2020–21 Annual Performance Statements of the Entity are prepared, in all material respects, in accordance with the requirements of Division 3 of Part 2-3 of the Public, Governance, Performance and Accountability Act 2013 (the Act).

3.33 The three qualified audit conclusions related to 16 of the 107 (14.9 per cent) performance measures across DSS’, AGD’s and DVA’s 2020–21 performance statements.

Attorney-General’s Department

3.34 The Attorney-General’s Department’s 2020–21 performance statements reported on 22 performance measures underpinned by 64 targets. The auditor’s report on the AGD’s annual performance statements was qualified with respect to 12 targets relating to six of the 22 performance measures.

Misleading result from a composite measure

3.35 AGD’s performance measure relating to managing ‘significant legal issues and arrangements for Australian Government legal services’ is a composite measure containing five targets. The ANAO assessed that four of these five targets — all of which AGD had assessed as ‘achieved’ — did not meet the requirements of section 16EA of the PGPA Rule.

3.36 Specifically, the ANAO assessed these targets as ‘activities’ and as such, they do not meet the requirement in subsection 16EA(d) of the PGPA Rule for measures of outputs, effectiveness and efficiency. The one target that AGD assessed as ‘not achieved’, the ANAO did assess as meeting the requirements of section 16EA. The auditor’s report noted:

The inclusion of targets not complying with section 16EA(e) resulted in the Entity’s reporting of the performance measure…being misleading as it has the effect of changing the overall result for the achievement of this measure. If [these] targets … had not been included in the determination of the overall result it would have been assessed as not achieved.

Qualitative analysis through case studies — measures not free from bias

3.37 There were 34 targets based on work samples or surveys contributing to the overall assessment of 18 performance measures. The department could not demonstrate that the work samples or survey participants were selected in a way that was free from bias (subsection 16EA(c) of the PGPA Rule) for seven of those targets relating to five performance measures. The Auditor-General could not conclude whether the results for these measures accurately reflected the performance of the key activities to which the measures relate.

3.38 AGD had 27 targets relating to 18 performance measures where methodology and supporting evidence was provided to the ANAO which sufficiently mitigated the risk of bias. For the seven targets relating to five measures⁶⁴ that formed the basis for the qualification, the department could not provide this methodology. Further, it could not explain why pieces of work or survey participants were excluded from the selected population.

3.39 In its guidance on developing performance information, the Department of Finance explains that the method to support case study-based measures should be made clear to the reader in the corporate plan:

Case studies and details of information to be collected should … be decided at the time the corporate plan is developed and before information collection occurs. Sufficient information should be included in the corporate plan to provide confidence to the reader that the selection of case studies and reviews are unbiased.⁶⁵

Department of Social Services

3.40 The auditor’s report on the DSS 2020–21 annual performance statements was qualified with respect to eight out of 42 performance measures.

Inappropriate performance measures

3.41 DSS’ 2020–21 annual performance statements contain the key activity ‘Family Safety’ and a performance measure aimed at contributing to a reduction in violence against women and their children. The target for this measure relates to the ‘successful implementation of departmental actions’ to contribute to this goal. The auditor’s report states:

The target is a measure of the Entity’s activity and does not relate directly to the achievement of the Entity’s purposes related to [the] performance measure … which is contributing to a reduction in violence against women and their children. Furthermore, I was unable to obtain sufficient appropriate evidence over the completeness and accuracy of the Entity’s records in regard to the reported achievement of this measure.

3.42 In part, this example demonstrates that entities need to be able to assess what the measure seeks to achieve. In this case, where the measure aims to contribute to a reduction in violence against women and their children, the target must be able to assess whether and how this occurred. In addition to this problem with the construct of the measure, DSS could not provide evidence to explain the methodology it used to assess the ‘successful implementation of departmental actions’.

3.43 DSS’ performance measure relating to volunteer grants recipients in the ‘Volunteering and Community Connectedness’ key activity is based on a survey of grants recipients. The survey question asks: ‘Did the grant make it easier for your organisation to achieve its goals?’. However, prior to the survey volunteer grant recipients were required to sign an agreement with DSS to confirm that the grant funding would be used to purchase items ‘consistent with the purpose of your organisation’. The auditor’s report concludes:

Given that the terms of the agreement under which the grant payments were made required recipients to commit to using grant funds to achieve their goals, I have assessed that the survey question and target is not free from bias.

Ineffective supporting frameworks to develop, gather, assess, monitor, assure and report

3.44 DSS’ 2020–21 annual performance statements contain six performance measures that rely on information provided by third party service providers who are in receipt of grant funding from the department. The auditor’s report noted that DSS did not have adequate internal controls in place to validate the accuracy of the results reported by service providers and mitigate the risk that the reported results were not unbiased. The ANAO was not able to perform alternative audit procedures to determine whether the results reported against these six measures were accurate and complete, and supported by appropriate records.

3.45 Four of these six measures are based on information from DSS’ Data Exchange (DEX). Third party service providers enter information into DEX based on their judgment of their clients’ progress or their clients’ requests for assistance. DSS does not ensure that these judgments or data are accurate. Without controls in place to validate the accuracy of the results reported by service providers, there is a risk that the reported results may contain bias (subsection 16EA(c)).

3.46 Two of the six measures are based on surveys conducted by peak bodies. DSS does not assess the survey methodology prior to the survey being conducted, nor does it receive the source data from the peak bodies to ensure that the result provided is accurate. For these measures, DSS collates the results received from surveys conducted by different organisations using different survey questions and structures to report a single result. They are not an unbiased basis for measuring and assessing a result (subsection 16EA(c)).

Department of Veterans’ Affairs

3.47 The auditor’s report on DVA’s 2020–21 annual performance statements was qualified with respect to two out of 43 performance measures.

Inability to obtain sufficient evidence to support the result

3.48 DVA’s 2020–21 performance statements contain a measure relating to students’ progress through their education or career training. DVA’s records to support the result were incomplete with the department unable to provide evidence to support the current or previous year enrolment details for sampled students. The reported result was based on the presumption that students had progressed through the relevant level of education.

3.49 For the performance measure relating to support through Open Arms (a telephony contact centre), DVA was unable to provide records to show it had appropriate controls to assure the reliability of the underlying systems from which the performance measure was reported. The ANAO was therefore unable to obtain sufficient appropriate audit evidence to assess the completeness and accuracy of those records, and unable to conclude whether the results reported against this measure were accurate and complete, and supported by appropriate records.

4.1 This chapter highlights the progress that entities have made during the 2020–21 audits and the lessons for the sector as a whole in how to prepare and report their performance statements. The chapter then notes the progress that the ANAO made in 2021 to prepare for the expansion of the performance statements audit program, and the lessons the ANAO has gained to improve its processes and its engagement with the auditee.

Progress made by entities in 2021

Audit plans and performance measure documents

4.2 Entities were better prepared to prepare their 2020–21 annual performance statements than in the previous reporting period. In 2021 each entity prepared a plan and developed profiles (manuals) for their performance measures (chapter 3). These documents were important for producing higher quality performance statements for the Parliament. This is a positive development and gave entities greater confidence in the process of preparing the performance statements.

4.3 The performance measure profiles that were developed during the 2020–21 performance statements audits offer ongoing benefits for both the entity and the audit.

• Performance measure profiles will serve as a central source of performance information for the entity to plan and prepare its performance statements and to assist divisions to plan internally. The specific detail of each measure’s data sources, methodology and assurance processes should feed into the preparation plan’s schedule for the calculation of results, assurance activities and delivery of information to the ANAO. Entities’ preparation plans need more detail of this nature.
• For the audit, the receipt of this performance documentation at the outset of the engagement will enable the ANAO to provide its interim assessment sooner. This in turn will provide the entity with more time to consider revising its corporate plan in response to the ANAO’s findings.

Areas for entities to improve

4.4 In addition to the themes identified in chapter 3, the following are areas for entities to consider when preparing performance statements.

Governance and preparation processes

4.5 CFOs of Australian Government entities have a clearly defined role and set of responsibilities. The function and authority of the CFO is well-established within entities and is expressed clearly in Finance’s guidance. To fulfill their responsibilities, CFOs have a range of professional support networks to develop and build their skills and experience.

4.6 For performance management and reporting, there is no CFO-equivalent function. While there is generally a senior officer responsible for preparation of the entity’s annual performance statements, the authorising environment and systems and support structures are not as well-defined as for financial reporting. To provide Parliament and the public with high quality performance information, the performance reporting function and supporting systems will need to develop further and play a more proactive role in strategic planning and quality assurance. This would include:

• engaging with policy areas when programs are being designed to ensure quality performance measures are developed during the design process that meet the requirements of the Commonwealth Performance Framework;
• providing clear and consistent direction to business areas about the framework’s requirements for records to support results; and
• implementing and executing formal quality assurance processes of the draft performance statements and supporting evidence pack before this material is provided to the ANAO.

Record keeping

4.7 The Public Governance, Performance and Accountability Act 2013 (PGPA Act) requires the entity to gather and retain accurate records to support the preparation of an entity’s annual performance statements.⁶⁶ Each of the three entities received an audit finding relating to inadequate record keeping.

4.8 Entities’ plans should consider the risks associated with records being unavailable, incomplete or unreliable. The experience of the 2020–21 audits shows that these risks will be higher where a measure is based on:

• data and records administered by other entities or service providers;
• a jointly administered program that is due to close or closes during the period;
• an assessment by a third party, particularly where there is no agreement in place to provide records to the entity; and
• record-keeping without adequate controls in place, such as spreadsheets without clear protocols on use.

4.9 To alleviate these risks, entities should construct performance measures with record keeping requirements in mind. Measures reliant on third party data should be based on an agreement with the third party to provide the entity with timely access to records. Performance measure profiles should identify the processes and agreements in place to receive and verify the records to support each performance measure in time for the preparation of the performance statement.

Internal assurance processes

4.10 Entities should have processes in place to gain assurance over the reliability and verifiability of the data source and methodology, and the completeness and accuracy of results. The 2020–21 performance statements audits have highlighted the need for entities to improve their internal processes to gain this assurance.

4.11 Entities should keep records to demonstrate why and how internal assurance processes are undertaken. These records should assess whether the final data to inform the reported result was a complete and accurate reflection of the records in source systems.

4.12 Entities with performance measures that rely on third party systems and third-party assessments need to consider the risks associated with reporting an incomplete or inaccurate result against these measures in their performance statements. The risks will be higher where the third party is reporting (wholly or in part) on its own performance and where there are few, if any, checks of its assessments. Internal assurance records should set out these risk assessments.

The role of audit committees to drive improved performance reporting

4.13 The ANAO’s pilot program and the findings issued to the entities have highlighted the important role of audit committees in assurance of performance statements audits. Consistent with their role for financial statements, audit committees are well placed to question:

• whether each performance measure in the corporate plan meets requirements of the Public Governance, Performance and Accountability Rule 2014 (PGPA Rule), paying particular attention to the findings in the Interim Management Letter (IML) and the themes in chapter 3 of this report, and ensuring that they are addressed prior to the end of the year;
• the adequacy of the entity’s record keeping (as above);
• the entity’s quality control processes (as above); and
• the entity’s processes to provide assurance over the completeness and accuracy of results.

4.14 In considering these matters, it is important that audit committees also take into account final management and internal audit assurances.⁶⁷

Efficiency measures and the use of proxies

4.15 The 2019–20 and the 2020–21 performance statements audits have highlighted the difficulty that entities have in identifying and presenting efficiency measures. Most of the efficiency measures that have been presented have been ‘proxies’ for efficiency — they have not been measures of inputs invested for the outputs produced. Timeliness was a common efficiency proxy measure. Consistent with findings in the ANAO’s performance audit work, the development of efficiency measures is an opportunity for improvement across the public sector.

4.16 Entities need to consider how they can construct efficiency measures for outputs or results, which could involve linking money spent and resources applied to the results achieved. In the short-term, this may be based on the input and output data that is currently available to calculate a result. In the medium-term, entities could focus on constructing efficiency measures that relate to those parts of their business that are most important to driving the entity’s efficiency.

4.17 Finance guidance states that where proxies for efficiency measures are used, they should be identified as proxies in the corporate plan along with an explanation of why they are being used and why a proxy is suitable. In addition, the corporate plan should outline the entity’s approach to developing effectiveness and/or efficiency measures in the longer term.⁶⁸

Progress made by the ANAO in 2021

Applying an updated methodology and issuing timely auditor’s reports

4.18 Following the introduction of section 16EA of the PGPA Rule in February 2020, the ANAO incorporated these requirements into its audit methodology. As paragraph 1.36 noted, the 2020–21 performance statements audits demonstrated that, based on its audit methodology, an accessible and understandable independent assurance report can be issued.

4.19 The ANAO will continue to refine its audit methodology to enable the Auditor-General to provide the auditee with clear, concise and timely findings and Independent assurance reports in accordance with the framework. In particular, the ANAO will build into the methodology an assessment of the efforts that the entity is making toward resolving previous year findings where these findings are likely to take more than one reporting period to resolve. It will also include managing the audit process to ensure that Independent assurance reports are available to be included in entity annual reports.

4.20 The ANAO has established a new service group to build the performance statements audit practice. The Performance Statements Audit Services Group (PSASG) will build capability within the ANAO to deliver the phased implementation of the annual performance statements audit program and continue to refine the audit methodology.

Materiality

4.21 Materiality is an important consideration for both financial statements and performance statements audits. Information is material where, if omitted, misstated or not disclosed, it has the potential, individually or in aggregate, to influence the decisions of the users of the performance statements.

4.22 ANAO Auditing Standards require the auditor to consider materiality when planning and performing the audit engagement and evaluating whether the performance statements comply with the PGPA Act and the PGPA Rule.

4.23 While there is an accepted approach to assessing materiality for the purpose of preparing and auditing financial statements, the concept requires further consideration in relation to performance statements with experience. This recognises that performance reporting involves a mixture of measures (outputs, efficiency and effectiveness) over key activities (the activities that address the purposes of the entity) such that there is no central single driver of performance (unlike financial reporting where the overall performance and position of the entity can be summarised in monetary terms).

4.24 Notwithstanding, for the purposes of forming the Auditor-General’s audit conclusion, every performance measure identified in the PBS and the corporate plan that relates directly to the entity’s purposes or key activities is presumed to be material to the performance statements, unless otherwise specified by the entity in its PBS or corporate plan. This recognises that the entity has informed the Parliament of its purposes and intended results through the budget process.

Areas for the ANAO to improve

The need for the ANAO to establish communication protocols

4.25 The 2020–21 audits showed that the ANAO needs to increase awareness within the sector of its methodology to audit performance statements. In part, the current shortfall in awareness reflects the absence of a dedicated performance information officer function within entities and the relatively new requirements for entities’ performance information in sections 16E and 16EA of the PGPA Rule.

4.26 The 2020–21 audits also indicated that the ANAO needs to improve the way it communicates with entities on areas of potential qualification. Where qualifications are presented to the accountable authority, they should not come as a surprise. The onus is on the audit team to communicate the issues with the entity’s preparation team well in advance.

4.27 The ANAO will put in place communication protocols to ensure the entity understands how to address interim findings, particularly significant (A) and moderate (B) level findings. In the final phase of the audit, the protocol will direct ongoing ANAO communication with the preparation team to indicate whether the entity’s efforts to remediate the issues are adequate to resolve the findings.

Independent assurance reports in the entity’s Annual Report

4.28 Performance statements audits will have a timeline consistent with financial statements audits such that the Independent assurance report can be tabled in the auditee’s annual report by the end of October each year. Achieving this timeline will require:

• ongoing communication with the entity about what is required to meet each milestone;
• early engagement with the entity to flag areas of potential concern with the structure and content of the performance information in the corporate plan. This will enable the entity’s current (and the forthcoming) corporate plan to be amended;
• early receipt of performance measure profiles from the entity;
• an IML provided to the entity in March;
• communicating with entities to ensure they provide a quality assured evidence pack containing draft performance statements and supporting records; and
• the ANAO to indicate a milestone of signing the audit opinion by 30 September.

The need for the sector to collaborate

4.29 To date, there has been limited communication between entities about how to develop good performance information and how to plan and engage with a performance statements audit. However, the Department of Social Services (DSS) provided support and examples of program profiles to the Attorney General’s Department (AGD) as part of the 2020–21 cycle. DSS’ development of performance measure profiles was better practice and the support provided to AGD a leading example within the Australian public sector. It is important that entities have a forum where they are able to discuss these matters with other entities.

4.30 Finance and the ANAO have discussed the establishment of a performance statements Community of Practice. A Community of Practice will drive improvement in the effectiveness and the efficacy of the process and improve the profile and professionalism of the performance function within the sector.

4.31 A performance statements Community of Practice would be a forum for entities to discuss:

• constructing performance measures and targets;
• preparing an audit plan and performance measure profiles;
• providing assurance over the completeness and accuracy of results;
• keeping complete and accurate records;
• addressing the themes identified in chapter 3 of this report and the areas for improvement discussed in this chapter; and
• the best governance arrangements to engage in a performance statements audit.

The possibility of an annual JCPAA inquiry

4.32 For more than a decade, an annual Joint Committee of Public Accounts and Audit (JCPAA) inquiry into the prior year’s Major Projects Report (MPR) — prepared jointly by the Department of Defence and the ANAO — has enabled the Parliament to understand the progress and the risks associated with the largest Defence acquisition projects. Parliamentary scrutiny has also developed the quality of the MPR as a product for the Parliament.

4.33 The performance statements audit program and methodology are still in an early stage. The Finance Minister has noted that a JCPAA inquiry to review the audit methods and outcomes each year during the roll-out of the performance statements pilot program would inform incremental improvements in the program and practice and inform the design of legislation going forward.⁶⁹

4.34 Any findings on the approach and scope of performance statements audits arising from the JCPAA inquiries could inform the design of legislation to permanently embed them as part of the audit program.

Appendix 1 PGPA Act requirements to prepare key documents

Appendix 2 The Commonwealth Performance Framework

Source: Department of Finance, Commonwealth Performance Framework. Reproduced in Portfolio Budget Statements 2020–21

Appendix 3 Central agency guidance on performance reporting

1. Central agencies in the UK, New Zealand and South Africa have produced handbooks to guide entities through the performance statements process. This includes guidance on how to design compliant performance measures, how to ensure the quality of performance data and how to evaluate policies and programs.

2. In the UK, for example, the Public Value Framework⁷⁰ guides public organisations in their planning, everyday decision-making and reporting processes. Instead of focusing on quantifying the inputs and outputs of departments and observing the relationship between them, the framework focuses on the process of improving the value of public money expenditure.

3. The objective of the South African National Treasury’s Handbook is to ‘provide guidance to improve the appropriateness, availability and quality of programme performance information (PI).’⁷¹ At a minimum, organisations must comply with National Treasury Regulations which require a quarterly report on programme performance to be delivered to the Executive.⁷²

4. Central agencies in Canada and the USA are developing policy and program support for the legislative framework. In Canada this has taken the form of the Treasury Board of Canada Secretariat amending its directives in 2016 to introduce a Policy on Results and a Directive on Results. The objective of this Policy is ‘improving the achievement of results across government; and enhancing the understanding of the results government seeks to achieve, does achieve, and the resources used to achieve them.’⁷³

5. In the USA, the Office of Management and Budget (OMB) worked with 35 government agencies to support the process of implementing the Government Performance and Results Modernization Act of 2010.⁷⁴ This resulted in the publication of a Federal Program Inventory that centrally collates information on government programs with the goal of achieving greater transparency.⁷⁵

6. The USA and Canada have prioritised publication and the making of individual entity performance and whole-government performance information accessible to the public. This includes the Canadian government’s interactive website InfoBase⁷⁶ which collates and reports every performance measure result.⁷⁷

Appendix 4 The performance statements audit process

Appendix 5 Audit findings rating scale

Appendix 6 Findings by theme — 2020–21 audits

Findings by theme — 2020–21 audits

Theme of qualification		Significant	Moderate	Minor	Main areas of weakness
Construct of measure	Wording and construct of the measure and supporting methodology are inherently biased				Target and supporting methodology unable to support an objective assessment of the measure as it is worded
	Unclear alignment of purposes, key activities and measures				There is not a clear and understandable link between the purpose, the key activities that assist in achieving the purpose and the performance measures.
	Use of composite measures without adequate explanation				Multiple targets reported against a performance measure and there is no additional information provided to the reader to outline how the targets will be assessed and weighted.
	Vague wording of measures and targets				Reference to the word timely without providing expected timeframes to be assessed against. Target is not providing a reader with an understanding of what is required to achieve performance.
	Measure and target not aligning — target not able to assess the aim of the measure				Performance measure and the target do not align. Wording of measure and target are unable to be supported by evidence.
	Explanation and disclosure of efficiency proxies				Failure to articulate the proxies that are being used and demonstrate why the proxy measure is suitable and outline the entity’s approach to the development of effectiveness and/or efficiency measures in the longer term.
Methodology	16EA(b) Reliable				Inadequate controls in place to validate the accuracy of the results reported by third parties and mitigate the risk that the reported results may contain bias. The basis for measurement and assessment is therefore unreliable. Failure to identify the population. Failure to implement a suitable method for selecting case study parameters increases the risk of bias, as does poorly defined targets. Inadequate processes to manage and maintain records
	16EA(b) Verifiable
	16EA(c) Unbiased
	16EA(d) Qualitative and Quantitative Measures				Absence of qualitative and quantitative measures, and efficiency and effectiveness measures at an identified level (Outcome level, key activity level). Need to replace efficiency proxies with efficiency measures.
	16EA(e) Measures of Outputs, Effectiveness and Efficiency
Performance measures not meeting the intent of the framework	16EA(e) Measures assessed as activities				Composite measures containing targets that are assessed as activities rather than outputs.
Disclosure and Presentation	Non-disclosure of survey response rates and methodologies				Non-disclosure of survey response rates and methodologies. Failure to identify population and sampling approaches for survey-based measures.
	Presentation of performance information not meeting requirements of the Rule				Performance statements not consistent with the ‘concise requirement’ of PGPA Rule subsection 17AC(2). Presentation of performance statements would benefit from use of tables, graphs, diagrams, charts.
	Inadequate processes for corporate plan revision				Inadequate processes to identify need for, and timing of, variation to corporate plan.
Completeness and Accuracya	Inability to provide assurance over completeness and accuracy of results				Inability to provide assurance over the accuracy and completeness of the result and ANAO unable to perform alternative audit procedures to determine whether the results reported were accurate and complete. Inadequate risk-based quality assurance framework to ensure reported results are reliable, verifiable and unbiased.
	Inability to provide assurance over systems				Inability to identify systems used by third parties to report results. Inability to provide assurance over systems used by third parties to provide results.
	Inability to provide sufficient and timely evidence to support result				Insufficient records to verify the completeness and accuracy.
	Poor record keeping processes including inadequate oversight by management				Inadequate evidence that management has overseen record keeping and processes to establish the methodology for performance measures.

Note a: Completeness and accuracy relates to the audit procedures for testing the reported results

Source: ANAO analysis

Appendix 7 Qualified measures by theme, 2020–21 audits

Note a: Single measures may have been qualified under multiple themes

Note: Some measures are composite measures and are represented twice on the basis of assessments of multiple targets