Introduction

1. The Organisation for Economic Co-operation and Development (OECD) has observed that:

Measuring government performance has long been recognised as necessary for improving the effectiveness and efficiency of the public sector. Following the fiscal and economic crisis that began in 2008, however, accurate and timely data are needed more than ever to help governments make informed decisions about how and where to prioritise spending, reduce costs and promote innovation in the public administration.¹

2. Adequate performance information, particularly in relation to program effectiveness, allows entities to assess the impact of policy measures, adjust management approaches as required, and provide advice to government on the success, shortcomings and/or future directions of programs. This information also allows for informed decisions to be made on the allocation and use of program resources. In addition, performance information and reporting enables the Parliament and the public to consider a program’s performance, in relation to both the impact of the program in achieving the policy objectives of the government and its cost effectiveness.

3. Performance reporting regimes have been in place in many OECD countries, including Australia, since the mid-1980s. Over time, there has been a trend to move away from a narrow focus on reporting on financial inputs, towards integrated models that are intended to provide a clearer picture of the results or outcomes that have been achieved from the expenditure of public money—in other words, whether the outcomes or the impacts sought by government are being realised.

4. As the demands for better quality information about public sector performance increase, the need for the ongoing development and refinement of performance measurement and reporting frameworks has continued. The Australian Government’s Outcomes and Outputs framework was introduced as part of the 1999–2000 budget process, requiring entities to specify intended outcomes, and to measure and report on actual performance. While the requirement for entities to clearly specify and report against ‘outcomes’, ‘programs’, ‘deliverables’, and ‘key performance indicators’ has been maintained over the last decade, implementation within entities continues to require more focus and attention. The Outcomes and Outputs framework was replaced in 2009–2010 by the Outcomes and Programs framework, with the new emphasis being on entities identifying and reporting against the programs that contribute to government outcomes over the Budget and forward years. The transition to a programs focus was intended to improve entity reporting and to clearly demonstrate entities’ achievement against pre-defined program objectives.

The Outcomes and Programs framework

5. The Australian Government Outcomes and Programs framework requires entities to firstly identify, and secondly report against, the programs that contribute to government outcomes over the Budget and forward years. A central aspect of this approach is the development of clearly specified outcomes, program objectives and appropriate key performance indicators (KPIs).

6. KPIs are expected to enable users to assess the achievements of the entity against the stated program objectives and collectively, their contribution to stated outcomes.

7. The Outcomes and Programs framework provides a focus on KPIs that measure the effectiveness of government programs. It is not expected that KPIs, within the context of the Outcomes and Programs framework, will measure inputs to a program (resources provided to administer the program), or the outputs (that is, quantity and quality indicators which are related to the deliverables²).³

Previous ANAO reports

8. The Australian Government performance measurement and reporting framework has been the subject of various Australian National Audit Office (ANAO) performance audits. Audit reports have identified that the development of performance indicators, including implementing effective systems and practices to capture, monitor and report against them, is an area that requires ongoing management attention, particularly with respect to the clarity of the specification of outcomes and program objectives and the measurement of entities’ impact.

9. In ANAO Audit Report No.5 2011–12 Development and Implementation of Key Performance Indicators to Support the Outcomes and Programs Framework (ANAO Audit Report No.5 2011–12), the ANAO provided an assessment of the development and implementation of KPIs by government entities. The findings of the audit indicated that many of the entities reviewed continued to find it challenging to develop and implement effectiveness KPIs which are intended to provide quantitative and measurable information, to allow for an informed and comprehensive assessment and reporting of achievements against stated objectives.⁴

10. Other performance audits undertaken by the ANAO have also highlighted the need to strengthen program performance measurement. This includes establishing frameworks within entities to guide performance measurement activity, the importance of developing appropriate baselines and benchmarks early in program implementation⁵, informing internal and external stakeholders on progress and outcomes, and strengthening program management and accountability.⁶

The Joint Committee of Public Accounts and Audit inquiry into the Auditor-General Act 1997

11. In February 2009, the Joint Committee of Public Accounts and Audit (JCPAA) resolved to ‘review whether the provisions of the Auditor-General Act 1997 remained adequate in the modern public sector environment⁷, noting at the time that eight years had passed since the Committee’s last such review. A key recommendation of this inquiry was directed at ‘enhancing the Auditor-General’s role in reviewing the adequacy of agencies’ performance indicators’.⁸

12. In light of the report by the JCPAA, in December 2011, Parliament amended the Auditor-General Act 1997 to provide the Auditor-General with explicit authority to undertake audits of both the appropriateness of entities’ KPIs and the reporting by entities against those indicators.

13. The amendments to the legislation reflect Recommendation 3 of the JCPAA Report 419 Inquiry into the Auditor-General Act 1997, which stated:

That the Act be amended as necessary to enable the Auditor-General to review an agency’s compliance with its responsibilities for a sub-set of performance indicators. Proposed performance indicators should be identified annually by the Auditor-General and forwarded to the Parliament, via the JCPAA for comment, in a manner similar to the annual performance audit work program for the ANAO (p. 25).

14. To respond to Parliament’s interest in KPIs, ANAO performance audits will continue to include coverage of the performance measurement arrangements that have been established for the area being reviewed.9 In addition, as a consequence of the amended legislation, the ANAO has also invested in a pilot project, the subject of this report.

KPI audit pilot project

15. The KPI audit pilot project (the Pilot) was designed to assess the status of the Australian Government performance measurement and reporting framework as a basis for implementation of a future program of audits of entities’ KPIs, and to develop a suitable audit methodology. At the time of conducting the Pilot the ANAO recognised that the development of a full KPI assurance audit methodology and its implementation will take some time, and financial support.

16. To date the Pilot has been funded by the ANAO from its existing resource base. The Pilot project is expected to conclude in 2013–14. The future development and implementation of a broader ANAO KPI audit work program will require the ANAO to be appropriately resourced and in light of proposed enhancements to the Australian Government performance measurement and reporting framework, as presented by the Department of Finance and Deregulation (Finance) Commonwealth Financial Accountability Review (CFAR) discussion paper (March 2012) and subsequent position paper (November 2012).

17. This report presents the ANAO’s response so far to the legislative amendments of the Auditor-General Act 1997. The Pilot included:

• assessing the Australian Government performance measurement and reporting framework as a basis for implementation of a future program of audits of entities’ KPIs;
• reviewing the approaches taken by other relevant jurisdictions in the implementation of KPI audit methodologies;
• working with the responsible Australian Government central agencies; and
• testing a KPI audit approach and methodology within three entities.

18. Four entities participated collaboratively in the Pilot. The Department of Finance and Deregulation contributed in view of its role in being responsible for administering the Outcomes and Programs framework. The Australian Taxation Office, the Department of Education, Employment and Workplace Relations, and the Department of Health and Ageing, contributed as entities with experience in applying the principles of the Outcomes and Programs framework. For these three entities, one outcome and program from each was selected for assessment.

19. The Pilot also benefited from the experiences of a number of audit offices that have work programs with a performance information focus. In particular, the Office of the Auditor General for Western Australia and the Office of the Auditor-General of New Zealand have provided information and support.

Overall conclusion

20. Non-financial performance information allows entities to assess the impact of policy measures, adjust management approaches as required and provide advice to government on the success, shortcomings and/or future directions of programs. Non-financial performance information also assists the Parliament and the public in assessing whether policy goals have been achieved and how effectively the public sector has performed. The capacity to undertake this analysis is dependent on the provision of high-quality and robust performance information. Australian Government entities currently report against some 650 government programs that are measured by approximately 3 500 KPIs. However, it is still difficult to get an accurate picture of the performance of programs across the public sector.¹⁰

21. The results of the Pilot, conducted in the light of the amendments to the Auditor-General Act 1997, reinforced the issues identified in ANAO Audit Report No.5 2011–12 and other ANAO performance audits. The assessment of entity performance measurement and reporting identified that entities continue to experience challenges in developing and implementing meaningful KPIs, and that the administrative framework supporting the development and auditing of KPIs remains problematic. This is similar to the experience of some other jurisdictions where a robust system of KPI reporting and auditing has taken some years to develop.

22. The broad range of roles and activities undertaken by Australian Government entities highlights that while some entities’ programs suit the development of relatively straightforward performance information, the less tangible nature of the objectives of some other programs is more of a challenge.¹¹ A homogenous framework for application by all Australian Government entities, without recognition of the variety of entity activity/ies, has compounded the challenges that entities have in implementing the Australian Government performance measurement and reporting framework. For example, in a purchaser/provider model one entity will be responsible for the oversight of a policy initiative/s and be better suited to measuring the impact through KPIs, while another entity is responsible for the delivery of services and will be better suited to a focus on measuring deliverables. The development of a framework that accommodates this diversity, providing entities the ability to develop and report appropriate performance information regardless of their role, is critical.

23. It is also apparent that the Outcomes and Programs framework would benefit from consideration of intermediate objectives where an overall outcome can only be achieved over the longer-term. In these circumstances it may be necessary to relate targets associated with effectiveness indicators to milestones that demonstrate progress towards the program objective. This has been an issue which has been raised by the ANAO over many years, but where greater emphasis is required to obtain a stronger focus on the impact of programs.

24. Additionally, a more comprehensive model for performance measurement and reporting in the Australian Government would include consideration of the development and implementation of ‘efficiency’ indicators to complement the ‘effectiveness’ indicator focus within the current model.

25. The promulgation of updated, comprehensive guidance will be central to supporting the development and implementation of appropriate KPIs by public sector entities. It is also clear from the Pilot that the current framework and accompanying guidance does not provide an effective framework against which entities’ KPIs can be reliably evaluated through an assurance audit process, as it does not specify clear standards or criteria that KPIs should satisfy. That said, it does need to be recognised that the current framework was not designed with this specific purpose in mind.

26. Performance reporting is most effective in informing the government, the Parliament, and the public when based on clearly expressed outcome statements, program objectives, deliverables and KPIs. The Pilot found there was room for strengthening the Pilot entities’ performance reporting frameworks which may require a further investment in resources in order to improve the quality of the performance information contained in the entities’ annual reports.

27. The Pilot also confirmed the need for focused and clear outcome statements and well defined program objectives as important in allowing the development of appropriate KPIs. Entities’ periodic review of both outcomes and objectives, together with adherence to Finance guidance, will contribute to meeting the need for focus, clarity and well defined outcome statements and program objectives. The entities engaged in the Pilot were receptive to the ANAO’s feedback and planned to revisit their current approaches where required.

28. Implementation of a systematic assurance audit of the appropriateness of entities’ KPIs, and the completeness and accuracy of their reporting is a process that will take time as entities continue to develop and refine their KPIs. The focus of the ANAO in this area through both performance audits and the specific audits of KPIs can be expected to lead to improvements in the quality of performance information provided to the government, the Parliament, and the public in the longer-term.

29. The Australian Government is taking steps to address the long‑standing performance measurement and reporting challenge. On 8 December 2010 the Minister for Finance and Deregulation announced the CFAR initiative with the purpose to ‘analyse the Commonwealth financial framework from first principles and consider options to ensure the framework supports high quality resource management’.¹² CFAR provides the opportunity to consider how to position the current performance measurement and reporting framework to respond to a range of contemporary issues in public administration.

30. Our audit reports and the Pilot show it is time for greater attention, investment and resourcing to be given to the quality and integrity of KPIs used by public sector entities to inform decisions about the performance of government programs. This requires a stronger and sustained focus by entities to enhance KPIs, and support provided by Finance through improved guidance. Entity leadership will be critical to success here. Encouragement to achieving better performance measurement can also be given by the Government and the Parliament through reviews, inquiries and the questions asked about the changes being brought about by specific programs having regard to the program’s objectives.

31. The timetable for the development of a suitable approach for the formal audit of KPIs is subject to consultation with the JCPAA, the Government and the appropriate resourcing of the ANAO. In 2013–14, the ANAO will continue to invest resources in developing and refining the approach for the systematic audit of the appropriateness of entities’ KPIs, as part of the pilot project.

Footnotes

[1] Organisation for Economic Co-operation and Development, Government at a Glance, 2011.

[2] Program deliverables (outputs) are the goods and services produced and delivered through the implementation of a program.

[3] Department of Finance and Deregulation, Guidance for the Preparation of the 2011–12 Portfolio Budget Statements March 2011 [Internet], Finance, p. 29. available from http://www.finance.gov.au/budget/budget-process/docs/ Guidance_for_the_Preparations_of_the_2011-12_Portfolio_Budget_Statements.pdf [accessed 21 March 2013].

[4] ANAO Audit Report No.5 2011–12, Development and Implementation of Key Performance Indicators to Support the Outcomes and Programs Framework, p.17.

[5] ANAO Audit Report No.2 2012–13, Administration of the Regional Backbone Blackspots Program, p. 23.

[6] ANAO Audit Report No.30 2010–11, Digital Education Revolution Program––National Secondary Schools Computer Fund, p. 100.

[7] Joint Committee of Public Accounts and Audit, Report 419 Inquiry into the Auditor-General Act 1997, JCPAA, Canberra, December 2010, p. vi.

[8] ibid.

[9] ANAO, Audit Work Program July 2012 [Internet], ANAO, p. 14. available from http://www.anao.gov.au/About-Us/~/media/Uploads/Audit Work Program/2012 Audit work program.pdf [accessed 10 April 2013].

[10] Department of Finance and Deregulation, Commonwealth Financial Accountability Review, position paper, Sharpening the Focus: A Framework for Improving Commonwealth Performance, November 2012, p. 16.

[11] ANAO Audit Report No.5 2011–12, Development and Implementation of Key Performance Indicators to Support the Outcomes and Programs Framework, p. 52.

[12] Department of Finance and Deregulation, Commonwealth Financial Accountability Review, Introduction [Internet], Finance, available from http://www.cfar.finance.gov.au/2012/03/22/introduction/#more-39 [accessed 10 April 2013].