Introduction

1. The Organisation for Economic Co-operation and Development (OECD) has observed that:

While measuring government performance has long been recognised as playing an important role in increasing the effectiveness and efficiency of the public administration, following the economic crisis and fiscal tightening in many member countries, good indicators are needed more than ever to help governments make informed decisions regarding tough choices and help restore confidence in government institutions.¹

2. Performance reporting regimes have been receiving increasing attention in many OECD countries, including Australia, since the mid-1980s. Over time, there has been a trend to move away from a narrow focus on reporting on financial inputs, towards integrated models that are intended to provide a clearer picture of the results or outcomes that have been achieved from the expenditure of public money—in other words, whether the outcomes or the impacts sought by government are being realised. With the current focus on budget and policy priorities, performance information, particularly concerning the impact of government policies where such information is available, can be expected to be a key input into decisions by both government and government agencies.

3. Measuring the impact of programs, or outcomes, can have many benefits and provides performance information that measurement of inputs (resources invested) and outputs (deliverables) alone cannot. Importantly, outcome measurement provides information about the effectiveness of programs or services and supports the longer-term evaluation of programs. In addition, performance information can also inform decisions on the efficiency of delivery models adopted to achieve desired policy outcomes.

4. In essence, performance measurement can:

• help clarify government objectives and responsibilities;
• promote analysis of the relationships between agencies and between programs, enabling governments to coordinate policy within and across agencies;
• make performance more transparent, and enhance accountability;
• provide governments with indicators of their policy and program performance over time;
• inform the wider community about government performance; and
• encourage ongoing performance improvements in service delivery and effectiveness, by highlighting improvements and innovation.²

5. Performance measurement and reporting is expected to be an integral component of agencies’ governance arrangements, which provides information for both internal management purposes and external accountability. Dr Ian Watt AO, Secretary of the Department of the Prime Minister and Cabinet, has highlighted that:

... we do need to embrace opportunities to be a more effective and smarter public service, and a more stringent fiscal situation is a powerful signal to us for change ...

But we cannot, and should not, simply wait for the [National] Commission [of Audit]³ to do the thinking for us. We should already be adjusting priorities to meet the current fiscal environment and a new government and thinking about major changes that might come. We should be not only assessing the way we work, but also what we are working on; we should be reviewing our critical functions; and looking at what we can, and should, stop doing.⁴

6. At a time when the demands on government exceed available resources, there is a premium on leadership, effective governance and longer-term planning by government agencies in order to provide sound advice for government and efficiently deliver government policies and programs.

The Outcomes and Programs framework

7. Since the mid-1980s in Australia, public sector management frameworks have emphasised the importance of measuring program performance. While the frameworks have changed over the years, the fundamental goals have remained largely consistent—to be able to measure and assess the impact of government programs; some frameworks have also included a focus on program efficiency.

8. In 2009–10, the Australian Government’s Outcomes and Programs framework replaced the previous Outcomes and Outputs framework, which was originally introduced as part of the 1999–2000 budget process.⁵ The additional emphasis under the new framework being on agencies identifying and reporting on the impact of the programs that contribute to government outcomes over the Budget and forward years. A central aspect of this approach is the development of clearly specified outcomes, program objectives, deliverables and appropriate key performance indicators (KPIs) to enable users to assess an agency’s progress towards the stated program objectives and collectively, their contribution to stated outcomes. It is not expected that KPIs will measure inputs to a program (resources provided to administer the program), or the outputs (that is, quantity and quality indicators which are related to the deliverables⁶) as in the previous framework.⁷

Public Governance, Performance and Accountability Act 2013

9. In 2013 the Australian Government introduced the Public Governance, Performance and Accountability Act 2013 (PGPA Act) as part of broader reforms to the Commonwealth’s financial framework implemented following the Commonwealth Financial Accountability Review. The PGPA Act includes a number of requirements with respect to planning, budgeting and non-financial performance assessment and reporting. These include:

• the measurement and assessment of the performance of agencies against their corporate plan;
• the preparation of annual performance statements (to be included in agencies’ annual reports); and
• a responsible Minister, or the Finance Minister, may request the Auditor‑General to examine and report on the agency’s annual performance statement (the report is to be tabled in Parliament).

10. While the Department of Finance (Finance) is still developing the rules to underpin the PGPA Act, they would be expected to continue the focus on measuring program effectiveness, and potentially be complemented by measures to assess the efficiency of program delivery.

Previous ANAO reports

11. The Australian Government performance measurement and reporting framework has been the subject of various Australian National Audit Office (ANAO) performance audits. ANAO Audit Report No.5 2011–12 Development and Implementation of Key Performance Indicators to Support the Outcomes and Programs Framework, provided an assessment of the development and implementation of KPIs by government agencies. The findings of the audit indicated that many of the agencies reviewed continued to find it challenging to develop and implement effectiveness KPIs to allow for an informed and comprehensive assessment and reporting of progress towards stated objectives.⁸

12. Other performance audits undertaken by the ANAO have also highlighted the need to strengthen program performance measurement. Over the last two financial years (2011–12 and 2012–13), 48 per cent of ANAO performance audit reports included recommendations which focused on the need for better program effectiveness measures.⁹ In addition, audit findings and recommendations have highlighted the importance of developing an evaluation strategy at the outset of program implementation¹⁰; the need for a methodology for collecting and reporting on performance data early in program implementation, including establishing baselines and benchmarks¹¹; ensuring the availability of timely, accessible and comparable data¹²; and the development of interim performance indicators to inform internal and external stakeholders on the progress of a program towards achieving planned outcomes.¹³

13. In December 2011, amendments to the Auditor-General Act 1997 provided the Auditor-General with the explicit authority to conduct audits of the appropriateness of agencies’ key performance indicators and the completeness and accuracy of their reporting. As a consequence, the ANAO initiated a pilot project to audit KPIs (2011–12 Pilot), which is the subject of ANAO Report No.28 2012–13 The Australian Government Performance Measurement and Reporting Framework, Pilot Project to Audit Key Performance Indicators (ANAO Report No.28 2012–13), tabled on 23 April 2013.

14. The 2011–12 Pilot concluded that agencies continued to experience challenges in developing and implementing meaningful KPIs and that the administrative framework supporting the development and auditing of KPIs remained problematic. The report highlighted that the key areas where the administrative framework required further consideration included:

• the suitability of a homogenous framework for application by all Australian Government agencies, without recognition of the variety of agency activity/s;
• cross-agency performance reporting, where one agency is responsible for the development of the policy initiative, and actual delivery of services is carried out by a separate agency;
• the development of intermediate objectives where an overall outcome can only be achieved over the longer-term;
• the development and implementation of ‘efficiency’ indicators to complement the ‘effectiveness’ indicator focus within the current model;
• the need for focused and clear outcome statements and well defined program objectives; and
• the need for greater rigour within the framework to support provision of independent assurance over agencies’ performance measurement and reporting.

15. The 2011–12 Pilot also observed that there remained multiple sources of policy and guidance, rather than a single reference document, that consolidated the requirements for the Outcomes and Programs framework.¹⁴

16. In relation to agencies’ implementation of the framework, the 2011–12 Pilot also found there was room for strengthening the 2011–12 Pilot agencies’ performance measurement and reporting, in order to improve the quality of the performance information contained in the agencies’ annual reports, and confirmed that clear outcome statements and well defined program objectives are important for the development of appropriate KPIs.¹⁵

The Joint Committee of Public Accounts and Audit

17. The Joint Committee of Public Accounts and Audit (JCPAA) examines the reports of the Auditor-General which have been tabled in Parliament.¹⁶ The JCPAA’s examination includes reviews of reports. The selection is based on a range of factors including: program significance in relation to government policy; public interest and impact on the community; the significance of the Auditor-General’s findings; and the value of program expenditure.¹⁷

18. As part of the JCPAA’s ongoing focus on the measurement of government performance, and in light of the reforms introduced through the Commonwealth Financial Accountability Review, the Committee reviewed ANAO Report No.28 2012–13. In June 2013, the JCPAA tabled its findings in Report 439, Review of Auditor-General’s Reports Nos.11 to 31 (2012–13).

19. The Committee reinforced the importance of KPIs for: informing broader evaluations; continuous reporting; and project management discipline, while highlighting the need for other mechanisms such as independent evaluations and capability reviews.¹⁸ The Committee made two recommendations outlined below.

Source: Joint Committee of Public Accounts and Audit, Report 439, Review of Auditor-General’s Reports Nos.11 to 31 (2012–13), JCPAA, Canberra, 2013, p. 30.

20. At the time of preparing this report, a response had not been provided to the JCPAA regarding the recommendations made in Report 439. However, Finance advised the ANAO on 13 February 2014, that the department is progressing the development of a more coherent performance management framework through the Public Management Reform Agenda.¹⁹ Further, Finance advised that the department:

... will continue to consult with entity and government representatives on the development of relevant rules and associated guidance materials which will be finalised during 2014. This will include guidance for measuring and assessing entity performance, guidance and templates for entities to produce both Corporate Plans and Annual Performance Statements, and revamped policies for the elements of the Outcomes and Programmes Framework. This material will be presented to the government for consideration during the second half of 2014. Together these near-term measures will lead to an enhanced Australian Government monitoring and evaluation framework. The process of developing these materials will itself focus agencies on their current practices, and on bringing these up to a higher standard.²⁰

Pilot project to audit key performance indicators

21. In 2012–13, the ANAO pilot project to audit KPIs was continued with the objective of conducting a review of framework developments, both in terms of the clarity of the policy and guidance issued by Finance and the performance of agencies in applying this policy and guidance, as a basis for implementing a future program of audits; and to further develop and test an audit methodology to address the practical challenges of assessing the appropriateness of KPIs, and their complete and accurate reporting. The pilot project included:

• a cross-agency survey assessing Australian Government agencies’ approaches and attitudes towards performance measurement and reporting more broadly, including internal governance arrangements and the extent to which performance measurement is used in agencies’ management decision-making processes; and
• an examination of the 2012–13 and 2013–14 Portfolio Budget Statements of the three agencies participating in the pilot project, and the completeness and accuracy of the KPI data and disclosures in the agencies’ annual reports.

22. In 2012–13, 16 agencies participated in the cross-agency survey. These agencies were selected from the major Australian Government departments, prescribed FMA Act agencies and included those who participated in the continuation of the pilot project. The results of the survey are discussed in Chapters 2 and 3 of this report.

23. Additionally, three agencies participated collaboratively in the continuation of the pilot project. The Australian Taxation Office, the Department of the Environment, and the Australian Federal Police contributed as agencies with experience in applying the Outcomes and Programs framework. This allowed the ANAO to continue to review different approaches to the implementation of the Outcomes and Programs framework. During the course of the pilot project, Finance was also consulted on range of matters in view of the department’s role in administering the Outcomes and Programs framework.

Overall conclusion

24. Adequate performance information, particularly in relation to program effectiveness, allows agencies to assess the impact of policy measures, adjust management approaches as required, and provide advice to government on the success, shortcomings and/or options for revisions to current policies. This information also allows for informed decisions to be made on the allocation and use of public resources. In addition, performance measurement and reporting enables the Parliament and the public to consider a program’s performance, in relation to both the impact of the program in achieving the policy objectives of the Government and its efficiency.

25. ANAO performance audits have consistently reported that agencies have fallen short in delivering and reporting performance measures that demonstrate the extent of progress against stated program objectives. ANAO Report No.28 2012–13 The Australian Government Performance Measurement and Reporting Framework, Pilot Project to Audit Key Performance Indicators, reinforced the issues identified in ANAO performance audits over the last decade. The report concluded that the administrative framework supporting the development and auditing of KPIs remains problematic and that agencies continued to experience challenges in developing and implementing meaningful KPIs.

26. The continuation of the pilot project observed little change within the guidance promulgated by Finance for the 2013–14 financial year and observed that agencies’ implementation of performance measurement and reporting requires further development. The need for clearer guidance and greater support for agencies was also a theme in the cross-agency survey undertaken by the ANAO as a part of the pilot project. Consequently, the key findings previously made by the ANAO and the JCPAA remain largely unaddressed at the time of this report.

27. The PGPA Act, which takes effect on 1 July 2014, introduces new requirements for agencies to produce a corporate plan, and to measure and assess their performance and prepare an annual performance statement. While the first of the underpinning rules for the PGPA Act are scheduled to be submitted to the JCPAA in early 2014 for review, they are not expected to take full effect until the 2016–17 financial year. In the interim, to improve the standard of performance measurement and reporting, there would be considerable benefit in Finance providing enhanced guidance for the Outcomes and Programs framework, noting that agencies will continue to be required to report on performance on an annual basis under the current framework. It would be expected that, in the light of the implementation of the PGPA Act, any revised guidance would support the longer-term developments.

28. The results from the cross-agency survey undertaken by the ANAO support previous findings that there remains substantial work by agencies to implement the performance measurement and reporting framework and that agencies have difficulty developing KPIs that measure the impact or effectiveness of a program’s contribution to achieving government outcomes. Agencies identified that the most readily available performance information covered either activities or processes to support the delivery of programs or the quantity of outputs delivered, and not the effectiveness KPIs required by the Outcomes and Programs framework.

29. The results from the cross-agency survey also highlighted that the most commonly identified factors that contributed to an effective Outcomes and Programs framework included: alignment of agencies’ frameworks with the objectives of government; having an established, stable and well understood framework; and flexibility to accommodate uncertainty and changes in the operating environment. The most common message in achieving effective implementation of the framework was the engagement of senior leadership. Other areas for improvement identified by agencies included: the need for improved guidance, training/workshops and tools/templates to support agencies in implementing frameworks; the importance of conducting regular reviews of frameworks; careful up-front planning and selection of outcomes and deliverables; and the need for transparent targets that are openly reported.

30. The continuation of the pilot project has confirmed that implementation of performance measurement and reporting requires more focused attention, with mixed results from the review of agencies’ frameworks and varied levels of maturity in agencies’ development of their own internal processes and reporting. Irrespective of when the full set of requirements being established by the PGPA Act and associated rules come into effect, much more needs to be done by agencies to operationalise the framework and ensure that implementation of any changes result in a tangible improvement, when compared with existing arrangements.

31. The results of the cross-agency survey and the pilot project work performed in conjunction with agencies, highlighted that active engagement by the senior executives within agencies can provide positive results in the area of performance measurement and reporting. Further, stronger outcomes were achieved when focus and appropriate expertise was devoted to bring about improvements.

32. While much has been written on the importance of KPIs to inform the assessment of program performance, making a difference requires leadership, effective governance, and a desire to understand the impact of government programs and how even better outcomes may be achieved. Both government and government agencies have important roles in improving current practices.

33. Although supporting guidance and agency preparedness issues remain largely unaddressed, the completion of the pilot project has resulted in the development of a preliminary methodology to support ongoing audits of KPIs. However, the extent of future work undertaken by the ANAO will need to be agreed and resourced appropriately in order to implement a broader regime of KPI audits. In the longer-term, once an enhanced performance measurement and reporting regime has been established, greater assurance will be provided if agency performance information is subject to audit. Separately, the ANAO performance audit program will continue to include coverage of the performance measurement arrangements that have been established for the program or area being reviewed.

34. We appreciated the involvement of the Australian Taxation Office, the Department of the Environment, the Australian Federal Police and the Australian Government agencies that responded to the cross-agency survey in this pilot project and their contribution to informing the development of our audit processes, and also to informing their own initiatives to improve their performance measurement and reporting. We also appreciated Finance’s engagement on matters concerned with the performance measurement and reporting framework.

Footnotes

[1] Organisation for Economic Co-operation and Development, Government at a Glance 2013 [Internet], OECD Publishing, 2013, available from <http://www.oecd-ilibrary.org/governance/government-at-a-glance-2013_gov…; [accessed 9 January 2014].

[2] Steering Committee for the Review of Government Service Provision, Report on Government Services 2014, Volume A: Approach to performance reporting, Productivity Commission, Canberra, 2014, p. 1.4.

[3] The National Commission of Audit was announced by the Treasurer, and the Minister for Finance, on 22 October 2013.

[4] Dr Ian Watt AO, IPAA National Conference Address, Remarks given in response to Professor Christopher Pollitt’s address: ‘The Public Service: Dissolution, Revolution, Evolution?’ [Internet] 20 November 2013,available from <http://www.dpmc.gov.au/media/speech_2013-11-20.cfm&gt; [accessed 19 February 2014].

[5] The Outcomes and Outputs framework required agencies to specify intended outcomes and to measure and report on agencies’ actual performance in the delivery of their designated outputs.

[6] Program deliverables (outputs) are the goods and/or services produced and/or delivered by a program.

[7] Department of Finance and Deregulation, Guidance for the Preparation of the 2012–13 Portfolio Budget Statements, Finance, Canberra, 2012, p. 37.

[8] ANAO Audit Report No.5 2011–12 Development and Implementation of Key Performance Indicators to Support the Outcomes and Programs Framework, p. 17.

[9] Mr Ian McPhee, PSM, Presentation to the Risk Management Institute of Australia; Australasian Compliance Institute at the 2nd Annual GRC Conference, Public Sector Management—A Scorecard on the Journey [Internet],1 November 2013, available from <http://www.anao.gov.au/Publications/ Speeches> [accessed 9 January 2014].

[10] ANAO Audit Report No.1 2013–14 Design and Implementation of the Liveable Cities Program, p. 27.

[11] ANAO Audit Report No.19 2012–13 Administration of New Income Management in the Northern Territory, p. 26.

[12] ANAO Audit Report No.12 2012–13 Administration of Commonwealth Responsibilities under the National Partnership Agreement on Preventative Health, p. 25.

[13] ANAO Audit Report No.44 2012–13 Management and Reporting of Goods and Services Tax and Fringe Benefits Tax Information, p. 23.

[14] ANAO Report No.28 2012–13 The Australian Government Performance Measurement and Reporting Framework, Pilot Project to Audit Key Performance Indicators, p. 54.

[15] The International Public Sector Accounting Standards Board’s Exposure Draft 54—Recommended Practice Guideline on Reporting Service Performance Information, and the Steering Committee for the Review of Government Service Provision’s Report on Government Services 2014 are also additional sources of useful information to assist agencies in enhancing KPIs.

[16]Public Accounts and Audit Committee Act 1951 (Cth) s. 8(1)(c).

[17] Joint Committee of Public Accounts and Audit, Report 439, Review of Auditor-General’s Reports Nos.11 to 31 (2012–13), JCPAA, Canberra, 2013, p. 1.

[18] Joint Committee of Public Accounts and Audit, Report 439, Review of Auditor-General’s Reports Nos.11 to 31 (2012–13), JCPAA, Canberra, 2013, pp. 28–9.

[19] The Public Management Reform Agenda was announced in December 2010. The Commonwealth Financial Accountability Review and the enactment of the PGPA Act are discussed further in paragraphs 9–10 and in Chapter 2, pp.54–55.

[20] Finance’s comments are included in full at Appendix 1.