Introduction

1.1 The Public Governance, Performance and Accountability Act 2013 (PGPA Act) provides the basis for the Commonwealth performance framework (the framework). The framework consists of the PGPA Act, the accompanying Public Governance, Performance and Accountability Rule 2014 (PGPA Rule) and guidance issued by the Department of Finance (Finance). It is principles-based and requires entities to publish planned performance information, to allow an assessment of entities’ progress against their purposes when reported at year-end.

1.2 The Independent review into the operation of the Public Governance, Performance and Accountability Act 2013 and Rule, commissioned by the Minister for Finance, notes:

Citizens have a right to know how their money is used and what difference that is making to their community and the nation - what outcomes are being achieved, how, and at what price. ¹¹

1.3 By requiring Commonwealth entities to publish planned financial and non-financial performance information, the framework aims to provide more transparent and meaningful information to the Parliament and the public. The aim is to provide users with a greater understanding of how entities plan to, and have, utilised resources, not just in producing outputs, but also the entity’s impact and efficiency in delivering outcomes.

1.4 The PGPA Act commenced on 1 July 2014, and entities were required to present their first corporate plans and performance statements from 1 July 2015.¹² July 2018 marked the commencement of the fourth year of Commonwealth entities’ implementation of the framework, and October 2018 the conclusion of the third performance measurement and reporting cycle.

Commonwealth performance framework

1.5 The Commonwealth performance framework came into effect with the commencement of the PGPA Act. It has three inter-dependent elements — purposes, operating context and performance information — used to demonstrate the achievement of purposes. Under the PGPA Act and Rule, an entity’s purpose/s may be their objectives, functions or role, which are undertaken to achieve their outcomes.

1.6 An important element of the framework is the clear alignment of the purpose/s to the planned outcomes of the entity¹³, and that the performance information provided by the entity provides the Parliament and the public with information to assess an entity’s progress towards achieving their purposes. Alignment of all of the elements of the framework is intended to improve the line of sight between the use of public resources and the results achieved by entities. The key accountability documents produced under the framework, including the expected alignment and/or information flows, are demonstrated in Figure 1.1.

Figure 1.1: Commonwealth performance framework — 2017–18

 

 

Source: ANAO analysis of the PGPA Act, PGPA Rule, and Department of Finance guidance.

1.7 PBSs set out planned financial performance, and are required to describe at a strategic level, the planned outcomes intended to be achieved with the funding appropriated by the Parliament. Entities are also required to clearly map the outcomes, programs and performance measures from their PBS to their corporate plan purposes, to ensure a clear read between the documents.¹⁴ The performance information in an entity’s PBS must be able to be read across to an entity’s corporate plan. Aligning the corporate plan and PBS provides readers an insight into the expenditure expected to achieve that performance and aligns planned financial and non-financial performance.

1.8 As the primary planning document, the corporate plan sets out planned non-financial performance and provides the reader with an understanding of how an entity intends to measure and assess its actual performance. This includes a description of the entity’s purposes, and the planned inputs, key activities, outputs and the outcomes or impacts to be achieved by the entity, against their purposes. To make this information meaningful, and allow users to form their own expectations, entities must include measurement methodologies and targets or baselines against which progress can be assessed.

1.9 The results the entity has achieved against the performance measures and targets set out in the entity’s PBS and corporate plan, and accompanying analysis, are presented in the subsequent performance statements.¹⁵ Performance statements, included in an entity’s annual report, provide the Parliament and the public with the entity’s assessment of the extent to which it has progressed in achieving its purposes, as set out in the corporate plan.¹⁶ This information can then be used by the Parliament and the public to assess whether the entity’s strategies and activities have been successful in delivering the expected outcome.

1.10 Since its inception, Finance has reinforced the framework’s intention to give entities the flexibility to apply a fit-for-purpose approach to their performance measurement and reporting.¹⁷ Resource Management Guide No. 130: Overview of the Enhanced Commonwealth Performance Framework notes:

Each entity needs to consider how best, and by what method, to gather the necessary information to tell its performance story. The types of information needed will depend on the nature, size and complexity of the entity, its purposes and the characteristics of its activities – and may differ across different entities.

1.11 To support this aim, entities have been encouraged to broaden their performance measurement frameworks to generate better qualitative and quantitative information through tools such as benchmarking, peer reviews, and comprehensive evaluations.¹⁸ Guidance from Finance also advises that good performance information can be categorised by how it communicates accountability, strategic, tactical, or management related information, and how the information presented at these levels interact to inform decision-making¹⁹, as demonstrated in Figure 1.2.

Figure 1.2: Performance information hierarchy

 

 

Source: ANAO analysis of the Department of Finance, Resource Management Guide No.131: Developing Good Performance Information, April 2015, pp. 12–13, 30 & 45.

1.12 As noted earlier, the intended users of the PBS, corporate plan, and performance statements are the Parliament and the public. It is for this reason that entities should consider whether the level and volume of performance information presented meets users’ needs.

1.13 Accountability performance information demonstrates whether the use of public resources is making a difference and delivering on government objectives. This is the level of performance reporting that is the focus of the PGPA Act. Performance reporting for accountability purposes is of most interest to the Parliament and the public, and should be balanced with the other levels of the performance information hierarchy. ²⁰

1.14 Well-presented and easily interpreted accountability information is essential to enable governments to coordinate policy, clarify objectives, enhance transparency and accountability, improve service delivery, and keep the wider community informed. While strategic, tactical, and management performance information are also important, they should be used to support and advance accountability information, rather than replace it. Performance measures that address these lower levels of information, without sufficient connection to accountability information, may not be appropriate to include in the corporate plan and performance statements. ²¹

1.15 As the key accountability documents in the performance framework, an entity’s PBS, corporate plan, and performance statements should provide the Parliament and the public with sufficient information to determine what level of performance is being planned and communicated for each measure, and the insights this might provide. If a user is required to rely on an in-depth understanding of an entity, the basic information needs of the Parliament and public are not being met. In this circumstance, the performance measures and the PBS, corporate plan and/or performance statements should be reviewed.

Joint Committee of Public Accounts and Audit review

1.16 Improving the Commonwealth performance framework has been a long-term focus of the Joint Committee of Public Accounts and Audit (JCPAA). ²² On 6 December 2017 the JCPAA released Report 469: Commonwealth Performance Framework. The report included ten recommendations, five of which related to the committee’s consideration of ANAO Report No.58 2016–17 Implementation of the Annual Performance Statements Requirements 2015–16. Recommendation 6 was that ‘The Australian Government amend the PGPA Act, and the accompanying rules and guidance as required, as a matter of priority, to enable mandatory audits of performance statements by the Auditor-General …’.²³ The Committee also referred this matter to the attention of the ‘Independent Review of the PGPA Act’, and is discussed further below.

1.17 The report also included observations in regard to entity performance measurement and reporting relevant to this audit, including the need:

• to use a mix of qualitative and quantitative performance information, along with relevant contextual information and analysis, to focus on entity impacts and outcomes (reflecting the move away from key performance indicators based solely on measuring inputs and outputs);
• for narrative utilised as part of qualitative performance information to be evidence-based, reliable and robust;
• for further work on measurement methodologies for qualitative performance information, drawing on local and international research and practice in this area;
• for further collaborative work on measuring and articulating performance outcomes, to build consistency and maximise reporting efficiencies; and
• for methodologically robust attribution of entity activities to outcomes that makes accountabilities clear.²⁴

Review of the PGPA Act and Rule

1.18 In September 2017, the Minister for Finance appointed Mr David Thodey AO and Ms Elizabeth Alexander AM to undertake the review of the PGPA Act and Rule (PGPA Review) in accordance with section 112 of the PGPA Act. The review included consultation with the JCPAA, ANAO, Department of Finance, and other Commonwealth entities, with the following objective:

• To examine whether the operation of the PGPA Act and Rule is achieving the objects of the PGPA Act in a manner consistent with the guiding principles;
• To identify legislative, policy or other changes or initiatives, to enhance public sector productivity, governance, performance and accountability arrangements covered by the PGPA Act; and
• To examine whether policy owners’ implementation of the PGPA Act and Rule has appropriately supported their operation in Commonwealth entities.

1.19 The final report was released in September 2018²⁵, outlining 52 recommendations, fourteen of which relate to the performance framework.²⁶ In response to the JCPAA’s recommendation (see paragraph 1.16 above), the reviewers supported the ‘Auditor-General getting legislative power to conduct mandatory assurance audits of annual performance statements’. However, they considered that it is ‘too early to put this arrangement in place’, and that ‘practice across the Commonwealth is not mature enough to support systematic assurance audits of annual performance statements’.²⁷ Consequently, the reviewers directed the following recommendation to the Finance Minister [and the government]:

8. The Finance Minister, in consultation with the Joint Committee of Public Accounts and Audit, should request that the Auditor-General pilot assurance audits of annual performance statements to trial an appropriate methodology for these audits. The Committee should monitor the implementation of the pilot on behalf of the Parliament.

1.20 The Government is yet to respond to the report and accompanying recommendations. Until a request as outlined in Recommendation 8 is made, the Auditor-General will continue to position the ANAO to transition to a program of mandated annual audits of entity performance statements, as previously flagged to the JCPAA²⁸. Such a program would provide a similar level of assurance to the Parliament and the public as provided by mandatory annual audits of financial statements. This ongoing focus on performance statements is reflected in the Auditor-General’s Annual Audit Work Program 2018–19, which includes proposed audits of the Implementation of annual performance statements requirements in 2018–19, and Commonwealth performance reporting and the ‘clear read’ principle.²⁹

Roles and responsibilities

Minister for Finance and the Department of Finance

1.21 The Minister for Finance is the Minister responsible for administering the PGPA Act, and is supported by the Department of Finance (Finance). Finance is responsible for the whole-of-government administration of the Commonwealth performance framework and related legislation.

1.22 To assist entities in implementing the performance framework, Finance has published written guidance, including Resource Management Guides (RMG) and lessons learned papers. During 2017–18 Finance re-issued the following RMGs relevant to the framework:

• RMG No. 134 Annual performance statements for Commonwealth entities, July 2017;
• RMG No. 202 Guide for non-corporate commonwealth entities on the role of audit committees, and an accompanying model audit committee charter, May 2018; and
• RMG Nos. 135; 136; and 137 Annual reports for non-corporate, corporate, and Commonwealth companies, May 2018.

1.23 Finance also released the 2017–18 Corporate Plans Lessons Learned paper in November 2017, and the 2016–17 Annual Performance Statements Lessons Learned paper in April 2018. Finance also hosts Performance Community of Practice forums to facilitate the sharing of expertise and better practice. Forums were held during 2017–18 in December 2017, at multiple locations, to discuss the 2017–18 Corporate Plans Lessons Learned paper. Further forums were held in November and December 2018 focussed on recommendations arising from the PGPA Review, and Finance’s proposed approach to addressing Recommendation 4 of the JCPAA’s Report 469: Commonwealth Performance Framework30.

1.24 In considering the quality of performance reporting as part of the PGPA Review, the reviewers directed the following recommendation to Finance:

The Department of Finance should continue to develop guidance on performance reporting to assist Commonwealth entities to meet the requirements of the PGPA Act and Rule and develop high-quality performance reports. This will also assist audit committees to review performance reporting.

1.25 The ANAO noted that each of the selected entities had engaged with Finance to seek feedback and/or guidance during 2017–18. This was through direct contact with representatives of the department, and/or participation in Finance’s Community of Practice forum. In discussion with the ANAO, the selected entities noted that an increased focus on the development and use of qualitative performance measurement methods was a particular area where there would be value in enhancing existing guidance.

Responsible Ministers

1.26 Under the PGPA Act, a responsible Minister may:

• access the records kept about the performance of the entities within their portfolio (section 37); and
• request that the Auditor-General examine and report on the performance statements of entities within their portfolio (section 40).

Accountable authorities

1.27 Accountable authorities are responsible for the implementation of the requirements of the performance framework in their entities. Part 2-3 of the PGPA Act — relating to planning, performance and accountability — sets out the requirements of accountable authorities. The requirements relevant to this audit include:

• preparing a corporate plan each reporting period that complies, is published, and is provided to the responsible Minister and Finance Minister in accordance with any requirements prescribed by the PGPA Rule;
• keeping records about the entity’s performance in accordance with any requirements prescribed by the Rule;
• measuring and assessing the entity’s performance and complying with any requirements prescribed by the Rule; and
• preparing performance statements about the entity’s performance that comply with any requirements prescribed by the Rule, and including these statements in the annual report. ³¹

Audit committees

1.28 Audit committees are appointed by the accountable authority of an entity. The functions of an audit committee are prescribed by section 17 of the PGPA Rule, and must be set out by the accountable authority in a written charter. The required functions of an audit committee are detailed in the box below.

1.29 In May 2018, Finance released its Resource Management Guide No. 202: A guide for non-corporate Commonwealth entities on the role of audit committees, accompanied by a Model audit committee charter. The guide is intended to inform accountable authorities of the requirements and expectations of audit committees, as set out by the PGPA Rule, and provides guidance on the skills and knowledge required of members.

1.30 While the following excerpt relates to performance reporting, the guide establishes equivalent expectations across an audit committee’s four functions set out by the Rule:

Consistent with the requirements on accountable authorities under the PGPA Act regarding records about performance (section 37), advice to the accountable authority from the audit committee should be documented in the form of a written statement of its view on the appropriateness of the accountable authority’s performance reporting. An audit committee’s charter should specify the content and level of detail expected in a written advice.

The audit committee should communicate their view to the accountable authority and not merely state that it does not know of anything that would indicate the accountable authority’s performance reporting for the entity is not appropriate. The statement adds value by providing comfort to the accountable authority that their performance reporting is appropriate, and by providing references and suggestions for systems and process improvement.

1.31 On the basis of the above, there is an expectation that audit committees document their conclusion in relation to the appropriateness of the accountable authority’s performance reporting. Such a statement would be strengthened by also documenting the steps taken to reach this conclusion, for example with reference to the types of activities suggested by the guidance from Finance.³²

Rationale for undertaking the audit

1.32 The PGPA Act commenced on 1 July 2014, and entities were required to present their first corporate plans and performance statements from 1 July 2015. July 2018 marked the commencement of the fourth year of Commonwealth entities’ implementation of the framework, and October 2018 the conclusion of the third performance measurement and reporting cycle.

1.33 As noted by the Joint Committee of Public Accounts and Audit (JCPAA), the framework ‘seeks to improve public sector performance information to strengthen accountability’.³³ Observations from previous audits of entities’ performance statements, and recommendations from the JCPAA, indicate the need for sustained attention in this area to meet this aim, particularly with regard to:

• the development of performance measures that are relevant, reliable and complete;
• the provision of meaningful information to the Parliament to demonstrate progress against an entity’s purpose and meet the requirements, and the objects, of the PGPA Act;
• consideration of an entity’s efficiency in meeting its purpose; and
• review by audit committees of their accountable authority’s performance reporting.

1.34 This performance audit is the ANAO’s third audit of entities’ progress in the implementation of the performance statements requirements under the PGPA Act. Given the time elapsed, entities should have fully embedded the principles of the framework into their organisational processes, to support the presentation of meaningful performance information to the Parliament and the public.

1.35 The ANAO’s continuing focus in this area is expected to assist in keeping the Parliament, the government, and the public informed on implementation of the framework and to provide insights to entities to encourage improved performance.

1.36 To date, the ANAO’s audits of performance statements have considered entities where the achievement of purposes were driven by a focus on program-delivery, and the use of quantitative measures was more prevalent. In contrast, the entities selected for this audit are intended to provide an opportunity to consider the performance measurement and reporting frameworks of entities whose activities are more focused on policy development, coordination and leadership. These types of activities typically generate more qualitative, than quantitative, information, and entities may, or may not be, taking advantage of the flexibility encouraged by the framework in the development of their own performance measurement and reporting frameworks in response.

Audit approach

Audit objective, criteria and scope

1.37 The objective of the audit was to continue to examine the progress of the implementation of the performance statements requirements under the PGPA Act and the PGPA Rule by the selected entities. The audit was also designed to:

• provide insights to entities more broadly, to encourage improved performance; and
• continue the development of the ANAO’s methodology to support the possible future implementation of annual audits of performance statements.

1.38 To form a view against the audit objective, the following high level criteria were adopted:

• entities complied with the requirements of the PGPA Act and PGPA Rule and, in doing so, met the objects of the PGPA Act;
• the performance criteria presented in the selected entities’ budget statements, corporate plans and performance statements were appropriate; and
• entities had effective supporting frameworks to develop, gather, assess, monitor, assure and report performance information.

1.39 The audit involved assessment of the appropriateness (relevance, reliability and completeness) of the performance criteria, and the completeness and accuracy (fair presentation) of reporting. This was completed for a subset, or all of the performance criteria presented by the selected entities in their 2017–18 Performance Statements.

1.40 The audit considered the performance criteria established by each of the selected entities to demonstrate progress against the following elements of their corporate plans:

• Attorney-General’s Department’s performance criteria for ‘Strategic Priority 3 — Justice’ and ‘Strategic Priority 5 — Rights’;
• Department of Education and Training’s performance criteria for ‘Outcome 1 — Quality early learning and schooling’;
• Department of Foreign Affairs and Trade’s performance criteria for ‘Priority Function 1: Promoting a stable and prosperous regional and global environment’ and ‘Priority Function 2: Improving market access for Australian goods and services, attracting foreign investment and supporting business’; and
• Department of the Prime Minister and Cabinet’s performance criteria for all purposes.

1.41 The performance criteria considered for each entity are presented in Appendix 2.

1.42 Under the PGPA Act and Rule, an entity’s purpose may be their objectives, functions or role. This has led to variability in the language used by entities when labelling elements of their corporate plans, as demonstrated in Figure 1.3, and reduces inter-entity comparability for readers. This may be an area where further consideration or guidance by Finance may assist in providing additional clarity across entity performance reporting.

Figure 1.3: Comparison of corporate plan elements

 

 

Source: ANAO analysis.

Audit methodology

1.43 The audit included reviewing:

• the selected entities’ 2017–18 PBS, Corporate Plans and Performance Statements;
• internal systems, processes, and procedures, including the governance and oversight put in place by the selected entities to support their development of the performance statements;
• records, and interviews of staff of the selected entities; and
• the selected entities’ 2018–19 PBS and Corporate Plans to identify any further opportunities for improvements to performance measurement and reporting that may be addressed in the 2018–19 Corporate Plans and/or Performance Statements.

1.44 The audit also included reviewing Finance’s role in whole-of-government administration of the framework.

1.45 The audit was conducted in accordance with the ANAO Auditing Standards at a cost to the ANAO of approximately $585,000.

1.46 The team members for this audit were Jennifer Hutchinson, Kara Ball, Jillian Blow, Alicia Vaughan and Michael White.

Have the selected entities complied with the requirements of the Public Governance, Performance and Accountability Act 2013 and accompanying Rule?

 

2.1 Table 2.1 outlines the PGPA Act and Rule requirements for the presentation of annual performance statements. To determine whether entities met the requirements of the PGPA Act and Rule, the ANAO reviewed whether:

• the entities had prepared, and tabled in Parliament, performance statements in accordance with the PGPA Act and Rule; and
• the statements, results and analysis presented in those performance statements met the requirements of the PGPA Act and Rule.

2.2 The results of this assessment is summarised in Table 2.1, and the following sections.

Table 2.1: Compliance with PGPA Act and PGPA Rule requirements

Source: ANAO analysis against PGPA Act and Rule requirements.

Preparation and tabling

2.3 Each of the selected entities complied with subsection 39(1) of the PGPA Act to prepare performance statements and include a copy in their tabled 2017–18 Annual Reports. Table 2.2 below presents a summary for each entity of the date the performance statements were approved by the accountable authority, and subsequently tabled in the Parliament.

Table 2.2: Approval and tabling of the selected entities’ performance statements

Source: ANAO analysis.

2.4 Across the four entities, the time elapsed between the end of the reporting period (30 June 2018) and the approval of the performance statements by the accountable authority varied between two and three months. The entities’ annual reports, including the performance statements, were all tabled within five weeks of the accountable authority’s approval of the performance statements.

2.5 The PGPA Review included consideration of the timeliness of entity annual reporting and noted ‘Current arrangements for presenting annual reports to the Parliament do not ensure they receive adequate scrutiny by the Parliament.’ Three recommendations were made intended to improve the timeliness and scrutiny of annual reports, including Recommendation 30:

[Subject to the implementation of Recommendation 31] Annual reports should be presented to the Parliament on or before 30 September. This would ensure the Parliament has annual reports available before the Senate Supplementary Budget Estimates hearings. Annual reports should be presented to the responsible minister no later than seven days before this date.³⁴

2.6 As demonstrated above, none of the selected entities tabled its annual report before 30 September 2018. Entities would benefit from considering how existing processes supporting the preparation of performance statements may be adapted to facilitate timelier reporting in the future. The processes supporting each entities’ preparation of the performance statements are discussed further in Chapter 4 of this report.

Statements

2.7 Each of the selected entities’ 2017–18 Performance Statements included the required statements by the accountable authority under subsection 16F(1) of the PGPA Rule. The selected entities also included an additional statement that the performance statements were ‘based on properly maintained records’. This reflects the requirement under section 37 of the PGPA Act for Commonwealth entities to keep records that properly record and explain the entity’s non-financial performance.

Results

2.8 As set out in Table 2.1, under subsection 16F(1) of the PGPA Rule, an entity must:

measure and assess the entity’s performance in achieving the entity’s purposes in the reporting period in accordance with the method of measuring and assessing the entity’s performance in the reporting period that was set out in the entity’s corporate plan, and in any Portfolio Budget Statement, Portfolio Additional Estimates Statement or other portfolio estimates statement, that were prepared for the reporting period.

2.9 The following section assesses whether the selected entities’ met this requirement by acquitting, or presenting results against, all of the performance measures originally presented in the PBS and corporate plan.³⁵ Entities’ are also required to report in accordance with the methods set out in PBS and corporate plan. An assessment of these methods forms part of the assessment of the appropriateness of the selected entities’ performance criteria in Chapter 3.

2.10 The performance measures presented in the selected entities’ 2017–18 corporate plans were all reported in the 2017–18 performance statements. Education’s and PM&C’s performance statements also reported results against all of the performance measures presented in their 2017–18 PBS, in accordance with the PGPA Rule. In reviewing AGD’s and DFAT’s performance statements, it was unclear how the results presented addressed certain PBS measures³⁶.

2.11 For example, two performance measures and accompanying targets³⁷ were presented in AGD’s 2017–18 PBS (under Program 1.9: Royal Commissions) that were aligned to Strategic Priorities 1 and 3 in the corporate plan. These measures and targets were not presented in AGD’s performance statements. There was information related to Royal Commissions presented in a section of the performance statements, however there was no clear connection to the original measures and/or targets, affecting a user’s assessment of AGD’s performance in this instance.

2.12 The consistency and completeness of the presentation of performance criteria and targets across the PBS, corporate plan and performance statements is important to establish a clear read. Finance guidance notes that the Finance Secretary’s Direction ‘does not necessarily require entities to publish a line-by-line acquittal, however, the reader should be able to clearly discern the entity’s performance against all of the proposed measures.’³⁸ It is important for entities to ensure the performance statements make clear how the results presented relate to measures originally presented in the corporate plan and PBS.

2.13 The accuracy of results presented against the measures in the entities’ performance statements were also reviewed and concluded to be materially correct, and supported by appropriate records. This aligns with the statements made by each accountable authority under subsection 39(1) of the PGPA Act that the performance statements ‘accurately present the entity’s performance in the reporting period’. Chapter 4 discusses the systems and processes supporting the entities’ performance statements, including an assessment of the results and accompanying records in more detail.

Analysis

2.14 Subsection 16F(1), Item 3: Analysis requires an entity to provide an analysis of factors that may have contributed to the entity’s performance in achieving its purposes. To fulfil this requirement, and the object of the PGPA Act to provide meaningful information, an entity’s analysis should establish for a reader the connection between:

• the results from individual performance measures;
• the internal or external environmental influences that affected those results; and
• how these form the basis for the entity’s assessment of its progress against the overarching purpose.

While each of the selected entities presented a form of analysis in their performance statements, there are opportunities for improvement by each to fully meet the PGPA Rule.

Attorney-General’s Department

2.15 AGD presents an analysis section in the performance statements under each strategic priority (the level below its purpose — see Figure 1.3). These sections provide an assessment of the department’s progress towards completing key activities set out in the corporate plan relevant to each strategic priority. This is supported by expanded discussion of the outputs and actions arising from those activities, however, there is no assessment by the department of how these contribute to an assessment of achievement of the strategic priority, or the overarching purpose.

2.16 Similarly, the analysis does not provide the reader with an understanding of how the different elements presented in the performance statements (KPIs, key activities, results and analysis) should be interpreted together to determine AGD’s overall performance. There is also an opportunity for the department to expand its analysis of specific results in the performance statements to assist a reader in understanding the factors affecting the department’s performance, and what is being done in response.

2.17 For example, under Strategic Priority, Australia’s ranking under Factor 4 of the Word Justice Project’s Rule of Law Index remained at position 13 of 113 countries. The performance statements include the following statement ‘The 2017 Index noted the greatest decline globally over the past 12 months is in Factor 4, with 71 of the 113 countries that are included having a lower rating. Australia’s ranking remains at 13 of 113 countries.’ However, it does not explain why the target (position 10) was not met, or provide the department’s view of what might have contributed to it. Expanding this analysis would improve the meaningfulness of the information being presented.

Department of Foreign Affairs and Trade

2.18 DFAT presents an overview and analysis section in the performance statements for each priority function (the level below its purpose — see Figure 1.3). This section provides discussion and analysis of the department’s operating context and key risks, including outlining significant international issues and activities undertaken by the department. This type of analysis assists the reader to understand the rationale behind, and importance of, DFAT’s activities.

2.19 This could be improved by including analysis of how these issues impacted on DFAT’s performance, and outlining actions taken to address them. That is, rather than presenting information on circumstances and events that could impact performance, the reader would benefit from an explanation of if, and how, these did impact performance and how DFAT responded.

2.20 In addition, the analysis does not establish a sufficient connection between the results presented against each measure of ‘met’ or ‘partially met’, and the accompanying case studies. For example, under Priority Function 1 the performance criterion ‘Our ability to shape outcomes which reflect Australia’s interests, including through coalition building with international partners’ was rated as partially met. The case study notes that ‘Australia and Timor-Leste successfully concluded an agreement on maritime boundaries and resources development’, and ‘despite best efforts, the conciliation process was not able to secure an agreement between Australia, Timor-Leste and relevant companies on the best model for developing the Greater Sunrise Gas Fields’. In the absence of further discussion or analysis of what other activities or actions of the department undertook during the reporting period, beyond the Timor-Leste case study, the reader is left with limited information to determine whether the partially met rating is appropriate for the performance criterion.

2.21 Further, DFAT does not provide an overarching assessment of its progress against the priority function, or the department’s purpose. DFAT would benefit from making clear to a reader how the different sections of the performance statements (overview and analysis, and results) collectively demonstrate achievement of the department’s priority functions and overarching purpose.

Department of Education and Training

2.22 Education’s performance statements are structured to reflect its purpose ‘Maximising opportunity and prosperity through national leadership on education and training’, and two outcomes: ‘Outcome 1 Quality early learning and schooling’; and ‘Outcome 2 World-class tertiary education training and research’. Analysis, key performance results (corporate plan performance measures) and additional results (PBS measures) are presented for each Outcome in the performance statements.

2.23 The analysis presented for Outcome 1 is largely a summary of activities and positive results presented in the key performance results section, rather than an assessment of the department’s progress against the outcome itself. The analysis also does not address the only key performance result under Outcome 1 that was reported as less than ‘Achieved’ — ‘Greater proportion of students achieve at or above minimum standards for reading, writing and numeracy under the National Assessment Program.’³⁹

2.24 The analysis presented in the performance statements also does not provide an assessment of the department’s overall progress against its purpose, on the basis of the measures and results presented across the two outcomes. Presenting this overarching assessment in the performance statements, in compliance with the PGPA Rule, would assist users to assess Education’s progress towards its purpose.

Department of the Prime Minister and Cabinet

2.25 PM&C’s performance statements are structured to reflect its three purposes, which the department notes ‘help to achieve our mission of advancing the wellbeing of all Australians’:

• Purpose 1: Supporting the Prime Minister, as the head of the Australian Government, the Cabinet and portfolio minsters;
• Purpose 2: Providing advice on major domestic policy, national security and international matters; and
• Purpose 3: Improving the lives of Indigenous Australians.

2.26 Corporate plan activities, KPIs, performance measurements and key results, including case studies, are presented under a combined section for Purposes 1 and 2 and labelled ‘Supporting and advising’. This is then followed by a section for PBS performance criteria and results relevant to the two purposes. The same structure is adopted for Purpose 3.

2.27 Preceding the purposes, PM&C also presents an ‘Analysis of performance against our purposes’ section which contains the following statement:

During 2017–18 PM&C generally performed effectively in delivering our three purposes. Independent surveys of stakeholders found that we provided quality advice across a broad range of policy areas; that we collaborated effectively with public and non-government stakeholders; and we provided effective services to the Cabinet.

2.28 This is accompanied by a section on the operating environment, highlighting the climate in which the department operated during the year, its response and the impact this had. An additional ‘Areas for improvement’ section outlines, for those activities or areas where the department’s survey results were less than expected, how the department intends to respond in the future. Establishing a more direct link between these two sections would enhance a users’ understanding of the factors influencing the department’s performance during the period. For example, the department notes a ‘spike’ in official visits and events stretched the department’s resourcing and capacity, however it doesn’t point to specific activities or results that were affected by this.

2.29 In addition, where PM&C relied on case studies as a form of measurement, there is no analysis in the performance statements to establish a connection to the results presented against each measure of ‘Achieved’. As noted above for DFAT, without this, a reader is unable to assess whether the reported result is appropriate.

2.30 The analysis presented in the performance statements also does not provide an assessment of the department’s overall progress against its mission, on the basis of the measures and results presented across the purposes. Presenting this overarching assessment in the performance statements, in compliance with the PGPA Rule, would assist users to form a judgement on PM&C’s overall progress.

Did the selected entities’ annual performance statements fulfil other annual reporting requirements?

2.34 The purpose of the performance framework is to enhance the transparency and accountability of the public sector.⁴⁰ Finance guidance highlights the aim of the PGPA Act is to improve the line of sight between what is intended and what is delivered.⁴¹ To support this aim, sections 17AC, 17BD and 28D of the PGPA Act include provisions for ‘Plain English and clear design’, in relation to commonwealth entities’ and companies’ annual reports, including:

• annual reports must be prepared having regard to the interests of the Parliament and any other persons who are interested in the annual report; and
• requiring information in the annual report to be relevant, reliable, concise, understandable and balanced, including through clear design and defining technical terms.

These requirements provide for clear interpretation of the annual report, including the performance statements, by users.

2.35 Overall, all four entities’ performance statements were structured to support a reader’s understanding of the content. In particular, Education’s presentation of its results within the performance statements provides readers with a clear understanding of a measure’s information source, limitations and the relevant PBS program alignment to assist informed decision-making. However, as noted earlier, the need for clearer alignment between the results presented by entities in the performance statements and the original measures, and improving the quality of analysis remain areas for improvement.⁴²

2.36 As noted by Finance guidance, it is important to reinforce the connection between the corporate plan, PBS and performance statements to enable a ‘clear read’ of the three documents. Each entity adopted a slightly different approach in meeting this aim. Commonly, each of the four entities set out their performance statements in the same structure as the corporate plan, reinforcing the connection between the documents. The effectiveness of the entities’ approaches to establishing this same connection between the performance statements and the PBS varied.

2.37 As discussed in paragraph 2.10, it was not clear how some of the performance criteria presented in AGD and DFAT’s 2017–18 PBS were addressed in the performance statements. This is despite both entities providing mapping demonstrating the alignment between the PBS programs and corporate plan strategic priorities/priority functions in the performance statements. This highlights the importance of an entity providing clear signs to a reader of how specific measures have been transposed from the PBS and corporate plan to the performance statements.

2.38 PM&C and Education both present the performance measures and accompanying results from their PBSs separately in the performance statements to those in the corporate plans. This assists a reader confirm the completeness of the actual results in the performance statements compared to the expectations set by the PBS and corporate plan measures. However, there are opportunities for both departments to better integrate the PBS performance measure results with those from the corporate plan, particularly where the PBS performance measures appeared to overlap with the results already presented against the corporate plan measures.

2.39 For example, the following results were repeated in two different sections of Education’s performance statements due to the similarity of the PBS and Corporate Plan measures:

The Child Care Subsidy System (CCSS) went live on 2 July 2018. 1.02 million families completed their Child Care Subsidy assessment. Of 6,053 existing child care providers, 6,040(12,732 of 12,745) successfully transition to the new arrangements.⁴³

2.40 Neither department drew attention to, nor attempted to explain, where an overlap occurred. Given the framework intention is for a clear read across the PBS, corporate plan and performance statements, where measures overlap or present the same, or similar, results, introducing and/or explaining this to a reader would assist in meeting this aim.

Did the selected entities’ corporate plans support performance measurement and reporting in the annual performance statements?

3.1 As demonstrated in Figure 1.1, an entity’s corporate plan, alongside the PBS, sets out an entity’s planned performance through the description of purposes, activities and performance criteria. The corporate plan is intended to be an entity’s primary planning document, and the key source of information for the Parliament and the public to form an expectation of an entity’s performance.

3.2 This expectation may then be compared to the actual performance set out by an entity in its performance statements through results and accompanying analysis. The Parliament and the public use this comparison to assess the extent to which the entity has met that expectation. As a result, it is important for an entity’s corporate plan to provide a clear basis to not only support an entity’s assessment of its performance, but also the Parliament’s and the public’s.

Purpose

3.3 Section 16E of the PGPA Rule requires an entity’s corporate plan to state the entity’s purpose/s over the next four years. The PGPA Act defines purpose/s as including the objectives, functions or role of an entity. The aim of the purpose/s statement is to give context to the significant activities that the entity will pursue over that period, and should be stated in a relevant and concise manner. Finance guidance notes that:

Well-expressed purpose statements make it clear who benefits from an entity’s activities, how they benefit and what is achieved when an entity successfully delivers its purposes. Essentially, purposes describe the value an entity seeks to create or preserve.

3.4 The purpose presented in the corporate plans of AGD, Education and DFAT demonstrate the required elements of a ‘well-expressed’ purpose statement. Each purpose was readily identifiable, makes clear who will benefit from the entities’ respective activities, how the entities will deliver that benefit and the impact intended to be achieved in delivery against their purposes. PM&C’s purposes require improvement to meet the characteristics set out by Finance guidance, as described below.

3.5 Education and PM&C’s 2016–17 Corporate Plans were both included in Auditor-General Report No.54 2016–17 Corporate Planning in the Australian Public Sector 2016–17. As part of this audit, the ANAO observed that Education’s purpose could be made more readily identifiable, and PM&C’s purposes were expressed as actions or activities rather than as an outcome or result to be achieved.⁴⁴

3.6 When comparing Education’s 2016–17 and 2017–18 corporate plans, the department replaced its vision and mission with a purpose statement. By removing the vision and mission, the purpose statement can be easily identified by a user as the definitive expression of the department’s intended purpose. This was also supported by an explanatory statement which assists to further describe who will benefit, and how the benefit is intended to be delivered by the department through achievement of its purpose. It also expands on the department’s intended impact, leading to an improved purpose statement, and addressing the observations made in the report.

3.7 In contrast, changes made to PM&C’s 2017–18 purposes from the previous corporate plan did not address the observations made in Auditor-General Report No.54 2016–17. The purposes continue to be activities based and do not describe the intended impact, with the exception of Purpose 3. However, PM&C did introduce the following ‘Mission’ in its 2017–18 Corporate Plan:

The Department of the Prime Minister and Cabinet (PM&C) advances the wellbeing of Australians by delivering high-quality support to the Prime Minister and Cabinet.

We take a whole-of-nation approach, working closely with communities, stakeholders and across all areas of government and acknowledge the special place of the first peoples of our nation. We lead the APS in rigorous and collaborative policy development, implementation and program delivery.

3.8 For comparison, the ANAO assessed the mission against the characteristics of a well-expressed purpose. The department’s mission addresses all of these characteristics, where its purposes do not. The mission sets out who will benefit (Australians), how they will benefit (supporting the Prime Minister, working with communities, rigorous and collaborative policy development, and implementation and program delivery) and the intended impact (advances the wellbeing of Australians).

3.9 Finance acknowledges that entities may wish to include strategic statements such as a Mission, however notes that ‘Strategic statements should not be included as a substitute for a clear statement of purposes.’⁴⁵ Presenting the mission above the labelled purposes may limit a reader’s understanding of whether it is intended to be read as the purpose, part of the other purposes, or excluded altogether for the means of performance reporting. On this basis, users of the corporate plan would benefit from PM&C relabelling the mission as its purpose and the purposes as objectives or priorities. This would make clear to a user which element is intended to have priority and how they should be interpreted.

Activities

3.10 Finance guidance notes that:

An entity’s corporate plan does not need to describe everything it does to deliver its purposes. It should focus on the high-level activities through which the results captured by its performance frameworks are achieved. A discussion of activities should provide a reader some insight and understanding of how purposes are pursued. Each activity should be explicitly linked to a purpose, together with the contribution it makes to achieving an entity’s purposes.⁴⁶

3.11 The ANAO assessed the activities listed by the selected entities in their 2017–18 Corporate Plans against the following criteria, sourced from Finance guidance⁴⁷:

• Are the activities readily identifiable?;
• Do the activities align with the entity’s purpose?;
• Do the activities clearly describe what actions the entity will undertake to achieve its purpose?; and
• Are the activities identified at an appropriate level?

3.12 Overall, all four of the entities’ activities presented in their corporate plans met, or mostly met the assessment criteria. DFAT’s, Education’s and PM&C’s activities could all be improved, through the use of more specific language, to more clearly describe the actions that were to be undertaken by the entity.

3.13 For example, Education’s activities frequently referred to the term ‘supporting’ and can reflect a number of different actions, such as: financial support in the form of payments or subsidies; or direct engagement or delivery of services; or a combination of the two. Similarly, DFAT used terms such as ‘lead’, ‘shape’, ‘advance’ and ‘contribute effectively’. Without further detail a reader is unable to determine the specific actions the departments are undertaking to pursue their purposes. Further, adjectives such as ‘vigorously’, ‘energetically’ and ‘proactively’ are unnecessary in conveying the activities to be undertaken.

3.14 In the absence of a clearer description of the activities, the degree of alignment between DFAT’s, Education’s and PM&C’s activities and purposes was not always clear. However, all activities were determined to address the entities’ purposes to some extent. An exception was the section in PM&C’s Corporate Plan outlining measures relevant to its corporate services. As noted above, the purpose of the corporate plan is to report an entity’s intentions to the Parliament and the public.

3.15 As such, the information presented should be tailored to these stakeholders and framed at an appropriate level. PM&C may consider whether the corporate services activity is presenting information of most relevance to the Parliament and the public, or whether it would be better communicated through another setting.

Alignment to Portfolio Budget Statements

3.16 As noted in paragraph 1.6, alignment across the elements of the Commonwealth performance framework is intended to improve the line of sight between the use of public resources and the results achieved by entities. Finance guidance notes:

To demonstrate the achievement of its purpose(s) entities will need to clearly map (or attribute) the performance information from the Portfolio Budget Statements to the entities’ purpose(s). This mapping will serve to establish a clear read between the entity’s corporate plan, relevant Portfolio Budget Statements, annual performance statements and the annual report, and ensure it is clear how (and how well) the entity is fulfilling its purposes.⁴⁸

3.17 All four entities followed Finance’s suggested presentation of performance information in the PBS, including mapping of the program/s to an accompanying purpose in accordance with the Finance Secretary’s Direction issued on 3 March 2017. However, given performance information has, in most cases, been set at a level lower than the purpose, the alignment between AGD’s, DFAT’s and PM&C’s financial and non-financial performance is not clear to a reader and could be improved.

3.18 The ANAO has previously noted that establishing links between the funding reported in the PBS and the performance information presented in corporate plans and performance statements can be difficult⁴⁹. This includes ensuring an entity’s funding and organisational structures facilitate the development and collection of meaningful performance information and reporting. This is particularly the case where an entity’s key activities are broadly captured by one or two PBS programs, or the nature of the entity’s role is one where coordination and collaboration activities cut across multiple programs.

3.19 AGD’s corporate plan provides a table which outlines the linkages between the PBS programs and the strategic priorities. As presented, a number of PBS programs address multiple strategic priorities. Strategic Priorities 3 and 5, the focus of this audit, are each aligned to four PBS programs and of those, three are the same. This makes it difficult for a reader to establish a clear read between the two documents, including determining the extent to which each program is attributable to the relevant strategic priorities.

3.20 Similarly, DFAT’s corporate plan provides a ‘plan on a page’ which outlines the connection between the department’s PBS Outcomes, and the priority functions presented in the corporate plan. It does not, however, provide any further insight as to how the department’s PBS programs align to the priority functions. DFAT advised the ANAO Program 1.1 was attributed to both Priority Functions 1 and 2. Without providing clear alignment, it is difficult for a reader to determine the extent of the program’s attribution to either function and, as a result, the connection between the department’s financial and non-financial performance.

3.21 Users of the performance statements would benefit from AGD and DFAT considering how the alignment between PBS programs and the lower level corporate plan elements may be simplified, to improve the line of the sight between financial and non-financial performance. Finance may also consider whether the Finance Secretary’s Direction is assisting to establish this alignment, or if requiring entities to map their PBS program performance information to a level lower than the purpose, such as objectives or activities, would improve this line of sight.

3.22 PM&C’s corporate plan notes that it should be read “in conjunction with the associated Portfolio Budget Statements and Portfolio Additional Estimates Statements for 2017–18…”, however there is no further direction to the reader as to how the Corporate Plan and relevant budget statements should be interpreted alongside one another. In comparing the two documents, it was unclear whether the PBS performance criteria were intended to stand alone, or if they presented a summary of all of the activities under the relevant purposes that would be measured. For example, one of the Program 1.1 performance criteria is:

Quality and timely policy advice, support and services to the Prime Minister, the Cabinet and key stakeholders

3.23 The criterion mentions three activities ‘policy advice’, ‘support’, and ‘services’ each of which are separately presented in the corporate plan, with multiple performance indicators and measurements. As a result, it is unclear how results against this measure were intended to be presented in the performance statements at year-end. There may be benefit in PM&C enhancing the corporate plan to make mapping of its activities and performance information to the PBS clearer for a reader.

Were the selected entities’ performance criteria relevant, reliable and complete?

Performance information hierarchy

3.24 As noted at paragraph 1.13, performance information should aim to address the accountability level of the performance information hierarchy. This level of information is essential to enable government to coordinate policy, clarify objectives, enhance transparency and accountability, improve service delivery, and keep the wider community informed.

3.25 Tactical and strategic information, which are focused on the activities and outputs that are intended to lead to fulfilment of an entity’s purpose, may be used to support higher level accountability information. Management information is an important input to determining whether an accountable authority is addressing their duties to govern an entity as set out in the PGPA Act. However, there is limited benefit for the Parliament and public where this information is solely used to demonstrate performance at the purpose level, as the connection can be too remote.

3.26 Table 3.1 summarises the mixture of management, tactical, strategic and accountability level information presented by each entity’s performance criteria. As noted in paragraph 1.15, an assessment of which level of the performance hierarchy is addressed by a performance measure is influenced by the information presented in an entity’s corporate plan. This may lead to differences in the categorisations applied when further contextual information outside the corporate plan is considered.

3.27 Of the four entities, AGD, Education and PM&C had the highest proportion of measures that addressed the two lowest levels of the performance information hierarchy, and would benefit from considering whether this mix is appropriate. Education and PM&C also had the highest number of total performance indicators — 43 and 37 respectively.

3.28 Finance guidance notes the quality of performance information should be emphasised over quantity, recommending a small set of measures that is sufficiently comprehensive to cover those factors that affect an entity’s performance.⁵⁰ Education and PM&C would benefit from reviewing the number of indicators to improve the meaningfulness of performance information provided to the Parliament and the public. Both entities have reduced the total number of performance indicators presented in their 2018–19 corporate plans and PBSs.

Table 3.1: Assessment of the selected entities’ performance information hierarchy

Source: ANAO analysis, as provided to entities during the progress of the audit.

3.29 The 13 performance criteria that addressed the management level of the hierarchy were excluded from the ANAO’s assessment for appropriateness in the following section. These measures were excluded as they would not significantly change an assessment of the entity’s performance, and in turn are not expected to significantly influence the decision making of the Parliament or public.

Appropriateness of entities’ performance criteria

3.30 As demonstrated by Figure 1.1, the PBS and corporate plan are the originating sources of an entity’s performance criteria. The corporate plan is also expected to ‘set the foundations upon which a reliable performance narrative can be built’⁵¹, and appropriate performance criteria assists an entity in meeting this expectation.

3.31 Guidance from Finance notes that ‘appropriate’ performance information is ‘relevant, reliable and complete’. The ANAO assessed the selected entities’ performance criteria for these characteristics. The basis for this assessment is drawn from the characteristics of ‘good’ performance information as defined by Finance. The detailed criteria can be found at Appendix 5.

3.32 A summary of the ANAO’s assessment of whether the characteristics of ‘relevant’ and ‘reliable’ and ‘complete’ were suitably addressed by the selected entities’ performance measures is presented in Table 3.2.⁵² The scale used to rate the performance measures was:

• displayed all of the characteristics of the criterion (Yes);
• displayed most of the characteristics of the criterion (Mostly);
• displayed in part the characteristics of the criterion (Partly); and
• did not display the characteristics of the criterion (No).

3.33 Each of the entities’ performance criteria require improvement to fully meet the characteristics of relevant, reliable and complete. The majority of performance criteria were relevant, or mostly relevant, however most did not meet the reliability criterion and require improvement. The completeness of performance criteria is also a particular area requiring improvement, including developing measures of efficiency, and demonstrating an entity’s intended progress across the life of the corporate plan and beyond.

Table 3.2: Summary of ANAO assessment of the appropriateness of the selected entities’ performance criteria

Source: ANAO analysis.

Relevant — benefit, focus and understandable

3.34 In applying the ‘relevant’ criterion, the ANAO assessed whether each of the selected entities’ performance measures under review:

• clearly indicated who benefited and how they benefited from the entity’s activities;
• was focused on a significant aspect/s of the entity’s purpose/s, via the activity/ies, and the attribution of the entity’s activities to it is clear; and
• was understandable, that is, it provided sufficient information in a clear and concise manner.

3.35 More than three quarters of the selected entities’ performance measures were assessed as either demonstrating all, or most of, the characteristics of relevance. The summarised results of the ANAO’s assessment of the relevance of the selected entities’ performance measures are presented in Table 3.3.

Table 3.3: Summary of ANAO assessment of the relevance of the selected entities’ performance criteria

Source: ANAO analysis.

3.36 For AGD, DFAT and Education, the majority of measures were either relevant or mostly relevant. Where the measures did not fully meet the characteristics of relevance, it was more commonly the result of the measure not adequately defining the beneficiary or benefit intended to be delivered, or not reflecting a significant aspect of the purpose.

3.37 Nine, or one-third, of PM&C’s performance criteria were assessed as not relevant. This was the result of measures not adequately fulfilling any of the characteristics of benefit, focus or understandability. For example the measure ‘At least 70 per cent of funded activities within this program met the mandatory KPI on the extent of compliance with Project Agreement terms and conditions’ does not make clear the beneficiary or benefit. It is not clear if it is Indigenous Australians, or PM&C and the government, receiving the benefits associated with compliance. This also affects a reader being able to determine whether the measure significantly addresses the purpose of improving the lives of Indigenous Australians (focus), or understand the intended result (understandable). The same measure was repeated for five Outcome 2 PBS programs.

Reliable — measurable and free from bias

3.38 In applying the ‘reliable’ criterion the ANAO assessed whether each of the selected entities’ performance measures under review were accompanied by sufficient information in the corporate plan, or PBS, to be:

• measurable, that is, it used and disclosed information sources and methodologies (including a basis or baseline for measurement or assessment, for example a target or benchmark) that were fit-for-purpose; and
• free from bias, allowing for clear interpretation and an objective basis for assessment of the results.

3.39 Compared to relevance, a far higher proportion of the selected entities’ performance measures did not meet, or only partly met, the characteristics of reliable, as the information presented in the corporate plan was insufficient for readers to form a view of the expected performance of the entity. The summarised results of the ANAO’s assessment of the reliability of the selected entities’ performance measures are presented in Table 3.4 below and accompanied by further commentary for each entity.

Table 3.4: Summary of ANAO assessment of the reliability of the selected entities’ performance criteria

Source: ANAO analysis.

3.40 Finance guidance states a corporate plan should include a description of performance measures, when they will be reported on, the data collection techniques to be used and any targets the performance measures will be assessed against.⁵³ This information can then be used by the Parliament and the public to assess the results presented by the entity in its performance statements at year-end.

3.41 This does not necessarily mean an entity needs to describe in intricate detail every element of the method of assessment or methodology to be applied. However, there should be sufficient information for a reader to make an assessment of the reliability of that method and the intended result. The method should also be underpinned by appropriate internal quality frameworks, developed contemporaneously, that support the reliability of the reported result.

3.42 For example, a corporate plan may set out a criterion and accompanying target and/or baseline that intends to measure the satisfaction of a particular stakeholder cohort. To fully address the requirements, the corporate plan would note that the method of assessment is an annual survey conducted by a qualified survey provider. This would be further supported by the entity’s internal policies and procedures for the engagement of that provider and delivery of the survey, including quality assurance mechanisms that provide confidence in the reported result.

3.43 Finance guidance notes the minimum content for corporate plans includes the planned performance of the entity including ‘details of the methodology, data and information that it will use to measure and assess its performance’.⁵⁴ It is important for a reader to be able to understand the basic methodology supporting a measure for them to determine whether it can be relied upon.

Attorney-General’s Department

3.44 This was not demonstrated by four of the eight AGD performance measures assessed by the ANAO. For example, the performance measures ‘Civil justice policy advice, program work and legislative changes’ and ‘Policy advice, program work and legislative changes’ have the same target which is ‘Work is completed on time, within budget and in compliance with relevant guidelines’.

3.45 The broadness of the measure and targets means a reader is unable to determine what the intended result is to then enable an assessment against that expectation at year-end. The data collection technique is also not disclosed to enable a reader to determine whether it is fit for purpose or verifiable. The data collection technique was also absent from the two performance measures of stakeholder satisfaction.

3.46 An absence of targets, and/or only setting static targets, limits the ability of a reader to understand what an entity is aiming to achieve, and how incremental progress against that aim is expected to be measured over time. As noted above, the target ‘Work is completed on time, within budget and in compliance with relevant guidelines’ is used for two measures. Given there are three different assessments noted (time, cost and compliance), without a clear assessment method and defining the parameters of the assessments a reader cannot form an expectation of what ‘on time, within budget and in compliance’ looks like in advance, and as a result determine the potential for bias in either the expected, or reported, results.

Department of Foreign Affairs and Trade

3.47 Of the measures assessed, only one of DFAT’s performance measures demonstrated the characteristics of ‘measurable’. Six were intended to be measured through case studies, two planned reliance on reviews and two did not state a specific measurement method (those presented in the PBS).

3.48 As noted in paragraph 3.40, an entity’s corporate plan should describe the methodology and data intended to be used to measure performance. The department did not define, or provide the parameters for the case studies or reviews. This is demonstrated by the following measure:

Review the effective implementation of Australia’s FTAs, including commercially meaningful outcomes resulting from the General Review of the Association of Southeast Asian Nations (ASEAN)-Australia-New Zealand FTA (AANZFTA) and reviews of Australia’s bilateral North Asia FTAs.

3.49 Without the scope of the review, targets, benchmarks or timeframes, the reader is not informed of what constitutes ‘effective implementation’, or ‘commercially meaningful outcomes’, and how a conclusion about this will be formed. Furthermore, due to the use of the word ‘including’, it was unclear if the assessment of ‘commercially meaningful outcomes’ would be limited to the subset of FTAs mentioned, or all FTAs.

3.50 Consistent with the assessment for ‘measurable’, only one of the performance measures was determined to be ‘free from bias’. Without adequate disclosure of the basis for measurement, the potential for bias cannot be ruled out. For example, the measure of DFAT’s ‘whole-of-government coordination and leadership to advance Australia’s interests internationally’ is open to different interpretations. The absence of clear definitions for these terms — ‘coordination’ and ‘leadership’ — could lead to the result being biased towards a more favourable outcome, and it is unclear what conditions would lead to the measure not being met.

3.51 DFAT’s reliance on case studies and reviews, selected ex-post, provides further potential for bias. Where used, case studies can provide context to the department’s activities and achievements. However, case studies should not be relied upon as a stand-alone measurement, unless the scope is predetermined, activities clearly stated and measurement methods detailed in advance. The expected impacts can then be considered by readers in advance, and progress towards outcomes assessed on the basis of results, particularly in relation to the completeness of the reporting against program objectives.

Department of Education and Training

3.52 Education described the methods of assessment for measures in the corporate plan at a basic level, for example ‘qualitative assessment’ or ‘reporting’. Without being specific as to the method of measurement that will be used, a reader cannot reliably determine what will be measured, or whether reported results can be relied upon to assess Education’s progress in meeting its purposes.

3.53 Similarly, a number of PBS measures did not adequately describe the method of assessment that was intended to be used to determine the results. The department would benefit from providing further detail to assist a reader in better understanding and assessing the reliability of the information sources intended to be used to support its performance reporting.

3.54 Education has presented accompanying targets for all measures presented in the corporate plan and PBS. The use of targets provides some indication to a reader of the intended result, and provides a basis for comparing expected and actual performance, improving the reliability of the measures. However, there are opportunities for the department to further improve the reliability of measures and targets by better defining the parameters or benchmarks that would be used to assess performance.

3.55 For example, for the target ‘Eligible services and families transition from the Community Support Program and Budget Based Funded programs to new arrangements’, it is unclear whether the transition of all, some, or a proportion of, eligible services and families transitioning would see the target assessed as having been met.

3.56 Similarly, using language such as ‘ready to…’ when describing an intended result doesn’t provide a reader with sufficient understanding of the parameters that will be applied by the department when determining whether the target was met or not. For example, ‘Community Child Care Fund grants program is ready to start on 2 July 2018’ — does this mean the grant program is ready to accept submissions on 2 July 2018, all submissions are received by this date for assessment, or all recipients have been approved and payments will commence from 2 July. Without providing this specific detail, there is a potential for bias in the reported result as the reader has been unable to set an expectation against which to assess the department’s reported result.

Department of the Prime Minister and Cabinet

3.57 The design of some of PM&C’s measures provided an indication of the measurement method intended to be used, such as ‘feedback’ or ‘responses to request for briefs’. More commonly, PM&C has presented multiple methods without defining the particular one intended to be used. This is best demonstrated by ‘Use of case studies, independent panels or providers that show…’, which is repeated against multiple measures in the corporate plan. Without being specific as to which method of measurement will be used, a reader cannot reliably determine what will be measured, or whether reported results can be relied on to assess PM&C’s progress in meeting its purposes.

3.58 Of the 22⁵⁵ performance criteria presented in PM&C’s corporate plan, only six present accompanying targets. As noted earlier, the absence of a specific target affects a reader’s ability to determine the potential for bias in the reported result. This is demonstrated by PM&C’s performance criterion ‘Feedback from the Prime Minister, portfolio ministers, the Cabinet, ministerial officers and the Executives shows a high level of satisfaction with the quality and timeliness of advice received’. A ‘high level of satisfaction’ was not defined, and no accompanying target presented, in the corporate plan. As a result, PM&C could decide what constituted a ‘high level of satisfaction’ at any point up to publishing the performance statements, increasing the potential for bias.

3.59 PM&C provided targets for all measures presented in its 2017–18 PBS (nine⁵⁶), as required by the Department of Finance Secretary’s Direction.⁵⁷ However the PBS is only required to present one high-level indicator for each existing program, and cannot be relied upon to present a complete picture of an entity’s performance. As a result, targets should also be presented against performance criteria in PM&C’s corporate plan to improve the reliability of measures.

Completeness — collective and balanced

3.63 The PGPA Act requires accountable authorities to govern their entities in a way that promotes the proper use and management of public resources, which is defined by the Act to mean efficient, effective, economical and ethical, and to measure and assess the performance of the entity in achieving its purposes.⁵⁸ The Act also requires entities to provide meaningful information to the Parliament and the public to assist it in understanding how entities are performing, and how they are using the resources that have been entrusted to them. ⁵⁹

3.64 This is fulfilled by entities meeting the requirements for corporate plans to include statements of how the entity will achieve its purposes, and how they will be measured and assessed, the results of which are reported in the entity’s performance statements. This requires the inclusion of relevant measures demonstrating the proper use of resources in achieving the entity’s purposes in the corporate plan.

3.65 Finance guidance outlines the critical considerations for developing good performance information include using an understanding of an entity’s purposes to identify a set of measures that demonstrate the extent to which those purposes and activities are being delivered efficiently and effectively. As it is rare for a single measure to be able to adequately determine the effectiveness of an activity, Finance guidance advises that good performance information will draw on multiple sources and the quality of performance information should be emphasised over quantity. The guidance recommends a small set of measures that is sufficiently comprehensive to cover those factors that affect an entity’s performance.⁶⁰

3.66 The guidance also notes that in some cases, effectiveness may not be measurable, due to cost or a lack of complete information. In these cases, other measures may be used as proxies for effectiveness. Entities are advised that in these circumstances, they should be clear on why effectiveness cannot be measured and how the proxy measures are suitable. Finance notes that activities often work on different timeframes, and information on one or more timeframes may not be available at the time of reporting. It is therefore recommended that the performance story reflect the outcome that can be reasonably expected from the relevant activities at that time.

3.67 In assessing the selected entities’ performance criteria for completeness, the ANAO considered whether the performance criteria present a basis for a collective and balanced assessment of the entity against its purpose. In particular, the ANAO considered whether the selected entities’ performance criteria:

• collectively address the entity’s purpose through the activities identified in the corporate plan (collective);
• provide a basis for assessment of the efficiency and effectiveness of the entity in fulfilling its purpose either directly or through the use of proxies (balanced);
• relied on a mixture of quantitative and qualitative measurement bases (balanced); and
• assess a mixture of short, medium and long-term objectives (balanced). ⁶¹

3.68 None of the four entities presented measures of efficiency in their 2017–18 PBSs or corporate plans. As noted above, entities should identify a set of measures that demonstrate both efficiency and effectiveness. This is to support the Parliament and the public in the assessment of how well Commonwealth entities are performing, including how they are using the resources that have been entrusted to them.

3.69 With the exception of Education (refer paragraph 3.82), entities are also not realising the full potential of the minimum four year horizon of the corporate plan, by developing performance measures that assess a mixture of short, medium and long-term objectives. Commonly, where targets were set by an entity they were constant across the four years of the corporate plan and assessed as short-term measures as a result. A users’ understanding of an entity’s proposed performance would be enhanced by including in the corporate plan a description of an entity’s rationale for setting targets. In particular, where the target has been historically met, or is static across the corporate plan, and how incremental improvement is expected to be demonstrated over time.

Attorney-General’s Department

3.70 Most of AGD’s performance measures were able to be aligned to the activities outlined in the corporate plan, providing a collective basis for assessing progress against the purpose. In reviewing the alignment between the key activities presented for Strategic Priority 3 and 5 and accompanying performance criteria, the broad description of measures required a reader to assume the connection to the activities.

3.71 For example, ‘Civil policy advice, program work and legislative changes’ does not explicitly demonstrate alignment to ‘develop and implement reforms to the family law system’, or ‘undertake international parental child abduction casework’, however it is reasonable to assume a reader would make this connection, due to the proximity of information on the activities. There would be benefit in AGD considering how the presentation of performance criteria may be improved to more specifically address its key activities and collectively improve their completeness.

3.72 AGD presents four different categories of ‘KPI’ in its corporate plan: ‘effectiveness’, ‘efficiency’, ‘professionalism, skills and commitment’ and ‘impact’. The measures presented against the ‘effectiveness’ (stakeholder satisfaction) and ‘impact’ (Rule of Law Index) KPIs have been designed to reflect its categorisation — either effectiveness or impact. The measures categorised as ‘efficiency’ KPIs do not reflect measurements of efficiency, rather the activities or outputs of the department. For example, the measures ‘justice policy advice, program work and legislative changes’ and accompanying targets ‘Work is completed on time, within budget and in compliance with relevant guidelines’ are not measures of efficiency as presented. The remaining measures also considered activities or outputs of the department.

3.73 The majority of AGD’s performance measures are qualitative in nature (four of seven). Observations made in regard to the reliability of AGD’s measures in the previous section also impacted an assessment of whether they represented qualitative or quantitative information. The only quantitative measures were those presented in the PBS for Program 1.9. The department would benefit from considering whether this mix is appropriate.

3.74 None of AGD’s measures under Strategic Priority 3 or 5 presented targets across the four years of the corporate plan. All measures instead presented a target, without an explanation of whether they applied to all, or only some, of the future years of the corporate plan. The measures were assessed as short-term as a result. AGD would benefit from reviewing how to demonstrate to readers the expected timeframes of activities, and expected incremental progress over the forward years, through its performance measures.

Department of Foreign Affairs and Trade

3.75 Overall, it is unclear whether the performance measures presented in DFAT’s corporate plan and PBS for Priority Functions 1 and 2 provide a complete basis to assess the department’s progress in achieving these priority functions. Most of DFAT’s major operational activities were able to be mapped to an accompanying performance measure, however it could not be determined whether these presented a complete picture of performance.

3.76 This was largely due to the absence of clear methods of assessments as noted in paragraphs 3.47 to 3.51. As a result, the measurement basis and/or timeframe for four of the 13 measures assessed could not be determined. For example DFAT noted it would use case studies to assess:

‘Our promotion and protection of Australia’s economic interests in bilateral, regional, multilateral and plurilateral outcomes’

3.77 No further information about the focus, type or number of case studies intended to be used to assess the performance criterion was provided. As a result it could not be determined whether the basis for assessment would be qualitative, quantitative or a mixture of the two. As noted above, this also influenced an assessment of the extent to which the criterion would address DFAT’s activities.

3.78 Of those that had sufficient information, there appeared to be a mixture of type and bases — effectiveness, activity and output and qualitative and quantitative. However, none presented in Priority Function 1 or 2 were determined to measure the department’s efficiency. Two measures were also presented that were relevant to 2020–21, reflecting a medium term outlook, however the remainder appeared to be an annual assessment.

Department of Education and Training

3.79 Education’s purpose is ‘maximising opportunity and prosperity through national leadership on education and training’, however, there are no measures that seek to demonstrate the department’s ‘national leadership’. Similarly, while all of Education’s measures were able to be mapped to an activity in the corporate plan in some way, none appeared to fully address the activity ‘Strengthening the national evidence base and lifting outcomes in Australian schools’. The activity’s accompanying delivery strategies listed a number of key actions of the department, including:

• Contributing to national policy on the most effective teaching and learning strategies; and
• Leading efforts to improve our national education evidence base and data for schools and preschools.

3.80 While a number of measures appear designed to measure ‘lifting outcomes’, none provided an assessment of the department strengthening the national evidence base, or contributing to national policy. The department would benefit from considering how this may be addressed in future reporting periods to collectively improve the completeness of their measures.

3.81 The majority of Education’s corporate plan criteria measure effectiveness, while the PBS criteria largely address activities and/or outputs. None of Education’s performance measures address the department’s efficiency. The group of performance criteria also provide a balance of qualitative and quantitative information, however there were some measures where the measurement basis could not be determined due the method for assessment not being clear.

3.82 The department has also presented a balance of short, medium and long-term measures. In particular, the corporate plan presents the department’s planned progress over the four years of the corporate plan for particular activities. For example, for the criteria ‘Capacity and capability of child care services to include children with additional needs’, the department has an accompanying target for 2018–19 of ‘establish baseline data through new reporting tool’. The final two years of the corporate plan then both present the target of ‘Improved inclusion for children with additional needs’. Showing the progression of the department’s intended measurement over the life of the corporate plan provides the reader with a more holistic understanding of what the department is aiming to achieve, and the steps being taken in the short term and medium term to do so.

Department of the Prime Minister and Cabinet

3.83 PM&C’s corporate plan clearly outlines activities with accompanying performance criteria and measurements. Overall, the activities, criteria and measurements collectively address the department’s purposes.

3.84 More than half of PM&C’s criteria measure effectiveness, while the remaining criteria measure outputs or activities. None of PM&C’s criteria measure efficiency. The following criterion was presented in the department’s 2017–18 PBS, ‘Efficient department support to the six Indigenous Advancement Strategy programs’ and the accompanying target of ‘At least 70 per cent of key performance measures in the Corporate Plan are met or are on track’, however it is unclear how the accompanying target is intended to demonstrate the department’s efficiency.

3.85 The group of performance criteria provides a balance of qualitative and quantitative information, however, as noted in paragraph 3.58, in most cases no targets have been set against the performance criteria in the current year, or in the future periods covered by the corporate plan. As a result, all performance criteria were assessed as only addressing the short-term timeframe. The exception was the COAG target ‘Halve the gap for Indigenous students in year 12 (or equivalent) attainment rates by 2020’ which addresses the medium term.

Have the selected entities made improvements to their proposed performance measurement and reporting for 2018–19?

3.94 An entity is required to publish a corporate plan, relevant to the current and three future reporting periods, by 31 August each year. Subsection 16E(6) of the PGPA Rule also allows an accountable authority to vary an entity’s corporate plan at any time. Finance guidance provides examples of circumstances where an accountable authority may wish to do so including ‘new activities that warrant inclusion in the corporate plan; significant new performance criteria, targets or tools that will be used to measure or assess performance; and key changes in the capability of the entity or in its risk management approach’.⁶²

3.95 The timing of this audit provided an opportunity to review the selected entities’ 2018–19 PBS and proposed corporate plans to identify any further opportunities for improvement to the performance measurement and reporting. The ANAO provided each entity with feedback during the course of the audit that could then be used in the entities’ development of the 2018–19 Corporate Plans.

3.96 Each of the entities improved elements of their corporate plans in some way by addressing ANAO feedback, or matters already highlighted through the entities’ own improvement processes. Specific elements of Education’s and PM&C’s 2018–19 corporate plans, when compared to the previous year, do not provide an improved basis for performance measurement and reporting.

3.97 AGD incorporated feedback provided by the ANAO during this audit to improve its 2018–19 Corporate Plan, including:

• adding a section for methodology that describes the different sources that support the performance information and how it is collected;
• including a performance measure in the corporate plan focused on the departments’ work in regard to royal commissions; and
• presenting targets across the four years of the corporate plan.

3.98 Similarly, DFAT improved its corporate plan by mapping performance measures to specific delivery strategies, and presenting more succinct and understandable descriptions of the priority functions, which assisted the reader to understand the type of work that DFAT undertakes. These were also supported by the identification of DFAT’s delivery partners for each priority function. A new feature of the 2018–19 Corporate Plan was the inclusion of a table for each priority function, which listed the relevant PBS programs for that priority function. This establishes an alignment between the PBS and corporate plan at a high level.

3.99 Further improvements could still be made by AGD and DFAT to their performance measures to fully meet the characteristics of ‘relevant’, ‘reliable’ and ‘complete’. However, the changes made by both entities to the 2018–19 Corporate Plans provide an improved basis for performance measurement and reporting.

Department of Education and Training

3.100 Education provided its 2018–19 Corporate Plan to the ANAO on 21 August 2018. The department published the corporate plan on its website on 24 August 2018.

3.101 In reviewing the 2018–19 Corporate Plan, there are areas where improvements had been made by the department, including seeking to better describe the key elements of its purpose. This was achieved through the introduction of ‘themes’ that sought to explain what ‘maximising opportunity and prosperity’ meant. For example ‘Access and participation — everyone in Australia has access to and opportunity to participate in quality education’.

3.102 A ‘National Leadership’ section has also been introduced which outlines five key areas where the department intends to deliver national leadership. These included areas such as policy excellence and relationships and networks and collaboration. However, in reviewing the performance measures presented in the 2018–19 Corporate Plan, this particular aspect of the department’s purpose is still not reflected.

3.103 An area where the department’s corporate plan has not improved is in regard to its activities. As noted in paragraph 3.13, the activities presented in Education’s 2017–18 Corporate Plan could have more clearly described the specific actions the department is undertaking to pursue its purpose. The 2018–19 Corporate Plan no longer presents activities. The ‘Sustainability and Efficiency’ theme goes some way by discussing the department’s funding initiatives and compliance arrangements, however the others are presented as detailed objectives, providing only a very broad indication as to what the department intends to do to fulfil its purpose.

3.104 Finance guidance acknowledges that a corporate plan does not need to describe everything an entity does, however it recommends that ‘It should focus on the high-level activities through which the results captured by its performance frameworks are achieved. A discussion of activities should provide a reader some insight and understanding of how purposes are pursued.’ In the absence of key activities, a reader is left with a limited understanding of what the department will do to fulfil its purpose. It is also difficult to determine whether the measures presented will be representative of Education’s performance.

3.105 In responding to the above, the department noted that the activities had not been focused on early enough in the planning process with the senior leadership team, and it was decided that the previous year’s activities would not roll forward into the 2018–19 Corporate Plan. Instead the themes noted above were used, which better link to the department’s purpose, with the intention of presenting activities under these headings in the future. The department acknowledged that this approach ‘comes at the expense of clear attribution re-how we achieve this purpose’.

3.106 The ANAO notes the department’s ongoing efforts to improve its performance measurement and reporting. However, 2018–19 is the fourth year of the Commonwealth performance framework, and it is expected that entities have by now embedded adequate processes to support the timely consideration and development of their corporate plan.

Department of the Prime Minister and Cabinet

3.107 PM&C provided its 2018–19 Corporate Plan to the ANAO on 27 August 2018. The department published the corporate plan on its website on 31 August 2018.

3.108 The mission and purposes of the department have not changed in the 2018–19 Corporate Plan. As noted in paragraph 3.7, the department’s purposes are activity-based, with the exception of Purpose 3, and do not reflect the intended result. The mission on the other hand contains these characteristics and better describes the department’s ‘purpose’ as required by the framework. Reviewing the presentation of PM&C’s mission and purposes, and making clear the intended relationship between these elements in the corporate plan, would assist a users’ understanding.

3.109 When comparing the ‘Purposes’ sections between the two corporate plans, the 2018–19 plan provides clearer, and more concise, descriptions of the role/activities of the department. This has improved the readability of the corporate plan, and assist a reader’s understanding of what the department actually does when it refers to ‘support’ or ‘provide advice’.

3.110 The performance section of the 2018–19 Corporate Plan has also been changed by the department and is presented on a single page — a significant reduction compared to the eight pages of the 2017–18 Corporate Plan. However, this has not led to an improvement to the appropriateness of the performance criteria.

3.111 The 16 key performance indicators and accompanying 27 measurements presented in the department’s 2017–18 Corporate Plan have been replaced by the following five ‘assessment’ points:

• the Australian Government relies on our expertise and innovative advice
• our advice is timely, of high quality and impactful
• the public service and key stakeholders capitalise on our whole of government perspective to best deliver priorities
• we achieve our objectives under key focus areas identified in this Corporate Plan
• our people demonstrate the qualities articulated under our four pillars of transformation.

3.112 Given the measures are presented at a high level, their relevance to the department’s purposes, when considered alongside the activities, is demonstrable in most circumstances. However, the reliability of the measures is limited. The measures are accompanied by a footnote that states assessments are measured annually, and lists a number of methods. As noted in paragraph 3.57, by referencing multiple potential assessment methods, readers are unable to determine the standard against which the department’s performance is to be measured. As a result, there continues to be limited insight for a reader as to how the department will demonstrate its performance.

3.113 In addition, without knowing the specific measures that will be used to measure performance, a reader cannot determine whether the corporate plan enables a complete assessment of the department’s intended performance. For example, for the measure ‘We achieve our objectives under key focus areas identified in this Corporate Plan’, there are 22 key focus areas across the three purposes, the majority of which are framed as ongoing departmental objectives.

3.114 Within the focus areas and the measure there is no indication of timeframes for achievement, including how, and whether all or only some of, the focus areas will be reported against each year. There is also no longer an indication of the measurement bases to determine a balance of quantitative versus qualitative information, and no demonstration of progress/improvement over time as the points are described as being measured annually — presenting a static view across the four years of the corporate plan.

3.115 As noted in paragraph 3.30, the role of the corporate plan is to ‘set the foundations upon which a reliable performance narrative can be built’. To fulfil this role, it is important that a corporate plan provides a reader with a clear basis from which to set an expectation of an entity’s performance. PM&C’s 2018–19 Corporate Plan does not sufficiently fulfil this requirement and should be reviewed by the department.

Were the selected entities’ processes to inform the coordination and collation of the annual performance statements effective?

4.1 Corporate planning and performance reporting should be viewed by entities as a continuous process, rather than annual and distinct processes. Finance guidance notes that entities should design processes that support the complete cycle of planning, performance measurement and reporting. This would include the preparation and publication of the PBS, corporate plans, performance statements and annual reports, evaluating lessons learnt and incorporating these into the following cycle to contribute to continuous improvement.⁶³

4.2 The ANAO considered whether the processes implemented by the selected entities reflected a whole-of-cycle approach to performance measurement and reporting. The ANAO also considered whether those processes were supported by clearly defined roles and responsibilities and adequate structures to facilitate management engagement and support decision making.

Roles and responsibilities

4.3 All four entities had assigned responsibility for the coordination and collation of budget statements, corporate plans and performance statements to a central coordination area. AGD and PM&C’s sections reported to the Chief Internal Auditor or equivalent, DFAT’s were assigned within the Executive Branch, and Education’s reported to the Chief Financial Officer.

4.4 Each of the entities adopted a similar approach to the coordination and collation of the PBS, corporate plan and performance statements. The central coordination areas were responsible for developing and providing policy, guidance and templates to assist business areas in submitting information. This information was then collated and reviewed by the central coordination areas, before feedback or queries were provided for action to the business areas. The central coordination areas were also responsible for monitoring risks, timelines and engagement with the Audit Committee and other Executives.

4.5 The entities assigned responsibility for particular measures to a single person who was required to assure any submissions accompanying that measure. In Education, additional ‘data owners’ were also assigned to each performance criteria presented in the corporate plan and PBS. This was to assist in the identification of relevant data sources, and accompanying assurance processes and/or limitations, for each performance criterion and measure in advance of the performance statements being compiled.

Monitoring

4.6 As an entity’s primary planning document, the corporate plan is designed to signal the intended outcomes of an entity, the actions that will be taken and how the entity intends to measure its progress. It is intended to be used by the Parliament, the public and the entity to inform decision making.

4.7 Each of the entities’ corporate plans were supported by business or divisional plans. These plans were intended, in all circumstances, to align to the entity’s corporate plan, reflecting the activities and measurements that were expected to contribute to the achievement of the corporate plan.

4.8 AGD, Education and PM&C established formal monitoring of PBS and Corporate Plan measures from the beginning of 2017–18. Education and PM&C were on a quarterly basis, while AGD reported bi-annually. DFAT had existing business reviews which occurred on an annual basis, but these did not necessarily reflect the corporate plan or PBS measures. DFAT subsequently implemented quarterly reporting to the departmental Executive on progress against targets in the corporate plan and PBS in late 2017–18, with the first report compiled in June 2018. This supported efforts by the Secretary to establish performance reporting as a departmental priority. For example, in a communication to all staff in June 2018, the Secretary stated:

My objective this year is to ensure the Annual Report delivers a robust, credible performance story… demonstrating to the Parliament and the Australian people, what we do, how well we do it, why we do it, what our impact is, and who benefits from our successes.

4.9 In reviewing Education’s quarterly reporting, all elements of the corporate plan were required to be included, such as capability; activities and strategies; criteria and targets; and PBS program performance. The explanations provided against the different statuses provided by performance owners for these elements were also detailed. By including all elements of the corporate plan and PBS in the quarterly reports, rather than only performance criteria, it is more likely to identify broader issues that may influence decision making at the executive level. This appeared to provide a more comprehensive basis for Executive monitoring of the department’s performance and may be considered by other entities when designing their own monitoring processes.

4.10 The availability of information should also be a key consideration in the development of performance monitoring processes. For example, a number of AGD’s and PM&C’s corporate plan measures rely on annual stakeholder surveys, resulting in information on that aspect of the department’s performance not being available to inform or influence decision-making on a more regular basis.

4.11 Similarly, DFAT’s June 2018 quarterly performance report focused on activities rather than providing an assessment of progress against the corporate plan measures or purpose. This can in part be attributed to the department’s reliance on case studies selected ex-post, which cannot typically be used to progressively demonstrate a result and cannot be reliably extrapolated across a broader population. Reducing the department’s reliance on case studies to demonstrate its performance would improve the meaningfulness of information presented as part of future quarterly reporting processes. DFAT’s 2018–19 Corporate Plan does not refer to case studies as a method of measurement, instead referring to methods such as assessments, reviews or surveys.

Whole-of-cycle approach

4.12 Each entity intended for the coordination and compilation of the performance statements to leverage from the more regular reporting that was already established. The extent to which the entities could leverage from other reporting varied, however it was clear that some benefit was derived by each, and would improve in the future as processes matured. In particular, PM&C used its May 2018 report to compile performance statements that provided an early indication of the intended structure and layout. The report also included a list of draft case studies submitted by the divisions for inclusion in the 2017–18 performance statements. This provided an opportunity for the department to determine where there may be potential issues to address prior to year-end.

4.13 As discussed in Chapter 3, each entity had incorporated, or was seeking to incorporate, lessons learned in future reporting periods. In particular, DFAT advised the ANAO of its intention to align the release of its Corporate Plan with the PBS in future reporting periods to assist in reinforcing the connection between the two documents, and improving the department’s internal planning and performance monitoring processes.

4.14 Each of the selected entities had also engaged with the Department of Finance to seek feedback and/or guidance during 2017–18. This was through direct contact with Finance, and/or participation in Finance’s Communities of Practice. Each of the entities also undertook internal audit activity during the reporting period which was intended to contribute to continuous improvement (refer to paragraph 4.75).

Were the selected entities’ systems and methodologies to collect and report performance information effective?

4.15 In designing appropriate performance criteria, an important consideration for entities is the information the entity will need to collect to report against those measures. Finance’s RMG No. 131 Developing good performance information, sets out that an entity should consider the following for information used to report performance:

• Availability — is information readily accessible or feasible to access;
• Suitability — is the information accurate, auditable and reliable;
• Timeliness — the information is collected on a timescale relevant to the measure; and
• Cost — the effort required to collect and analyse information should be commensurate with the benefit it provides.⁶⁴

4.16 One focus of this audit was whether the selected entities had established systems and methodologies to support the collection of performance information that could be relied upon as a complete and accurate reflection of the results. In assessing the approaches adopted by the entities, the ANAO considered the following:

• the method and frequency used to collect the data for each criterion and whether it was suitable;
• the known limitations of the data;
• how data is recorded and stored and what controls exist to ensure it is not inadvertently or purposely manipulated;
• how data is extracted from systems and what processes are applied to reach the reported result; and
• whether adequate documentation of policies and procedures to support consistent data outcomes exist.

4.17 A range of information sources were used by the selected entities in support of the results presented in the 2017–18 performance statements. These sources consisted of information systems, internally developed case studies, stakeholder surveys and collation of management information.

Information systems

4.18 Education relies on multiple information systems owned by the department, or other government entities, to source its performance information. These include:

• Schools Education Management Information System (SEMIS);
• the Department of Social Services’ (Social Services) grants system, govGPS; and
• the Department of Human Services’ (Human Services) data warehouse (Data Warehouse) and Child Care Subsidy System (CCSS).

The interrogation and extraction of data from govGPS and Data Warehouse is also performed using the Department of Jobs and Small Business’ (Jobs and Small Business) Statistical Analysis Software (SAS).

4.19 PM&C also relied on two key information systems for its performance measurement and reporting: the Parliamentary Document Management System (PDMS); and CabNet. PDMS is a whole-of-government system which automates the storage, management and control of the workflow of Parliamentary documents. CabNet is a whole-of-government system that facilitates the management, storage and distribution of Cabinet information.

4.20 Given the reliance by Education and PM&C on these systems for the purposes of performance reporting, the ANAO reviewed the effectiveness of key controls established by each entity to ensure data integrity. The use of IT systems or applications, when not adequately controlled or monitored, can increase the risks posed to the accuracy and completeness of information being relied upon to produce reports. The existence of effective controls mitigates these risks. The ANAO focused on testing the following key controls for data capture, quality assurance and reporting integrity:

• User access restriction to systems;
• Data input validation and quality assurance;
• Monitoring and reporting of changes and/or deletion of records; and
• Extraction and transformation of data.

4.21 The ANAO adopted the following criteria to assess the information system controls:

• users are uniquely identifiable, authenticated and authorised;
• user access is limited to current employees and contractors whose job requires them to have that access;
• privileged user accounts are restricted in the activity that can be performed, and privileged user activity is monitored to prevent and detect any inappropriate activities;
• only authorised changes are made to the systems, programs and data; and
• authorised changes are correctly reflected in the system and do not adversely impact on other systems and processes.

Department of Education and Training

4.22 Education has implemented entity wide information system controls for managing access and changes to SEMIS⁶⁵ to reduce the risks of unauthorised access and changes. Education places similar reliance on information system controls at Social Services, Human Services and Jobs and Small Business to reduce the risks of unauthorised access and changes being performed on CCSS, govGPS and SAS.

4.23 No significant weaknesses were identified in the relevant information system controls at any of the above departments, as they related to Education’s performance reporting for Outcome 1. An opportunity for Education to improve controls supporting its use of Statistical Analysis Software (SAS) exists, and is discussed below.

Statistical Analysis Software (SAS)

4.24 The complexity of the data collection and transformation applied following extraction from key management systems typically influence the tools used to ensure the completeness, accuracy and validity of data. The main tool used by Education for complex analysis was SAS.

4.25 SAS provides the environment for Education to collate data submitted by state and territory governments under school funding arrangements, and perform analysis of data extracted from govGPS and Data Warehouse. SAS is owned by Jobs and Small Business and resides within the Jobs and Small Business infrastructure. SAS adheres to Jobs and Small Business’ information system controls for managing access and system changes. As noted earlier, no significant weaknesses were identified in the relevant controls at Jobs and Small Business.

4.26 In reviewing the extraction and transformation of data, the ANAO considered the result for the performance criteria ‘Maintain or increase the proportion of all children attending child care who are in priority groups’ and accompanying target ‘3% Indigenous children’. To determine this result, data is extracted from govGPS and the Data Warehouse by Education personnel on a quarterly basis using internally developed SAS programs.

4.27 The SAS programs contain business rules that automatically extract and transform the data required for performance reporting. Requests for access to SAS must be approved by an appropriate Education delegate. Approved requests are then forwarded to Jobs and Small Business for processing as part of that department’s information system controls.

4.28 The SAS program code is stored in a location within Education’s network and not within the SAS program. Education restricts access to this network location through access groups. However, changes to the SAS program code are not managed through Education’s information system controls and no formal change management process has been established. The personnel responsible for developing and initiating the SAS program code have access to update and run the code at any time. A copy of the SAS program code used each quarter is stored as part of an archive of each quarterly report, however the SAS program code can still be modified within the archive.

4.29 The absence of formal change management controls increases the risk of unauthorised and inappropriate changes occurring to the SAS program code. Although access to SAS program code is restricted to authorised staff, the location of the SAS program code does not allow for changes to be logged or monitored.

4.30 The ANAO was advised by the department that to mitigate this risk, Education performs quality checks to confirm the completeness and accuracy of the transformed data. These quality checks include variance and statistical analysis, investigation of missing data, and benchmarking against current policies. If required, Education will also liaise with Social Services and Human Services as the data and systems owners. Education rely on these checks to detect any errors in the SAS program code.

4.31 Rather than relying on the manual detective controls outlined above, there would be benefit in the department considering implementing preventative system-based controls to address this risk. These may include storing the SAS program code within the SAS environment, which may provide the capability to log and monitor changes, and segregate developer access to production code and systems.

Department of the Prime Minister and Cabinet

Parliamentary Document Management System (PDMS)

4.32 PM&C relies on PDMS to determine and report the timeliness of the department’s response to requests for briefs and ministerial correspondence. PDMS is owned by the Department of Finance (Finance) but resides within the Department of Jobs and Small Business (Jobs and Small Business) infrastructure. PM&C is one of the client entities who uses PDMS to manage Parliamentary documents. No significant weaknesses were identified in the relevant information system controls for PDMS at Finance, Jobs and Small Business, or PM&C.

4.33 Dates relied upon by PM&C for the calculation of the timeliness measures are automatically recorded by the system upon workflow through the document creation, quality assurance and approval stages of PDMS. For the purposes of performance reporting, this data is then extracted by PM&C personnel on a quarterly basis, including year-end. The data is extracted via customised reports generated by using filters, and imported into a Microsoft Excel spreadsheet that relies on a combination of formulas, pivot tables, direct entry and manual checks to produce the timeliness data.

4.34 This is relied on as the primary record to determine the results reported in the performance statements. The ANAO has previously noted⁶⁶ the risks arising from using spreadsheets as a primary form of evidence. Spreadsheets lack formalised change/version control and reporting, increasing the risk of error. This can make spreadsheets unreliable corporate data handling tools, as accidental or deliberate changes can be made to formulae and data, without there being a record of when, by whom, and what change was made.

4.35 The ANAO identified minor discrepancies between this spreadsheet and the results presented in the performance statements approved by the Secretary on 3 September 2018. These were subsequently corrected by the department prior to the tabling of the annual report. PM&C has advised the ANAO that it is seeking to automate the production of quarterly reports relied on for the purpose of performance measurement and reporting through PDMS. This automation is expected to mitigate risks arising from the manual extraction practices outlined above.

CabNet

4.36 CabNet is relied on by PM&C to support the presentation of results against the measure ‘timeframes are met for 90% of Cabinet and Cabinet committee minutes for approval’ under Purposes 1 and 2. The result reflects the analysis of Cabinet and Cabinet committee minute submission dates, automatically captured and recorded as part of a system workflow, and extracted directly from CabNet on a quarterly basis.

4.37 Testing of CabNet key controls, as outlined in paragraphs 4.20 and 4.21, identified that there is no formal monitoring of the two CabNet privileged users’ access, or use, by PM&C. Similarly, there are no standardised processes to manage, log, approve or document changes to the CabNet application and its data. In addition, the ANAO observed privileged users can both develop and migrate changes into production — to achieve effective control, these roles would typically be segregated.

4.38 The absence of adequate monitoring of the two privileged users, and formalised change management processes, increases the risk of accidental or deliberate changes to the program, data and/or reports, without there being a record of when, by who, and what change was made. This increases the risk of inaccurate reporting in the department’s performance statements.

4.39 The department advised the ANAO that it relies on business processes external to CabNet to identify any potentially inappropriate and/or unauthorised changes to data held in the system. For example, ongoing senior executive involvement and oversight of the preparation and finalisation of documentation supporting Cabinet decisions.

4.40 PM&C also noted that CabNet was not relied on, or referred to, as the formal record of Cabinet decisions — these were retained in paper form outside of the system. As such the department viewed the matters outlined above as posing a low risk to the integrity of Cabinet information held in, or drawn from, CabNet. A formal risk assessment that identified or incorporated these risks had not been provided to the ANAO by the department at the time of the audit.

4.41 PM&C has advised the ANAO that a new system intended to replace CabNet is currently being developed. The ANAO recommends the department consider the above observations in the design of any replacement system.

Case studies

4.42 As noted in Chapter 3, both DFAT and PM&C relied on case studies to report their performance in 2017–18. Case studies can be narrowly focussed examples of particular circumstances and without significant investment in appropriate upfront design, cannot typically be used to demonstrate a result that can be reliably extrapolated across a broader population.

4.43 As noted in paragraph 3.41, an entity’s performance measures should be underpinned by appropriate internal quality frameworks, developed contemporaneously, that support the reliability of the reported result. This expectation applies equally to case studies. If an entity wishes to rely on case studies as a form of measurement, some better practice characteristics are outlined in the box below, to assist in designing case studies that can reliably contribute to performance reporting.

Department of Foreign Affairs and Trade

4.44 DFAT relied on case studies for six of the ten measures considered by this audit⁶⁷. The above characteristics were not evident in the department’s determination of case studies or reviews reported under Priority Functions 1 and 2 in the 2017–18 Performance Statements. In all instances, DFAT’s documentation could have been improved to identify at the corporate planning stage the parameters used in the identification and development of case studies presented in the performance statements.

4.45 The introduction of quarterly performance reports to the Executive in June 2018, provided an opportunity for divisions to consider what may be reported as part of the 2017–18 Performance Statements. However, as noted in paragraph 4.11, there was insufficient detail presented in those reports to determine what would be reported. For example, the June 2018 quarterly report included titles of proposed case studies and topic sentences. However, no other information was presented as to how the case studies were selected, the information sources that would be used to compile the case studies, or how they were intended to demonstrate the relevant performance criterion.

4.46 As noted in Chapter 3, for the Priority Functions reviewed, the ANAO was unable to conclude whether the department’s proposed performance measures would enable a complete assessment of DFAT’s performance at the corporate planning stage. This was largely due to the department not having established a framework for the development and selection of case studies that reflected the above characteristics.

Department of the Prime Minister and Cabinet

4.47 Similarly to DFAT, the above better practice characteristics were not evident in PM&C’s determination of case studies or reviews reported in the performance statements. PM&C also referred to multiple measurement methods in its corporate plan, rather than identifying a particular one and providing sufficient information for a reader to form an expectation of the intended result.

4.48 Where case studies were presented in PM&C’s performance statements they typically provided an example of an output arising from the department’s activities, rather than a robust and broader qualitative assessment against the performance measures and accompanying ‘Achieved’ result. There were circumstances where the department presented multiple case studies, such as under the ‘Policy advice’ activity, which may have been sufficient to materially support the result of ‘Achieved’ for ‘show PM&C significantly influenced decision-making by government’, however there was no accompanying analysis. This affected the meaningfulness of the analysis presented to the Parliament and the public in the performance statements (Chapter 2).

Stakeholder surveys

4.49 Both AGD and PM&C relied on stakeholder surveys for the purposes of presenting results in the 2017–18 Performance Statements. In considering the appropriateness of the stakeholder surveys as reliable sources for performance reporting, the processes of each department in the determination and conduct of the survey were also considered.

4.50 AGD’s stakeholder survey is used to support the results presented against six of the 12 performance measures relevant to Strategic Priorities 3 and 5. PM&C uses its stakeholder survey to support the results presented against ten of the department’s 27 measures. Each measure seeks to quantify the satisfaction of different stakeholder cohorts with the activities of PM&C.

4.51 Both AGD’s and PM&C’s stakeholder surveys are conducted annually. Each department undertakes an open tender process for the selection of professional survey and statistics service providers each year. This approach promotes the transparency and independence of the survey process. The management of the survey process in each entity is also assigned to the same areas responsible for the collation of the department’s performance statements.

4.52 In considering whether the stakeholder surveys are an appropriate information source for the entity’s measures, the ANAO also considered whether they could be relied on to provide an accurate reflection of the measures as described. This was determined by considering the design of the measures, the underlying survey methodology including selection of stakeholders, and response rates.

Attorney-General’s Department

4.53 The following measures under Strategic Priorities 3 and 5 rely on AGD’s stakeholder survey to present results in the performance statements:

• two measure stakeholder or client satisfaction with the department’s effectiveness (KPI 1);
• two are intended to measure whether AGD’s ‘work is completed on time, within budget and in compliance with relevant guidelines’ (KPI 2); and
• two measure stakeholder or client satisfaction with the department’s capability (KPI 3).

4.54 It is clear that KPIs 1 and 3 are designed to report stakeholder satisfaction and the survey is an appropriate source for the intended results. This is not the case for KPI 2, which seeks to measure the department’s efficiency, among other things, and a more appropriate source would be the department’s internal monitoring of its timeliness, costs and compliance. AGD should consider addressing this matter by:

• reviewing the performance criteria so the result intended to be reported is clear to a reader; or
• using an information source that leads to measurement of the performance criteria as described; and
• clearly documenting the information source and methodology to support a reader’s understanding of the intended result.

4.55 In determining who to survey, AGD business units were all requested to submit the contact information of stakeholders and/or clients they had engaged with in the previous 12 months. These submissions formed the total pool of persons asked to respond to the survey. No weighting was applied to the stakeholders as part of the analysis of responses or the reported results.

4.56 Responses were received from 626 stakeholders (of 1,684), across more than 250 different organisations — a response rate of approximately 37 per cent. In comparison to the 2016–17 survey, the response rate has declined by approximately 5 per cent. AGD advised that the response rate was affected by the departure of certain business groups following the 20 December 2017 Administrative Arrangements Order, and the response rate for the groups that remained were consistent with the prior year.

4.57 Given AGD’s reliance on the stakeholder survey, it is important that the department review and continue to monitor the processes supporting the nomination of stakeholders to reduce the potential for bias. Similarly, response rates should be monitored to ensure they remain at a sufficient level to reliably inform a representative assessment of the department’s performance.

Department of the Prime Minister and Cabinet

4.58 The measures presented by PM&C have been clearly designed to report stakeholder satisfaction. For example, ‘Feedback from the Prime Minister, portfolio ministers, the Cabinet, ministerial officers and the Executive shows a high level of satisfaction with the quality and timeliness of advice received.’ Each activity in the corporate plan had an accompanying satisfaction measure, as a result the stakeholders surveyed formed three different groups: internal; ministerial; and external.

4.59 Under the methodology, certain stakeholders carried heavier weightings when considered in the analysis, relevant to the area being surveyed. For example, responses relevant to the Indigenous Affairs Group provided by the Minister for Indigenous Affairs carried a heavier weighting than other Ministers. Similarly, external stakeholders were categorised based on a three tier scale, self-assessed by divisions, of the stakeholders’ criticality to the success of their performance. This categorisation then carried through to the weightings applied to responses. PM&C has established processes to promote the consistency of these categorisations, including holding workshops between staff and the survey provider to consider the appropriateness of categorisations applied.

4.60 The overall response rate to PM&C’s survey by all stakeholder groups was 42 per cent — representing 284 respondents from a total stakeholder pool of 671. The response rates of the Executive and Ministerial groups were higher at 67 per cent. The department extended the original survey response period to ensure a response rate of greater than 40 per cent. Given PM&C’s reliance on the stakeholder survey, it is important for the department to continue to monitor response rates and ensure they remain at a sufficient level to continue to reliably reflect a representative assessment of the department’s performance.

Management information

4.61 When relying on internal assessments of management information, the development and documentation of measurement methodologies is important to support the collection of performance information that can be relied upon as a complete and accurate reflection of the results. There are opportunities for improvement to AGD’s, DFAT’s and Education’s consideration and documentation of internal management information in support of the entities’ performance measurement and reporting.

Attorney-General’s Department

4.62 The measure ‘Evaluation and design review process’ and accompanying target ‘For all major projects’ is presented under Strategic Priority 5 and relies on an assessment by AGD. The department did not have a clear idea of the major projects it intended to subject to evaluation and design review processes when it released the 2017–18 Corporate Plan. The exception was the ‘Optional Protocol Convention Against Torture and other Cruel, Inhuman or Degrading Treatment of Punishment’ which was listed as a key activity in the corporate plan.

4.63 There was also no evidence of documentation that outlined what an ‘evaluation’ or ‘design review process’ would or should entail. It is important for the department to have a clear and documented approach to how a project will be identified, and what processes should be undertaken for it to be considered as part of the reported result. Without this, the risk of biased and/or inaccurate and inconsistent reporting in the performance statements is heightened.

Department of Foreign Affairs and Trade

4.64 While a range of information sources were used in support of the ‘Overview and analysis’ and ‘Results’ sections of DFAT’s performance statements, reliance on observations by DFAT officers was common. This led to the performance statements including statements such as:

• Our efforts helped to galvanise the defence of the global, rules-based international system, including bolstering key countries’ commitment to continue fighting protectionism;
• Australia’s special and wide-ranging strategic partnership with Japan continued to develop and influence regional stability and prosperity; and
• Our efforts in Jakarta were critical to achieving consensus with the ASEAN Committee of Permanent Representatives.

4.65 Officer observations can be subject to interpretation and as a result may, in some circumstances, present a higher risk of presenting inaccurate or incomplete information. For this reason, it is important to ensure observations are also supported by more direct evidence demonstrating how and why a conclusion was reached.

4.66 There is a well-established process for the consolidation, communication and validation of officer observations in regard to DFAT foreign aid investment outcomes through the preparation of annual Aid Quality Checks. A sample of which are then independently assessed by the Office of Development Effectiveness.⁶⁸ While it may not be feasible to establish a similar function to evaluate the reliability of outcomes attributed to the department’s other activities, the benefits of applying similar principles of a quality assurance framework should be considered by DFAT.

Department of Education and Training

4.67 A small number of Education’s measures relied on administrative data held by the department, or submitted by third party providers under contract arrangements. Typically monitoring of this data was managed through MS Excel spreadsheets. As noted in paragraph 4.34, spreadsheets lack formalised change/version control and reporting, increasing the risk of error.

4.68 Where an arrangement was subject to contractual obligations, the reports submitted by providers were typically accepted as being complete and accurate. The department has made this limitation clear in the performance statements, noting where measures rely on third party reporting. However, while this provides a reader of the performance statements with a clear understanding of the potential limitations, there remains a risk of inaccurate reporting.

4.69 The ANAO observed in discussions with program areas varying approaches to reviewing information submitted by third party providers. Some for example were wholly reliant on the provider, and their history of fulfilling contractual obligations, in the absence of being able to negotiate access to the provider’s underlying records.

4.70 In other cases, areas had established processes to increase their confidence in the data relied on for performance reporting. For example, requesting the provider’s complete database to accompany its regular higher level progress reporting. This enabled the program area to compare consistency of the reporting to the underlying data set, compare to previous datasets, and identify and query/confirm any unexpected movements or inconsistencies. Through checks of this nature, the department has a higher level of confidence in the completeness and accuracy of the data being provided and should be considered for other third-party reporting.

Were the selected entities’ processes to obtain assurance over the annual performance statements effective?

4.71 Section 16F of the PGPA Rule requires an accountable authority to certify that their performance statements accurately present the entity’s performance for the reporting period. An equivalent certification by the accountable authority is required by section 42 of the PGPA Act for the annual financial statements. It is important that an accountable authority has confidence that the entity has adequate processes in place to provide sufficient assurance to support these certifications.

4.72 The requirement for entities to prepare and publish their financial statements, including mandatory annual audits by the ANAO, has been in place for a number of decades. As a result, the financial statements preparation processes of Commonwealth entities are mature, and supported by established assurance frameworks. Entities may leverage from these frameworks and implement similar processes to provide an accountable authority with sufficient assurance to certify the performance statements. Relevant processes may include:

• documented procedures and processes supporting the performance reporting cycle;
• management reporting of progressive results to internal stakeholders such as the accountable authority and audit committee;
• effective direction of internal audit resources;
• management certifications supported by suitable quality assurance of results; and
• detailed audit committee review and endorsement.

4.73 Earlier sections of this chapter considered the entities’ processes and procedures, including progressive management reporting and monitoring to support the performance statements. The following sections focus on the entities’ use of internal audit, management certifications and audit committee involvement and endorsement, as assurance mechanisms for their accountable authority’s certification.

Internal audit

4.74 It is important for an entity to consider how their internal audit function might be utilised to support the accountable authority, and audit committee, in improving an entity’s performance measurement and reporting. This contributes to the confidence of an accountable authority in the appropriateness, and completeness and accuracy, of the entity’s performance reporting.

4.75 All four entities had undertaken an internal audit, with specific consideration of elements of the entities’ performance measurement and reporting frameworks, in the current or previous financial year. The objective and scope of the internal audits varied across the entities. Each of the entity’s internal audit functions identified areas of improvement consistent with observations made elsewhere in this report.

4.76 The ANAO also observed that PM&C was the only entity to carry out an internal audit ‘in-flight’ to provide assurance of the results intended to be reported as at 30 June 2018. This is a practice other entities may consider to provide confidence to their accountable authority in the completeness and accuracy of the performance statements.

Attorney-General’s Department

4.77 AGD’s internal audit function conducted an audit of AGD’s 2016–17 performance framework in December 2017. The objective of the audit was to ‘review the adequacy of the Department’s existing framework for reporting against the performance criteria outlined in the corporate plan.

4.78 The report concluded that ‘the department’s performance reporting framework was generally fit for purpose’, and made recommendations in regard to the following areas for improvement:

• Methodology — the breadth of evidence used to assess performance;
• Record keeping — in relation to the Standard Operating Procedure guidance [explicitly commenting] on the type of evidence that needs to be maintained; and
• Continuous improvement — through the development of a formal process for identifying and applying lessons learned from the performance reporting for future business planning.

4.79 The recommendations were agreed to by management in March 2018. Since this time, AGD has implemented processes to address the recommendations for the 2018–19 performance reporting cycle.

Department of Foreign Affairs and Trade

4.80 DFAT’s internal audit function finalised an audit of DFAT’s 2016–17 Performance Statements in August 2018. The objective of the audit was ‘to provide assurance to the Audit and Risk Committee (ARC) that the 2016–17 annual performance statement was supported by appropriate and sufficient records.’

4.81 The report identified areas of good practice, and also opportunities for improvement although no recommendations were made. This was on the basis that ‘Changes to the annual performance statement preparation process have been implemented for 2017–18, including more clearly defined processes and guidance for divisions and posts, and formalisation of quality assurance processes.’ As noted in paragraph 3.47, the department’s 2017–18 Performance Statements relied largely on case studies or reviews selected ex-post. This limits the department’s ability to rely on internal audits of this nature (accuracy of records) across years, given the case studies are subject to change each year.

Department of Education and Training

4.82 Education carried out an internal audit in May 2018 in support of its 2017–18 performance measurement and reporting. The objective of the audit was to ‘determine the adequacy of the department’s existing processes for reporting against performance criteria outlined in its PBS and Corporate Plan’.

4.83 The internal audit made three recommendations agreed to by management aimed at documenting methodologies and assurance processes and clearly explaining any inherent limitations to the performance measures in the 2017–18 Performance Statements. The ANAO observed the department had already taken steps toward addressing these recommendations in the preparation of the 2017–18 Performance Statements.

Department of the Prime Minister and Cabinet

4.84 PM&C completed two specific pieces of internal audit work in regard to the department’s 2017–18 performance measurement and reporting. The objectives of each engagement are reproduced below:

• review the adequacy of the Department’s 2017–21 Corporate Plan and 2018–19 PBS Key Performance Indicators (KPIs) in meeting the requirements of the Commonwealth performance reporting framework.
• assess the accuracy of results reported in the APS [Annual Performance Statements] against a sample of performance criteria outlined in the Corporate Plan. The reliability of underlying processes for collecting performance information and the quality assurance activity conducted on reported results was reviewed.

4.85 Three recommendations were made across the first audit, which were agreed to by management. These recommendations focused on improving the department’s development of future measures, documenting a strategy for reporting against revised criteria, and specifically enhancing criteria and targets of programs under Outcome 2 in the development of the 2018–19 Corporate Plan.

4.86 The second internal audit, focusing on the accuracy of results, was intended to assist the audit committee’s consideration of the performance statements. The audit did not identify any issues, but did note areas where ‘practices could be improved’ such as improving the consistency of assurance processes and standardising reporting from PDMS.

4.87 PM&C initiated a further review engagement to ‘assess the extent to which a sample of potential case studies to be used in the 2017–18 Annual Performance Statement are supported by evidence’. ⁶⁹ The review concluded that the majority of case studies were well supported by underlying evidence. PM&C removed those case studies that were not well-supported from the performance statements prior to their approval by the Secretary.

Management certifications

4.88 Written certifications provide a formalised model for the communication of assurance between an entity’s management and its accountable authority. The value of certifications are influenced by the robustness of the processes underpinning them, their completeness, and also the attention and/or action taken in resolving matters identify by those certifications.

4.89 All four entities adopted multi-level management certifications as part of assurance processes for the 2017–18 Performance Statements. The ANAO did not observe any significant deficiencies in the entities’ certification processes, beyond the issues raised in this report that indicate the entities’ overall performance measurement and reporting could be improved.

4.90 For example, as noted in paragraph 4.35, the ANAO identified minor discrepancies in PM&C’s performance statements compared to records provided by the department after they were presented to the Secretary for approval on 3 September 2018. The brief recommended the Secretary approve the performance statements on the basis of advice that the performance statements were supported by sign off by Executive Board members as to the accuracy of the statements, assurance by internal audit, and advice from the Chair of the Audit Committee. The changes made to the performance statements following these certifications indicate that, in some instances, the assurance processes underpinning these certifications could be improved.

The role and function of audit committees and entity performance reporting

4.91 The accountable authority of a Commonwealth entity is required to ensure that the entity has an audit committee and to determine by written charter the functions of this committee. The minimum functions of an audit committee are set out in the PGPA Rule and include reviewing the appropriateness of the accountable authority’s performance reporting.

4.92 As noted in paragraph 1.29, in May 2018, Finance released Resource Management Guide No. 202: A guide for non-corporate Commonwealth entities on the role of audit committees, accompanied by a model audit committee charter. The guidance reiterates the audit committee’s role in relation to performance reporting, noting that to fulfil its function:

an audit committee must review the entity’s performance information, systems and framework and the completeness and appropriateness of performance reporting. The review would include information provided in the Corporate Plan, the Portfolio Budget Statements and the Annual Performance Statements

4.93 Finance guidance also notes that an audit committee’s advice to the accountable authority should be ‘documented in the form of a written statement of its view on the appropriateness of the accountable authority’s’ performance reporting’. The guidance also notes that a committee should ‘not merely state that it does not know of anything that would indicate the accountable authority’s’ performance reporting for the entity is not appropriate’. ⁷⁰

4.94 All of the entities’ audit committees provided written advice to the relevant accountable authorities outlining their view in accordance with their charter. Each committee’s advice contained varying levels of detail setting out the basis for their view. The ANAO observed that, at a minimum, each of the audit committees:

• incorporated a standing agenda item on performance reporting matters, including development of performance measures and the corporate plan;
• received briefings and/or presentations on the processes adopted by the entities to compile and support the accuracy of the performance statements, including internal audit reports, where completed; and
• received the draft performance statements, and assurance from management representatives of the compliance, appropriateness and accuracy of the performance statements.

4.95 The level of assurance an audit committee is able to provide to an accountable authority is influenced by the expertise and experience of its members, and how that is utilised in evaluating the design and implementation of an entity’s performance reporting framework. Audit committees are also typically able to influence and/or direct the work of internal audit and, as evidenced by the examples provided from paragraph 4.77, may provide a valuable resource in contributing to an audit committee forming its view on the appropriateness of an entity’s performance reporting.

4.96 Entities and audit committees should not confuse providing assurance with confirming the entity’s performance reporting is entirely appropriate. An audit committee may have identified an issue that affects the appropriateness of the accountable authority’s performance reporting and, while still providing assurance, have drawn it to the attention of the accountable authority in its advice. This was demonstrated by PM&C’s audit committee which noted in its advice to the Secretary that it had ‘suggested a number of improvements’ but these ‘do not prevent you from signing the Performance Statement’.

4.97 The observations made in other sections of this report regarding compliance with the PGPA Rule, and the appropriateness of the selected entities’ measures, systems, and methodologies indicate that there is still some way to go in the maturity of audit committees’ assurance and advice in this area.

Did the selected entities’ annual performance statements accurately present their performance?

4.98 An accountable authority is required to state, under section 16F of the PGPA Rule, that the performance statements accurately present the entity’s performance in the reporting period. This is reinforced by section 37 of the PGPA Act that sets out the requirement for Commonwealth entities to keep records that properly record and explain the entity’s non-financial performance.

4.99 In determining whether the selected entities’ performance statements accurately presented their performance the ANAO reviewed if:

• the results enabled a complete assessment of the entities’ performance on the basis of the assessment of their appropriateness (Chapter 3); and
• records retained by the entities supported the reported results.

Attorney-General’s Department

4.100 For Strategic Priorities 3 and 5, AGD’s performance statements accurately presented their performance. AGD also retained adequate records to support the results presented for those measures under Strategic Priorities 3 and 5 in the performance statements. However, for one measure, the ANAO was unable to fully conclude whether the reported result ‘Met’ was appropriate based on the evidence provided.

4.101 As noted in paragraph 4.62, AGD did not establish at the corporate planning stage the method for assessing the measure ‘Evaluation and design review processes’ and accompanying target of ‘For all major projects’. AGD reported a result of ‘Met’ in the performance statements, and this was on the basis of some projects referring to evaluations or reports that have been completed, while others simply stated what the department intends to do. The variation amongst the projects considered, their statuses and associated evaluation or review activities made it difficult to determine how the reported result of ‘Met’ is appropriate.

4.102 In the absence of a documented methodology for ‘evaluations’ or ‘design review processes’, the ANAO is unable to conclude whether the reported result is appropriate, as the completeness of the result could not be confirmed.

Department of Foreign Affairs and Trade

4.103 As set out in the previous sections, DFAT’s performance criteria relied on case studies or reviews. As previously noted, case studies can be examples of particular circumstances and without significant investment in appropriate upfront design, cannot typically be used to demonstrate a result that can be reliably extrapolated across a broader population. This is the case for the results presented by DFAT for Priority Function 1 and 2 — it is unclear whether they can be relied on as being representative of the department’s broader performance.

4.104 As a result, the ANAO’s assessment of the accuracy of the performance statements is limited to whether or not the department had sufficient records to support the information presented in the performance statements, rather than whether they could be relied on by a reader to present an accurate picture of the entity’s performance. Of the results considered by the ANAO, DFAT provided sufficient records to support their reporting for the Priority Functions considered.

Department of Education and Training

4.105 Education’s performance statements, for Outcome 1, accurately reflected the department’s performance. Education relied on a combination of information sources, including information systems, publicly available statistical data and program information compiled by third party providers to record and collate results presented in the performance statements. As noted in paragraph 4.68, the department’s performance statements acknowledge limitations that may affect the suitability of some of the information sources used. However, results presented against the performance measures were supported by records.

Department of the Prime Minister and Cabinet

4.106 As noted earlier, PM&C relied on three key information sources to record and collate information used for the performance statements: a stakeholder survey; IT systems; and case studies. As noted for DFAT above, case studies cannot typically be relied on to make broader extrapolations of performance unless specific criteria, such as those outlined on page 69, are met. However, PM&C was not solely reliant on the case studies to demonstrate its performance. This provided a more comprehensive basis for the ANAO to assess whether PM&C’s performance statements accurately presented the department’s performance.

4.107 Overall, PM&C’s performance statements accurately presented the department’s performance, on the basis of the records retained. However, as noted in paragraphs 4.34 to 4.41, the ANAO observed opportunities for PM&C to improve systems, processes and documentation in support of its performance reporting.

Appendix 1 Entity responses

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix 2 Selected entities’ performance criteria

1. Each of the selected entities’ performance criteria, considered as part of this audit, are presented in the following tables. As demonstrated by Figure 1.3, each entity has adopted different labels for elements of the corporate plan, including performance criteria. The presentation of performance criteria in the following tables reflect this variation.

Table A.1: AGD performance criteria

Table A.2: DFAT performance criteria

Table A.3: Education performance criteria

Appendix 3 Requirements for annual performance statements

1. Performance statements under the PGPA Act are required to be published in entities’ annual reports. Specific requirements for the presentation of performance statements are outlined in the box below.

2. Section 37 of the PGPA Act sets out the requirement for Commonwealth entities to keep records that properly record and explain the entity’s non-financial performance as outlined in the box below.

Appendix 4 Department of Finance Quick Reference Guide – RMG 131 Developing good performance information

1. The Department of Finance’s Quick Reference Guide – RMG 131 Developing good performance information was released in September 2016 to assist entities in the development of good performance information.

Table A.5: Quick Reference Guide – RMG 131 Developing Good Performance Information

Source: Department of Finance, Quick Reference Guide – RMG 131 Developing good performance information, September 2016.

Appendix 5 ANAO criteria for the assessment of the appropriateness of performance information

1. To undertake an assessment against the Department of Finance’s Quick Reference Guide – RMG 131 Developing good performance information (refer to Appendix 4), the ANAO has applied the following audit criteria. This criteria has been applied for audits of performance information since Auditor-General Report No.58 2016–17 Implementation of the Annual Performance Statements Requirements 2015–16. The assessment characteristics and explanations have been updated over time to reflect the ANAO’s methodology development.

Table A.6: Criteria for the assessment of the appropriateness of performance information