Background

1. Performance reporting frameworks have been in place in the Australian public sector for several decades, to enable the measurement and assessment of the impact of government programs. The current performance measurement and reporting requirements for Commonwealth entities are established under the Public Governance, Performance and Accountability Act 2013 (PGPA Act) and the accompanying Public Governance, Performance and Accountability Rule 2014 (PGPA Rule). These are supported by the enhanced Commonwealth performance framework, which took effect on 1 July 2015.

2. The framework aims to provide financial and non-financial information to the Parliament and the public, and improve the line of sight between the use of public resources and the results achieved by government entities. It differs from previous frameworks in that it broadens the performance information collected and reported by entities, and introduced the requirement for entities to publish corporate plans at the beginning of each reporting period. The framework also introduced the requirement for entities to publish annual performance statements in their annual report. Annual performance statements provide an assessment of entities’ progress in achieving their purpose/s, as set out in their corporate plans and aligned to the Portfolio Budget Statements.

3. Portfolio Budget Statements, which are updated throughout the year¹, are required to describe at a strategic level, the outcomes intended to be achieved with the funding appropriated by the Parliament. The performance criteria presented in the Portfolio Budget Statements are required to be a strategically focused subset of the performance information reported in an entity’s corporate plan.² This positions corporate plans as the primary document for setting out an entity’s planned non-financial performance and provides the reader with an understanding of how this will be measured and assessed.

4. The Department of Finance (Finance) is responsible for the whole-of-government administration of the enhanced Commonwealth performance framework, and related legislation. Accountable authorities³ are responsible for the implementation of the framework within their entities. To assist entities in implementing the enhanced Commonwealth performance framework, Finance has provided various forms of guidance and support.

5. The ANAO plays a role in advising the Parliament, and the Joint Committee of Public Accounts and Audit (JCPAA), on the implementation of the reforms to the enhanced Commonwealth performance framework. One aspect of this role includes undertaking audits to provide feedback on, and influence the development, implementation and operation of, an effective performance framework.

6. The Auditor-General’s responsibilities, as set out in the Auditor-General Act 1997, include auditing the annual performance statements of Commonwealth entities in accordance with the PGPA Act. The PGPA Act does not require the Auditor-General to conduct annual audits of performance statements unless requested by either the Minister for Finance or the responsible minister. This means that the Parliament does not receive assurance, as a matter of course, on performance statements included in annual reports, as it does over financial statements, where an independent audit is mandatory.

7. The ongoing implementation of the Commonwealth’s resource management framework, including the enhanced performance framework, will continue to be a focus in future ANAO audit work programs.

Audit objective and criteria

8. The objective of the audit was to examine the implementation of the annual performance statements requirements under the PGPA Act and the enhanced Commonwealth performance framework.

9. To form a conclusion against the audit objective, the following high level criteria were adopted:

• the selected entities met their obligations to publish annual performance statements;
• the performance criteria were appropriate and were reported against in the selected entities’ performance statements for 2015–16;
• the selected entities had effective supporting frameworks and processes to gather, assess, assure and report information included in their annual performance statements; and
• sufficient records were retained to support the results reported by the entities against their non-financial performance measurement frameworks.

10. The ANAO reviewed one purpose in the 2015–16 performance statements of the Australian Federal Police and the Department of Agriculture and Water Resources (the selected entities), including the supporting systems and processes.

11. This performance audit is one of three audits in the ANAO’s current work program that address key aspects of the implementation of the PGPA Act. These audits have been identified by the JCPAA as priorities of the Parliament. This will assist in keeping the Parliament, government and the community informed on implementation of the resource, risk and performance management frameworks introduced by the PGPA Act.

Conclusion

12. The Australian Federal Police (AFP) and the Department of Agriculture and Water Resources (Agriculture) met the minimum requirements for the preparation and publication of the first annual performance statements under the PGPA Act and the PGPA Rule. For both entities, the performance statements included reporting against the entities’ purposes, activities, and performance criteria reviewed as part of the audit.

13. The performance criteria were mostly relevant to the activities undertaken by the selected entities. Alignment of entity activities to performance criteria and measurement of the attribution of specific activities to the achievement of the entities’ purposes could be enhanced.

14. Both entities’ performance criteria mostly provided a reliable method of assessing the entities’ progress in fulfilling their purposes. Addressing any potential bias in the reported results should also be considered. In addition, describing the methodology for measurement and basis for assessment, including through a target or baseline, needs to be addressed to improve the reliability of the entities’ performance criteria.

15. As a whole, the performance criteria for both entities were substantially complete, collectively providing a balanced basis for assessing the entities’ progress in fulfilling their purposes. The selection of performance criteria will require ongoing effort by both entities to identify opportunities to clarify or increase the overall alignment of performance criteria to the purpose. The entities’ balance of performance criteria—for example qualitative, quantitative, efficiency-focused and short, medium and long term timeframes—should be reviewed.

16. Both entities established or adapted existing systems and processes to meet the requirements of the PGPA Act and the PGPA Rule. These remained in development during the audit, with further work being undertaken in 2016–17, to support the quality of information reported in future performance statements.

17. The selected entities established assurance processes to certify that the reported performance information accurately reflected entity performance. Planning and assurance processes for the entities should mature over time. As part of this process entities should give further consideration to the role and function of their respective audit committees, to ensure that the intent and requirements of the framework are met, as neither audit committee could fully demonstrate compliance with the PGPA Rule.

18. The majority of results presented in the selected entities’ annual performance statements were supported by complete and accurate records as required by the PGPA Act and PGPA Rule. Both entities could improve record-keeping to better demonstrate the calculations and analysis applied to raw data to produce results, and to support the analysis in the annual performance statements.

Supporting findings

Measurement and reporting of performance

19. Both of the 2015–16 performance statements reviewed as part of the audit complied with the PGPA Act and PGPA Rule, in relation to their publication. The performance statements were published as part of the entities’ 2015–16 Annual Reports. They included the required statements, results and analysis against the performance criteria outlined in the corporate plan and Portfolio Budget Statements reviewed as part of the audit.

20. The analysis section of both entities’ annual performance statements included some consideration of the entities’ operating environment and were supported in some cases by case studies and trend information. However the quality of the analysis could be improved, in particular, by providing further discussion of how the entities’ activities, through the results of the performance criteria, had contributed towards the achievement of their purpose/s and the external factors which impacted performance.

21. Both entities built on their existing external performance measurement and reporting framework, to meet the requirements of the enhanced Commonwealth performance framework under the PGPA Act. The information published by the entities in their corporate plan and Portfolio Budget Statements provides a foundation for reporting in the annual performance statements, although there was scope for both entities to improve how material was presented to achieve a clearer ‘line of sight’.

22. The performance criteria reviewed as part of the audit enabled the reporting of, and accountability for, the progress of each entity towards fulfilling their purposes.

23. The performance criteria for both entities were mostly relevant, providing a basis to make decisions on the entities’ progress in fulfilling their purposes. One of the AFP’s performance criteria required improvements to assist the reader to identify the benefit or beneficiary measured by the performance criterion and its link to the AFP’s activities. Agriculture’s performance criteria were mostly relevant, however the benefit or beneficiary was often not clear, or the focus of the measure was not clearly attributable to the entity.

24. The performance criteria for both entities were mostly reliable, providing a basis for reasonably consistent assessment of the entities’ progress in fulfilling their purposes. Improvements to two of the AFP’s performance criteria are required to limit the level of potential bias in the reported results. The majority of Agriculture’s performance criteria did not describe the method or basis for measurement, or provide a target or baseline, impacting the reliability of the performance criteria.

25. As a whole, the performance criteria for both entities were substantially complete, collectively providing a balanced basis for assessing the entities’ progress in fulfilling their purposes. To improve the completeness of the performance criteria, the selection of performance criteria will require ongoing effort by both entities to identify opportunities to clarify or increase the overall alignment of performance criteria to the purpose, and present a greater balance of performance criteria across the different forms of performance information and their timeframes.

Systems and processes to support performance measurement and reporting

26. Each entity adapted pre-existing processes for the preparation of the Annual Report, to facilitate the coordination and collation of information for the annual performance statements. In addition, both entities developed a project plan outlining the roles and responsibilities, risks and mitigating controls, milestones for delivery, and assurance mechanisms to guide the preparation of the performance statements. Neither entity completed a comprehensive pre-assessment of the processes of producing performance statements as part of their planning. The incremental development of performance reporting by both entities was focused on identifying lessons learnt.

27. Both entities had established, or leveraged from existing systems and methodologies to collect and report performance information for the purposes of the annual performance statements. The AFP would benefit from considering the frequency and extent of reviews of the methodology supporting one performance criterion, and strengthening processes for the validation of information sourced outside of management systems. Further consideration of data availability is required by Agriculture to establish a system to support the consistent collection, analysis and reporting of non-financial performance information.

28. Processes were established by both entities to provide assurance that the results reported in the annual performance statements were an accurate representation of performance. Further refinement of these arrangements is required by the entities. This would include documenting: guidance on the assurance process; and the review and endorsement of the annual performance statements to ensure that evidence of management and audit committee assurance is recorded and retained.

29. Each entity relied on management certifications over the selected performance criteria and the completeness and accuracy of underlying records. The entities’ audit committees also received regular briefings on the preparation of the annual performance statements, including details on the management certification processes. Additionally, Agriculture’s audit committee commissioned an internal audit on the Key Performance Indicators (performance criteria) and Performance Reporting, to inform the committee’s review responsibilities. The audit committee also monitored the implementation of key recommendations. However, the audit committee’s sign off to Agriculture’s accountable authority was limited, and did not meet the requirements of the department’s audit committee charter or the PGPA Rule and its intent. The AFP was unable to locate the final certification by its audit committee to the accountable authority over the performance statements, limiting an assessment against the audit committee charter, or the PGPA Rule and its intent.

30. There would be benefit in both entities further considering the role of the audit committee as a source of independent assurance to the accountable authority and how their audit committee charters and processes establish a basis to provide this assurance. The PGPA Rule provides that an audit committee’s functions must include reviewing the appropriateness of an accountable authority’s performance reporting. This function would necessarily involve the committee forming a view on how the entity should measure its performance. As a result, an audit committee’s charter, and any certification by the audit committee to an accountable authority discharging their performance reporting function, should reflect this requirement.

31. Records were largely available and supported the results and analysis sections reported in the annual performance statements for both entities. The AFP maintained complete and accurate records for all but one measure. Agriculture was unable to provide complete records for one performance criterion, and relied on an absence of advice as confirmation of compliance for another. Both entities could further improve record-keeping to demonstrate the calculations and analysis applied to raw data to produce results, and to support analysis in the annual performance statements.

Opportunities for improvement and key learnings

32. The ANAO recognises that this is the first year of published performance statements under the PGPA Act and has taken this into account in the conduct of this audit. It is expected that entity processes will take some time to mature. On this basis the ANAO has not made any recommendations in this audit, but has highlighted a range of matters which warrant further attention by the AFP and Agriculture.

33. In addition, guidance from Finance is being incrementally updated as lessons are learnt, through its role as policy owner in the Commonwealth. The need for further clarity in guidance, in particular for audit committees, has been acknowledged by the Department of Finance in recent discussions with audit committee chairs.⁴ Finance has previously advised the JCPAA that section 112 of the PGPA Act provides for an independent review of the framework in 2017.⁵ This review of the framework would include the PGPA Act and the rules, and presents an opportunity for Finance to further consider the accompanying guidance.

34. Below is a summary of key learnings identified in this audit report that may be considered by other Commonwealth entities in preparing their annual performance statements.

Summary of entities’ responses

35. Summary responses from the selected entities are provided below, while the full responses are provided at Appendix 1.

Department of Finance

The Department of Finance supports the findings of the report.

Australian Federal Police

The AFP welcomes the ANAO’s findings and acknowledges the assessment provided on the performance statement and on processes and systems related to performance measurement and reporting. The consolidated list of key learnings in the report will be useful for continued improvement.

This audit has assisted the AFP to focus efforts in ongoing performance measurement reform across the entire cycle of planning, monitoring, analysis and reporting.

The AFP will continue active participation in the Department of Finance performance community of practice and also maintain a localised law enforcement performance group to promote best practice.

Department of Agriculture and Water Resources

The Department of Agriculture and Water Resources 2015–16 annual performance statements were the first prepared under the enhanced Commonwealth Performance Framework. The department’s work under the framework is an evolving process and, as acknowledged in this report, the department has already made a range of changes to its performance framework. This audit has been an opportunity to identify further improvements.

The department agrees with most of the ANAO’s findings indicating areas in which its performance measurement and reporting can be improved. These findings will inform its continued efforts.

The department does not accept the finding that its Audit Committee did not meet the requirements of its charter, or the requirements and intent of the PGPA Rule, in providing assurance of the certification process.¹ The committee undertook a range of work to meet its charter, and the department considers the assurance provided was consistent with advice from the Department of Finance on the role of audit committees in the certification process.² The department is committed to establishing processes to ensure the Audit Committee meets the requirements of the PGPA Rule.

ANAO comments on the Department of Agriculture and Water Resources’ summary response

1. The Audit Committee’s Charter required the committee to review and provide independent advice and assurance about the appropriateness of the department’s performance reporting (see paragraph 3.49). The ANAO concluded that the advice provided to the Accountable Authority by the Audit Committee did not provide assurance about the appropriateness of the department’s performance reporting (see paragraph 3.51).

2. See paragraph 33. The Department of Finance has acknowledged the need for further clarity in guidance to audit committees.

Introduction

1.1 Since the mid-1980s, public sector management frameworks have emphasised the importance of measuring program performance. While the focus of the frameworks has changed over the years, the fundamental goals have remained largely consistent—to be able to measure and assess the impact of government programs and not just measure the activities, deliverables and associated costs. Some frameworks have also included a focus on program efficiency.

1.2 The current performance measurement and reporting requirements for Commonwealth entities (corporate and non-corporate) are established under the Public Governance, Performance and Accountability Act 2013 (PGPA Act) and the accompanying Public Governance, Performance and Accountability Rule 2014 (PGPA Rule).

1.3 The Auditor-General’s responsibilities, as set out in the Auditor-General Act 1997, include auditing annual performance statements of Commonwealth entities in accordance with the PGPA Act. The PGPA Act does not require the Auditor-General to conduct annual audits of performance statements unless requested by either the Minister for Finance or the responsible minister. This means that the Parliament does not receive assurance, as a matter of course, on performance statements included in annual reports as it does over financial statements where an independent audit report is mandatory.

1.4 The ongoing implementation of the Commonwealth’s resource management framework, including the enhanced performance framework, will continue to be a focus in future ANAO audit work programs.

Enhanced Commonwealth performance framework

1.5 The enhanced Commonwealth performance framework (the framework) took effect on 1 July 2015, to provide both financial and non-financial information to the Parliament and the public, under the PGPA Act.⁶ The aim is to improve the line of sight between the use of public resources and the results achieved by entities. A key change from the previous framework was to broaden the performance information collected and published by entities, in demonstrating progress against their purposes and outcomes.

1.6 The framework introduced the requirement for entities to publish corporate plans at the beginning of each reporting cycle from 2015–16. Plans must set out the entities’ strategies for achieving their purpose/s and determining how progress will be measured. In addition, the framework introduced the requirement for entities to prepare annual performance statements (performance statements) at the end of the reporting period. These are to be included in entities’ annual reports. Performance statements provide an assessment of the extent to which the entity has progressed in achieving its purpose/s, as set out in the corporate plan and aligned to the Portfolio Budget Statements (PBS).

1.7 PBSs, which are updated throughout the year⁷, are to describe at a strategic level, the outcomes intended to be achieved with the funding appropriated by the Parliament. The performance criteria presented in the PBSs are a strategically focused subset of the performance information included in an entity’s corporate plan. Entities are expected to clearly map the performance information from the PBSs to the entities’ purpose/s published in their corporate plans.⁸ This positions the corporate plan as the primary document for setting out planned non-financial performance and provides the reader with an understanding of how an entity intends to measure and assess its performance.⁹ Key elements of the enhanced Commonwealth performance framework, and the broader Commonwealth Resource Management Framework, are set out in Figure 1.1.

Figure 1.1: Elements of the Commonwealth Resource Management Framework

Source: ANAO analysis of guidance from the Department of Finance, incorporating relevant legislation.

Roles and responsibilities

Minister for Finance and Department of Finance

1.8 The Minister for Finance is the Minister responsible for administering the PGPA Act. The Finance Minister may, by legislative instrument, make rules prescribing matters required or permitted by the PGPA Act to be prescribed by the rules, including those relating to performance.¹⁰ On 25 April 2015, the Finance Minister amended the PGPA Rule to reflect the requirements for entities to publish corporate plans and annual performance statements from 2015–16.

1.9 As noted previously, under section 40 of the PGPA Act, the Minister for Finance or a responsible Minister also has powers to request the Auditor-General to examine and report on the annual performance statements of an entity.

1.10 The Minister for Finance is supported by the Department of Finance (Finance). Finance is responsible for the whole-of-government administration of the enhanced Commonwealth performance framework and related legislation. Finance has previously advised the Joint Committee of Public Accounts and Audit that section 112 of the PGPA Act provides for an independent review of the framework in 2017.¹¹ This review of the framework would include the PGPA Act and the rules.

1.11 To assist entities in implementing the enhanced Commonwealth performance framework, Finance has:

• published written guidance in the form of Resource Management Guides (RMGs)¹²;
• reviewed draft corporate plans and performance statements on behalf of entities, when requested;
• established the Performance Community of Practice (Performance CoP) to provide a forum to share expertise and examples of better practice and establish a feedback loop between framework design, implementation and results. Four Performance CoP performance statements workshops were conducted in 2017¹³;
• released regular Public Management Reform Agenda newsletters and emails to keep entities informed of emerging better practice and other developments;
• published lessons learned papers in February 2016 and January 2017, based on Finance’s assessment of a selection of 2015–19 and 2016–20 corporate plans, including examples of better practice; and
• released a draft lessons learned paper in March 2017 and published the final paper in May 2017, based on Finance’s assessment of a selection of 2015–16 performance statements, including examples of better practice.

1.12 Finance also completed a review of entities’ compliance with the PGPA Act and PGPA Rule in regard to the preparation and publication of annual performance statements for 2015–16. Finance’s review concluded that all Commonwealth entities required to produce annual performance statements for 2015–16 did so per section 39(1) of the PGPA Act. In addition, 82 per cent of entities were fully compliant with Section 16F of the PGPA Rule. The primary compliance matters identified through Finance’s review were minor issues in entities’ expression of the accountable authority’s introductory statement.

Joint Committee of Public Accounts and Audit

1.13 The Joint Committee of Public Accounts and Audit (JCPAA) resolved to inquire into and report on the development of the enhanced Commonwealth performance framework on 26 March 2015. Through the inquiry, the Committee sought to ensure that the rules, directions and guidance that underpin the framework provided clarity to entities regarding performance monitoring and reporting expectations, and also facilitated scrutiny.

1.14 The JCPAA released Report 453 Development of the Commonwealth Performance Framework in December 2015 and made the following observations relevant to this audit:

• The importance of a ‘clear read’ of performance information, ensuring it is presented clearly and consistently across reports produced by an entity within an annual reporting period, and across several cycles. This would also include the comparability of performance information across entities.¹⁴
• The need for strong and sustained leadership, from all levels, including all senior leadership teams within entities.¹⁵
• Active planning, monitoring, reporting and evaluation initiatives could allow programs and policies to be tracked and adjusted in real-time to improve results.¹⁶
• Central monitoring, reporting and evaluation of performance across an entity may allow specific and systemic issues to be identified and addressed.¹⁷

Portfolio Ministers

1.15 Under the PGPA Act, a Portfolio Minister has powers to:

• access the records kept about the performance of the entities within their portfolio (section 37); and
• request the Auditor-General to examine and report on the annual performance statements of entities within their portfolio (section 40).

1.16 A Portfolio Minister who makes a request of the Auditor-General, is required to table the report in each House of Parliament with the accompanying annual performance statements, as soon as practicable after the report is received.

Accountable Authorities

1.17 Accountable authorities are responsible for the implementation of the requirements of the enhanced Commonwealth Performance framework in their entities. Part 2–3 of the PGPA Act—relating to planning, performance and accountability under the Act—sets out the detailed requirements. Key requirements relevant to this audit include:

• prepare a corporate plan each reporting period that complies, is published, and is provided to the responsible Minister and Finance Minister in accordance with any requirements prescribed by the PGPA Rules;
• cause records to be kept about the entity’s performance in accordance with any requirements prescribed by the Rules;
• measure and assess the entity’s performance and comply with any requirements prescribed by the Rules; and
• prepare annual performance statements about the entity’s performance that comply with any requirements prescribed by the Rules and include it in the annual report.

Audit Committees

1.18 Audit committees are appointed by the accountable authority of an entity. The functions of an audit committee are prescribed by section 17 of the PGPA Rule, and must be set out by the accountable authority in a written charter. The required functions of an audit committee are detailed in Box 2.

Source: PGPA Rule.

Requirements for annual performance statements

1.19 The first performance statements under the PGPA Act were required to be published in entities’ annual reports in late 2016.¹⁸ Specific requirements are outlined below:

Source: PGPA Rule.

1.20 Section 37 of the PGPA Act sets out the requirement for Commonwealth entities to keep records that properly record and explain the entity’s non-financial performance as outlined below.

Source: PGPA Act.

1.21 As noted above, the accountable authority of a Commonwealth entity must also ensure that the entity has an audit committee. The committee must perform functions in accordance with the requirements prescribed by the Rules (Box 2).

Audit coverage

1.22 This performance audit is one of three audits in the ANAO’s current work program that address key aspects of the implementation of the PGPA Act. The other two audits are:

• Corporate Planning in the Australian Public Sector. This performance audit is the second in a series of audits that is assessing progress in implementing the corporate planning requirement under the PGPA Act. The first in the series was ANAO Audit Report No.6 2016–17 Corporate Planning in the Australian Public Sector; and
• The Management of Risk by Public Sector Entities. This audit is assessing how effectively selected public sector entities manage risk, including compliance with the Commonwealth Risk Management Policy.

1.23 These audits have been identified by the JCPAA as priorities of the Parliament¹⁹ and will assist in keeping the Parliament, government and the community informed on implementation of the resource, risk and performance management frameworks introduced by the PGPA Act.

Audit approach

1.24 The objective of the audit was to examine the implementation of the annual performance statements requirement under the PGPA Act and the enhanced Commonwealth performance framework. To form a conclusion against the audit objective, the following high level criteria were adopted:

• the selected entities met their obligations to publish annual performance statements;
• the performance criteria were appropriate and were reported against in the selected entities’ performance statements for 2015–16;
• the selected entities had effective supporting frameworks and processes to gather, assess, assure and report information included in their annual performance statements; and
• sufficient records were retained to support the results reported by the entities against their non-financial performance measurement frameworks.

1.25 The audit involved:

• reviewing one purpose/s statement from each entity’s performance statements, and all performance criteria established to demonstrate progress against the following strategic objectives²⁰:
  • Australian Federal Police – Federal Policing and National Security (see Table 1.1); and
  • Department of Agriculture and Water Resources – Building successful primary industries (see Table 1.2);
• reviewing internal systems, processes and procedures, including the governance and oversight put in place by entities, to support their development of the annual performance statements; and
• reviewing records and interviewing staff of the two selected entities.

1.26 The audit scope also included reviewing Finance’s role in whole-of-government administration of the annual performance statements requirements.

1.27 The ANAO recognises that this is the first year of published performance statements under the PGPA Act and has taken this into account. It is expected that entity processes will take some time to mature. Finance is incrementally updating guidance as lessons are learned. On this basis the audit seeks to share key learnings and point to areas for improvement.

1.28 The audit was conducted in accordance with the ANAO Auditing Standards at a cost to the ANAO of approximately $390,700.

1.29 The team members for this audit were Corinne Horton, Jennifer Hutchinson, Kara Ball, Alicia Vaughan and Michael White.

Table 1.1: Audit scope: performance framework for the Australian Federal Police²¹

Note a: Supplementary notes from the original publications have been removed.

Note b: This reflects the AFP’s PBS Programme 1.1.

Source: Australian Federal Police 2015–19 Corporate Plan.

Table 1.2: Audit scope: performance framework for the Department of Agriculture and Water Resources²²

Note a: Supplementary notes from the original publications have been removed.

Note b: This reflects Agriculture’s PBS Programme 1.5: Horticulture Industry, Programme 1.6: Wool Industry, Programme 1.7: Grains Industry, Programme 1.8: Diary Industry, Programme 1.9: Meat and Livestock Industry and Programme 1.10: Agricultural Resources.

Source: Department of Agriculture and Water Resources 2015–19 Corporate Plan.

Have entities met the requirements of the Public Governance, Performance and Accountability Act 2013 and accompanying Rule?

2.1 Table 2.1 outlines the ANAO’s summary assessment of the selected entities’ compliance with the PGPA Act and Rule for the presentation of performance statements. Both entities complied with section 39 of the PGPA Act, however improvements could be made to address the requirements of section 16F of the PGPA Rule more comprehensively.

Table 2.1: Compliance with PGPA Act and PGPA Rule requirements

Note a: Compliance with subsection 16F(1) of the PGPA Rule is linked to compliance with subsection (2), Item 2: Results.

Source: ANAO analysis.

Results

2.2 The selected entities addressed the requirement, under section 16F of the PGPA Rule, to provide the results of the measurement and assessment of their performance in achieving their purposes. The performance statements included results against each performance criterion and references to the location of each criterion within the entities’ PBSs and corporate plans.

2.3 Subsection 16F(1) of the PGPA Rule also requires that results are presented in accordance with the method set out in the corporate plan and PBS. Finance guidance indicates that a corporate plan should include a description of performance measures, when they will be reported on, the data collection techniques to be used and any targets the performance measures will be assessed against. Agriculture’s 2015–19 Corporate Plan did not include information on the data collection techniques that would be used to measure results. Data sources were identified in the performance statements for six of the 11 performance criteria examined.

Analysis

2.4 The analysis sections of the selected entities’ performance statements contained background information to enhance the reader’s understanding of the results, including graphical representations of historical trends for some criteria. Both entities included some analysis of the factors contributing to the reported performance, in accordance with the requirements of sub-section 16F(2) of the PGPA Rule. The analysis could have been improved by further identifying and clarifying links between the reported results, and internal or external factors that influenced these results.

Australian Federal Police

2.5 The AFP’s analysis section consisted of high level statements regarding the results against each performance criterion, and how these reflected achievement of the AFP’s purpose. Where results did not meet the expected targets, the AFP identified factors contributing to the outcome and noted actions it had taken to address these factors. There was limited discussion of changes to the environment during the reporting period which may have contributed to the AFP’s performance.

2.6 The AFP provided more detailed analysis against the individual measures in the results section. A page was dedicated to each measure, detailing historical quantitative results and comparative analysis of the previous reporting period. For some criteria, relevant case studies were also listed in the performance statements and detailed later in the annual report.

Department of Agriculture and Water Resources

2.7 Agriculture discussed its results under six sub-headings, and provided case studies and references to other publications from which information was sourced. The analysis section included consideration of the broader environment, comprising the Australian economy, the overall market sector, and seasonal conditions, which may have affected the reported results.

2.8 There was limited analysis to demonstrate how the department’s activities had contributed towards the reported results. For example, the sections on ‘Managing industry levies’ and ‘Rural research and development’ advised how much revenue was collected and distributed to research and development corporations, but not how the levies or work undertaken by the corporations had contributed to improved farm-gate returns. The inclusion of case studies of specific activities that the department had undertaken throughout the year that contributed towards the reported results would have improved the quality of reporting.

Did the entities’ corporate plans support performance measurement and reporting in the annual performance statements?

2.9 For its 2015–19 Corporate Plan, the AFP built on its existing performance measurement and reporting framework as published in previous Portfolio Budget Statements (PBS). The AFP has since commenced a project to review and redevelop its performance measurement and reporting framework. The redeveloped framework will be subject to a phased implementation commencing in 2017–18.

2.10 Agriculture commenced re-development of its performance measurement and reporting framework in January 2015. This process combined the 15 programs reported in the PBSs into five objectives, identifying effectiveness, efficiency and output performance measures and deliverables. In February 2015, a Performance Reporting Project Board was established to oversee the re-development of the department’s non-financial performance reporting framework. The department reviewed its performance framework for 2016–17 and 2017–18, and advised that it intends to review the framework annually to drive continuous improvement. Performance measures from the 2015–19 Corporate Plan were revised and an amended set of measures and targets were included in the 2016–17²³ Corporate Plan. The department advised that changes included:

• clarification of language;
• inclusion of relevant evaluation processes; and
• inclusion of new measures and initiatives.

Purpose statements and activities

2.11 Section 16E of the PGPA Rule requires that an entity’s corporate plan state the entity’s purposes over the next four years. The PGPA Act defines purpose/s as including the objectives, functions or role of an entity. The aim of the purpose/s statement is to give context to the significant activities that the entity will pursue over the period covered by the plan.²⁴ Finance guidance notes that the purpose/s of an entity should be stated in a relevant and concise manner.²⁵

2.12 Finance has defined an activity as a distinct effort of an entity undertaken to achieve a specific result (fulfilling purpose/s). Activities should be aligned with the entity’s purpose/s, and be a focus for performance measurement and reporting.

2.13 As noted in paragraph 1.7, the PBS and corporate plan are intended to be complementary and provide the foundations for the performance story presented in entity performance statements.

Australian Federal Police

2.14 The 2015–19 Corporate Plan met the requirements of the PGPA Rule and provided a foundation for performance reporting.²⁶

2.15 The AFP’s 2015–19 Corporate Plan identified the following purpose:

to enforce Commonwealth criminal law and contribute to combating organised crime and protecting Commonwealth interests from criminal activity in Australia and overseas.

2.16 The purpose reflects the strategic objectives of the entity and is derived from the Australian Federal Police Act 1979 and associated Ministerial Direction issued in May 2014. The purpose also reflects Outcome 1 as set out in the AFP’s 2015–16 PBS. Outcome 2 is not covered by the AFP’s 2015–19 Corporate Plan, as it is subject to a separate purchasing agreement with the ACT Government. Readers are directed to separate reporting on this arrangement. The AFP’s outcomes and related programs for 2015–16 are outlined in Appendix 2.

2.17 The presentation and expression of the AFP’s activities could be improved to assist a reader to identify and assess the alignment of the performance criteria to the purpose. The AFP’s program objectives set out in the 2015–16 PBS are used as the basis for describing the AFP’s activities in the corporate plan. The activities described under Program 1.1 and 1.2 collectively address the AFP’s purpose. The activities for Program 1.1 are dispersed through narrative across the sub-sections on ‘Federal Policing’, ‘National Security’, and ‘Specialist and supporting capabilities’. Each sub-section also includes information on the AFP’s organisational structures, functions, and aspirations. As a result, it can be difficult for the reader to easily identify specific activities, and obtain a clear read of how the activities contribute towards the AFP’s purpose and objectives.

2.18 The AFP identified the presentation and expression of its activities as an area for improvement as part of its ongoing review of the performance measurement and reporting framework. As a result, the AFP’s 2016–17 Corporate Plan has been structured so that activities intended to be undertaken by the AFP to fulfil its purpose are clearly articulated. This will provide the basis for a clearer read between the 2016–17 Corporate Plan and the performance statements to be included in the 2016–17 Annual Report.

Department of Agriculture and Water Resources

2.19 The 2015–19 Corporate Plan met the requirements of the PGPA Rule and provided a foundation for the reporting of Agriculture’s annual performance. The key priorities and objectives published in the 2015–16 PBS were reflected in the 2015–19 Corporate Plan purpose statement, meeting the requirements of the PGPA Act. The plan’s purpose statement comprised five purposes, three outcomes and eight objectives. Each objective included performance measures and the key activities the department planned to undertake to achieve intended results. There was scope to better focus the purpose statement, to allow the intended results to be more readily identifiable. The structure of the purpose statement was improved in the 2016–17 Corporate Plan. The three outcomes of the department have been consolidated into a clear and concise purpose statement, including intended activities and results.

2.20 In re-developing the performance framework for the 2015–19 Corporate Plan, Agriculture also considered the outcomes, programs and performance measures already published in its PBS. The 2015–19 Corporate Plan has clear links to the outcomes and programs within the PBS. The PBS performance measures and targets are a subset of the performance measures included in the 2015–19 Corporate Plan, consistent with Finance guidance. Agriculture’s outcomes and related programs for 2015–16 are outlined in Appendix 2.

2.21 This audit focused on the following strategic objective included in the 2015–19 Corporate Plan:

Building successful primary industries – Improve the farm-gate returns for agriculture, fisheries, food and fibre industries.

2.22 The key activities from the 2015–19 Corporate Plan for the ‘Building successful primary industries’ objective are reproduced in Table 1.2.

2.23 The activities were clearly identified and reflect the government priorities listed in publications such as the Agricultural Competitiveness White Paper and on the department’s website. Improvements could be made to the identification of key activities by including the specific activities that the department will undertake, rather than broad references such as ‘in accordance with government agenda’. Broad references can hinder the development of meaningful performance criteria and reduce the reader’s ability to judge the extent to which intended results were achieved, and the factors that affected the delivery of those results.

Did the entities’ performance criteria support the reporting of progress against their purposes?

2.24 The ANAO assessed the selected entities’ performance criteria for relevance, reliability and completeness. The basis for assessment has been adapted from the criteria for the evaluation of the appropriateness of key performance indicators as set out in ANAO Report No.21 2013–14 Pilot Project to Audit Key Performance Indicators. The criteria have been updated to reflect Finance’s guidance to support the enhanced Commonwealth performance framework and can be found at Appendix 3.

Relevance and reliability of entities’ performance criteria

2.25 In applying the ‘relevant’ criterion, the ANAO assessed whether each performance measure adopted by the selected entities for the strategic objectives under review:

• clearly indicated who benefited and how they benefited from the entity’s activities²⁷;
• was focused on a specific aspect of the entity’s purpose/s and activity/ies, and the attribution of the result to the entity is clear ²⁸; and
• was easily understandable.²⁹

2.26 In applying the ‘reliable’ criterion the ANAO assessed whether each relevant performance measure:

• was measurable, that is, it used and disclosed information sources and methodologies (including a basis or baseline for measurement or assessment, for example a target or benchmark) that were fit-for-purpose and verifiable³⁰;and
• was free from bias, allowing for clear interpretation and an objective basis for assessment of the results.³¹

2.27 The characteristics of the ‘relevant’ and ‘reliable’ criteria set out above are linked and observations made during an assessment against those characteristics may contribute to reporting for more than one criterion. Tables 2.2 and 2.3 summarise the ANAO’s overall assessment for the selected entities’ performance criteria. The scale used to rate each entity’s performance criteria was:

• did not display the characteristics of the criterion (No—Not relevant/reliable);
• displayed in part the characteristics of the criterion (Partly relevant/reliable);
• displayed most of the characteristics of the criterion (Mostly relevant/reliable); and
• displayed all of the characteristics of the criterion (Yes—Relevant/reliable).

Australian Federal Police

2.28 The majority of the AFP’s performance criteria were assessed to meet the relevant and reliable criteria. A summary of the ANAO’s analysis is outlined in the following paragraphs and Table 2.2. Further detail of the analysis can be found at Appendix 4.

2.29 Eight of the nine performance criteria assessed by the ANAO met the ‘relevant’ criterion. One criterion, the ‘level of external client/stakeholder satisfaction’, partly met the ‘relevant’ criterion. The satisfaction measure does not define who stakeholders or clients are, and in what context they have engaged with the AFP to be able to assess satisfaction. This limits the understanding of the performance criterion and its focus, limiting an assessment of what aspect of the purpose or activities it relates to.

2.30 The corporate plan defines and/or directs readers to further information in relation to the methodologies the measures are based on. The methodologies range from accessing results from internal databases, to participant and stakeholder surveys, and calculations that rely on inputs from internal and external sources. All of the performance criteria were capable of measurement by these methodologies, and targets have been assigned across the four year corporate plan.

2.31 Of the AFP’s performance criteria presented in Table 2.2, ‘assets restrained’, mostly met the reliable criterion, and ‘counter-terrorism investigation outcomes’ did not meet the criterion at all. This assessment was made due to the variability of the measurement outcomes of ‘assets restrained’ and the inbuilt bias of ‘counter-terrorism investigation outcomes’.

2.32 The reliability of the ‘assets restrained’ measure is impacted by the variability in the size, value and number of restraints that may occur in a given year. The ANAO notes that an explanation of contributing factors to the movement in the results of this measure is included in the AFP’s performance statements. It is important that accompanying analysis acknowledges this variability and includes evaluation of the factors contributing to the level restrained in a particular year, to allow the reader to determine the reliability of the measure as an indicator of past and future performance by the AFP.

2.33 The ‘counter-terrorism investigation outcomes’ measure compares the number of finalised (closed) counter-terrorism investigations to those that resulted in a prosecution, disruption or an intelligence referral. These three variables represent the breadth of potential outcomes for a finalised counter-terrorism investigation and invariably return a result of 100 per cent. This limits the reliability of the measure as it is biased to a positive outcome. Entities should assess performance criteria for inbuilt bias, before inclusion in their corporate plan

2.34 The remaining performance criteria were assessed to be reliable. Further discussion in relation to the suitability of the information sources supporting the performance measurement methodologies is provided in Chapter 3.

Table 2.2: Assessment of AFP performance criteria relevance and reliability

Source: ANAO analysis.

Department of Agriculture and Water Resources

2.35 Agriculture’s performance criteria mostly met the ‘relevant’ and ‘reliable’ criteria. A summary of the ANAO’s analysis is outlined in the following paragraphs and Table 2.3 below. Further detail of the analysis is provided at Appendix 4.

2.36 For five of the performance criteria, the performance statements did not specify how the department’s activities contributed to the results reported, impacting their focus and relevance. The performance measures focus on the changes in the broader Australian economy and the primary industries sector, which can lead to improved farm-gate returns (meeting the department’s purposes). The outcomes are not always within the direct control or influence of the department (for example, Portfolio industries may record an increase in productivity), or are shared with another entity and do not clearly describe Agriculture’s contribution.³² This makes it difficult to attribute the results of a particular departmental activity to changes observed in the primary industries sector.

2.37 Where it is difficult to attribute the results of a particular departmental activity to changes observed in the primary industries sector and broader environment, there would be value in acknowledging this limitation in the corporate plan. The limitation could be further addressed by including specific case studies or details of activities Agriculture will undertake throughout the reporting period that influence the results being reported. For example, Agriculture could highlight the key projects or programs that are expected to contribute towards the results reported through the performance criteria and include an analysis of the progress, or otherwise, of the specific projects or programs.

2.38 To improve the relevance of the performance criteria, Agriculture should also ensure that it is clear to readers who is the beneficiary of the activity measured by the criterion, and how they will benefit. This was not clear for six of the performance criteria.

2.39 None of Agriculture’s performance criteria in 2015–16 disclosed the method and parameters (targets, timeframes or baselines) that would be used to collect data and measure performance to enable assessment by the reader. This affects the reliability of the performance criteria presented in Agriculture’s annual performance statements. For some measures, results have been informed by old data or trend analysis across a decade or more, which does not reflect progress in 2015–16. Other measures are not able to be easily measured and rely on assumptions or estimates. The performance measures included in Agriculture’s 2016–17 Corporate Plan have addressed some of these issues with the inclusion of targets, baselines or information on the basis for how results will be measured and assessed for future performance statements.

Table 2.3: Assessment of Agriculture performance criteria relevance and reliability

Source: ANAO analysis.

Completeness of the performance criteria in assessing the entities’ activities and purpose

2.40 Finance guidance notes that it is rare for a single measure to be able to adequately determine the effectiveness of an activity. It advises that good performance information will draw on multiple sources. The quality of performance information should be emphasised over quantity. The guidance recommends a small set of measures that is sufficiently comprehensive to cover those factors that affect an activity’s performance.

2.41 Finance guidance indicates that performance measures should be a mix of quantitative and qualitative data or information. It is stated that efficiency, output and input measures will typically be used to complement effectiveness measures. The guidance also notes that in some cases, effectiveness may not be measurable, due to cost or a lack of complete information. In these cases, input, output and efficiency measures can be used as proxies for effectiveness. Entities are advised that in these circumstances, they should be clear on why effectiveness cannot be measured and how the proxy measures are suitable.

2.42 The guidance also recognises that activities often work on different timeframes, and information on one or more timeframes may not be available at the time of reporting. It is therefore recommended that the performance story told at a particular time reflect the outcome that can be reasonably expected from the relevant activities at that time.

2.43 In assessing the selected entities’ performance criteria for completeness, the ANAO considered whether the performance criteria present a basis for a collective and balanced assessment of the entity against its purpose. In particular, the ANAO considered whether the selected entities’ performance criteria:

• clearly align to the entity’s purpose through the activities identified in the corporate plan (collective);
• provide a basis for assessment of the effectiveness and efficiency of the entity in fulfilling its purpose (balanced);
• relied on a mixture of quantitative and qualitative data (balanced); and
• assess a mixture of short, medium and long-term objectives (balanced).

Australian Federal Police

2.44 The performance criteria assessed as part of the audit are substantially complete, providing a collective and balanced basis for assessing the AFP’s progress in fulfilling its purpose. Areas for improvement include clearer alignment of performance criteria to the activities, program objectives and purposes, and presenting a mixture of performance criteria that focus on short, medium and long-term objectives. In the 2016–17 Corporate Plan, the AFP has described its approach to enhancing performance measurement and reporting over the next three years. The proposed approach is to include more qualitative discussion and evaluation of overall performance.

2.45 As discussed earlier, the presentation of the activities of the AFP could be improved in the corporate plan, to assist readers to align the performance criteria to the AFP’s purpose. The ANAO identified that ‘Specialist and supporting capabilities’ could not be directly linked to the performance criteria presented in the corporate plan or performance statements. It was also not clear whether ‘External client/stakeholder satisfaction’ linked directly to a single activity, multiple activities or all AFP activities. The AFP also relies on a high proportion of output measures, rather than measures of effectiveness to assess progress in fulfilling its purpose. While this approach can be appropriate in certain circumstances, the AFP should include an explanation of why measuring outputs provides a reasonable basis for assessing its effectiveness.

2.46 Further assessment of the quantitative results against the AFP’s performance criteria may assist the AFP to present a more meaningful performance story. For example, against ‘percentage of cases before court that result in conviction’, the AFP may consider cases where a conviction was not recorded, and include a discussion of the contributing factor/s and what measures the AFP has implemented to increase the likelihood of a conviction in future cases. This type of analysis would provide users with additional information to assist in evaluating the performance of the AFP.

2.47 The AFP’s performance criteria include a mixture of quantitative and qualitative data, providing a basis for a more comprehensive performance story. Each performance criterion is reported against annually, and static targets have been set over the four years of the corporate plan for eight of the nine measures reviewed by the ANAO. ‘Assets restrained’ reports against a target of an ‘increase’ against a rolling average of the previous five years’ results.

2.48 There would be benefit in the AFP including in the Corporate Plan a description of the rationale for setting static targets, particularly where the target has been historically met, and how incremental improvement is expected to be demonstrated over time. The ANAO notes that the AFP provided historical results in the annual report appendices for performance criteria with static targets. This enables a reader to assess the AFP’s performance against targets over the longer-term, however including more detailed analysis of longer-term performance in the performance statements would be beneficial.

2.49 Table 2.4 below details the AFP’s performance criteria against the ‘complete’ criterion characteristics set out in paragraph 2.43.

Table 2.4: ANAO assessment of AFP performance criteria—completeness

Note a: Performance criteria have been mapped to specific activities by the ANAO. Note that some performance criteria may relate to more than one activity.

Source: ANAO analysis.

Department of Agriculture and Water Resources

2.50 Agriculture’s performance criteria assessed as part of the audit are substantially complete, providing a collective and balanced basis for assessing the department’s progress in fulfilling its purpose. For future reporting, consideration could be given to ensuring that there are performance criteria that cover all identified key activities, to provide a more complete overview of the department’s performance in fulfilling its purposes.

2.51 As noted in Table 2.5, of the 11 performance criteria reviewed as part of the audit:

• four can be linked to measuring the activity of supporting advice to government on how to assist primary industries to remain competitive, productive and sustainable into the future;
• three provide a basis to assess the performance of the department’s administered programs supporting Research and Development Corporations;
• two relate to the collection of levies for research, development and marketing;
• one reflects the administration of reforms in industry regulations; and
• one could not be linked to the activities listed for the strategic objective or purpose.

2.52 The performance criteria ‘Improved availability of safe, efficient and effective agricultural and veterinary chemicals’ and ‘Access to water, land, forest and marine resources for primary production is maintained or improved’ were matched to specific activities by the ANAO. This required further investigation into the details of the relevant activities, and was not immediately clear from information provided in the corporate plan. The activity of ‘support current government priorities including country-of-origin labelling and the relocation of rural research and development corporations’ could not be clearly linked to the performance criteria.

2.53 The performance criteria were balanced, providing a mixture of qualitative and quantitative measures, and covering the four year term of the 2015–19 Corporate Plan (though not specific to any one year) along with specific measures for 2015–16. The majority of the performance criteria focussed on effectiveness (six of the 11 reviewed), one had a focus on efficiency, two were output measures, with the remaining two being input measures. The ANAO notes that in the 2016–17 Corporate Plan, the input measures have become input targets. The performance measures had a mixture of short, medium and long-term timeframes.³³

2.54 Table 2.5 details Agriculture’s performance criteria against the ‘complete’ criterion characteristics set out in paragraph 2.43.

Table 2.5: ANAO assessment of Agriculture performance criteria—completeness

Note a: Performance criteria have been mapped to specific activities by the ANAO. Note that some performance criteria may relate to more than one activity under the strategic objective of ‘Building successful primary industries’. In addition, some performance criteria reflect the aggregate of activities by government/s and industries, and include factors not directly attributable to Agriculture.

Source: ANAO analysis.

Note a: Department of Finance, Resource Management Guide No. 130: Overview of the enhanced Commonwealth Performance Framework, April 2015, p. 33.

Did the entities’ processes inform the coordination and collation of the annual performance statements?

3.1 In reviewing the processes implemented by the entities in developing their first annual performance statements, the ANAO considered whether the entities:

• established, or leveraged on, structured approaches to support the development of their performance statements;
• clearly defined roles, responsibilities and accountabilities;
• consulted internal and external stakeholders; and
• fully engaged their management.

3.2 Prior to the introduction of the requirements for all Commonwealth entities and companies to prepare performance statements, entities were required to publish the results of non-financial performance measures set out in their Portfolio Budget Statements, in the Annual Report. As a result, there were pre-existing processes for the preparation of the Annual Report in both entities. These were built upon to facilitate the coordination and collation of information for the preparation of the annual performance statements.

3.3 The Department of Agriculture and Water Resources (Agriculture) established a Planning and Reporting Working Group, comprising representatives of the divisions involved in performance reporting, and their divisional business areas. The Working Group provided oversight and monitoring of the department’s business planning and performance reporting arrangements, including:

• reviewing the department’s annual business planning and performance reporting cycle;
• identifying gaps in performance information and the ability to report on planned reporting of measures;
• approving arrangements to prepare each year’s corporate plan, annual performance statements and annual report, and performance information required for the Portfolio Budget Statements;
• reviewing arrangements for the divisional business planning process, including the preparation of divisional business plans, mid-year and end-of-year divisional performance reviews;
• approving arrangements to report against the Regulator Performance Framework³⁴; and
• reporting to the department’s Executive Management Committee on these matters.

3.4 The Australian Federal Police (AFP) has an equivalent group, the Organisational Performance team, within the then Internal Audit and Business Analysis branch.³⁵ The team has been established for a number of years and is responsible for the collation of data and monitoring of organisational performance for the AFP, including performance criteria set out in the PBS, corporate plan and annual performance statements. This team was also responsible for:

• engagement with the AFP’s National Managers, Executive Leadership Committee and Commissioner, on the form and content of the performance statements;
• consultation with the audit committee;
• participation in the Department of Finance’s Communities of Practice; and
• participation in a peer group of other law-enforcement and regulatory entities to discuss learnings and future collaboration regarding performance criteria.

3.5 Both entities developed a project plan outlining the roles and responsibilities, risks and mitigating controls, milestones for delivery, and assurance mechanisms to support the preparation of their performance statements. The plans provided a basis for monitoring the progress of the performance statements and to assist relevant stakeholders understand the assurance framework underpinning the preparation processes.

3.6 Agriculture’s plan was finalised in March 2016 and the AFP received endorsement from the Audit Committee of its plan in May 2016. To be able to identify or address issues in the design or underlying processes supporting the annual performance statements, the timing of both entities’ project plans could be moved forward. A comprehensive pre-assessment of the processes of producing performance statements as part of their planning would also assist.

3.7 In addition, the entities should consider how current and planned future processes can support their annual cycle of planning and performance reporting. Corporate plans and annual performance statements are the bookends of the performance framework for Commonwealth entities³⁶, and should be viewed by entities as a continuous reporting process, rather than annual and distinct publications. Implementing processes that support objective planning and performance reporting, including the preparation of corporate plans and annual performance statements throughout the year and into the future, would also assist in standardising the assurance mechanisms that underpin entity planning and reporting.

Note a: Department of Finance, Resource Management Guide No. 130: Overview of the enhanced Commonwealth Performance Framework, reissued in July 2016, p. 9.

Were systems and methodologies established to collect and report performance information?

3.8 In designing appropriate performance criteria, an important consideration for entities is the information that the entity will need to collect to report against those measures. Resource Management Guide No. 131: Developing good performance information, issued by the Department of Finance, sets out that entities should consider the following for information used to report performance:

• availability—is information readily accessible or feasible to access;
• suitability—is the information accurate, auditable and reliable;
• timeliness—the information should be collected on a timescale relevant to the measure; and
• cost—the effort required to collect and analyse information should be commensurate with the benefit it provides.

3.9 One focus of this audit was whether the selected entities considered the above factors and established systems and methodologies to support the collection of performance information that could be relied upon as a complete and accurate reflection of the results reported in the performance statements. In assessing the approaches adopted by the selected entities, the ANAO considered the following:

• the method and frequency used to collect the data for each criterion and whether it was suitable;
• the known limitations of the data;
• how data is recorded and stored and what controls exist to ensure it is not inadvertently or purposely manipulated;
• how data is extracted from systems and what processes are applied to reach the reported result; and
• whether adequate documentation of policies and procedures to support consistent data outcomes exist.

3.10 As part of its review the ANAO also considered the extent of reliance placed on data extracted from IT systems for performance reporting, and the controls established by the entities to ensure the integrity of that data. Details of the ANAO’s assessment of the selected entities’ systems and methodologies is provided in Appendices 5 and 6. The following section provides a summary for each entity.

Australian Federal Police

3.11 The AFP maintains a KPI Business Rulebook (updated each reporting cycle) which summarises for each internal and external criterion reported:

• the data sources and collection arrangements;
• the frequency of their internal measurement or reporting; and
• any identified limitations of the data source or collection methods.

3.12 Frequent measurement, monitoring and interrogation of performance results increase the likelihood that issues affecting the completeness and accuracy of data supporting the performance criteria are identified. The frequency of the AFP’s reporting of results is high for all performance criteria, except those based on annual surveys, and is presented to senior management groups within the AFP each month. The emphasis placed on regular reporting of results by the AFP, and the engagement of its senior management groups in reviewing and considering the AFP’s performance internally, provides users with confidence in the consistency and accuracy of information presented externally by the AFP in the annual performance statements.

3.13 Six of the AFP’s performance criteria use data sourced from the AFP’s management systems, extracted through a corporate reporting module. Further discussion on the reliability of these systems can be found at paragraphs 3.22 to 3.26 of this audit report.

3.14 Only one of the criteria (return on investment) requires further analysis of the systems data to reach the reported result. The ANAO noted that the methodology and supporting inputs underpinning the performance criterion were developed over a number of years by the AFP and augmented with research work conducted jointly with the University of Queensland Institute for Social Science Research. The methodology was finalised in 2008 and 2009 and inputs to the methodology were updated in August 2013 following a review by the AFP.³⁷ No further updates to the methodology have been made since then. There would be benefit in the AFP considering the frequency and extent of these reviews given the complexity of the underlying methodology.

3.15 Two criteria are based on the results of annual surveys conducted through external research companies and the information used to report the results against these measures is drawn directly from the external reports. Engaging reputable research companies with expertise in the survey field provides a level of confidence in the quality and objectivity of the survey construction and analysis of results. The risk of potential misstatement of the results in the annual performance statements is also limited, as there is no need to perform further analysis or manipulation of the result as presented in the third party report.

3.16 The final criterion is based on a collation of responses to hardcopy questionnaires posted to the AFP from presentation participants. The ANAO noted that the calculated response rate, based on the number of participants and responses received, was 8.1 per cent³⁸, and there was no evidence of analysis of the impact of partial or incomplete responses. Without consideration of these factors, the analysis of the effectiveness of the AFP’s activity is limited.

3.17 There would be benefit in the AFP assessing the frequency and extent of reviews of the inputs for the return on investment methodology, and giving further consideration to the impact of response rates on the meaningfulness of the analysis of questionnaire responses.

Department of Agriculture and Water Resources

3.18 Documentation of the processes for collection, consolidation, validation and reporting of performance measures was not consistent, and remained at a high-level. Performance criteria data and information was drawn from operating systems across the department and a number of external sources, such as the Australian Bureau of Statistics. The performance criteria reflected a variety of relevant data collection techniques and methodologies. Data collection tools included surveys, internal business systems (including ICT data management systems), satellite imaging and advice from line areas. Agriculture has considered opportunities going forward to improve the quality and diversity of data sources and make greater use of tools such as benchmarking, stakeholder surveys, peer review and evaluation.

3.19 Agriculture would have benefited from further consideration of the availability of data to support the performance criteria in developing its methodologies. Results for two of the 11 performance criteria relied on alternative data sources to produce the result presented in the annual performance statements compared to sources identified earlier in the year. In addition, the results reported against five criteria did not fully reflect the original criterion due to the absence of available data or analysis that was expected to be available. This indicates a need for further development by Agriculture of the system for identifying and collecting performance information, including a revision of reporting frequency. Agriculture, in its review of the performance reporting framework in 2017, identified this need to reconsider the frequency of reporting of some performance criteria.

Reliance on IT systems

3.20 The use of IT systems or applications, when not adequately controlled or monitored, can increase the risks posed to the accuracy and completeness of information being relied upon to produce reports. The existence of effective controls mitigates these risks.

3.21 The ANAO focused its testing on the selected entities’ key controls for data capture, quality assurance and reporting integrity. The controls tested included:

• user access restriction to systems and any data warehouse systems;
• data input validation and quality assurance;
• monitoring and reporting of changes and/or deletions of records; and
• extraction and transformation of data.³⁹

Australian Federal Police

3.22 The AFP places reliance on its management systems (PROMIS⁴⁰ and CAD⁴¹) and supporting corporate reporting modules to record and extract data used to report results against a number of performance criteria.

3.23 The ANAO noted the maintenance of user manuals and delivery of user training to reinforce behaviours and assist in minimising errors and maintain data quality. The AFP has established an overarching National Guideline on the Security of ICT System Access (the Guideline). The Guideline defines system user obligations to securely manage access to all AFP ICT systems, including key controls for how access is granted, maintained and audited. The ANAO’s assessment of user access controls in both PROMIS and the data warehouse systems did not identify any practice that deviated from the Guideline.

3.24 The regular performance measurement, reporting and internal monitoring referred to in paragraph 3.12 above, also includes quality assurance reviews of the data being drawn from the AFP’s corporate reporting systems. These reviews are undertaken by the Organisational Performance team prior to analysis. This process includes:

• confirming the dataset has been extracted using the fields required in the KPI Business Rulebook;
• reviewing the consistency of related fields within PROMIS, for example that status reasons align to case notes;
• investigating any incompatible fields through follow up with individual case officers; and
• reviewing fields to confirm only relevant data is included in the report.

3.25 Where the Organisational Performance team identifies that data supporting performance criteria has been input incorrectly, the relevant case officer is notified and a correction is made to PROMIS. Changes cannot be made retrospectively by an operator to the CAD system,and all errors are manually confirmed and updated in the performance reporting data. These changes are recorded by the Organisational Performance team throughout the year, providing a snapshot of potential error rates arising from inaccurate data input. No formal analysis of the potential error rate and impact on the data is performed by the team.

3.26 The ANAO did not identify any IT control weaknesses that would affect the integrity of data extracted from AFP systems to support performance reporting in the performance statements.⁴²

Department of Agriculture and Water Resources

3.27 Agriculture leveraged on its existing data and management systems (Phoenix⁴³, EXDOC⁴⁴, SARA and DAISY⁴⁵, and E5⁴⁶) to generate the results presented in the performance statements, against the following performance criteria:

• high level of efficiency in collecting and distributing levies to fund research and development in research and development corporations;
• inspections of levy agent records cover at least 30 per cent of levy revenue; and
• less than 5 per cent of quota allocations are rejected because of quota certification failures.

3.28 Access to these systems is managed by Agriculture through an established control framework. This framework includes controls restricting access to management systems and to the supporting ICT environment, including monitoring any changes to system records. The ANAO’s assessment of key user access controls for the relevant management systems did not identify any control weaknesses that would affect the integrity of data extracted to support performance reporting in the performance statements.

3.29 Existing systems’ controls were relied on by Agriculture to reduce the risk of inaccurate and incomplete information being produced and used for performance reporting. The systems’ controls comprised data input validation, automated calculations and data cleansing⁴⁷ programs. For example, where a system control identified an incomplete record, it would prompt the user to seek and input additional information. These changes are then subject to automated revalidation by the system. The ANAO did not identify any IT control weaknesses related to these systems’ controls that would affect the integrity of the results presented in the performance statements.

3.30 The ANAO noted that, in some instances, data was required to be extracted and then transformed by Agriculture staff to generate the results used for performance reporting. This was the result of the systems not being originally designed to present data in the required manner for performance reporting. Clearly documenting the methodology for performing additional analysis over data extracted from systems enables a party, either internal or external to Agriculture, to re-perform the analysis supporting the annual performance statements.

Were processes established to obtain assurance over the annual performance statements?

3.31 Section 16F of the PGPA Rule requires an accountable authority to certify that their annual performance statements accurately present the entity’s performance for the reporting period. An equivalent certification by the accountable authority is required by section 42 of the PGPA Act for the annual financial statements. It is important that an accountable authority has confidence that the entity has adequate processes in place to provide sufficient assurance to support these certifications.

3.32 The requirement for entities to prepare and publish their financial statements has been in place for a number of decades. As a result, the financial statements preparation processes of Commonwealth entities are mature, and supported by established assurance frameworks including mandatory annual audits by the ANAO. Entities may leverage from these frameworks and implement similar processes to provide an accountable authority with sufficient assurance to certify the annual performance statements. Relevant processes may include:

• documented procedures and processes supporting consistent reporting;
• assessment and documentation of risk and risk mitigation strategies;
• management reporting of progressive results to internal stakeholders such as the accountable authority and audit committee;
• management certifications; and
• audit committee involvement and endorsement.

3.33 Earlier sections of this chapter considered the selected entities’ processes and procedures, including risk identification and management, and the frequency of management reporting to support the annual performance statements. The following sections focus on the AFP and Agriculture’s implementation of management certifications and audit committee involvement and endorsement, as assurance mechanisms for their accountable authority’s certification of the annual performance statements.

Management certifications

Australian Federal Police

3.34 The key management certification relied on by the AFP was provided by the National Manager’s Forum on 12 September 2016. National Managers were assigned responsibility for the AFP’s performance criteria included in the corporate plan in May 2016. This responsibility included reviewing and endorsing the performance criteria relevant to their business area, as set out in the KPI Business Rulebook referred to in paragraph 3.11 above, that would be reported against in the annual performance statements.

3.35 National Managers received regular updates of the progress of their performance criteria as part of the reporting referred to in paragraph 3.12 of this audit report. This was in addition to the quality assurance reviews of the data being drawn from the corporate reporting systems by the Organisational Performance team, referred to in paragraphs 3.24 and 3.25 above. On this basis, the National Managers:

endorsed that the annual performance statements are based on properly maintained records, accurately reflect the AFP’s performance in 2015–16 and comply with subsection 39(2) of the PGPA Act.

3.36 In accordance with the Performance Statements Preparation Plan, the performance statements were then circulated to the Executive Leadership Committee by the Deputy Commissioner Capability. The Deputy Commissioner Capability advised the Manager Capability Development of the endorsement of the performance statements by the Executive Leadership Committee on 21 September 2016. The AFP was unable to provide formal minutes recording this endorsement. The performance statements were subsequently recommended to the Commissioner for signing on 23 September 2016.

3.37 The performance statements were signed by the Commissioner and presented in the AFP’s 2015–16 Annual Report on 10 October 2016.

Department of Agriculture and Water Resources

3.38 Advice to Agriculture’s Secretary centred on a risk based approach and the specific assurance process followed prior to signing the necessary certifications was at the discretion of each First Assistant Secretary. The assurance processes varied, from meeting with senior managers to gain assurance, to email correspondence and reliance on results published in internal and external reports.

3.39 The ANAO reviewed the certification statements for the 11 performance criteria examined as part of this audit. Five First Assistant Secretaries (FASs) provided signed certifications across the 11 measures. A summary of the signed certifications was provided to the Secretary prior to the signing of the annual performance statements. The certifications were limited to the performance results tables in the performance statements and did not include the supporting analysis sections presented in the performance statements.

3.40 Two of the five certifications included caveats. The caveats were footnoted to the tables in the summary provided to the Secretary, but the summary table itself did not draw attention to them. The effect of the caveats on the results was not disclosed in the certifications or the summary, however, Agriculture did disclose the caveats in the annual performance statements. Agriculture also provided additional information in the form of graphs and analysis in the performance statements to further support the reported result. This additional information assists users to make an assessment of the reliability of the result. In future, the certification provided by the FASs should state the impact of any caveats and these should be clearly brought to the attention of the Secretary. Agriculture should also, where possible, limit the use of information which may require caveats in the future.

3.41 In its 2016–17 Annual Report implementation plan, the department noted plans to ensure certification of the entire document by relevant FASs, to provide additional assurance of compliance with the PGPA Rule. While the precise process was yet to be determined, it was proposed that FASs certify reporting in both the results tables and analysis sections for their divisions, and that the department’s Regulatory Reform Unit certify industry reporting for which it is responsible.

3.42 The proposed process could be refined by developing a check list for each FAS to complete prior to signing the certification. This could cover aspects such as: the availability of records to support the results reported in the performance statements; documentation of any known limitations to the data or information reported; and when and how to adequately and meaningfully caveat the certifications. As discussed above, all caveats should be clearly brought to the attention of the Secretary, with sufficient documentation to validate the effect of the caveat on the results.

The role and function of audit committees and entity performance reporting⁴⁸

3.43 The accountable authority of a Commonwealth entity is required to ensure that the entity has an audit committee⁴⁹ and to determine by written charter the functions of this committee.⁵⁰ The minimum functions of an audit committee are set out in the PGPA Rule and include reviewing the appropriateness of the accountable authority’s performance reporting (Box 5, emphasis added).

3.44 An information sheet issued by the Department of Finance in August 2015, Audit Committees and performance reporting, provided further guidance, including:

• This function⁵¹ would necessarily involve the committee forming some view on how the entity should measure its performance and how these measures can be weaved into an effective performance story.
• Audit committees might also choose to look at the alignment of corporate plans with annual performance statements, compliance with rule minimum requirements, and the types of performance methodologies able to be used or integrated into forward plans.
• Another potential role for audit committees could be to give assurance on support systems that sit behind performance reporting. This could involve testing the robustness of results and providing assurance that claimed results are valid and supportable.

Australian Federal Police

3.45 The AFP’s audit committee charter, endorsed by the Commissioner on 2 July 2015, sets out the committee’s functions in regard to performance reporting in accordance with the PGPA Rule as follows:

Review the appropriateness of the AFP’s performance reporting: including the framework for developing, measuring and reporting key performance indicators and the AFP’s annual performance statements.

3.46 The ANAO’s review of AFP’s Audit Committee meeting minutes from 2015–16 noted the following actions by the Audit Committee on the AFP’s performance measurement and reporting:

• discussion of the potential benefit of Internal Audit involvement to provide assurance to the Commissioner around the preparation of the AFP’s Performance Statement. (No internal audit on this topic was completed, although it has been included as a potential topic in the 2016–2019 Internal Audit Work Program);
• requested and received advice from the Internal Audit and Business Analysis section on how the performance criteria were derived; what risks there were to the accuracy of underlying data; and what assurance AFP management had that those risks were acceptable;
• requested the development and presentation of a Performance Statements Preparation Plan, including a timeline, to demonstrate the assurance process supporting the preparation of the performance statements; and
• enquired as to the knowledge and understanding of management representatives of their role and responsibility for assuring information contained within the performance statements.

3.47 The audit committee was scheduled to review and recommend the signing of the annual performance statements at the 6 September 2016 meeting through a formal minute to the Commissioner. At the time of that meeting the management certification of the annual performance statements by the National Managers and Executive Leadership Committee had not occurred.

3.48 The draft annual performance statements were provided to the audit committee for review as part of the papers distributed in advance of the 6 September 2016 meeting. The Chair advised members that any significant change to the content or process of the annual performance statements provided to the Committee would be distributed to members out of session for information. The minuted outcome from the 6 September 2016 meeting was that the Audit Committee endorsed the process without specific reference to the appropriateness of the performance reporting, which would not meet the requirements of the audit committee’s charter discussed at paragraph 3.45 above. The signed recommendation to the Commissioner from the Chair of the Audit Committee could not be located for ANAO review. This prevents assessment against the PGPA Rule and its intent.

Department of Agriculture and Water Resources

3.49 Agriculture’s audit committee charter sets out the committee’s functions in regard to performance reporting in accordance with the PGPA Rule as follows:

review and give independent advice and assurance about the appropriateness of the department’s:

b. performance reporting – including the framework for developing, measuring and reporting key performance indicators and the department’s annual performance statement.

3.50 The Audit Committee received quarterly briefings during 2015–16 on the process for the preparation of the 2015–16 Annual Performance Statements. In addition, to address its responsibility under the PGPA Rule and its charter, the audit committee:

• reviewed the framework for the selection of performance measures;
• commissioned an internal audit (which was completed in March 2016) on the Key Performance Indicators (performance criteria) and Performance Reporting, to inform the committee’s review responsibilities, and monitored the implementation of key recommendations of the audit;
• reviewed the proposed process for completing the performance statement, including the assurance mechanisms (sign-offs); and
• noted in March 2017 the department’s review of the performance reporting framework which included the department’s proposed next steps for the 2017–18 PBS and corporate plan, and 2016–17 Annual Performance Statements.

3.51 The committee provided a report to the Secretary on 8 September 2016, advising ‘that the performance framework and assurances adequately supported management’s recommendation’ that the Secretary sign the 2015–16 Performance Statements Certificate. This limited advice does not meet the requirements of the charter discussed at paragraph 3.49 above, which requires the committee to provide assurance about the appropriateness of the department’s performance reporting. As the department’s charter reflects the requirements and intent of the PGPA Rule, these framework requirements have not been met.

Summary

3.52 There would be benefit in the selected entities further considering the role of the audit committee as a source of independent assurance to the accountable authority and how their audit committee charters and processes establish a basis to provide this assurance. This function would necessarily involve the committee forming a view on how the entity should measure its performance. An audit committee’s charter, and any certification by the audit committee to an accountable authority discharging their performance reporting function, should reflect this requirement.

Comparison to the financial statements assurance process

3.53 As noted in paragraph 3.31 of this audit, the certification required by an accountable authority of their performance statements is equivalent to that of the financial statements. Both entities’ audit committees noted that ‘reporting of performance information is not necessarily subject to the same established framework of internal controls or accounting standards which apply in respect of the annual financial statements’. As the entities’ non-financial performance reporting frameworks mature, management structures and systems of internal control for performance statements can be expected to mirror those of the financial statements. In addition, audit committee charters and certifications should reflect the requirements of the PGPA Rule and clearly enunciate the level of assurance provided to accountable authorities.

3.54 Chief Financial Officers have responsibility for the preparation, and joint certification with the accountable authority, of the financial statements. To meet this responsibility, there is an expectation that the Chief Financial Officer has the relevant knowledge, skills and experience of accounting to prepare the financial statements. Similar expectations were not evident within the performance statements certification processes. In addition, audit committee charters did not highlight that there must be a mix of relevant skills and experience, including performance measurement and reporting, within the committee.⁵²

3.55 Agriculture’s Audit Committee charter requires all members to be conversant with financial management reporting, and that at least one member of the committee should have accounting or related financial management experience and/or qualifications, and a comprehensive understanding of accounting and auditing standards. The AFP’s Audit Committee charter also includes a requirement that at least one member possess these skills. There are no similar requirements for performance reporting in either entity’s charter for the audit committee.

3.56 Agriculture has no single internal unit dedicated to continuous development and monitoring of organisation-wide performance, but has established a working group (as noted in paragraph 3.3). The AFP has a section dedicated to monitoring organisational performance and is staffed by personnel with relevant skills and experience in performance measurement.

Have entities maintained records to support the results and analysis included in the annual performance statements?

3.57 Section 37 of the PGPA Act sets out the requirement for Commonwealth entities to keep records that properly record and explain the entity’s non-financial performance. Box 6 provides the specific requirements.

3.58 The ANAO reviewed documentation made available by both entities to record and explain the selected performance criteria results and analysis presented in the annual performance statements.

3.59 Records were available to support the majority of results reported in the performance statements of both entities. The ANAO’s review, discussed below, indicates that detailing the methods used for calculations and analysis based on datasets, would assist to properly explain the entities’ performance, maintain records integrity and ensure consistent methodologies are applied in future reporting periods.

Australian Federal Police

3.60 The AFP collated and maintained all results centrally by a team responsible for organisational performance.

3.61 The ANAO identified inconsistencies in the underlying dataset for the results of only one of the nine performance criteria reported in the annual performance statements. The calculation and workings applied to the raw data to achieve the reported result were also not retained. These inconsistencies did not lead to a material difference in the overall result reported by the AFP.

3.62 Excluding the performance criterion noted above, and noting matters raised in paragraph 3.48 in regard to maintaining records of endorsements, the AFP maintained adequate records to support results reported in the annual performance statements.

3.63 The ANAO’s observations of the AFP’s record-keeping supporting the annual performance statements are summarised in Table 3.1 below.

Table 3.1: ANAO assessment of AFP record-keeping

Note a: The total questionnaire responses reported in the 2015–16 Annual Performance Statement was 814, compared to participant numbers of more than 10 000 parents, carers and teachers. This reflects an approximate response rate of 8 per cent.

Source: ANAO analysis of AFP records.

Department of Agriculture and Water Resources

3.64 Agriculture’s record-keeping was dispersed, with relevant records retained by the responsible areas within the department.

3.65 Agriculture records were available to support the majority of results reported in the performance statements. This ranged from regular internal reports, external reporting and analysis of existing data and a selection of case studies. Most performance criteria reviewed as part of the audit were based on information that was publicly available and therefore subject to public scrutiny. This information was often provided by professional sources, such as the Australian Bureau of Statistics (ABS) and Australian Bureau of Agricultural and Resource Economics and Sciences (ABARES).

3.66 As noted in paragraph 3.19, Agriculture relied on different data sources to those identified in internal documentation to produce the results presented in the annual performance statements for two performance criteria. In addition, the results of three criteria did not fully reflect the original criterion due to the absence of available data or analysis.⁵³ In these cases, the ANAO has assessed the completeness and accuracy of records used to support the result as reported in the annual performance statements.

3.67 Agriculture was unable to provide complete records to support the result reported for marine resources against the performance criterion ‘Access to water, land, forest and marine resources for primary production is maintained or improved’. Records for the reported result of access to land and forest resources were publicly available.⁵⁴

3.68 For the performance criterion ‘100% of allocated funding under the Research and Development for Profit program expended in accordance with the agreed timetable’, the funding details of the overall program are reported on the department’s website. This includes the total approved spending per project for each round. It does not detail the agreed timetables or expenditure breakdown against these timeframes. The department was unable to provide further evidence to support the reported result of ‘projects progressing in accordance with agreed timeframes’.

3.69 The result reported against the performance criteria ‘100% of rural research and development corporations are compliant with statutory and contractual requirements’, noted that ‘Pending formal notification for 2015–16 of material compliance, no research and development corporation has advised of any events [of non-compliance]’. As the result is based on an absence of advice, the ANAO has concluded only on the basis that no evidence that this statement is not accurate came to our attention.

3.70 For the remaining performance criteria, the relevant records were able to be obtained from the responsible areas within the department. To enable consistent record-keeping across responsible areas, guidance outlining procedures for producing and storing records should be considered. The establishment of a dedicated repository may assist in this.

3.71 The ANAO’s observations of Agriculture’s record-keeping supporting the annual performance statements are summarised in Table 3.2 below.

Table 3.2: ANAO assessment of Agriculture record-keeping

Source: ANAO analysis of Agriculture records.

Appendix 1 Entity responses

Department of Finance

Australian Federal Police

Department of Agriculture and Water Resources

ANAO Comments

1. The Audit Committee’s Charter required the committee to review and provide independent advice and assurance about the appropriateness of the department’s performance reporting (see paragraph 3.49). The ANAO concluded that the advice provided to the Accountable Authority by the Audit Committee did not provide assurance about the appropriateness of the department’s performance reporting (see paragraph 3.51).

2. The Department of Finance has acknowledged the need for further clarity in guidance to audit committees (see paragraph 33).

Appendix 2 Summary of the selected entities’ outcomes and programmes

1. The following tables summarise the 2015–16 outcomes and programmes of the AFP and Agriculture.

Table A.1: AFP’s 2015–16 outcomes and related programmes

Source: Australian Federal Police Portfolio Budget Statements 2015–16.

Table A.2: Agriculture’s 2015–16 outcomes and related programs

Source: Department of Agriculture and Water Resources Portfolio Additional Estimates Statements 2015–16.

Appendix 3 Criteria for the assessment of the relevance, reliability and completeness of performance information

2. The following criteria have been adapted from the evaluation of the appropriateness of key performance indicators as depicted in ANAO Report No.21 2013–14 Pilot Project to Audit Key Performance Indicators. The criteria have been updated to reflect Finance’s guidance to support the enhanced Commonwealth performance framework.

Table A.3: Criteria for the assessment of the relevance, reliability and completeness of performance information

Source: Adapted from ANAO Report No.21 2013–14 Pilot Project to Audit Key Performance Indicators, incorporating Department of Finance Resource Management Guides No: 130; 131; 132; and 134.

Appendix 4 Assessment of entities’ performance criteria relevance and reliability

1. The following tables details the results of the ANAO’s assessment of the relevance and reliability of the Australian Federal Police, and the Department of Agriculture and Water Resource’s performance criteria reviewed as part of this audit.

Table A.4: Assessment of AFP’s performance criteria relevance and reliability

Source: ANAO analysis.

Table A.5: Assessment of Agriculture’s performance criteria relevance and reliability

Source: ANAO analysis.

Appendix 5 Summary of entity’s sources, collection arrangements, measurement frequency and limitations of data for performance reporting

1. The following tables summarise the sources, collection arrangements, measurement frequency and limitations of data used by the Australian Federal Police, and the Department of Agriculture and Water Resources for reporting against selected performance criteria in the annual performance statements.

Table A.6: AFP performance criteria data sources, collection arrangements, measurement frequency and limitations

Note a: ‘Cube’ refers to reports generated by the AFP, built via a Statistical Analysis System (SAS) tool, for the purpose of extracting specific data fields from management systems to support internal and external reporting.

Note b: PROMIS (Police Realtime Online Management Information System) is the key case management information system used by the AFP.

Note c: Computer Aided Dispatch (CAD) is a system developed to record the AFP’s response to reported incidents. This includes recording the time the incident was reported, who was dispatched in response and when, and how the incident was resolved.

Source: ANAO analysis.

2. To examine Agriculture’s assessment approach for 2015–16, the ANAO determined the data sources, measurement frequency and limitations identified by the department. Information provided in the 2015–16 Annual Performance Statements, the department’s data collection summaries and advice from the department is summarised in Table A.7 below.

Table A.7: Agriculture performance criteria data sources, collection arrangements, measurement frequency and limitations

Source: ANAO summary of the 2015–16 Annual Performance Statements, the department’s data collection summaries and advice from the department.

Appendix 6 Assessment of entities’ methods for data collection and collation of results

1. The following tables summarise the ANAO’s observations of the data collection methodologies used by the Australian Federal Police, and the Department of Agriculture and Water Resources, including collation of results, for reporting against selected performance criteria in the annual performance statements.

Table A.8: Review of AFP’s methods for data collection and collation of results

Note a: Participants consisted of “more than 10,000 parents, carers and teachers” (AFP 2015–16 Annual Performance Statements, p. 33).

Source: ANAO analysis.

Table A.9: Review of Agriculture’s methods for data collection and collation of results

Note a: The Australian broadacre sector includes industries for wheat and other crops, and livestock such as sheep and cattle. Department of Agriculture and Water Resources, Australian farm survey results 2013-14 to 2015-16, April 2016, p. 8.

Source: ANAO analysis.