This edition of audit insights focuses on ANAO insights into assessing efficiency in the Australian Public Service. Knowing whether your organisation is efficient or becoming more efficient, relative to prior years or to other organisations, is important in ensuring the organisation remains sustainable over time. The ANAO has a focus on efficiency as it is a requirement of the Public Governance, Performance and Accountability Act 2013 (PGPA Act). The PGPA Act requires accountable authorities to govern Commonwealth entities in a way that promotes the proper use and management of public resources. Proper in this context means efficient, effective, economical and ethical.

The Commonwealth performance framework focuses on entities providing sufficient performance information to users, including program managers, accountable authorities, ministers, the Parliament and the public, to allow an assessment of progress against intended objectives. The aim of the framework is to provide users with a greater understanding of how entities plan to, and subsequently have, utilised resources (inputs) not just in producing outputs, but also the impact and efficiency in delivering outcomes.

The concept of efficiency can be defined in various ways depending on the context. In a public sector auditing context, efficiency is primarily about entities making the most of available resources—that is, minimising inputs used to deliver the intended policy outputs in terms of quality, quantity and timing. Efficiency is generally relative to some standard, not absolute. Identifying a suitable reference point, benchmark or comparator is an important step in measuring efficiency.

The ANAO has undertaken targeted audits examining efficiency; considered efficiency as a general concept in audits with a broader scope; and looked at measuring efficiency in the context of our audits of annual performance statements.

These audits have shown that while entities develop many indicators of effectiveness they rarely develop indicators of efficiency.

No single approach to auditing efficiency will be appropriate in all situations. Nevertheless, a basic analytical model involves three main steps:

• identifying efficiency measures, for the entity or subject matter;
• comparing the measures against suitable comparators; and
• analysing the results of the comparison.

An important step in our audit process is to determine whether the entity has themselves established efficiency measures that it assesses and reports on over time.

Where an entity has established efficiency indicators, our audit will assess whether these are appropriate and, if so, use them to report on agency performance.

In the absence of appropriate efficiency measures identified by the entity, our methodology includes the following steps for identifying appropriate measures:

1. Identify the relevant inputs and outputs for the entity or program being audited. Some examples of inputs include direct staff costs, contractor and consultant costs, funding streams, buildings, cash and other assets, revenue collected and other resources. Examples of outputs include grant dollars approved, paid and acquitted, telephone queries effectively handled, licences issued and complaints handled.
2. Identify the outcomes sought by government. Examples might include reduced environmental impact, reduction of long-term unemployment, and the protection of threatened species.
3. Assess the relationship between inputs and outputs, or inputs and outcomes to determine the appropriateness of the measure, or set of measures. A common approach to developing efficiency measures is to calculate ratios of inputs to outputs. Calculating ratios of inputs to outcomes provides measures of cost effectiveness.

The decisive test of appropriateness is whether the measurement of inputs/outputs/outcomes provides meaningful insights into the performance of the entity or program being audited.

Where inputs or outputs are not identified, or suitable data that directly measures the activity is not available, we identify whether proxy indicators are available or can be developed. A proxy measure is an indirect measure of the activity, which is strongly correlated with the activity.

Once appropriate measures of efficiency are identified or developed, an appropriate comparator against which to assess the measures is determined for the entity or program. Two possible approaches are: comparison across a time series of the agency specific data to test if efficiency is improving; and/or comparison with external benchmark data from, if possible, a wide range of relevant entities.

A key factor to remember when considering the use of external benchmarks is that the comparator does not need to be identical. This argument is sometimes made by those trying to explain why efficiency measurement should not apply to them.

In this context, assessing efficiency is like assessing value for money in procurement. In many procurements, value for money assessments need to compare products with differing prices and functionality. Benchmarking is particularly important in sole sourced procurements where identifying alternative products can be essential to maintain competitive pressure and achieve value for money. In these circumstances the fact that the comparator is not identical does not remove its utility. As long as the purchaser knows the differences in functionality, the price/functionality relationship between the two can be determined and this used to assess comparative value for money. This may be particularly useful where the procurement involves a partnership in product development and the price/functionality of the product is likely to vary through the procurement process.

Similarly, when assessing efficiency the comparator benchmark does not need to be the same, particularly if you know what the differences are and the benchmark can be used to assess reasons for variations through time.

Audits from July to September 2018

During the July to September 2018 quarter, the ANAO finalised one performance audit that used our efficiency methodology, and other audits considered efficiency alongside effectiveness. The audits undertaken in the quarter identified scope to improve measurement of efficiency either through benchmarking against other providers, or through developing an understanding of how resources are best deployed to deliver on government and consumer expectations in a way that is sustainable over time.

The Operational Efficiency of the Australian Commission for Law Enforcement Integrity audit identified through key learnings that:

• Efficiency involves making the best use of available resources to achieve required outputs. Selecting performance measures that focus on the final outputs to be produced rather than interim processes, is likely to provide a more robust indicator of performance.
• An appropriate set of efficiency measures can help to inform accountable authorities on the allocation of entity resources in pursuit of the best outcomes for government and the community. Including efficiency measures in Annual Performance Statements support transparency and accountability for entity performance.
• Efficiency can be relative rather than absolute. Identifying suitable comparators to assess efficiency against provides a useful indicator of performance. Comparators can include: past performance; organisations with comparable functions or processes; or appropriate targets and benchmarks.

The Implementation and Performance of the Cashless Debit Card Trial audit found that the key performance indicators developed to measure the performance of the trial were relevant, mostly reliable but not complete because they focused on evaluating only the effectiveness of the trial based on its outcomes and did not include operational and efficiency aspects of the trial.

The Management of Commonwealth Leased Office Property audit, tabled just outside the quarter on 9 October 2018, found that:

• While the Operation Tetris program was intended to realise efficiencies in office space utilisation and associated costs of office leases, the methodology used to estimate the savings was not robust and the efficiencies realised were not transparent. Savings were retained within agencies but actual outcomes were not verified or compared with the estimates.
• Further, Finance had not established all the elements of a fit-for purpose performance framework to assess and report on its program of work on property efficiency. Broad objectives have been set but are not well-defined, and current performance metrics are incomplete and not properly integrated – addressing occupational density but not cost or other relevant indicators.

Audits from previous periods

The Efficiency through Contestability Programme audit found that substantial Budget savings have arisen from the program, although entities had not demonstrated the extent of efficiency and performance improvements supporting those Budget savings. The audit noted that performance measures should be established in the design phase, including baseline data, benchmarks and access to reliable sources of data, to capture improvements in efficiency over time.

The Efficiency of Veterans Service Delivery by the Department of Veterans’ Affairs (DVA) audit sought to assess whether DVA was efficiently delivering services to veterans and their dependents.

The audit found that the majority of DVA rehabilitation and compensation services are being delivered to veterans and their dependents within DVA’s time based performance targets, however a minority of claims take an excessively long period to process due to inefficient handling. These delays can have significant impacts for affected veterans.

The audit identified weaknesses in DVA’s business systems and processes, which do not adequately support the efficient delivery of services.

Rehabilitation and compensation processes are designed as an end to end process but are not designed to maximise efficiency. For example, the approach to the assessment of claims is resulting in the unnecessary disaggregation and re-aggregation of components within some claims. This indicates that efficiencies could be identified and incorporated to improve the ratio of outputs (claims processed) to inputs (human resource time spent assessing claims).

The Design, Implementation and Monitoring of Health’s Savings Measures audit found that the Department of Health lacked appropriate arrangements to monitor the implementation and impact of departmental efficiency savings. The department was unable to demonstrate that efficiencies had been realised beyond a reduction in the department’s overall appropriation. This means that the department’s executive does not have visibility over the service quality impact of savings measures, nor can it make informed, evidence-based decisions over which business areas can improve efficiency relative to a benchmark or comparator. Particular findings were that:

• In designing departmental savings measures Health did not develop costings for all efficiency measures.
• Oversight arrangements for departmental savings were undermined by a lack of appropriate measures to monitor the progress of implementation (realisation of efficiencies) and impact.
• The integration of efficiency measures into the department’s broader change program reduced visibility of the department’s progress in achieving the efficiency outcomes.

The audit of the Administration of Medicare Electronic Claiming Arrangements examined the effectiveness of Medicare electronic claiming arrangements and found that:

• In relation to the claiming and payment strategy, the Department of Human Services introduced a benefits realisation framework intended to identify financial benefits as well as elements of efficiency and effectiveness.
• Human Services does not currently track the relative costs of maintaining each claiming channel (meaning the relative efficiency of each channel is not known). There would be benefit in Human Services developing the capability to better understand the costs of each channel, as well as the administrative savings and other benefits that have been realised, to support decisions about future directions for, and investments in, electronic claiming channels.

The audit of Australia Post’s Efficiency of Delivering Reserved Letter Services considered whether Australia Post was meeting its community service and international obligations efficiently and the effectiveness of Commonwealth shareholders in monitoring value for money. The audit found that:

• Australia Post’s strategies to improve its efficiency focussed on process optimisation and automation along with labour force flexibility, all with the objective of improving labour productivity.
• Australia Post had improved its efficiency over time, however these improvements were relatively slow compared to its international peers, particularly in relation to its management of operating costs.
• While Australia Post monitors and evaluates the efficiency with which it meets its obligations, there is scope for Australia Post to provide more strategic information on the long-term sustainability of the letters business; changes in its performance over time; and the assumptions driving key forecasts that underpin the enterprise valuation.

The audit of the Efficiency of the Australia Council’s Administration of Grants considered whether selected grant programs were being administered efficiently by the Australia Council in relation to suitable comparators. The audit found that:

• The Australia Council had not established grant administration metrics to support the measurement and benchmarking of its efficiency in administering grant funding against suitable organisations or its own performance over time.
• The Australia Council captured data about inputs and outputs but it did not use this data to calculate measures of efficiency.
• The Australia Council had not measured changes in its efficiency over time, or set a target(s) to work towards.
• Consistent with its legislative obligations, the Australia Council had sought to be efficient in its grants administration although, in the absence of an efficiency baseline, measures and trend analysis, it was not clear if the steps taken have improved efficiency.

In addition, the following audits also made observations which are relevant to measuring efficiency:

• Costs and Benefits of the Reinventing the ATO Program
• Defence’s Implementation of the First Principles Review
• The Integration of the Department of Immigration and Border Protection and the Australian Customs and Border Protection Service

Audits of annual performance statements

The audit report Implementation of the Annual Performance Statements Requirements 2016–17 drew a number of conclusions in relation to measuring efficiency, including:

• The completeness of performance criteria is a particular area requiring consideration, including increasing the use of effectiveness and efficiency measures or, where appropriate, making clear the intention to use input, activity and/or output measures as proxies. Entities are also not realising the full potential arising from the minimum four-year horizon of corporate plans, by developing performance criteria that assess a mixture of short, medium and long-term objectives.
• Finance guidance outlines that the considerations that are critical to developing good performance information include using an understanding of an entity’s purposes to identify a set of measures that can demonstrate the extent to which those purposes and activities are being delivered effectively and efficiently.
• The then Department of Employment was the only entity to include a measure (cost per employment outcome) that, when considered over time, may be used to demonstrate its efficiency in achieving its purpose. As noted above, entities should identify a set of measures that demonstrate both effectiveness and efficiency. This is to support the Parliament and the public in their assessment of how well Commonwealth entities are performing, including how they are using the resources that have been entrusted to them.

These findings supported a recommendation that entities review their performance measurement and reporting frameworks to develop measures that provide the Parliament and the public with an understanding of their efficiency in delivering their purposes.

These findings expanded on one of the key learnings of the Implementation of the Annual Performance Statements Requirements 2015–16 audit:

• Complete performance criteria should collectively address the entity’s purpose, and provide a balanced examination of the entity’s effectiveness and efficiency across the different forms of performance information and their timeframes.

These examples show that measuring efficiency is an integral part of good financial management – which is why this requirement is part of the PGPA Act accountabilities. If accountable authorities are not identifying appropriate benchmarks and comparators for efficiency, it is not clear how accountable authorities gain assurance that they're minimising inputs to deliver a given set of outputs. It is also not clear how entities can continue to deliver an efficiency dividend, and meet productivity saving targets, if these comparisons are not being made. Efficiency measurement is not a well-developed capability across government. The audits mentioned above are a useful resource to start a conversation in entities about measuring efficiency.

The ANAO will continue to audit how entities measure efficiency, and there are two audits in-progress on this topic:

• Efficiency of the investigation of transport accidents and safety occurrences
• Efficiency of the processing of applications for citizenship by conferral

The ANAO’s annual audit work program also contains a number of potential audits that consider efficiency, including:

• Case management efficiency of the Office of the Commonwealth Director of Public Prosecutions
• Collection of visa revenue
• Efficiency of Australia’s primary overseas aid delivery providers
• Efficiency of the airport master plan approval process
• Border Intelligence Fusion Centre