1.1 The Department of Defence’s Capability Acquisition and Sustainment Group (CASG) is responsible for purchasing and maintaining military equipment and supplies in the quantities and to the service levels that are required by Defence and approved by government.⁴ Defence’s forecast expenditure for 2018–19 is $8.6 billion for acquisition projects, and $6.0 billion for sustainment products. As at 16 May 2019, CASG was managing 204 major and minor acquisition projects and 112 sustainment products.

The Quarterly Performance Report

1.2 The Capability Acquisition and Sustainment Group prepares a Quarterly Performance Report focusing on key major acquisition projects and key sustainment products, with the purpose of providing:

senior stakeholders within Government and the Department of Defence with a clear and timely understanding of emerging risks and issues in the delivery of capability to our Australian Defence Force end-users. These risks and issues are highlighted so that stakeholders can respond in a coordinated manner to guide the conduct of remediation actions.⁵

1.3 Internal guidance on the Quarterly Performance Report further states that:

In keeping with the primary goal of the QPR, the focus remains on highlighting Projects and Products of Interest. By identifying and remediating issues early on in the Capability Life Cycle the QPR aims to prevent the capability from becoming a Project of Concern (PoC).⁶

1.4 The Capability Acquisition and Sustainment Group’s Business Framework states that:

the Deputy Secretary of Capability Acquisition and Sustainment Group confirms business assurance through the Quarterly Performance Report provided to the Minister, Secretary of Defence and Chief of the Defence Force.

1.5 The Quarterly Performance Report includes information about CASG’s performance in delivering acquisition projects and sustainment products costing approximately $15 billion (2018–19 forecast expenditure).⁷

Process for preparing the report

1.6 Figure 1.1 outlines the Capability Acquisition and Sustainment Group’s performance reporting cycle to measure its progress against its business plan and delivery of service to its customers.

Figure 1.1: Capability Acquisition and Sustainment Group’s performance reporting cycle

Source: Adapted from the Capability Acquisition and Sustainment Group Business Framework, 25 July 2018.

1.7 The Quarterly Performance Report is prepared by CASG’s Program Approvals and Agreements Directorate. Defence has not determined the overall cost of producing the report (each year or each quarter). Defence advised that production of the report involves:

Three staff members make up the Strategic Reporting team and include an EL1, APS 6 and APS 5 who primarily work on the compilation of QPR, including the Projects of Concern reporting. Their responsibilities are spread across other functions of the team, dependent on work peaks and troughs. Additionally, the Director of the team will spend a portion of time on the compilation of the QPR.

Outside of the team, there is the Major Program Control team of around four who review the draft report and provide input, and one staff member (on rotation) will assist with the QPR analysis (for part of one day usually).

It is harder to quantify outside of the directorate as there are the project and product management staff and CASG Division management, as well as the Defence Finance Group staff who would work on both an ‘as usual’ and ‘as required’ basis. GBM [Group Business Manager] CASG and DEPSEC [Deputy Secretary] CASG and their staff work on the ‘compilation’ during the clearance process.

1.8 The Directorate compiles the Quarterly Performance Report manually using data primarily sourced from Defence’s Monthly Reporting System⁸ and the Sustainment Performance Management System.⁹ Additional data is sourced from Capability Acquisition and Sustainment Group’s Systems Program Offices (responsible for the management of sustainment products), Project Offices (responsible for the management of acquisition projects), and Independent Assurance Reviews. Information on projects and products is cleared through Division Heads in the Capability Acquisition and Sustainment Group. The high level process and data sources for the Quarterly Performance Report are outlined in Figure 1.2 (on the following page).

1.9 The report is approved by the Deputy Secretary, Capability Acquisition and Sustainment and provided to the Secretary and Chief of the Defence Force (CDF) for endorsement.

Report distribution

1.10 The Secretary and CDF provide the report to the Ministers for Defence and Defence Industry. The report is also provided to the Defence Investment Committee for noting and feedback. The Defence Investment Committee includes representatives from the Department of the Prime Minister and Cabinet and the Department of Finance.¹⁰

1.11 The report is also provided for information to the Australian Industry Capability team, the Capability Acquisition and Sustainment Group Statutory Reporting team, and the Program Integration and Interoperability Office in Joint Systems Division.¹¹ In addition, the following committees¹² within Defence also receive a copy of the Quarterly Performance Report, or an Executive Summary for information:

• Defence Committee;
• Defence Audit and Risk Committee;
• Capability Acquisition and Sustainment Group Executive Advisory Committee; and
• Capability Acquisition and Sustainment Group Risk Advisory Committee.

Figure 1.2: High level process and data sources for the Quarterly Performance Report

Note: In June 2018, the Defence Investment Committee’s role in the process changed. Prior to June 2018, the Committee cleared the Quarterly Performance Report. The Committee’s role is now to note and provide feedback on the report.

Source: ANAO analysis of Defence information.

What is reported in the Quarterly Performance Report?

1.12 The structure of the report has been similar across the eight reports prepared for the quarters ending March 2017 to December 2018. Each report has included information about:

• Projects of Concern — a status update and broad description on the progress of remediation activities.
• Acquisition projects — there are two sets of reports:
  • For each of the key acquisition projects (that is, the ‘Top 30’ acquisition projects as set out in the department’s Portfolio Budget Statements as well as projects appearing in the annual Major Projects Report) the analysis includes comments and performance information about the project’s capability, schedule and cost and ‘traffic light’ ratings based on quantitative data.¹³ The report also includes a comment on relevant Australian Industry Capability.
  • For each acquisition Project of Interest, the analysis provides: a project overview; summary of risks to capability, schedule and costs; ‘traffic lights’ reflecting a qualitative assessment of these risks based on criteria; and the remediation strategy for the project.
• Sustainment products — there are two sets of reports:
  • For each of the Top 30 sustainment products (as set out in the department’s Portfolio Budget Statements) the analysis includes comments, performance information and ‘traffic light’ ratings in terms of availability and expenditure as well as a comment on relevant Australian Industry Capability.
  • For each sustainment Product of Interest, the analysis provides: a product overview; a summary of risks to availability and cost; ‘traffic lights’ reflecting a qualitative assessment of these risks based on criteria; and the remediation strategy for the product.

Coverage of operational output of the Capability Acquisition and Sustainment Group

1.13 The Quarterly Performance Report provides performance information for approximately 30 per cent of projects and products being managed by the Capability Acquisition and Sustainment Group.

1.14 The key acquisition projects and ‘Top 30’ products represent approximately 70 per cent of the Major Capital Equipment budget and 70 per cent of the sustainment program budget as reported in the Quarterly Performance Reports for June, September and December 2018. The number and value of acquisition projects and sustainment products reported on in the June 2018 Quarterly Performance Report, which is the focus of this audit, is outlined in Table 1.1 (on the following page).

1.15 A useful feature for the reader is that there are separate sections designed to draw attention to the projects/products that have been identified as requiring additional management attention due to risks to delivery. These sections include all the projects/products that are ‘of concern’ and ‘of interest’ regardless of whether they are key acquisition projects and ‘Top 30’ sustainment products. As a consequence though, some projects and products may be discussed twice in the report.¹⁴

Table 1.1: Number and value of acquisition projects and sustainment products reported on in the June 2018 Quarterly Performance Report

Note a: Figures for acquisition projects are the total budget. Figures for sustainment products are the annual budget for 2017–18.

Note b: Post second pass acquisition projects only.

Source: ANAO analysis of Defence information. Capability Acquisition and Sustainment Group June 2018 Quarterly Performance Report.

Previous ANAO and Parliamentary review of the Quarterly Performance Report

ANAO

1.16 Auditor-General Report No.2 2017–18 Defence’s Management of Materiel Sustainment identified issues with the data quality, completeness and timeliness of the Quarterly Performance Report¹⁵ and recommended that ‘Defence institutes a risk-based quality assurance process to ensure the accuracy, completeness and relevance of the information included in the Defence Quarterly Performance Report’.¹⁶ Defence records indicate that the recommendation was closed on 7 September 2017. Chapter two of this audit report discusses implementation of the recommendation.

Joint Committee of Public Accounts and Audit Report 470: Defence Sustainment Expenditure

1.17 The Parliament’s Joint Committee of Public Accounts and Audit (JCPAA) conducted an inquiry into Auditor-General Report No.2 2017–18 Defence’s Management of Materiel Sustainment. The Deputy Secretary, Capability Acquisition and Sustainment Group advised the Committee that Defence was undertaking the changes necessary to address the ANAO findings and recommendation on the Quarterly Performance Report:

I think the fact that we are, effectively, changing the entire model as we go through … would indicate that I did have concerns about the accuracy, completeness of that report. And that’s why we are changing the model and making sure that model is data drawn directly and that the domain heads who sign that off for me are getting accurate information that’s not filtered through the system, so that we can be assured, I can be assured, the capability manager can be assured and government can be assured that they’re getting timely and accurate information.¹⁷

1.18 The Deputy Secretary further advised the Committee that Defence would have a ‘much better reporting system’ by the end of 2017.¹⁸ In particular, by:

implementing the SPMS [Sustainment Performance Management] system, so that we can actually get that consistency across, take out the individual interpretation and make sure that it is data driven — that is the whole purpose of moving from an old, effectively manual system into an automated system, which draws directly from the finance data from the Defence system, from the project schedules and from the data that’s actually being referred to.¹⁹

1.19 In its report on the inquiry, Report 470: Defence Sustainment Expenditure, the Committee stated that it hoped that:

Defence’s recent IT improvements will lead to a better reporting system, but notes that technology should not be Defence’s sole focus. Defence’s internal reporting regime should be open to the Minister and the Secretary, and report using plain language. Terms like ‘overachievement’ should not be used to describe overspending. More broadly, the culture in the Department of Defence will need to continue to change to reinforce reform across this area.

1.20 Recommendation 5 of the Committee’s report was that:

… the Auditor-General consider reviewing the Department of Defence’s new Monthly Reporting system and documentation including advising whether there have been improvements in reporting to the Minister and Secretary, and that the information provided via reporting is timely and accurate.²⁰

1.21 In response to the recommendation, the Auditor-General wrote to the Committee in September 2018 advising that he would consider expanding the audit outlined in the ANAO’s 2018 Annual Audit Work Program on the Sustainment Performance Management System to assess the governance, use and quality of data within the performance management and reporting systems for both acquisition and sustainment that inform the preparation of the Quarterly Performance Report.

Rationale for undertaking the audit

1.22 The Quarterly Performance Report is an area of interest to the Parliament. Quality performance reporting supports the accountable authority and those charged with governance in their oversight of an entity.²¹ Regular reporting on performance to senior management also provides incentives for local line management to improve performance.

1.23 The Quarterly Performance Report is prepared to facilitate oversight by Defence senior leaders, help identify projects and products that require additional senior leadership attention and escalation, and to support informed decision making. The report is also provided to Defence Ministers and key external stakeholders — the Department of the Prime Minister and Cabinet and Department of Finance — to support their understanding of Defence’s administration and their own decision-making.

Audit approach

Audit objective, criteria and scope

1.24 The objective of the audit was to examine the effectiveness of Defence’s Quarterly Performance Report as a mechanism to inform senior stakeholders on the status, risks and issues in the delivery of capability to the Australian Defence Force.

1.25 The high-level audit criteria were:

• The Quarterly Performance Report provides senior stakeholders with accurate and timely information about status and emerging risks and issues.
• The Quarterly Performance Report provides senior stakeholders with information they value, as an aid to monitoring status, risks and issues and to inform decision-making.

1.26 This audit has focused on the June 2018 Quarterly Performance Report — the most recent report available at the time the Auditor-General decided to conduct this audit. The following reports were also reviewed during the course of fieldwork — March 2017, June 2017, September 2017, December 2017, March 2018, September 2018 and December 2018.

Audit methodology

1.27 The audit method involved:

• examining and analysing data from Defence’s information technology systems including the Sustainment Performance Management System and the Monthly Reporting System, key reviews and documentation;
• testing the Quarterly Performance Report process by examining information reported in the June 2018 Quarterly Performance Report for a targeted sample of eight sustainment products²², with a focus on accuracy, completeness and timeliness;
• drawing on audit work conducted by the ANAO as part of the annual Defence Major Projects Report, to test the accuracy of reporting for the 26 major acquisition projects included in the Major Projects Report;
• drawing on audit work conducted by the ANAO as part of recent performance audits of acquisition projects and testing the accuracy, completeness and timeliness of reporting for a further acquisition project;
• analysing the eight responses received as part of an ANAO survey of 17 senior leaders who are routinely provided with the Quarterly Performance Report. The survey was sent to the Secretary of the Department of Defence, the Chief of the Defence Force, Defence Ministers and members of the Defence Investment Committee. The survey is included in Appendix 5; and
• discussions with Defence personnel involved in the Quarterly Performance Report’s preparation.

1.28 This audit was conducted in accordance with the ANAO Auditing Standards at a cost to the ANAO of approximately $338,000.

1.29 The team members were Natalie Whiteley, Clyde Muthukumaraswamy, Jordan Bastoni, Cherise Reed and Sally Ramsey.

Is the Quarterly Performance Report provided to senior stakeholders and Ministers in a timely manner?

Preparation of the report

2.1 Each Quarterly Performance Report notes that the next report will be circulated two months after the end of the quarter. The majority of the time taken to prepare the report (approximately six weeks) is focussed on the compilation of the information in the report. This includes: extracting data on acquisition projects and sustainment products from Defence’s Information Technology systems, seeking input from Systems Program Offices and Project Offices including sign-off on this information at the division head level, and collating the information into the format required for the report.

2.2 Figure 2.1 summarises the process and approximate timeframe for compiling the Quarterly Performance Report after the end of a quarter.

Figure 2.1: Process and timeline for Preparation of the Quarterly Performance Report

Source: ANAO analysis of Defence documentation.

2.3 Defence advised the ANAO in June 2019 that:

• the timeframe for the development and preparation of the Quarterly Performance Report is affected by the availability of data from performance reporting systems. For example, cash positions from financial systems are available on the third business day of the following month, and Division Head signoff on project and product information occurs in the middle of the following month;
• the Monthly Reporting System and the Sustainment Performance Management System are located on different servers, which adds time to the preparation of the Quarterly Performance Report; and
• the time required to centrally analyse data, conduct quality assurance and compile the draft Quarterly Performance Report is 11 days.

Timeliness of provision to Ministers

2.4 In 2014, the Minister requested that the report be provided within two months of the end of the reporting period. ANAO analysis (as shown in Figure 2.2) indicates that since December 2017, all reports have been provided to Ministers below the agreed benchmark of two months.

Figure 2.2: Number of calendar days taken to submit Quarterly Performance Reports to Ministers from the end of the quarter

Note: The dotted line represents the 16 August 2017 hearings of the Joint Committee of Public Accounts and Audit into Defence sustainment expenditure, discussed in paragraphs 1.17 and 2.6 of this audit report.

Source: ANAO analysis of Defence documentation.

2.5 The total days taken to submit the report to Ministers for each of the Quarterly Performance Reports produced over the last two years, from the end of the quarter, ranged from 44 days (1.5 months) to 107 days (3.5 months). Since December 2017 the time taken to submit reports has been between 44 and 54 days — an average of 48 days (1.6 months). This is an improvement on overall performance since 2017, when the average time taken to submit the report was 71 days.

2.6 The ANAO noted in Auditor-General Report No.2 2017–18 Defence’s Management of Materiel Sustainment that, for the Quarterly Performance Report reviewed (June 2017), the report was not timely as an aid to decision-making, as the information in it was nearly two months old by the time it was provided to Ministers. The timeliness of reporting was also noted at the Joint Committee of Public Accounts and Audit hearings in August 2017 into Defence sustainment expenditure, in the course of the Committee’s inquiry into that ANAO report. In that context, Defence committed to improving the timeliness of production.^{23 24} As shown in Figure 2.2, there has been an improvement in overall performance since late 2017.

2.7 Reporting is still impacted by the quality of data inputs. Since 2017, the Sponsor’s Papers for the Quarterly Performance Report provided to the Investment Committee noted that: ‘Timeliness of CASG [Capability Acquisition and Sustainment Group] reporting is currently characterised by IT [Information Technology] reporting systems that suffer from inconsistency in the quality of the data inputs from across the organisation.’ Available evidence indicates that this issue has not been discussed at the Investment Committee.

2.8 To mitigate the risk that the Secretary is unaware of escalating risks to acquisition projects and sustainment products, the Deputy Secretary, Capability Acquisition and Sustainment Group is expected to provide ‘exception’ reporting on any substantial variations or issues to the Secretary as they arise. Senior management may also access performance data to enable monitoring through separate reports, such as through fleet screenings.

Stakeholder views on the timeliness of the Quarterly Performance Report

2.9 The ANAO surveyed 17 senior stakeholders — who routinely receive the Quarterly Performance Report — on the timeliness of the report. Of the eight responses received, views were mixed as to whether the Quarterly Performance Report highlights risks and issues in a timely manner to support their decision making and whether they rely on the report to inform their decision making:

• four senior stakeholders indicated that the Quarterly Performance Report always, mostly or sometimes highlights key risks in projects and products in a timely manner to assist decision making;
• three senior stakeholders indicated that the report never does;
• one senior stakeholder commented that the time taken to collate the report means that it is not used to assist in making decisions as it is after-the-fact reporting;
• one senior stakeholder noted that the report covers events that have unfolded in the past, and that it has not been designed to support specific decision making processes; and
• another senior stakeholder also considered the accuracy of data to be reduced by the time taken to compile the report (meaning information can be out of date).

Is the information included in the Quarterly Performance Report accurate?

2.10 The following sections provide an assessment of the clarity and accuracy of information in the Quarterly Performance Report for acquisition projects and sustainment products.

Reporting on acquisition projects in the Quarterly Performance Report

2.11 To assess the accuracy of the information on acquisition projects in the June 2018 Quarterly Performance Report, the ANAO:

• Compared the project information in the Quarterly Performance Report (including the project maturity score, capability risks ratings, budget and expenditure information, and key milestone dates) for 26 acquisition projects to information reported in the 2017–18 Major Projects Report (as at June 2018)²⁵ and key data from the Monthly Reporting System (Defence’s primary performance reporting system for acquisition projects).
  • For two of the 26 projects, the ANAO also compared the results of two recent performance audits²⁶ with the information included in the June 2018 Quarterly Performance Report. This comparison focussed on whether the issues around availability, capability or cost identified in these reports had been identified in the Quarterly Performance Report.
• Reviewed the accuracy of performance reporting for one additional acquisition project (AIR 5428 Phase 1 Pilot Training System) to spot check reporting for a project not subject to additional scrutiny through the limited assurance review conducted by the ANAO as part of the annual Major Projects Report.

2.12 The ANAO found that the information included by Defence in the June 2018 Quarterly Performance Report was largely aligned with the information reported in the 2017–18 Major Projects Report (MPR)²⁷ and the budget and schedule data from the Monthly Reporting System. Reporting for the Army’s Protected Mobility Vehicle–Light and the F-35 Joint Strike Fighter was also found to be an accurate representation when compared to recent performance audits. For both projects, the budget identified in the Quarterly Performance Report does not reflect the budget identified in the performance audit reports because the Quarterly Performance Report does not include operating and support costs. These costs can be significant. For example, the Joint Strike Fighter has an overall budget of $15.5 billion with additional operating and support costs of $4.6 billion.²⁸

2.13 The ANAO’s review of the AIR 5428 Phase 1 Pilot Training System project indicated that a number of key risks, including risks to achieving Initial Operating Capability, were not reported in a timely manner through the Quarterly Performance Report. A summary of the results of this analysis is included in the box below.

Note a: Defence advice to the ANAO, June 2019.

Source: ANAO analysis of Defence documentation.

Reporting on sustainment products in the Quarterly Performance Report

2.14 To assess the accuracy of reporting for sustainment products, the ANAO reviewed eight sustainment products from the June 2018 Quarterly Performance Report. The targeted sample covered: a range of domains²⁹; risk ratings; and high, medium and low value products (within the Top 30 list). The list of sustainment products reviewed is included at Appendix 6.

2.15 The ANAO compared information reported for these products in four Quarterly Performance Reports (December 2017 to September 2018) with product data from the Sustainment Performance Management System, Independent Assurance Reviews, Materiel Sustainment Agreements and other reports, reviews and minutes.

2.16 A summary of the results of the ANAO’s review of the sample of eight sustainment products is included at Table 2.1.

Table 2.1: Sustainment product sample analysis summary

Note: The sample was CA 39 — ADF Clothing; CA48 — Multi Role Helicopters; CA59 — Explosive Ordnance Army Munitions; CAF12 — Air Traffic Management; CAF19 — Heavy Air Lift Weapons System; CAF21 — F/A18F Super Hornet Weapon System; CN09 — Armidale Class Patrol Boat; and CN40 — Hobart Class Destroyer.

Source: ANAO analysis of Defence documentation.

2.17 The following sections discuss the detailed results of the ANAO’s review of: the accuracy and completeness of information in the Quarterly Performance Report; and the clarity of information in the report.

Accuracy and completeness of information in the Quarterly Performance Report

2.18 A number of issues regarding accuracy and completeness of data relating to the sample were identified by the ANAO:

• Emerging risks and issues for the products were not adequately identified in the Quarterly Performance Report — For five of the eight products in the sample, there were a number of emerging risks that were identified in Defence documentation such as Independent Assurance Reviews and Sustainment Performance Management System reports for the products that were not consistently reported in the Quarterly Performance Reports. These included emerging risks with the budget, deficiencies with the procurement contract, and deficiencies with existing Defence policy for managing the product.
• Data inconsistencies between the Quarterly Performance Report and other reporting documents covering the same period — There were inconsistencies between information that appeared in the Quarterly Performance Reports and that which appeared in other reporting documents in two products from the sample. These were:
  • For CA48 — Multi Role Helicopter Weapon System, there was a focus in the four Quarterly Performance Reports on problems with contractor performance affecting aircraft serviceability and sustainment and on the use of additional contracted manpower in pursuit of serviceability targets. Internal Defence advice and an Independent Assurance Review for the product, in September and October 2017 respectively, had raised concerns about a range of workforce issues affecting aircraft serviceability and sustainment, including the structure of the Systems Program Office, the loss of commercial experience and the high turnover of staff.
  • For CA39 — ADF Clothing, the Commonwealth elected in June 2018 to apply liquidated damages to a supplier. This was included in a report to the Deputy Secretary of Capability Acquisition and Sustainment Group, but was not reported on in the June 2018 Quarterly Performance Report.³⁰
• Incorrect or missing performance indicators in Defence’s information technology systems used to populate the Quarterly Performance Report — for two of the products in the sample, the performance indicators listed in the Materiel Sustainment Agreements (MSA) for the products did not match the performance indicators in the Sustainment Performance Management System. Defence recently reviewed the key performance indicators for sustainment products and identified discrepancies between the performance indicators listed in the MSA Product Schedules and those listed in the Sustainment Performance Management System. The review found a significant number of performance indicators reported on in the system were either not agreed by the Capability Manager or included in Defence records. The review also identified a number of misaligned baselines for the delivery of services.³¹

Clarity of information in the Quarterly Performance Report

2.19 The following issues regarding clarity of data in the sample were identified by the ANAO:

• Traffic light indicators can be confusing for the reader when reviewing reports for the same product in different sections of the Quarterly Performance Report — Four of the eight products in the sample have been identified as Products of Interest. For these products, as well as the report in the Top 30 sustainment product section, there is an additional Product of Interest report included in the Quarterly Performance Report, with traffic light indicators for risk to availability and cost. The appendix at the back of each Quarterly Performance Report notes that the availability and cost ratings are determined qualitatively for the Product of Interest report and quantitatively for the Top 30 Sustainment Product report.³² This difference in methodologies means that the traffic light indicators for availability and cost can differ between the Product of Interest report and the Top 30 Sustainment Products report. While these differences in traffic lights do not represent an error, having different ratings with similar headings for the one product in different sections of the report requires the reader to reference the appendix in the report to understand the different ratings and may be confusing for the reader.³³
• Traffic lights can be an inaccurate representation of a product’s availability and budget position — The traffic lights for cost in the Quarterly Performance Report may not be a clear and accurate representation of the product’s budget position. Despite significant budget overspends in 2017–18 for CA59 — Explosive Ordnance Army Munitions, the traffic light for year-end expenditure was rated ‘green’ in the June 2018 Quarterly Performance Report. The rating of ‘green’ was achieved due to changes in the target for the budget in the Sustainment Performance Management System.³⁴ The ANAO’s review of the Investment Committee Member Clearance Comments from June 2017 to December 2018 found instances where committee members had questioned the accuracy of the traffic light ratings for projects and products in the June 2017, December 2017 and September 2018 Quarterly Performance Reports.³⁵ The traffic light for cost also reflects baseline funding for sustainment products only, and does not include operating and support costs.³⁶
• A lack of clarity for the reader around how performance indicators are reported in the Quarterly Performance Report including underlying targets and evidence — The Quarterly Performance Report includes a performance indicator for availability and two performance indicators for cost (year to date and year end expenditure) for each of the Top 30 sustainment products and Products of Interest. For the eight sustainment products in the sample, the ANAO found a lack of transparency around the data underpinning these performance indicators. The performance indicator for availability is an assessment on the overall availability for performance by the Division head responsible for the product. When the Division head overrides or changes a traffic light rating for a product in the Sustainment Performance Management System, there is a commentary field in the system to document the change. However, there is no transparency in the Quarterly Performance Report and the explanation for a change in a performance indicator would not be apparent to the reader.

2.20 As noted at paragraph 1.19, in Report 470: Defence Sustainment Expenditure, the Joint Committee of Public Accounts and Audit stated that the Quarterly Performance Report should use plain language and avoid terms like ‘overachievement’ to describe overspending. While the ANAO did not identify any instances in the sample of the use of this term, the term ‘overachievement’ is still used in the Quarterly Performance Report. For example, the term was used to describe overspending and failure to meet performance indicators for four products in the December 2018 Quarterly Performance Report.

2.21 The results of the ANAO’s review — of the accuracy and completeness of information in the Quarterly Performance Report, and the clarity of information in the report — as well as the IT issues discussed in paragraph 2.7, indicate that while Defence has continued to make improvements to the Sustainment Performance Management System since 2017, it has not yet fully realised the commitment made to Parliament in August 2017 that it would implement an improved reporting system which addresses data quality in the Quarterly Performance Report.

2.22 As discussed in paragraphs 1.17 to 1.19 of this audit report, Defence advised the Joint Committee of Public Accounts and Audit in August 2017 that Defence would have a ‘much better reporting system’ by the end of 2017, including by implementing the Sustainment Performance Management System. In June 2019, Defence advised the ANAO that it considered that the commitment to Parliament in 2017 to implement an improved reporting system had been met and the processes to assure data quality issues in the Quarterly Performance Report would continue to be strengthened — while acknowledging that a number of data quality issues remain. In particular, Defence indicated that the Sustainment Performance Management System was implemented by the end of 2017 and all products were reporting in the system by mid-2018. Defence also advised the ANAO that it has continued to make Quarterly Performance Report related improvements in the Sustainment Performance Management System, including:

• Product Exception Report on a Page: provides three months of performance information on all red Product KPIs
• Product of Concern and Product of Interest Page: provides efficient and effective CASG Division level reporting and signoff
• Product Performance Trends: increased Product performance visibility can be easily accessed for the previous three months and 12 months respectively
• Attention Request Close Loop reporting: has been incorporated to better address issues, concerns and or risks highlighted by Product managers to senior executives
• KPI Remediation Estimate on Product Dashboards: this feature allows Product managers to estimate key performance remediation timeframes enhancing transparency of underperformance and performance remediation projections to senior executives and QPR team.³⁷

2.23 Defence further advised the ANAO, in May 2019, that from June 2019 trend performance data for sustainment products would be automated in the Sustainment Performance Management System.³⁸ In past reports performance indicators were presented as a point-in-time metric, and recording trend data for the products in the Quarterly Performance Report would make emerging risks and issues clearer to the reader. The lack of trend analysis was also identified as an issue in the stakeholder survey conducted for this audit. A senior Defence leader observed that: ‘a key gap for me is understanding trends. Without cross-referencing previous reports there is no ability to know if an amber or red traffic light has been a long standing issue or is a new pop up. Also, to help management know when to intervene there is no projection or prediction of future status provided.’

Has Defence established arrangements to provide assurance over the quality and consistency of information reported?

2.24 Auditor-General Report No.2 2017–18 Defence’s Management of Materiel Sustainment identified inconsistencies between data included in the Quarterly Performance Report and other Defence documentation.³⁹ The audit report included a recommendation that: ‘Defence institutes a risk-based quality assurance process for the information included in the Defence Quarterly Performance Report.’ Defence agreed to implement the recommendation.⁴⁰

2.25 Defence informed the ANAO that it implemented this recommendation in September 2017, through the implementation of quality assurance processes including:

• cross-referring risks and remediation recommendations to Independent Assurance Review outcomes for Projects and Products;
• independent review by the CASG [Capability Acquisition and Sustainment Group] Major Program Control team to validate the inputs into the Quarterly Performance Report;
• initiating the shift in focus of QPR [Quarterly Performance Report] to Initial and Final Operational Capability instead of Initial and Final Materiel Release, with the aim of reflecting a holistic capability;
• automating the data gathering process in order to minimise human errors as much as possible; and
• consulting with CASG Executives, Defence Groups and Capability Manager Representatives in the development of [the] Quarterly Performance Report.

2.26 Defence included advice to senior stakeholders in the June 2017 Quarterly Performance Report that the recommendation had been fully implemented by 30 June 2017. Defence had developed and endorsed the Management Action Plan for the recommendation on 26 June 2017.⁴¹

2.27 Time for quality assurance checking has been allocated into the production process for the Quarterly Performance Report.⁴² Defence provided evidence indicating that the Strategic Performance Reporting team receives copies of Independent Assurance Reviews for projects and products. These copies are to be used to cross-reference the advice provided in the Quarterly Performance Report. The ANAO also identified instances of Division Heads in CASG being consulted and clearing relevant sections of the report as it is being produced.

2.28 Defence advised that part of the quality assurance process for the Quarterly Performance Report was the cross-referring of risks and remediation recommendations to Independent Assurance Review outcomes for projects and products. As discussed in paragraphs 2.14–2.19 of this audit report, the ANAO identified inconsistences between the Quarterly Performance Report and other Defence reporting in its sample, as well as delays in reporting on emerging risks (for example, risks identified by contemporary review activity such as Independent Assurance Reviews). These findings indicate that the cross-referencing process envisaged as part of the quality assurance program is not yet functioning effectively.

2.29 To automate the data gathering process for the report, Defence introduced a Quarterly Performance Report page in the Sustainment Performance Management System in July 2018. Prior to the development of this page, staff compiling the report would undertake an analysis of information from the Sustainment Performance Management System, manually compile the data and confirm the information with the Systems Program Offices and Division Heads. The ANAO’s review of Defence’s record management system for the December 2018 Quarterly Performance Report indicates that despite changes to the Sustainment Performance Management System, the compilation of the report is still a largely manual process, particularly for information on acquisition projects.⁴³ As previously noted in paragraph 2.7, the Sponsor’s Papers for the Quarterly Performance Report to the Investment Committee note that the timeliness of reporting is impacted by the quality of data from the reporting systems.

Has Defence sought feedback from senior stakeholders about the Quarterly Performance Report to establish whether the information reported is valued?

3.1 The Deputy Secretary, Capability Acquisition and Sustainment has invited recipients of the report to provide feedback on the format and content of the Quarterly Performance Report through the foreword of each report reviewed by the ANAO.

3.2 The ANAO reviewed the available signed briefs returned by Defence Ministers for the last two years. There was no feedback provided on the Quarterly Performance Report from the Ministers during this period. Defence advised the ANAO that the most recent feedback received from the Ministers on the Quarterly Performance Report was in 2014. This was when the Minister for Defence requested that the report be provided within two months of the end of the reporting period.

Survey responses

Senior stakeholders’ feedback on improving the report

3.3 In the ANAO survey of senior stakeholders undertaken as part of this audit, four of the eight respondents indicated awareness that their feedback had been sought for improvements to the Quarterly Performance Report, and these respondents further indicated that this feedback had been incorporated into the report. The remaining four respondents indicated that their feedback had not been sought for improvements to the report, with one of these respondents indicating that the question was not applicable to them. One respondent, who indicated that they had not been consulted for feedback, stated that they were aware of plans to consult them in the future.

Senior stakeholders’ feedback on value of the report

3.4 The majority of respondents to the ANAO survey indicated that it is a valued report, with five respondents advising that: the report was logically structured and easy to read; and information was mostly or always sufficient, accurate and of sufficient quality. Four respondents advised that the report mostly or always provides useful information on the status of acquisition projects and sustainment products.

3.5 Five respondents commented that timeliness affects the usefulness of the report in highlighting key risks or informing senior stakeholders’ decision-making. For example, one respondent commented that the report includes information on the status of projects and products in the past, and that it is not designed to support decisions under the capability life cycle. Another respondent noted that they did not use the document to assist in decision-making as it was ‘reporting after the fact’.

Internal feedback on the report

3.6 The ANAO reviewed the signed minutes returned by the Secretary and Chief of the Defence Force for the last two years. The Secretary and Chief of the Defence Force have not recently provided feedback on the value of the Quarterly Performance Report. There had been a request from the Secretary in 2016 for further advice about an activity in a Quarterly Performance Report.

Defence Investment Committee

3.7 In parallel with the clearance by the Secretary and Chief of the Defence Force prior to Ministerial submission, the report is provided to the Defence Investment Committee⁴⁴ for noting and feedback (as noted previously in Figure 2.1). The report includes a covering brief that includes some comments⁴⁵ and asks for feedback.

3.8 The ANAO reviewed the comments and feedback on clearance briefs by the Committee (for Quarterly Performance Reports for quarters ending in June 2017 to December 2018) to assess the feedback provided on the report. Committee feedback was provided on specific projects or products. Examples of the feedback provided includes:

• corrections to delivery dates of projects;
• questions about the accuracy of information provided such as traffic light ratings and consequences of budget information reported; and
• requesting further advice on risks identified by Committee members.

3.9 Feedback from the Committee also indicates that there remains scope for the Capability Acquisition and Sustainment Group to improve its consultation with stakeholders in the development of the report. Issues have been raised internally regarding capability managers receiving adequate time to provide input to the report, and the Investment Committee being provided adequate time to review the report. The current practice is that Committee feedback is incorporated into the next Quarterly Performance Report rather than correcting or amending the current report in light of the Committee’s observations.

3.10 A substantive discussion of the report at a future meeting of the Investment Committee was suggested in February 2019, to assist Defence to respond to the risks identified by the Committee⁴⁶ in a coordinated manner including to guide the conduct of remediation activities. Defence informed the ANAO in April 2019 that the Deputy Secretary, Capability Acquisition and Sustainment Group would provide an update to the Investment Committee on 17 April 2019. In June 2019 Defence further advised that at the 17 April 2019 meeting, the Deputy Secretary had committed to reviewing the performance reporting framework to design an improved enterprise reporting system for all capability projects.

Defence Audit and Risk Committee

3.11 The Defence Audit and Risk Committee raised an action item at its August 2018 meeting which called for useful information about specific projects covered by the report to be included in future updates to the Committee. This action item was closed at the March 2019 meeting.

Other indicators on value of the report

3.12 Other indicators of the value placed on the Quarterly Performance Report by Defence include the:

• continuing production of the report and the effort and allocation of resources to do this (see paragraph 1.7);
• investment in systems and quality assurance mechanisms (paragraphs 2.25 to 2.29);
• circulation of the report to Defence Ministers and to senior committees including the Investment Committee and the Defence Audit and Risk Committee for information;
• use of the report by several senior committees to inform discussion on performance;
• development of other enterprise reports⁴⁷ based on the Quarterly Performance Report’s approach and format; and
• agreement to use the report to implement recommendation 2.12 from the First Principles Review. This decision is discussed further in Box 1 below.

Adding further value to the report

3.13 The Capability Acquisition and Sustainment Group takes steps to add further value to its reporting by providing a brief to Ministers on the key elements of the Quarterly Performance Report. The covering brief identifies key points in the report including an overview of its contents, whether the acquisition projects are being delivered to scope and within cost, and whether the sustainment products are meeting availability and cost targets.

3.14 The brief identifies key risks and issues. For example, the September 2018 Quarterly Performance Report identified the key risk for acquisition projects as schedule performance. In the case of sustainment products it identified workforce availability, availability of platforms and the ageing nature of platforms resulting in long-term sustainment funding pressures. The brief also listed the changes since the last Quarterly Performance Report, with a focus on projects and products added to, or removed from, the Projects of Concern and Projects or Products of Interest lists.

3.15 The reports (and the briefs) reviewed by the ANAO did not include trends in performance data. In addition, emerging candidates for listing as projects/products of interest/concern that have been identified through Independent Assurance Reviews or being actively considered by Capability Acquisition and Sustainment Group senior management are not reported.

3.16 The extent to which the Quarterly Performance Report addresses its purpose⁴⁸ can be improved by including:

• trend performance data for sustainment products. Defence advised the ANAO that from June 2019 trend performance data will be automated for sustainment products⁴⁹, which would support the reporting of trends in sustainment product performance to senior stakeholders; and
• reporting on emerging candidates for Defence’s Projects of Concern regime and Projects and Products of Interest list. The report should include those projects and products recommended as candidates by Independent Assurance Reviews and those under active consideration by Defence senior management.

3.17 The inclusion of this advice would be particularly useful in relation to projects and products that are not part of the group of materially significant key acquisition projects and Top 30 sustainment products that are routinely reported on in the Quarterly Performance Report.

Has Defence used feedback from stakeholders to improve reporting to better address their needs, and does the report have an impact on decision-making?

3.19 As discussed, the Deputy Secretary, Capability Acquisition and Sustainment has invited recipients of the report to provide feedback on the format and content of the Quarterly Performance Report through the foreword of each report reviewed by the ANAO.

3.20 Defence advised the ANAO that feedback:

is responded to commensurate with the seniority, seriousness or formality of the feedback received. This includes directing feedback to the relevant area to respond directly or in consultation with the area of enquiry.

Steps taken to capture observations and improve the report

3.21 Defence advised the ANAO in December 2018 that over the last two years improvements have been made to the Quarterly Performance Report as a result of feedback from stakeholders, and that the improvements included:

• streamlining the content of the report;
• reducing Defence-centric jargon⁵⁰;
• incorporating Australian Industry Capability information; and
• shifting the focus of reporting to Initial and Final Operational Capability instead of Initial and Final Materiel Release to reflect an holistic view of capability as recommended by the First Principles Review.

3.22 The ANAO’s review of Quarterly Performance Reports from March 2017 to December 2018 identified two further changes to the report, namely:

• the addition of Project Maturity Scores to performance summaries of acquisition projects as part of the response to Recommendation 2 of the Joint Committee of Public Accounts and Audit’s Report 468: Defence Major Projects Report 2015–16⁵¹; and
• incorporating a background page for both acquisition projects and sustainment products that provides information to assist the reader with the detailed summary reports on projects and products.

3.23 Stakeholder feedback has been used to incrementally improve the eight reports reviewed by the ANAO.

Impact of the report

3.24 The ANAO sought evidence that the Quarterly Performance Report had an impact and was being used actively by senior stakeholders — for example, prompting further inquiry or action, or demonstrably informing decision-making. The direct impact of the Quarterly Performance Report is unclear, as there is limited Defence documentation to inform an assessment. Similarly it is unclear whether the discipline of providing information for the Quarterly Performance Report itself provides an incentive for local line management to improve performance.

3.25 The following instances of active use of the report were identified in Defence documentation:

• There were three occasions (two in 2016 and one in 2017) where there is evidence of the report being discussed by the Defence Committee (chaired by the Secretary). On one occasion, the Chair sought further advice regarding an activity outlined in the report.
• The ANAO’s review of feedback from the Defence Investment Committee (discussed at paragraph 3.8–3.10) indicated that the Committee reviewed the information provided and considered the broader consequences, requesting additional advice and discussion where relevant. There was also evidence of the report prompting a number of requests for further information on project and product status, cost, budget and risks.
• As discussed in paragraph 3.6, there had been a request from the Secretary in 2016 for further advice about an activity in a Quarterly Performance Report.

3.26 Defence advised the ANAO in January 2019 that the Capability Acquisition and Sustainment Group was also using the report as the basis for broader performance discussions with senior managers.

3.27 Defence advised the ANAO in May 2019 that Ministers and senior stakeholders also have access to other briefings, discussions and information on projects and products to inform their thinking and decision-making. As outlined in paragraph 3.5, respondents to the ANAO’s survey indicated that the impact of the report was mitigated to some extent by the timeliness of reporting.

Appendix 1 Response from the Department of Defence

Appendix 2 Members of the Defence Investment Committee

Members of the Defence Investment Committee

Source: Defence documentation.

Appendix 3 Defence Committees that receive the Quarterly Performance Report

1. The table below outlines the committees in Defence that receive the Quarterly Performance Report, their responsibilities, how often they receive the Quarterly Performance Report, and their level of engagement with the Quarterly Performance Report.

Source: ANAO analysis of Defence documentation

Appendix 4 Explanation of traffic lights in the Quarterly Performance Report

Projects and products of Interest

1. For the section on Projects of Interest in the report, the traffic lights for acquisition projects and sustainment products are a qualitative assessment based on the criteria identified in the table below.⁵²

Source: June 2018 Quarterly Performance Report.

Key acquisition project traffic lights

2. Section 2a — Key Acquisition Project Dashboard and Section 2c — Performance Summaries for Key Acquisition Projects in the Quarterly Performance Report use a quantitative data-driven system of traffic lights, outlined in the table below.

Source: Annex A — Explanation of Traffic Lights and Major Milestones, June 2018 Quarterly Performance Report.

Top 30 sustainment product traffic lights

3. Section 3a — Top 30 Sustainment Products Dashboard and Section 3c — Performance Summaries for Top 30 Sustainment Products in the Quarterly Performance Report use a quantitative data-driven system of traffic lights, outlined in the following tables.

Note: For the purpose of the Quarterly Performance Report, CN10 — Collins Class Submarine is the only product reporting in the Monthly Reporting System.

Source: Annex A — Explanation of Traffic Lights and Major Milestones, June 2018 Quarterly Performance Report.

Note a: In cases where there is more than one Key Performance Indicator (KPI) for a product, an average or sum is calculated for the purpose of showing a single traffic light. This is done by applying a weighting against each KPI traffic light. The weightings are: 1 for Green, 2 for Amber and 3 for Red. These are then added together and divided by the number of KPIs to get an aggregate traffic light.

Note b: This indicator measures the year end product price baseline forecast against the year-end budget. Data reported is the year end actual (forecast).

Note c: This indicator measures the year to date achievement against product price baseline funding. Data reported is the year to date actual up to the current reporting period measured against the year to date phasings for the financial year.

Source: Annex A — Explanation of Traffic Lights and Major Milestones, June 2018 Quarterly Performance Report.

Appendix 5 Survey provided to recipients of June 2018 Quarterly Performance Report

1. Are you aware that the Department of Defence’s Capability Acquisition and Sustainment Group provides you with a copy of the Quarterly Performance Report on acquisition and sustainment?

(Yes/No)

2. Do you read the Quarterly Performance Report?

(Yes/No)

3. If you read the Quarterly Performance Report, do you find it to be logically structured and easy to read?

(Yes/No/Not Applicable)

The following questions seek your views on the sufficiency, accuracy and quality of the information reported in the Quarterly Performance Report.

4. Is enough information included in the Quarterly Performance Report about projects and products to assist you?

(Always/Mostly/Sometimes/Rarely/Never)

5. In your experience, is the data included in the Quarterly Performance Report accurate at the point in time the report is circulated?

(Always/Mostly/Sometimes/Rarely/Never)

6. Is the information reported in the Quarterly Performance Report of sufficient quality to assist you?

(Always/Mostly/Sometimes/Rarely/Never)

7. Overall, do you find the Quarterly Performance Report to be a useful and timely report?

(Always/Mostly/Sometimes/Rarely/Never)

The following questions seek your views about the usefulness and timeliness of the Quarterly Performance Report as a tool to support decision making.

8. In your experience, is the Quarterly Performance Report a useful source of information about the key risks and issues that are affecting the capability, schedule and cost of acquisition projects?

(Always/Mostly/Sometimes/Rarely/Never)

9. In your experience, is the Quarterly Performance Report a useful source of information about the key risks and issues that are affecting the availability and costs of sustainment products?

(Always/Mostly/Sometimes/Rarely/Never)

10. Does the Quarterly Performance Report highlight key risks in projects and products in a timely manner to assist you in making decisions?

(Always/Mostly/Sometimes/Rarely/Never)

11. Do you rely on the Quarterly Performance Report to inform your decision-making in relation to projects and products under the Capability Acquisition and Sustainment Group’s management?

(Always/Mostly/Sometimes/Rarely/Never)

The following questions seek to identify whether the Capability Acquisition and Sustainment Group has engaged with recipients of the Quarterly Performance Report to obtain feedback and improve reporting.

12. Has your feedback been sought on the design, content and/or preparation of the Quarterly Performance Report to improve reporting and to better address your information needs?

(Yes/No)

13. Has your feedback on the design, content and/or preparation of the Quarterly Performance Report been incorporated?

(Yes/No/Not applicable)

Thank you for taking part in the survey.

Are there any other comments you wish to make about the Quarterly Performance Report?

Appendix 6 Sustainment product sample

Source: Capability Acquisition and Sustainment Group Quarterly Performance Report June 2018.

Appendix 7 2017–18 Major Projects Report — list of acquisition projects

1. The table below outlines the 26 acquisition projects from the 2017–18 Major Projects Report that also reported on in the June 2018 Quarterly Performance Report.

Note a: The budget figures are rounded to full figures.

Source: Auditor-General Report No.20 2018–19 2017–18 Major Projects Report, available from https://www.anao.gov.au/sites/default/files/Auditor-General_Report_2018-2019_20.pdf [accessed 7 March 2019] and Capability Acquisition and Sustainment Group Quarterly Performance Report June 2018.