Due to the complexity of material and the multiple sources of information for the 2020–21 Major Projects Report, we are unable to represent the entire document in HTML. You can download the full report in PDF or view selected sections in HTML below. PDF files for individual Project Data Summary Sheets (PDSS) are also available for download.

.

Background

1. The Department of Defence’s (Defence) Capability Acquisition and Sustainment Group (CASG) manages the process of bringing new specialist military equipment into service for the Australian Defence Force (ADF). As at 30 June 2021, CASG was managing 174 active major and minor capital equipment projects worth $121.6 billion¹, with an in-year budget of $8.6 billion.² Defence capitalised some $9.0 billion from these projects in 2020–21.³

2. The Major Projects Report (MPR) comprises: Defence information and commentary on a selection of its major projects (the Major Projects); and assurance and analysis of that information by the Australian National Audit Office (ANAO). This is the fourteenth annual MPR.

3. Major Projects are selected for inclusion in the MPR based on criteria endorsed by the Parliament’s Joint Committee of Public Accounts and Audit (JCPAA).⁴ The projects represent a selection of the most significant major projects managed by CASG.

4. The total approved budget for the 21 Major Projects included in this report is approximately $58.0 billion, covering 48 per cent of the total budget of active major and minor capital equipment projects of $121.6 billion.

Selected projects

5. The Major Projects selected for review and their government approved budgets as at 30 June 2021 are listed in Table 1, on p.4. They comprise six AIR projects, eight SEA projects, five LAND projects and two joint (JP) projects.

Table 1: 2020–21 MPR — selected projects and approved budgets at 30 June 2021¹

Note 1: LAND 19 Phase 7B Short Range Ground Based Air Defence and AIR 2025 Phase 6 Jindalee Operational Radar Network are included in the MPR Program for the first time in 2020–21.

Note 2: These projects have been the subject of individual performance audits. See Table 8, on pp.42–45, for more information.

Note 3: Light Tactical Fixed Wing was previously called Battlefield Airlifter. For further discussion see the Light Tactical Fixed Wing PDSS in Part 3.

Source: The PDSSs in Part 3 of this report.

Rationale for undertaking the review

6. Defence’s major defence equipment acquisition projects continue to be the subject of parliamentary and public interest. This is due to their high cost and contribution to national security, the challenges involved in completing them within the specified budget and schedule, and to the required capability, and their contribution to industrial and employment policy objectives.

7. The JCPAA has stated that the objective of the MPR is ‘to improve the accountability and transparency of Defence acquisitions for the benefit of Parliament and other stakeholders.’⁵

Conduct of the review

8. Defence prepares information for ANAO review in accordance with guidelines endorsed annually by the JCPAA (included in Part 4 of this report).⁶ The status of the Major Projects selected for review is reported in the Statement by the Secretary of Defence (included in Part 3 of this report) and a Project Data Summary Sheet (PDSS) prepared by Defence for each of the Major Projects (included in Part 3 of this report).

9. The ANAO has reviewed each of the PDSSs prepared by Defence as a ‘priority assurance review’ under subsection 19A(5) of the Auditor-General Act 1997 (the Act), allowing the ANAO full access to the information gathering powers under the Act.

10. The ANAO’s review provides limited assurance⁷ and was undertaken in accordance with the applicable auditing standards. The ANAO’s review included an assessment of Defence’s systems and controls, including the governance and oversight in place, to ensure appropriate project management. The ANAO also sought representations and confirmation from Defence senior management and industry (through Defence) in relation to the status of the selected Major Projects.

11. The objective of this ANAO assurance engagement and the ANAO review procedures is to provide a basis for independent assurance by the Auditor-General over the status of the Major Projects selected for review. The Auditor-General’s summary conclusion is set out below. The full conclusion is found in the Auditor-General’s Independent Assurance Report in Part 3 of this report.

12. Certain forecast information found in the PDSSs is excluded from the scope of the ANAO’s review, such as forecast dates, expected capability delivery performance and future risks.⁸ Accordingly, the Auditor-General’s Independent Assurance Report does not provide any assurance in relation to this information. However, material inconsistencies identified in relation to this information are considered in forming the Auditor-General’s conclusion. These exclusions to the scope of the review are due to a lack of Defence systems from which to provide complete and accurate evidence⁹ in a sufficiently timely manner to facilitate the review. This has been an area of focus of the JCPAA over a number of years¹⁰, and it is intended that all components of the PDSSs will eventually be included within the scope of the ANAO’s review.

13. Separate to the formal assurance review, the ANAO has also undertaken an analysis of key elements of the PDSSs — including cost, schedule, progress towards delivery of required capability, and risks and issues. Longitudinal analysis across these key elements of projects has also been undertaken by the ANAO.

14. Defence also provides additional insights and context in: its commentary and analysis contained in Part 2 of the MPR; and information on significant events occurring post 30 June 2021 in the Statement by the Secretary of Defence contained in Part 3 of the MPR. This commentary and analysis is not included in the scope of the ANAO’s review.

Overall outcomes of the review

Auditor-General’s summary conclusion

15. The Auditor-General has concluded in the Independent Assurance Report for 2020–21 that ‘nothing has come to my attention that causes me to believe that the information in the 21 Project Data Summary Sheets in Part 3 (PDSSs) and the Statement by the Secretary of Defence, excluding the forecast information, has not been prepared in all material respects in accordance with the 2020–21 Major Projects Report Guidelines (the Guidelines), as endorsed by the Joint Committee of Public Accounts and Audit.’

Statement by the Secretary of Defence

16. The Statement by the Secretary of Defence was signed on 2 December 2021. The Secretary’s statement provides his opinion that the PDSSs for the 21 selected projects ‘comply in all material respects with the Guidelines and reflect the status of the projects as at 30 June 2021’.

17. In addition, the Statement by the Secretary of Defence details significant events occurring post 30 June 2021, which materially impact the projects included in the report, and which should be read in conjunction with the individual PDSSs. The Statement includes information on: Joint Strike Fighter, Future Frigates, Future Subs, MRH90 Helicopters, Offshore Patrol Vessel, Repl Replenishment Ships, Battle Comm. Sys. (Land) 2B, Maritime Comms, ANZAC Air Search Radar Repl and UHF SATCOM.¹¹

Summary of key observations from the ANAO’s review

18. The ANAO’s review (found in Part 1 of this report) includes Defence’s project management and reporting arrangements contributing to the overall governance of the Major Projects. A summary of key observations is provided below.

Status of JCPAA recommendations and requests

• Following JCPAA recommendations made in May 2014 and May 2016¹², Defence is yet to implement a system of materiel capability delivery performance/scope reporting, with a robust methodology applicable to materiel acquisition (see paragraphs 2.63 to 2.68).
• Following a JCPAA recommendation made in September 2018¹³, Defence advised the Committee in May 2020 that Predict! was the mandated risk management system.¹⁴ Defence advised the ANAO in October 2021 that Defence’s policy was updated in August 2021 to reflect that risks must be recorded in Predict!, although this is yet to be applied consistently across all projects (see paragraphs 1.76 to 1.80).
• Following a JCPAA request made in 2018¹⁵ ‘on how Defence major project cost variations and the costs of retaining project staff over time might be reported annually in future Major Projects Reports’, Defence advised that it is not yet in a position to provide the staff cost component of projects and its systems are not capable of calculating the cost of retaining project staff over time (see paragraphs 1.64 to 1.67).

Status of Auditor-General report recommendations

• Auditor-General Report No.34 2020–21 Implementation of ANAO and Parliamentary Committee Recommendations — Department of Defence tabled in April 2021 and included an assessment of four recommendations relevant to the MPR.¹⁶ Of these, one was assessed as implemented, one was largely implemented, and two were not implemented.
• In July 2020 Defence closed both recommendations from Auditor-General Report No.31 2018–19 Defence’s Management of its Projects of Concern. The ANAO assessed these recommendations as not implemented (see paragraphs 1.19 to 1.21).¹⁷
• Defence closed Recommendation No.1 from Auditor-General Report No.3 2019–20 Defence’s Quarterly Performance Report on Acquisition and Sustainment in March 2020, and this was considered in Auditor-General Report No.19 2019–20 Major Projects Report.18 The ANAO assessed this recommendation as implemented (see paragraphs 1.27 to 1.28).¹⁹
• Recommendation No.1 from Auditor-General Report No.40 2018–19 Modernising Army Command and Control — the Land 200 Program was assessed as largely implemented.²⁰
• Other relevant ANAO audits tabled in 2020–21 include: Auditor-General Report No.18 2020–21 Defence’s Procurement of Combat Reconnaissance Vehicles (LAND 400 Phase 2) and Auditor-General Report No.12 2020–21 Defence’s Procurement of Offshore Patrol Vessels SEA 1180 Phase 1. These audits included recommendations for improvement in Defence Independent Assurance Review processes. The Defence Audit and Risk Committee has accepted closure of the two recommendations from Auditor-General Report No.12 2020–21 Defence’s Procurement of Offshore Patrol Vessels SEA 1180 Phase 1. Reporting is yet to be provided to the Committee on the recommendation in Auditor-General Report No.18 of 2020–21 Defence’s Procurement of Combat Reconnaissance Vehicles (LAND 400 Phase 2) (see paragraphs 1.14 to 1.17).

Defence acquisition governance

19. In the course of its review of the PDSSs, the ANAO considered:

• Defence’s system for reporting on the status of acquisition projects through the Independent Assurance Review (IAR) process. Fifteen of the 21 projects included in this report had an IAR conducted during 2020–21²¹ (see paragraphs 1.11 to 1.13).
• Defence’s approach to entry and exit from the Projects of Interest and Projects of Concern lists (see paragraphs 1.18 to 1.23).
• The reporting provided to senior stakeholders within government and Defence with insight into the delivery of capability to the Australian Defence Force. The ANAO observed a gap in reporting activity (see paragraphs 1.26 to 1.33).
• The importance of capturing government decisions in internal Defence documentation and ensuring that Materiel Acquisition Agreements are appropriately aligned with these decisions (see paragraphs 1.36 to 1.42).
• Defence’s implementation of the Smart Buyer Framework to support strategic decision making in the acquisition of major projects. The framework was not utilised at the Second Pass government approval stage for projects in the current MPR (see paragraphs 1.43 to 1.44).
• Implementation of new business systems to report on the status of acquisition projects (see paragraphs 1.45 to 1.48).
• Projects’ use of contingency funds (see paragraphs 1.56 to 1.60). Two projects in the MPR, MRH90 Helicopters and Battle Comm. Sys. (Land) 2B, committed contingency funds in 2020–21. The MRH90 Helicopters project committed $34.4 million of contingency funds to manage supportability and performance risks. The Battle Comm. Sys. (Land) 2B project used $30.3 million of contingency funds to address delayed delivery and refinement of specifications.
• The status of CASG’s Risk Management Reform Program (initiated by the Deputy Secretary CASG in 2017) and establishment of the CASG Risk Management Framework (see paragraphs 1.68 to 1.80).
• A number of projects had not fully met the requirements of Defence’s Project Risk Management Manual Version 2.5 (PRMM V2.5) in relation to contingency allocation (see paragraph 1.60) and risk management (see paragraph 1.74).
• Defence updated its policy on Lessons Learned in May 2020 but is yet to fully implement the policy, including compliance monitoring arrangements (see paragraphs 1.81 to 1.83).
• Defence has not defined in its internal policies and procedures the terms ‘caveat’ or ‘deficiency’ relating to the declaration of significant capability milestones. The ANAO has continued to observe the use of these terms by Defence to represent exceptions to the achievement of significant milestones (see paragraphs 1.84 to 1.86).

ANAO analysis of project performance

20. In addition to its limited assurance review, the ANAO has undertaken an analysis of key elements of the Defence PDSSs — relating to cost, schedule, progress towards delivery of required capability, and risks and issues. The ANAO has also undertaken longitudinal analysis across these key elements of projects.

Cost

21. Cost management is an ongoing process in Defence’s administration of the Major Projects. While all projects reported that they could continue to operate within the total approved budget of $58.0 billion, the MRH90 Helicopters and Battle Comm. Sys. (Land) 2B projects were required to draw upon contingency funds to complete project activities.

22. The total approved budget for Major Projects included in this MPR has increased by $18.3 billion (33 per cent) since initial Second Pass Approval by government.

23. Budget variations greater than $500 million are detailed in Table 2, on p.11.²²

24. As the MPR predominantly focuses on the approved capital budget for acquisition, the ongoing costs of Project Offices²³, training, replacement capability, etc., are not reported here.

Schedule

25. Delivering Major Projects on schedule continues to present challenges for Defence, affecting when the capability is made available for operational release and deployment by the Australian Defence Force, as well as the cost of delivery.

26. Total schedule slippage²⁴ for the Major Projects, as at 30 June 2021, was 405 months when compared to the initial schedule. This represents a 23 per cent increase since Second Pass approval.

27. Across MPR projects that have experienced slippage — comprising 10 of the 19 projects with approved dates for Final Operational Capability (FOC)²⁵ — the average slippage is 40 months (3.38 years).

28. Table 3, on p.13, includes details of in-year and total schedule slippage by project. The table shows an increase of 73 months of in-year slippage during 2020–21.

29. The total slippage of 405 months in 2020–21 is 102 months lower than the total in 2019‍–‍20 of 507 months. This is due to:

• the exclusion of projects which have exited the MPR (AWD Ships, P-8A Poseidon, Growler, and Collins R&S), removing 175 months of slippage from the total reported in 2019–20 (see Table 4); and
• the increase of 73 months of in-year slippage described above.

Table 3: Schedule slippage from original planned Final Operational Capability¹

Note 1: Slippage refers to a delay in the current forecast date compared to the original government approved date of FOC. These figures exclude delays to a project’s schedule that do not result in slippage past the original government approved date, and schedule reductions over the life of the project.

Note 2: These projects have been identified by Defence as Projects of Interest (see paragraph 1.33).

Note 3: These projects’ FOC milestones had not been approved by government at 30 June 2021.

Note 4: Light Tactical Fixed Wing was previously called Battlefield Airlifter.

Note 5: These projects’ FOC forecasts are reported in their PDSSs as ‘TBA’. The delays are anticipated to be several years.

Source: ANAO analysis of the 2020–21 PDSSs.

30. Platform availability has contributed to the slippage experienced within some projects. For example, Maritime Comms and Collins Comms and EW have been impacted by changes to docking schedules of the ANZAC Class frigates and Collins Class submarines respectively.

31. Significant delays have also been experienced by projects with the most developmental content: MRH90 Helicopters, MQ-4C Triton, CMATS, and Battle Comm. Sys. (Land) 2B.

32. Table 4, on p.14, provides details of schedule slippage for projects which have exited the MPR. Compared to the 405 months total schedule slippage for the current 19 Major Projects with approved FOC milestones²⁶, the 33 projects which have exited the MPR²⁷ have reported accumulated schedule slippage of 1321 months, as at their respective exit dates.

33. Table 4 indicates that schedule slippage for projects which have exited the MPR was more pronounced in projects with the most developmental content.

Table 4: Schedule slippage for projects which have exited the MPR¹

Note 1: The Hornet Refurb and Battle Management System (BMS) projects are not included in this table as they did not have FOC milestones approved by government.

Note 2: Next Gen Satellite shows slippage in Figure 7a, on p.59, which related to the final capability milestones at the time. By the time it reached FOC, a new final capability milestone had been introduced and slippage was reduced.

Source: PDSSs in Major Projects Reports and ANAO analysis.

34. Additional ANAO analysis (refer to Figure 6, p.58) has compared project slippage against the Acquisition Type of projects — that is, Defence’s classification of projects as Military Off-The-Shelf (MOTS)²⁸, Australianised MOTS or developmental. These classifications are a general indicator of the difficulty associated with the procurement process.

35. Figures 7a and 7b (pp.59–60) provide analysis of projects either completed, or removed from the MPR review, and shows that a focus on MOTS acquisitions has assisted in reducing schedule slippage. Prima facie, the more developmental in nature a project is, the more likely it will result in a greater degree of project slippage. The data in Figures 7a and 7b was requested by the JCPAA in May 2014.²⁹

36. In this MPR, additional ANAO analysis has been included in relation to the Acquisition Categorisation (ACAT) level³⁰, due to the removal of reporting against the Acquisition Type of projects in the PDSSs (see Figures 8, 9a and 9b on pp.61–63).

37. Additional reporting against the ACAT level has identified that there has been an increase of projects at the ACAT I³¹ and ACAT II³² levels, and that ACAT I projects currently in the MPR are reporting significantly more slippage to FOC than ACAT II projects (Figure 8, on p.61). ACAT I projects carry a higher level of technical risk.

38. Longitudinal analysis indicates that while the reasons for schedule slippage vary, it primarily reflects the underestimation of both the scope and complexity of work, particularly for Australianised MOTS and developmental projects (see paragraphs 2.34 to 2.35).

Capability

39. The third principal component of project performance examined in this report is progress towards the delivery of capability approved by government. While the assessment of expected capability delivery by Defence is outside the scope of the Auditor-General’s formal review conclusion, it is included in the ANAO analysis to provide an overall perspective of the three principal components of project performance.

40. The Defence PDSSs report that 17 projects in this year’s report will deliver all of their key capability requirements. Two projects, Light Tactical Fixed Wing³³ and Battlefield Command System, report that they are unable to deliver all of the required capability by FOC. Two PDSSs (Future Frigates and Future Subs) do not include this reporting, as the projects did not have approved materiel capability to be delivered at 30 June 2021.

41. Defence’s assessment indicates that some elements of capability to be delivered by projects may be ‘under threat’, but the risk is assessed as ‘manageable’. The four project offices experiencing challenges with expected capability delivery (2019–20: five) are Joint Strike Fighter, MRH90 Helicopters, Overlander Medium/Heavy, and Battle Comm. Sys. (Land) 2B.

42. Table 5, on p.16, summarises expected capability delivery as at 30 June 2021, as reported by Defence.

Table 5: Capability delivery

Note 1: Defence advised in these years that AWD Ships would not deliver one element of capability at FOC (which equated to approximately one per cent). However, across all the Major Projects this percentage rounded to zero per cent.

Source: PDSSs in Major Projects Reports and ANAO analysis.

43. In addition to reporting on expected capability delivery, Defence has continued the practice of including in the PDSSs declassified information on contractual remedies for projects, including stop payments and liquidated damages. During 2020–21 Battlefield Command System negotiated contractual remedies involving stop payments, and Hawkei received $6.2 million in liquidated damages via a Contract Change Proposal.

Summary longitudinal analysis

44. Table 6, on p.17, provides summary data on Defence’s progress toward delivering the capabilities for the Major Projects covered in this report, and compares current data against that reported in previous editions of the MPR.

Table 6: Summary longitudinal analysis¹

Refer to paragraphs 19 to 42 in Part 1 of this report.

Note 1: The data for the 21 Major Projects in the 2020–21 MPR compares the data from projects in the 2019–20 MPR and 2018–19 MPR. The Major Projects included within each MPR are based on entry and exit criteria in the Guidelines, which have been included in Part 4 of this report. The entry and exit of projects should be considered when comparing data across years.

Note 2: Where a project has multiple Second Pass Approvals, the MPR has historically reported budget variations from the initial Second Pass Approval. The figures in this row are consistent with prior year reporting. See Table 2 for a breakdown of the major components of this variance, and Table 8 on pp.42–45 for all real variations.

Note 3: Where a project has multiple Second Pass Approvals, the budget at Second Pass Approval reported in the Header refers to the total budget as at the final Second Pass Approval. The figures in this row use this methodology.

Note 4: Slippage refers to a delay in the current forecast date compared to the original government approved date of FOC. Slippage can occur due to late delivery, increases in scope or at times can be a deliberate management decision.

Note 4a: As shown in Table 3 on p.13 and Table 11 on p.68 of this report, for the ten 2020–21 major projects which have experienced slippage, the range is 2 to 95 months of total slippage.

Note 5: The grey section of the table is excluded from the scope of the ANAO’s priority assurance review, due to a lack of systems from which to obtain complete and accurate evidence in a sufficiently timely manner to facilitate the review.

Note 6: The figures represent the combined number of open ‘high’ and ‘extreme’ risks and issues reported in the PDSSs across all projects. Risks and issues may be aggregated at a strategic level.

Note 7: These figures represent the average predicted capability delivery across all of the Major Projects. This method reduces the effect of any individual project’s size on the aggregate figure. Previously, these figures were calculated based on the number of distinct capability measures defined by each project and therefore projects with more capability measures had more of an effect on the aggregate figure.

Note 8: Defence advised in these years that AWD Ships would not deliver one element of capability at FOC (which equated to approximately one per cent). However, across all the Major Projects this percentage rounded to zero per cent.

Impact of COVID-19 on the selected Major Projects

45. Sixteen of the 21 Major Projects reported in their PDSS that they experienced an impact as a result of the COVID-19 pandemic.³⁴ Of these 16 projects, six reported impacts across multiple areas.

Cost / budget

46. Six projects reported an impact on project budget as a result of the COVID-19 pandemic. These six projects experienced an underspend, citing varying reasons for this (delay to training and support, overseas suppliers, shipyard closures and international travel restrictions). One of these projects has highlighted an impact to the budget as an emerging issue.

Schedule

47. The 16 projects that reported an impact on scheduling as a result of the COVID-19 pandemic included factors relating to:

• supplier disruption (supplier production and/or shipping delays);
• workforce limitations relating to travel (specialists and crew were due to travel both interstate and from other countries to work with/on the projects or to deliver/undertake training) and social distancing restrictions; and/or
• contractor delays (scope, delivery and certification delays).

Capability

48. No projects reported an impact to capability delivery as a result of the COVID-19 pandemic.

1.1 This chapter provides the Australian National Audit Office’s (ANAO) overview of the scope and approach adopted for its limited assurance review of the 21 Project Data Summary Sheets (PDSSs) prepared by the Department of Defence (Defence) for this year’s Major Projects Report (MPR). The chapter also includes information and commentary on developments in Defence’s acquisition governance processes, based on the ANAO’s review.

Review scope and approach

1.2 In 2012 the Parliament’s Joint Committee of Public Accounts and Audit (JCPAA) identified the ANAO’s review of Defence PDSSs as a priority assurance review, under subsection 19A(5) of the Auditor-General Act 1997 (the Act). This provided the ANAO with full access to the information gathering powers under the Act. The ANAO’s review of the individual PDSSs, which are included in Part 3 of the MPR, was conducted in accordance with the auditing standards set by the Auditor-General under section 24 of the Act through the incorporation of the Australian Standard on Assurance Engagements (ASAE) 3000 Assurance Engagements Other than Audits or Reviews of Historical Financial Information, issued by the Australian Auditing and Assurance Standards Board.

1.3 The following forecast information provided by Defence is excluded from the scope of the ANAO’s review: capability delivery; risks and issues; and forecast dates. These exclusions are due to the lack of Defence systems from which to provide complete and accurate evidence³⁵, in a sufficiently timely manner to complete the review. Accordingly, the Independent Assurance Report by the Auditor-General does not provide any assurance in relation to this information. However, material inconsistencies identified in relation to this information are required to be considered in forming the Auditor-General’s conclusion.

1.4 The ANAO’s work is appropriate for the purpose of providing an Independent Assurance Report in accordance with the ANAO Auditing Standards. However, the review of individual PDSSs is based on a limited assurance approach and is not as extensive as individual performance audits and financial statement audits conducted by the ANAO, in terms of the nature and scope of issues covered, and the extent to which evidence is required by the ANAO. Consequently, the level of assurance provided by this review, in relation to the 21 major Defence equipment acquisition projects (Major Projects), is less than that provided by the ANAO’s program of performance and financial statement audits.

1.5 Separately, the ANAO reviews developments in Defence’s acquisition governance processes (information and commentary on governance issues appears in this chapter) and undertakes analysis of key elements of the PDSSs (information and commentary on systemic issues and longitudinal analysis for the 21 projects reviewed appears in the next chapter).

1.6 The ANAO’s review was conducted in accordance with the ANAO Auditing Standards at a cost to the ANAO of approximately $1.8 million.

Review methodology

1.7 The ANAO’s review of the information presented in the individual Defence PDSSs included:

• examination and assessment of the governance and oversight in place to ensure appropriate project management;
• an assessment of the systems and controls that support project financial management, risk management and project status reporting within Defence;
• an examination of each PDSS and the documents and information relevant to them;
• a review of relevant processes and procedures used by Defence in the preparation of the PDSSs;
• discussions with persons responsible for the preparation of the PDSSs and management of the projects;
• analysis of project information, for example, cost and schedule variances;
• taking account of industry contractor comments provided on draft PDSS information;
• assessing the assurance by Defence managers attesting to the accuracy and completeness of the PDSSs;
• examination of the representations by the Chief Finance Officer supporting the project financial assurance and contingency statements;
• examination of confirmations, provided by the Capability Managers, relating to each project’s progress toward Initial Materiel Release (IMR), Final Materiel Release (FMR), Initial Operational Capability (IOC) and Final Operational Capability (FOC); and
• examination of the Statement by the Secretary of Defence, including significant events occurring post 30 June, and management representations by the Secretary of Defence.

1.8 The ANAO’s review of PDSSs also focused on project management and reporting arrangements contributing to the overall governance of the Major Projects. The ANAO considered:

• developments in acquisition governance (see paragraphs 1.10 to 1.48, below);
• the financial framework, particularly as it applies to the project financial assurance and contingency statements (see Section 2 of the PDSSs);
• schedule management and test and evaluation processes (see Section 3 of the PDSSs);
• materiel capability / scope delivery forecast assessments, including Defence statements of the likelihood of delivering key capabilities, particularly where caveats are placed on the Capability Manager’s declaration of significant milestones (see Section 4 of the PDSSs);
• changes due to Defence’s reform of the Defence Enterprise Risk Management Framework, and the completeness and accuracy of major risk and issue data (see Section 5 of the PDSSs); and
• the impact of acquisition issues on sustainment to ensure the PDSS is a complete and accurate representation of the acquisition project.

1.9 This review activity informed the ANAO’s understanding of the systems and processes supporting the PDSSs for the 2020–21 review period. It also highlighted issues in those systems and processes that warrant attention.

Acquisition governance

1.10 Consistent with previous years, the ANAO considered Defence’s Major Project acquisition governance processes when planning and conducting the review for the 2020–21 MPR. While some of these processes are now established, others continue to mature or require further development to achieve their intended impact.

Defence Independent Assurance Reviews

1.11 The Defence Independent Assurance Review (IAR) process provides the Defence Senior Executive with assurance that projects and products will deliver approved objectives and are prepared to progress to the next stage of activity. These management-initiated reviews consider a project’s status while sufficient time remains for corrective action to be implemented.^36,37

1.12 IARs are intended to commence at project initiation and are conducted through to FOC; for higher-complexity projects, ideally on an annual basis. They are an important input to key acquisition and sustainment decision points or milestones.³⁸

1.13 Fifteen of the 21 Major Projects had an IAR conducted during 2020–21³⁹, which formed key evidence for the ANAO’s review.

1.14 The ANAO recently published two performance audit reports which recommended improvements in Defence IAR processes.

1.15 Auditor-General Report No.12 2020–21 Defence’s Procurement of Offshore Patrol Vessels — SEA 1180 Phase 1 made two recommendations: that Defence plan the sequencing of IAR activity undertaken during a platform selection process in order to avoid conflicts with other processes and to ensure access to all relevant information; and that Defence maintain Commonwealth records, document and retain all evidence and advice regarding its decision-making in procurement.

1.16 Auditor-General Report No.18 2020–21 Defence’s Procurement of Combat Reconnaissance Vehicles (LAND 400 Phase 2) included a recommendation that Defence should review the process in place to provide assurance to its senior leadership that agreed IAR recommendations have been implemented appropriately and in a timely manner.

1.17 The assessment of whether these recommendations have been implemented by Defence is outside the scope of this review. The Defence Audit and Risk Committee has accepted closure of the two recommendations from Auditor-General Report No.12 of 2020–21 Defence’s Procurement of Offshore Patrol Vessels — SEA 1180 Phase 1. Reporting is yet to be provided to the Committee on the status of the recommendation in Auditor-General Report No.18 of 2020–21 Defence’s Procurement of Combat Reconnaissance Vehicles (LAND 400 Phase 2).

Projects of Concern

1.18 The Projects of Concern process is intended to focus the attention of the highest levels of government, Defence and industry on remediating problem projects.⁴⁰ As at 30 June 2021, one MPR project, MRH90 Helicopters, was a continuing Project of Concern. The project was placed on the list in November 2011 due to contractor performance relating to significant technical issues preventing the achievement of milestones on schedule.⁴¹ The project has progressed the materiel capability delivery relating to the Taipan Gun Mount, Aero-Medical Evacuation Equipment and the Common Mission Management System.⁴² FOC is scheduled for June 2022, six months later than stated last year. In October 2021, it was reported that Navy may replace its six MRH90 Helicopters with 12 additional Seahawk helicopters.⁴³ Defence advised the ANAO in November 2021 that while the United States Government had approved the sale of MH-60R helicopters to Australia, the Australian Government had not yet approved the purchase.

1.19 Auditor-General Report No.31 2018–19 Defence’s Management of its Projects of Concern assessed whether Defence’s Projects of Concern regime was effective in managing the recovery of underperforming projects. It concluded that, while the regime is an appropriate mechanism for escalating troubled projects to the attention of senior managers and ministers, Defence was not able to demonstrate the effectiveness of its regime in managing the recovery of underperforming projects. Moreover, the audit observed that the transparency and rigour of the framework’s application has declined in recent years. The ANAO recommended that:

• Recommendation No.1: Defence introduce, as part of its formal policy and procedures, a consistent approach to managing entry to, and exit from, its Projects of Interest and Projects of Concern lists. This should reflect Defence’s risk appetite and be made consistent with the new Capability Acquisition and Sustainment Group Risk Model and other, Defence-wide, frameworks for managing risk. To aid transparency, the policy and the list should be made public.
• Recommendation No.2: Defence evaluates its Projects of Concern regime.⁴⁴

1.20 In July 2020 Defence closed both these recommendations, advising that the Capability Acquisition and Sustainment Group (CASG) had developed a consistent approach to entry and exit from the Projects of Interest and Projects of Concern lists; that the Projects of Concern list was publicly available; and that CASG had evaluated the Projects of Concern regime and had effective assurance mechanisms in place, underpinned by IARs.⁴⁵

1.21 Auditor-General Report No.34 2020–21 Implementation of ANAO and Parliamentary Committee Recommendations — Department of Defence concluded that the two recommendations relating to the management of Projects of Concern had not been implemented. The ANAO reported that there was no evidence that Defence established a clear basis or criteria to ensure a consistent approach to entry to and exit from the Projects of Concern or Projects of Interest lists, and that no evidence of an evaluation was provided to the ANAO.⁴⁶

1.22 At the JCPAA’s September 2021 hearings on the 2019–20 Major Projects Report, the Deputy Secretary CASG stated that:

We are working to improve the way in which we’re able to measure the underperforming projects. Invariably, it’s data driven quite easily on cost and schedule against the documented milestones and loaded milestones and then the capability a little more to that. As we develop up the program report or the project and sustainment report that we’re doing to supplement the sequencing in between portfolio budget statements, portfolio additional estimates statements and from this major projects report itself, we will continue to mature that by feeding in capability manager assessments and information. That’s important because, ultimately, they are the first principles responsible for the capability delivery and we are the delivery agency but the operational effect is through the capability manager.⁴⁷

1.23 The ANAO noted during the preparation of the 2020–21 MPR that the Future Subs project was not listed as a Project of Concern or a Project of Interest.

1.24 The Future Subs project was included for the first time in the 2019–20 MPR. Since its entry to the MPR, the Future Subs PDSS has included a risk in Section 5.1 that: ‘There is a risk that our Program Partners will not adequately address issues and challenges (including technical risks) that arise during the course of the Program’.⁴⁸ Defence reported that this risk was being remediated as follows: ‘Contracted requirements exist on Program performance, behaviours and expectations and are supported by: ongoing engagement with CEO’s; bilateral and tripartite governance arrangements; and ongoing independent critical peer review by the Naval Shipbuilding Advisory Board and Submarine Advisory Committee.’ The ANAO was advised by Defence in 2020‍–‍21 that the Future Subs project was not considered as a candidate for inclusion as a Project of Concern or Interest because it was already subject to the most senior levels of governance and scrutiny. Defence further advised that this included more frequent reporting to ministers and government, which Defence advised was effectively consistent with the reporting expectations of a Project of Concern or Interest even if the project was not formally included in that framework.

1.25 On 16 September 2021 the Australian Government announced a change in strategic direction for the Future Subs project, stating that: ‘The pursuit of nuclear-powered submarine technology means that Australia will no longer proceed with the Attack class conventional submarine program with Naval Group’.⁴⁹

Quarterly Performance Report and Project and Sustainment Report

1.26 The aim of the Capability Acquisition and Sustainment Quarterly Performance Report (QPR) was to provide senior stakeholders within government and Defence with insight into the delivery of capability to the Australian Defence Force.⁵⁰ The report was provided to the Minister for Defence and the Minister for Defence Industry on a quarterly basis.⁵¹

1.27 In July 2019, the ANAO completed an audit on the effectiveness of the QPR in providing senior stakeholders with accurate and timely information on the status of projects and emerging risks and issues. It found the June 2018 QPR, reviewed by the ANAO, to be largely effective, contained mostly accurate information, and was valued by senior stakeholders.⁵² The ANAO recommended that Defence improve the QPR as a tool for senior leaders by reporting on:

1. trend performance data for sustainment products; and
2. emerging candidates for the Projects/Products of Concern list and Products/Projects of Interest list that have been recommended by an Independent Assurance Review or which are under active consideration by senior management.⁵³

1.28 During its review for the 2018–19 MPR, the ANAO observed that Defence’s June 2019 QPR reported on both improved and deteriorated performance for both acquisition and sustainment products since the previous QPR. This reflected a change in trend reporting consistent with the agreed ANAO recommendation. Additionally, the ANAO observed that Defence’s June 2019 QPR reported the emerging candidates for the Projects/Products of Concern list and Products/Projects of Interest list which had been recommended either by an IAR or which were under active consideration. This change was also consistent with the agreed ANAO recommendation.⁵⁴ Defence closed this recommendation in March 2020.⁵⁵

1.29 CASG ceased producing QPRs after June 2020, with the report superseded in February 2021 by the Project and Sustainment Report (PSR). For the 2020–21 MPR review, the ANAO examined the February 2021 PSR as part of the procedures for its limited assurance review of Defence’s PDSSs.⁵⁶ The February 2021 PSR covered the reporting period since the last QPR in June 2020, and contained detailed reporting on Projects/Products of Concern and Projects/Products of Interest, information on reform and governance activities, significant events, Domain overviews and performance analysis and trends comparing data to previous reporting periods, and indicative overviews of Defence’s Top 30 acquisition and sustainment projects.

1.30 During Budget Estimates hearings held on 1 June 2021, the Deputy Secretary CASG stated that the PSR was anticipated to be issued on a six-monthly basis.⁵⁷

1.31 A six-month gap in reporting activity introduces a risk of diminished information being available for decision making by senior leaders. Further, compared to the QPR, the PSR contains less information on acquisition projects and sustainment products that are not classified as a Project/Product of Concern or Project/Product of Interest.

1.32 Defence advised the ANAO in September 2021 that it has ‘management processes that ensure Capability Managers and Delivery groups are informing the Secretary of Defence and the Chief of Defence Force through weekly roundtable discussions and the Ministers are … informed on pertinent issues as they arise’. Defence also advised the ANAO that the next PSR was still in development and a draft would not be ready prior to the completion of the 2020‍–‍21 MPR.

1.33 In October 2021, Defence further advised the ANAO that the PSR was only an interim report, and that a new ‘Capability Report’ originally intended to replace the QPR was not sufficiently mature to be implemented. The new report was yet to be named, and would not be ready before the end of 2021.

1.34 Defence’s February 2021 PSR identified nine MPR projects as Projects of Interest⁵⁸:

• Joint Strike Fighter — the PSR notes that contingency plans have been enacted to counteract the impacts of COVID-19 to build capability and support systems.⁵⁹
• Future Frigates — included due to size, complexity, risk profile and media interest. The PSR notes that all scheduled activities have been progressing as planned, although some will need to be reassessed in light of COVID-19 restrictions.⁶⁰
• MQ-4C Triton — the PSR notes that the United States Navy announced a production funding suspension for its Triton program until 2023. The suspension will have capability, schedule and cost implications, and potential sustainment cost and capability impacts. The project is undergoing a fundamental review.⁶¹
• Hawkei — the PSR notes concerns around reliability, design and production delays, and that a brake related incident has further delayed the forecast IOC.⁶²
• Light Tactical Fixed Wing⁶³ — FOC was not met in December 2019 and the PSR notes that ‘residual’ activities remain outstanding, including fleet fitment and certification of Mode 5 Identification Friend or Foe and remediation of the Missile Approach Warning System. Air Force revised scope for the residual activities required to achieve FOC has been provided to government for consideration.⁶⁴
• JORN Upgrade — the PSR notes that engineering milestone delays may impact FOC.⁶⁵
• CMATS — the PSR notes some reduction of tower capability and the ongoing need for Airservices Australia to implement cost saving changes agreed with Defence.⁶⁶
• Battlefield Command System — the PSR notes schedule risks due to vehicle integration issues.⁶⁷
• UHF SATCOM — the PSR notes that the Network Control System experienced software development delays and security and integration issues.^68,69

1.35 These reported issues with the Projects of Interest align with the results of the ANAO’s review of the relevant PDSSs. Delays to progress have impacted the delivery schedule of MQ-4C Triton, Light Tactical Fixed Wing, and CMATS during 2020–21.⁷⁰

Project Directives and Materiel Acquisition Agreements

1.36 Project Directives (previously known as Joint Project Directives) state the terms of government approval, reflecting the approved scope and timeframes for activities, responsibilities and resources allocated, and key risks and issues.⁷¹ Project Directives have historically been used to inform internal Defence documentation such as Materiel Acquisition Agreements (MAAs) between CASG and the Service Chiefs.^72,73 Project Directives had previously been described as a key governance document under the Capability Life Cycle⁷⁴, intended to ensure that all parties in Defence are informed of government decisions.

1.37 Defence updated the Capability Life Cycle Manual in June 2020, no longer referring to Project Directives as a key governance document. The Capability Life Cycle Manual was superseded by the Defence Capability Manual in December 2020. The Defence Capability Manual also does not refer to Project Directives. Defence has advised the ANAO that government decisions are recorded in CapabilityOne, which records government decisions in relation to a project. In some cases, the Joint Force Authority may provide a specific documented directive. The ANAO has previously highlighted the importance of ensuring that Project Directives properly reflect the relevant government decision, and that MAAs are appropriately aligned with the relevant Project Directive.⁷⁵

1.38 Of the two new projects entering the 2020–21 MPR, only the JORN Upgrade project advised the ANAO that it had direct access to government approval documentation. The SRGB Air Defence project advised that it did not have direct access to government approval documentation. Last year, the Future Subs, Combat Recon. Vehicles and Battlefield Command System projects advised that they did not have access to government approvals.

1.39 In November 2020, Defence advised the ANAO that ‘the internal Cabinet Liaison Services section provides advice to Defence in relation to information pertaining to government approvals. Where a Project has not been identified as having a need to know, the Project can request access to relevant Cabinet documents via a business case.’

1.40 The risk of misalignment or error is reduced if Defence has appropriate access to government records, such as that previously provided by Project Directives. If projects can access original Cabinet documentation, there is no residual impact.

1.41 The ANAO requires access to original approval documents to validate the requirements of projects. Validation based on internal Defence documentation is not always possible or may not meet evidentiary standards.

1.42 First advised by Defence in July 2016⁷⁶, Product Delivery Agreements (PDAs)⁷⁷ were to be developed to replace the existing MAAs and Materiel Sustainment Agreements (MSAs). Defence advised the ANAO during preparation of the 2020–21 MPR that this initiative is still in the concept phase and will not apply until a PDA framework is approved and implemented. In October 2021, Defence advised the ANAO that in the absence of the PDA framework, Capability Managers and Delivery Groups continue to use the Materiel Acquisition Agreement and Materiel Sustainment Agreement Framework.

Smart Buyer Framework

1.43 The 2015 First Principles Review recommended the construction of a ‘smart buyer’ framework, with the aim of ‘[ensuring] Defence can make strategic decisions regarding the most appropriate procurement and contracting methodologies’. Defence has begun to conduct Smart Buyer assessments for acquisition projects at different stages of approval. None of the projects currently in the Major Projects portfolio have been approved under the Smart Buyer processes.

1.44 Defence advised the ANAO that no MPR projects have been considered by the Smart Buyer Framework in 2020–21. Of the two projects entering the MPR in 2020–21, neither was approved under the Smart Buyer process, although SRGB Air Defence participated in a pilot workshop while the Smart Buyer framework was still being defined, and JORN Upgrade conducted an informal, internal assessment against the Smart Buyer criteria. The ANAO also observed that Smart Buyer workshops were being held for future tranches and phases of projects in the MPR.

Business systems

1.45 Defence continues to review its business systems with the aim of consolidating them to provide a more manageable ICT environment. During 2019–20, CASG continued to report on the status of acquisition projects in the Monthly Reporting System (MRS), which formed part of the evidence for the ANAO’s review of the PDSSs. In July 2020, Defence discontinued the MRS and project reporting now occurs via the Monthly Reporting Module (MRM). Defence advised the ANAO that MRM replicates the functionality of MRS while delivering an updated platform and user interface. A second new system, the Project Performance Report Information Platform (PPRIP), delivers a platform for projects to also conduct monthly reviews of their project and enable the raising of risks and actions with line management.

1.46 As the MRM was implemented in the 2020–21 financial year, the ANAO reviewed Defence’s use of the MRM for the 2020–21 MPR. As the replacement for the MRS, the MRM should provide information that can be used to support disclosures in Defence’s PDSSs.

1.47 During the preparation of the 2020–21 MPR, Defence advised the ANAO that the MRM could be used as a source of evidence for the PDSSs; however, as the MRM is not entirely system generated, ANAO concerns remain regarding its consistency and accuracy. The ANAO identified errors within MRM reports and therefore their reliability was not sufficient as supporting evidence for the MPR. Alternate evidence was sourced to verify the PDSSs. With regard to the errors in MRM identified by the ANAO, Defence advised the ANAO in November 2021 that:

In relation to the internal processes to assess accuracy and completeness, the process has been: data checking and reconciliation work with DFG to ensure BORIS file uploads reflect the accrual accounting position (complete Oct 20); daily automated system checks to ensure that data flows are maintained and messages are provided to users when data is not up to date; prior to each MRM lockdown period reminders on data requirements are sent to reduce human error; after each lockdown period system statistics are used to drive lessons on sign off and identify areas of improvement; and to assure that in each reporting round if the data was accurate trend information over time is used to identify anomalies and drive improvements.

1.48 These checks do not appear to be fully effective as the ANAO’s review identified anomalies in the MRM data, and alternate evidence had to be obtained. The ANAO will continue to monitor the completeness and accuracy of data in MRM.

Results of the ANAO’s review

1.49 The following sections outline the results of the ANAO’s review. The results inform the overall conclusion in the Independent Assurance Report by the Auditor-General for 2020–21.

Financial framework

1.50 The project financial assurance statements were introduced in the 2011–12 MPR and have been included within the scope of the Independent Assurance Report by the Auditor-General since 2014–15. The contingency statements were introduced for the first time in the 2013–14 MPR and describe the use of contingency funding to mitigate project risks. Together, they are aimed at providing greater transparency over projects’ financial status.

1.51 A project’s total approved budget comprises:

• the allocated budget, which covers the project’s approved activities, as indicated in the MAA; and
• the contingency budget, which is set aside for the eventuality of risks occurring and includes unforeseen work that arises within the delivery of the planned scope of work.⁷⁸

1.52 In 2020–21, the ANAO reviewed the financial framework as it applied to managing project budgets and expenditure, including: project financial assurance, contingency, the reporting environment, and reporting cost variations and personnel costs.

Project financial assurance statement

1.53 The project financial assurance statement’s objective is to enhance transparency by providing readers with information on each project’s financial position (in relation to delivering project capability) and whether there is ‘sufficient remaining budget for the project to be completed’.⁷⁹ The project financial assurance statement is restricted to the current financial contractual obligations of Defence for these projects, including the result of settlement actions and the receipt of any liquidated damages, and current known risks and estimated future expenditure as at 30 June 2021.

1.54 In the 2018–19 MPR, the ANAO identified an instance where project scope worth $1.5 billion was transferred to other projects or other phases of that program without a corresponding transfer of funds out of the project budget.⁸⁰ The ANAO has not observed any such transfers of project scope in the 2020–21 financial year, however, the Light Tactical Fixed Wing project reduced its scope during 2020–21 without a corresponding Real Cost Decrease. The Light Tactical Fixed Wing project office was unable to quantify the value of this reduction in scope, but advised that it anticipated returning unspent funds at project closure. The Battlefield Command System project also expects to reduce its scope, following government consideration. As at 30 June 2021, discussions from the outcomes of a technical review and finance review were underway to determine the way forward for this project.

1.55 The Chief Finance Officer’s representation letter to the Secretary of Defence on the 2020‍–‍21 MPR’s project financial assurance statements was unqualified.

Contingency statements and contingency management

1.56 The purpose of the project contingency budget is to estimate the inherent cost, schedule and technical uncertainties of projects’ in-scope work.⁸¹ Defence policy requires that project managers ensure that all decisions in regard to a project’s contingency budget are included in the project’s contingency budget log to ensure ongoing transparency and traceability. Defence’s Project Risk Management Manual (PRMM version 2.5, p.105) requires that contingency be applied for identified risk mitigation activities.

1.57 Contingency provisions are approved by government as part of the total project budget, though are not programmed or funded in cash terms⁸² and projects are encouraged to meet contingency funding requirements from within their currently programmed cash funding. If this cannot be achieved, a project may propose to access contingency funding from the relevant capital program — the Approved Major Capital Investment Program (AMCIP), Facilities and Infrastructure Program (FIP) and ICT Capital Program. If this cannot be achieved, the contingency call will be presented to the Defence Investment Committee, which if agreed will potentially be met by budget offsets across the whole Integrated Investment Program.⁸³

1.58 Defence PDSSs are required to include a statement regarding the application of contingency funds during the year, if applicable, as well as disclosing the risks mitigated by the application of those contingency funds.

1.59 In 2020–21, two Major Projects, MRH90 Helicopters and Battle Comm. Sys. (Land) 2B, had accessed their contingency funds. The MRH90 Helicopters project committed $34.4 million of contingency to mitigate supportability and performance risks. The Battle Comm. Sys. (Land) 2B project used $30.3 million of contingency funds to address delay for the Defence Tactical Communications Network and refinement of specifications for the Headquarters on the Move capability.

1.60 The ANAO’s examination of project contingency logs as at 30 June 2021 highlighted that the clarity of the relationship between contingency allocation and identified risks continues to be an issue. Five projects (Future Frigates, Combat Recon. Vehicles, Offshore Patrol Vessel, Repl Replenishment Ships, and Pacific Patrol Boat Repl) did not explicitly align their contingency log with their risk log to ensure that the expected cost impact of risks is maintained effectively, as required by PRMM version 2.5. One project (JORN Upgrade) did not have an up to date risk log as the project is in the process of renegotiating a new approach to deliver the project with its prime contractor. Defence advised the ANAO that the JORN Upgrade risk log will be updated once the negotiations are finalised, and the associated Contract Change Proposals have been executed.

1.61 The ANAO will continue to monitor non-compliance with PRMM version 2.5 and the release of specific guidance following the implementation of the CASG Risk Management Framework, which is expected to be implemented in stages to 2022 (as discussed from paragraph 1.72).

Reporting on cost variations, project personnel numbers and costs

1.62 In May 2018, the JCPAA wrote to the Auditor-General to request that the ANAO report back to it ‘on how Defence major project cost variations and the costs of retaining project staff over time might be reported annually in future Major Projects Reports.’⁸⁴

Cost variations since Second Pass Approval

1.63 Table 8, at pp.42–45, shows all budget variations post initial Second Pass Approval for projects.

Project personnel numbers and costs

1.64 In terms of calculating the cost of retaining project staff, Defence advised the ANAO in November 2018 that its IT systems did not provide a direct mapping of personnel to projects. It noted that personnel often work on multiple projects and sustainment activities at any given time.

1.65 The ANAO observed during fieldwork in 2019 that several MPR projects had staff who worked concurrently on other projects, which included shared corporate staff. Some of these projects did not have systems in place to record accurately the proportion of time these shared staff attributed to the project. The ANAO also observed that MPR projects used different methods to record personnel data. These observations were repeated in 2020 and 2021.

1.66 In April 2020, the Defence Finance Group (DFG) indicated that it was possible to extract employee expenses (excluding contractors) from Defence’s personnel system, known as the Personnel Management Key Solution (PMKeyS). DFG advised the ANAO that it would need to work with Defence People Group to ensure that all relevant Department IDs within PMKeyS for a project have been captured, as well as ensuring that people are properly allocated to the correct Department IDs. In November 2020 Defence advised the ANAO that it continued to investigate whether PMKeyS could be used as a robust source to track employee costs by project.

1.67 In September 2021 Defence advised the ANAO that it is still not yet able to provide the staff cost component of projects and that its systems are not capable of calculating the cost of retaining project staff over time. Accordingly, Defence has not provided any data on the costs of project staff for projects in the MPR. The ANAO will continue to monitor Defence’s progress in recording project personnel numbers.

Enterprise Risk Management Framework

1.68 While major risks and issues data in the PDSSs remains excluded from the formal scope of the Auditor-General’s Independent Assurance Report⁸⁵, material inconsistencies identified in relation to this information are required to be detailed in the report. The following information is included to provide an overall perspective of how risks and issues are managed within Defence and the selected Major Projects.

1.69 Risk management has been a focus of the MPR since its inception. The CASG risk management environment consists of multiple policies and varying implementation mechanisms and documentation. There are multiple group-level (i.e. CASG), sub-group (i.e. Divisional) and project-level risk management documents. The primary focus of the ANAO’s examination of risk management is at the project level, to conduct its review of the PDSSs. At the Group level, the Deputy Secretary CASG issued a directive in May 2017 establishing a CASG Risk Management Reform Program to implement a risk management model within Defence’s risk management framework.

1.70 In June 2020 the Deputy Secretary CASG issued a directive establishing the CASG Risk Management Framework, which is the key deliverable of the CASG Risk Management Reform Program. The initiative includes:

• the framework, which is the primary policy and operating framework for the management of risk across the group; and
• the Group Risk Management Strategy 2020–22, which provides a structured pathway to implementing the remodelled approach to managing risk across the 2020–22 period.

1.71 The reform was initially planned to be concluded in June 2019. Defence concluded the contract with its industry partner in May 2020. Defence advised the ANAO in November 2020 that it had delivered all three phases of the reform, including the development of risk management policies and toolsets for use by projects. However, Risk Profiles for some CASG Domains remain in draft, and Risk Management Implementation Plans are still being updated. Further, as detailed in Table 7, on p.34, not all projects are using the mandated risk management software tool, Predict!.⁸⁶

1.72 Defence advised the ANAO in October 2021 that it had released tools to standardise risk practices across CASG, and that this includes the rollout of Predict! across CASG. Defence anticipates the rollout of these tools and risk practices to be completed by February 2022, which will conclude the third and final phase of the CASG Risk Reform as initially planned in the CASG Deputy Secretary’s Directive of 2017.

1.73 The ANAO will continue to monitor implementation of the Risk Management Framework, however, it will not be able to consider including risks and issues in the scope of the MPR until implementation of the framework is more mature.

1.74 In 2020–21, the ANAO again examined project offices’ risk and issue logs at the Group and Service level, which are predominantly created and maintained utilising spreadsheets and/or Predict! software. The key issues with risk management, as observed by the ANAO, related to:

• variable compliance with corporate guidance. While most of the 21 MPR projects had an approved Risk Management Plan, only the SRGB Air Defence and Collins Comms and EW projects have updated their risk management plan within six months as required by PRMM version 2.5⁸⁷;
• the visibility of risks and issues when a project is transitioning to sustainment;
• for one project (Joint Strike Fighter), acquisition and sustainment risks are managed together⁸⁸;
• the frequency with which risk and issue logs are reviewed to ensure risks and issues are accurate and complete, appropriately managed in a timely manner, and accurately reported to senior management;
• lack of quality control resulting in inconsistent approaches in the recording of issues within Predict!;
• lack of a clear link between allocations against risk in the contingency log and risk logs; and
• risk management logs and supporting documentation of variable quality, particularly where spreadsheets are being used.

1.75 The ANAO has previously observed that Defence’s use of spreadsheets as a primary form of record for risk management is a high-risk approach. Spreadsheets lack formalised change/version control and reporting, thereby increasing the risk of error. This can make spreadsheets unreliable corporate data handling tools as accidental or deliberate changes can be made to formulae and data, without there being a record of when, by whom, and what change was made. A significant amount of quality assurance is necessary to obtain confidence that spreadsheets are complete and accurate at 30 June, which is not an efficient approach to the review.

1.76 The JCPAA recommended in September 2018 that Defence plan and report a methodology to the JCPAA showing how acquisition projects can transition from the use of spreadsheet risk registers to tools with better version control.⁸⁹ In response, Defence advised the JCPAA in May 2020 that Predict! will be mandated as the risk management system.⁹⁰ Defence advised the ANAO in October 2021 that Defence policy was updated in August 2021 to mandate that risks must be documented in Predict!.

1.77 In October 2021 Defence further advised the ANAO that as at 30 June, 14 of the 21 MPR projects⁹¹ have a presence in Predict!, but are not necessarily using it to manage risks and issues. The ANAO’s review of documentation relating to CASG’s 21 project offices indicates that as at 30 June 2021:

• fourteen utilised Predict!;
• six utilised spreadsheets as their primary risk management tool;
• two (Joint Strike Fighter and CMATS) utilised a bespoke SharePoint based tool; and
• one (Future Frigates) used Predict! and Defence’s CapabilityOne.

1.78 Table 7 below lists the MPR projects’ use of the Predict! Risk Management System tool.

Table 7: MPR projects’ use of Predict! Risk Management System as at 30 June 2021

Note 1: Light Tactical Fixed Wing was previously called Battlefield Airlifter.

Source: ANAO

1.79 Defence advised the ANAO that CASG has an approved rollout plan to transition all CASG projects to Predict! by February 2022.

1.80 Implementation of Predict! is expected to improve the efficiency of Defence’s risk management, standardise reporting, and improve the reliability of supporting evidence for the annual MPR.

Lessons learned

1.81 In May 2020, CASG released a revised version of its Lessons Program Policy. The Policy is underpinned by a Defence Joint Directive which directs all ‘Groups and Services, as required, to establish and lead a whole-of-Defence Joint Lessons that provides centralised Lessons management and coordination’.

1.82 Version 2.0 of the CASG Lessons Program Policy states that the:

Deputy Secretary CASG expects the CASG leadership to share best practices and lessons. To facilitate this the CASG Lessons Program continually analyses lessons collected from programs, projects, products and governance review outcomes; and ensures they are readily available via the Defence Lessons Repository (DLR).⁹²

1.83 Defence is yet to fully implement a lessons learned framework and compliance monitoring process. The ANAO has observed that not all project lessons are available within the Defence Lessons Repository. Full implementation is expected to enable projects to review and apply lessons learned that are applicable to enable more consistent and improved project outcomes. The ANAO will continue to monitor Defence’s progress in implementing the lessons learned process for projects’ use and provide an update in the 2021–22 MPR.

Caveats and deficiencies

1.84 Defence has not defined, in its internal policies and procedures, the terms ‘caveat’ or ‘deficiency’ relating to the declaration of significant capability milestones. In November 2021, Defence advised the ANAO that caveats or deficiencies are used where a milestone (IMR, IOC, FMR, or FOC) has been achieved in principle, with outstanding actions to be rectified or mitigated.

1.85 The ANAO first observed the declaration of a major milestone with caveats in 2013–14, and Defence has continued to declare major milestones with caveats since then. In the 2017–18 MPR the ANAO noted advice from Defence that it discourages IARs recommending caveats at FOC.⁹³

1.86 In 2020–21, Defence declared the following caveats or deficiencies relating to projects in the MPR:

• Joint Strike Fighter — Defence declared IOC on 28 December 2020, acknowledging a number of known acceptable deficiencies with the aircraft and support systems, including some delays to weapons delivery and integration;
• Combat Recon Vehicles — Defence declared IMR in June 2021 with three exceptions relating to the integration of electronic counter measures, the completion of Functional Configuration Audit and Physical Configuration Audit, and transportability studies including air transportability and integration with other Army vehicles; and
• ANZAC Air Search Radar Repl — Defence declared IMR 1 in December 2020 with three caveats, and IMR 2 in April 2021 with four caveats. Of those, three caveats had been resolved by 30 June 2021.

2.1 Performance information is important in the management and delivery of major defence equipment acquisition projects (Major Projects). It informs decisions about the allocation of resources, supports advice to government, and enables stakeholders to assess project progress.

2.2 Project performance has been the subject of many of the reviews of the Department of Defence (Defence), and a consistent area of focus of the Parliament’s Joint Committee of Public Accounts and Audit (JCPAA) since the first Major Projects Report (MPR). This chapter progresses previous Australian National Audit Office (ANAO) analysis of project performance.

Project performance analysis by the ANAO

2.3 The major dimensions of project performance are:

• cost performance (discussed at pp.37–48) — this includes the percentage of budget expended (Budget Expended), changes in budget since Second Pass Approval, in-year changes to budget, and in-year expenditure;
• schedule performance (discussed at pp.49–65) — this includes the percentage of time elapsed (Time Elapsed), total schedule slippage, and in-year changes to schedule; and
• capability performance (discussed at pp.69–74) — this includes the key challenges faced by Defence in the delivery of key materiel capabilities.

2.4 This chapter provides ANAO analysis relating to the three principal dimensions of project performance noted above, drawing on the Defence Project Data Summary Sheets (PDSSs) for the 21 Major Projects. This work includes analysis of in-year information, longitudinal analysis and analysis of the results of project progress for the year-ended 30 June 2021.

2.5 Figure 1, on p.38, directly compares cost performance with schedule performance through two metrics, Budget Expended and Time Elapsed.⁹⁴

Figure 1: Budget Expended and Time Elapsed at 30 June 2021

 

 

Note 1: At 30 June 2021, Future Frigates and Future Subs did not have Final Operational Capability (FOC) milestones approved by government.

Note 2: JORN Upgrade FOC forecast is disclosed in its PDSS as ‘TBA’.

Source: ANAO analysis of the 2020–21 PDSSs.

2.6 Figure 1 shows that for most projects (17 of 18⁹⁵), Budget Expended was broadly in line with (within 10 per cent), or lagging, Time Elapsed. This relationship is generally expected in an acquisition environment predominantly based on milestone payments. However, due to the varying complexity, stages and acquisition approaches across the portfolio of projects, further analysis of these simple performance measures is required to provide a better understanding of key variances.

2.7 Where Budget Expended is significantly lagging Time Elapsed, the project schedule may be at risk — i.e. expenditure lags may indicate delays in milestone achievement. In 2020–21, the Budget Expended for two projects lagged Time Elapsed by at least 20 per cent. These were Light Tactical Fixed Wing and Joint Strike Fighter.

2.8 For Light Tactical Fixed Wing (Budget Expended 66 per cent, Time Elapsed 90 per cent), the expenditure lag is associated with project deliverables not yet on contract, a reduction in project scope without a corresponding reduction in budget, and the planned delivery of some requirements after Final Operational Capability (FOC). In 2020 the government agreed to an operational capability ‘pivot’ from Battlefield Airlifter to Light Tactical Fixed Wing, and to re-scoping and rescheduling activities resulting in an updated Materiel Acquisition Agreement (MAA). The revised FOC is anticipated to be achieved in 2021–22.

2.9 For Joint Strike Fighter (Budget Expended 58 per cent, Time Elapsed 82 per cent), the expenditure lag continues to reflect the transition from the aircraft development stage to the production stage, where relatively little budget was expended. In-year expenditure is increasing compared to prior years, with the gap between Budget Expended and Time Elapsed being 12 percentage points smaller than in 2019–20.

2.10 Where Budget Expended leads Time Elapsed, the project budget may be at risk — i.e. expenditure increases may indicate real cost increases. However, for the one project (ANZAC Air Search Radar Repl) where Budget Expended leads Time Elapsed by 10 per cent or more, the cause of the variance does not relate to insufficient project funds.

2.11 ANZAC Air Search Radar Repl (Budget Expended 75 per cent, Time Elapsed 57 per cent) spent approximately nine per cent of its budget prior to Second Pass government approval, to conduct a Risk Reduction Program and make early purchases of equipment to ensure the schedule would be met.⁹⁶

2.12 In each case of significant variance between Budget Expended and Time Elapsed, the performance information highlights projects that may require further attention. This is to ensure that unspent funds are returned to the Defence budget for re-allocation in a timely manner, the timing of key deliverables remains in focus, or planning focuses on bringing together all elements in a timely manner, as equipment is delivered.

Cost performance analysis

Approved budget at initial Second Pass Approval and at 30 June 2021

2.13 Figure 2, on pg.40, compares each project’s approved budget at initial Second Pass Approval and its approved budget at 30 June 2021. Four projects had variations of $500 million or more, with the following components:

• Joint Strike Fighter — net increase of $12.9 billion, comprising $10.5 billion for 58 additional aircraft in 2013–14, $2.0 billion for exchange rate variation and $0.4 billion for price indexation;
• MRH90 Helicopters — net increase of $2.8 billion, comprising $2.6 billion for 34 additional aircraft in 2005–06 and other minor scope changes, and $0.7 billion for price indexation, offset by a $0.3 billion decrease due to scope transfers for facilities, and a $0.1 billion decrease for exchange rate variation;
• Overlander Medium/Heavy — net increase of $0.8 billion, comprising $0.7 billion ‘project supplementation’ to reduce cost pressures and $0.1 billion exchange rate variation; and
• MQ-4C Triton — net increase of $1.0 billion, comprising $0.3 billion for an extra air vehicle in 2019–20, $0.8 billion for an additional air vehicle and interim support services for the first seven years in 2020–21, offset by a $0.1 billion decrease in exchange rate variation.

Figure 2: Approved project budgets at initial Second Pass Approval and at 30 June 2021 ($ million)

 

 

Note 1: 

 indicates that the budget for the project at 30 June 2021 is less than the original budgeted cost.

 

Source: ANAO analysis of the 2020–21 PDSSs. Previous MPRs have reported that budget variances since initial Second Pass Approval have resulted from: increasing the scope of a project via revised Second Pass Approvals, programmatic decisions, Real Cost Increases/Decreases, transfers to/from other projects, and budgetary adjustments. Project budgets may also be affected by price indexation⁹⁷ and foreign exchange variation.

2.14 The total budget for the 21 MPR projects at 30 June 2021 was $58.0 billion, a net increase of $18.3 billion when compared to the approved budget at initial Second Pass Approval of $39.8 billion. A summary of budget variations is at Table 2, on p.10, and a more detailed analysis of these budget variations is included in Table 8, pp.42–45.

Budget performance

2.15 The following figures and tables illustrate the budget performance of the 21 selected projects by way of:

• in-year budget variations by project (see Table 9, on p.47); and
• expenditure forecasting performance against actual expenditure for 2020–21 (see Figure 3 on p.49).

In-year budget variance analysis

2.16 Table 9, on p.47, sets out the in-year budget variations for each project. Overall, the approved budget for the selected projects as at 30 June 2021 decreased by $985.2 million (a 1.7 per cent decrease) compared to their approved budget as at 30 June 2020. This was driven by exchange rate variation decreases of $1808.5 million and net real increases of $822.9 million.

2.17 Exchange rate variations result from a project’s exposure to foreign currencies and movements in exchange rates against the Australian dollar.⁹⁸ Budget adjustments aim to maintain the relative buying power of the project budget. Movements in the United States dollar and the Euro were the main influences. Projects with larger movements in foreign exchange in 2020–21 included:

• Joint Strike Fighter — movement of minus $1000.6 million, or 0.6 per cent decrease in budget.
• Future Frigates — movement of minus $245.0 million, or 3.9 per cent decrease in budget.
• MQ-4C Triton — movement of minus $182.0 million, or 1.0 per cent decrease in budget.
• Combat Recon. Vehicles — movement of minus $106.3 million, or 1.8 per cent decrease in budget.

2.18 Real Variations⁹⁹ primarily reflect changes in the scope of projects, transfers between projects for approved equipment/capability and budgetary adjustments such as administrative savings decisions. In 2020–21 there was one project with a significant Real Variation:

• MQ-4C Triton — variation of $823.9 million reflecting budget transfers for Second Pass Approval for Tranche 3 (providing for one additional aircraft) and Tranche 4 (providing sustainment funding for the first 7 years).

Table 9: In-year (2020–21) budget variations by project

Note 1: The Total Variance and components for this project do not add up due to rounding differences.

Note 2: Previously referred to as Battlefield Airlifter.

Note 3: The difference between the total approved budgets for 2019–20 and 2020–21 is due to the projects entering the MPR in 2020–21 (SRGB Air Defence and JORN Upgrade) not contributing to the total budget figure for 2019–20).

Source: ANAO analysis of the 2019–20 and 2020–21 PDSSs.

In-year forecast and actual expenditure

2.19 Accurately forecasting and managing budget expenditure is an important element in the management of a portfolio of projects. Figure 3, on p.49, sets out the expenditure forecasting performance of each project against actual expenditure in 2020–21. In total, actual expenditure for the 21 Major Projects at 30 June 2021 was $6120.4 million. This is compared against an initial Portfolio Budget Statements (PBS) forecast expenditure of $6853.0 million, a mid-year Portfolio Additional Estimates Statements (PAES) forecast of $6396.0 million, and a final forecast of $6218.5 million (Final Plan, approved as at June 2021).

2.20 The Defence PDSSs report that the variances illustrated in Figure 3 reflect the following developments:

• Joint Strike Fighter (expenditure of $2565.9 million compared to $2430.6 million PBS, $2354.4 million PAES and $2252.9 million Final Plan estimates) — the overspend is attributed to early achievement of cooperative program deliverables associated with the air vehicle and engine.
• Future Frigates (expenditure of $508.5 million compared to $587.0 million PBS, $506.9 million PAES and $498.4 million Final Plan estimates) — the underspend is reported as reflecting reprogramming of activities against the Head Contract.
• Future Subs (expenditure of $630.8 million compared to $782.5 million PBS, $782.2 million PAES and $768.3 million Final Plan estimates) — the underspend is predominantly attributed to not entering the next contracted work scope as initially forecast with Naval Group, and Lockheed Martin Australia not achieving the Commonwealth’s expected labour levels. There was also lower than anticipated expenditure against other contractor support.
• Combat Recon. Vehicles (expenditure of $414.6 million compared to $566.1 million PBS, $501.4 million PAES and $488.7 million Final Plan estimates) — the underspend is reported as reflecting later than expected achievement of milestones due to technical difficulties and delays in the global Boxer program, some of which are a result of COVID-19 impacts on the supply chain and travel restrictions.
• Offshore Patrol Vessel (expenditure of $204.6 million compared to $285.1 million PBS, $249.2 million PAES and $248.9 million Final Plan estimates) — the underspend is reported as reflecting schedule delays as a result of COVID-19 in achieving contractual milestones and underspend on Project Office costs and government furnished equipment.
• Repl Replenishment Ships (expenditure of $150.4 million compared to $231.3 million PBS, $254.5 million PAES and $252.1 million Final Plan estimates) — the underspend is reported as reflecting the transfer of additional works from Spain to Australia and contract change proposal delays for final sparing deliveries.
• Battlefield Command System (expenditure of $67.5 million compared to $216.5 million PBS, $118.5 million PAES and $116.6 million Final Plan estimates) — the underspend is reported as reflecting the finalisation of a number of contract change proposals which updated the payment and delivery schedules for Battle Management System (BMS) and Tactical Communications Network (TCH) prime contracts and the inability to achieve the exit criteria for the Release 1.1 Software Release Review milestone.

Figure 3: In-year (2020–21) projects’ forecast expenditure performance compared to actual expenditure ($m)

 

 

Sources: ANAO analysis of the 2020–21 PDSSs and Defence Portfolio Budget Statements.

Schedule performance analysis

2.21 Defence data continues to show that schedule performance is a key issue in delivering and sustaining Defence equipment and capability.¹⁰⁰ Project schedule slippage can have the effect of introducing or exacerbating a capability gap or requiring an extension to the planned withdrawal date for those platforms being replaced.¹⁰¹

Schedule slippage and acquisition type by approval date

2.22 The ANAO compares project slippage against the classification of projects as Military Off-The-Shelf (MOTS), Australianised MOTS, or developmental, as these classifications are a general indicator of the difficulty associated with the procurement process.¹⁰² Prima facie, the more developmental in nature a project is, the greater the schedule risk and therefore the greater the need for more robust planning by Defence.^103,104

2.23 In 2020–21 the acquisition type, or classification, is no longer reported in the PDSS Header for the selected Major Projects. Project complexity is instead indicated by the Acquisition Categorisation (ACAT) level. Projects are graded into one of four (ACAT) acquisition categories¹⁰⁵:

• ACAT I — major capital equipment acquisitions that are normally the Australian Defence Force’s (ADF) most strategically significant. They are characterised by extensive project and schedule management complexity and very high levels of technical difficulty, operating, support and commercial arrangements.
• ACAT II — major capital equipment acquisitions that are strategically significant. They are characterised by significant project and schedule management and high levels of technical difficulty, operating, support arrangements and commercial arrangements.
• ACAT III — major or minor capital equipment acquisitions that have a moderate strategic significance to the ADF. They are characterised by the application of traditional project and schedule management techniques and moderate levels of technical difficulty, operating, support arrangements and commercial arrangements.
• ACAT IV — major or minor capital equipment acquisitions that have a lower level of strategic significance to the ADF. They are characterised by traditional project and schedule management requirements and lower levels of technical difficulty, operating, support and commercial arrangements.

2.24 This year, the ANAO has continued its longitudinal analysis based on acquisition type (MOTS, Australianised MOTS and developmental) to provide a transition to analysis based on ACAT levels.

ANAO analysis based on project acquisition type and acquisition category level

2.25 Table 10, below, provides information on the acquisition type and ACAT level of all 56 Major Projects included in the MPR since its inception, and the year of approval (generally Second Pass) for each Major Project.

2.26 In relation to the acquisition type for the 56 Major Projects, Table 10 indicates that:

• 14 projects (25 per cent) were developmental;
• 26 projects (46 per cent) were Australianised MOTS; and
• 16 projects (29 per cent) were MOTS.

2.27 In relation to the acquisition category for the 56 Major Projects, Table 10 indicates that:

• 14 projects (25 per cent) were ACAT I;
• 29 projects (52 per cent) were ACAT II;
• 12 projects (21 per cent) were ACAT III; and
• 1 project (2 per cent) was ACAT IV.

Table 10: Project year of approval, acquisition type and acquisition category

2.28 Table 10 (above) and Figure 4 (p.54) indicate that developmental projects became less common after 2005. Based on the findings of the Defence Procurement Review 2003 (Kinnaird Review)¹⁰⁶, in 2005 Defence began to increase the proportion of MOTS acquisitions, which are generally lower risk projects and therefore more likely to meet schedule timelines.¹⁰⁷ Table 10 and Figure 4 also indicate a continuing trend, where developmental projects have become more common since 2014. Of the 16 Major Projects which have received government approval since 2014:

• 7 projects (44 per cent) were developmental;
• 6 projects (38 per cent) were Australianised MOTS; and
• 3 projects (19 per cent) were MOTS.

2.29 Figure 5 (p.55) illustrates the proportion of ACAT I to IV projects over time. In common with Figure 4, Figure 5 indicates a continuing trend towards the approval of more complex projects at the ACAT I and II levels. Of the 16 Major Projects which have received government approval since 2014:

• 6 projects (38 per cent) were ACAT I;
• 9 projects (56 per cent) were ACAT II;
• 1 project (6 per cent) was ACAT III; and
• no projects were ACAT IV.

Schedule slippage by acquisition type and acquisition category

2.30 The challenge of gaining a full understanding of the complexities of developmental aspects of projects at Second Pass Approval is evident by the extent of slippage over time. Figure 6, on p.58, illustrates the total schedule slippage¹⁰⁸ since Second Pass Approval for 19 of the selected Major Projects.¹⁰⁹ It also depicts the acquisition type and places projects in order of government approval.

2.31 Figures 7a and 7b, on pp.59–60, illustrate the total schedule slippage for the 33 projects that have exited the MPR.¹¹⁰ Twenty post-Kinnaird projects (Figure 7a) and 13 pre-Kinnaird projects (Figure 7b) have exited the MPR. Total slippage of the 20 post-Kinnaird projects is 37.0 years. Total slippage of the 13 pre-Kinnaird projects is 79.6 years. One of the 20 post-Kinnaird projects was a developmental acquisition and four of the 13 pre-Kinnaird acquisitions were developmental.

2.32 In relation to the 20 exited post-Kinnaird MPR projects (see Figure 9a, on p.62):

• two were ACAT I with an average slippage of 37 months;
• eleven were ACAT II with an average slippage of 15 months; and
• seven were ACAT III with an average slippage of 12 months.

2.33 Of the 13 exited pre-Kinnaird MPR projects (see Figure 9b, p.63):

• one was ACAT I with slippage of 77 months¹¹¹;
• six were ACAT II with an average slippage of 89 months;
• five were ACAT III with an average slippage of 47 months; and
• one was ACAT IV with slippage of 107 months.¹¹²

2.34 Figures 7a and 7b indicate that the inclusion of MOTS acquisitions contributed, prima facie, to a reduction in schedule slippage in the Major Projects portfolio. For projects that have exited the MPR, MOTS projects reported an average of 13 months of slippage per project, Australianised MOTS projects reported an average of 38 months, and developmental projects reported an average of 96 months. Decisions on whether to undertake developmental projects should be considered on a risk basis.¹¹³ In this context, the consideration of risk should be holistic and weigh up the level of capability to be acquired while having regard to Defence’s past experience in managing the delivery of developmental projects.

2.35 Figures 6, 7a and 7b also illustrate that older projects have experienced the most slippage. These projects tended to be more developmental (complex) in nature and typically experienced schedule slippage in the past and have often continued to do so. This demonstrates an ongoing trend of slippage in historically late projects, which is more pronounced in older projects. This trend is also visible, but less prominent, in newer projects.

2.36 Figure 8, on p.61, indicates that for current projects graded at the ACAT I level, there was an average of 43 months slippage. Projects at the ACAT II level experienced an average of 21 months slippage. For projects that had exited the MPR (see Figures 9a and 9b, on p.62–63):

• ACAT I projects reported an average of 57 months slippage;
• ACAT II projects reported an average of 42 months slippage;
• ACAT III projects reported an average of 28 months slippage; and
• the ACAT IV project reported 107 months slippage.¹¹⁴

2.37 Three developmental projects, Hawkei, JORN Upgrade, and ANZAC Air Search Radar Repl, are yet to experience slippage to their FOC dates. However, these projects have experienced slippage to design reviews, test programs, or material release milestones.

2.38 Hawkei has experienced 24 months slippage to the Production Reliability Acceptance Test, leading to 17 months slippage to Initial Materiel Release (IMR) — which was declared in May 2020 with four caveats, which have now been resolved.¹¹⁵ Hawkei experienced an additional six-month delay pending resolution of a vehicle safety incident and a further five month delay due to vehicle integration issues.

2.39 JORN Upgrade experienced persistent lag in execution of the systems engineering program, leading to 17 months of slippage to the System Definition Review and at least two years of slippage to the Preliminary Design Review. FOC is expected to be affected; the amount of slippage to this milestone is not yet known, but the delay is anticipated to be several years. Key drivers for the delays are predominantly attributed to the underestimation of JORN systems engineering complexity and required design effort.

2.40 ANZAC Air Search Radar Repl experienced a total of 18 months of slippage to the original definition of IMR due to delays in receiving Identification Friend or Foe certification, which was impacted by COVID-19 travel restrictions. Early project milestones have also been delayed by manufacturing delays, delays in the contractor obtaining Environmental Qualification for equipment, limited numbers of test facilities and longer than anticipated test durations.

2.41 In contrast, a recent MOTS project, Pacific Patrol Boat Repl, has adhered more closely to the design and materiel release schedule with only minor variances. This indicates that although developmental projects currently in the MPR are not reporting significantly more slippage to FOC than MOTS projects, developmental projects still carry a higher level of technical risk.

2.42 Four¹¹⁶ of the eight current ACAT I projects and six¹¹⁷ of the 11 current ACAT II projects have experienced slippage to their FOC dates. There are no ACAT III or IV projects in the 2020–21 MPR.

2.43 The average slippage for ACAT I projects in this year’s MPR is 42 months, and the average for ACAT II projects is 21 months. This indicates that ACAT I projects currently in the MPR are reporting more slippage to FOC than ACAT II projects and carry a higher level of technical risk.

Schedule performance

2.44 The figures and tables that follow illustrate, for the selected Major Projects:

• the original and 30 June 2021 forecasts for achieving FOC;
• in-year schedule changes to achieving FOC; and
• total schedule slippage across the Major Projects.

Original and 30 June 2021 Final Operational Capability forecasts

2.45 Figure 10, on p.65, presents information on the selected Major Projects’ original and 30 June 2021 forecasts for achieving FOC. Total schedule slippage for the 21 Major Projects as at 30 June 2021 was 405 months compared to the initial prediction when approved by government.¹¹⁸ This represents a 23 per cent increase on the approved schedule.

2.46 Of the 21 projects in the 2020–21 report, ten have experienced schedule slippage (not including JORN Upgrade, which discloses its FOC forecast as ‘TBA’ and expects this milestone to slip by a currently unknown magnitude, primarily due to the underestimation of JORN systems engineering complexity and required design effort) and two did not have FOC dates approved by government at 30 June 2021.

2.47 Total schedule slippage across the Major Projects was 405 months in 2020–21. This is 102 months lower than the figure of 507 months reported in the 2019–20 MPR. The difference is mainly due to the exit of projects with significant slippage — including AWD ships, P-8A Poseidon, Growler and Collins R&S — which reduced the total accumulated slippage by 175 months. This was offset by in-year slippage for MRH90 Helicopters (associated with remediation of technical difficulties and reliability issues), MQ-4C Triton (associated with changes in the planned aircraft delivery schedule), Light Tactical Fixed Wing (with Defence revalidating the business case for the delivery of this project’s remaining scope), CMATS (associated with the replanning of deliverables), and Battle Comm. Sys. (Land) 2B (as a result of COVID-19 impacts). These five projects, combined, added 73 months to schedule slippage in 2020–21.

2.48 The reasons for schedule slippage often require a deep understanding of project technical elements and a realistic assessment of the capacity of the private sector to deliver in the expected timeframe. A project office’s ability to gain access to the platform for upgrade activities can also result in schedule delay (for example, Maritime Comms).¹¹⁹

2.49 A closer examination of the reasons for schedule slippage demonstrates the importance of initial assessments of project complexity. Experience indicates that a key factor is whether a project is MOTS, Australianised MOTS or developmental.¹²⁰ One project, MRH90 Helicopters¹²¹, was originally misclassified as MOTS. The project was reclassified by Defence to Australianised MOTS (i.e. more developmental) subsequent to Second Pass approval, and a Defence Independent Assurance Review of this project in December 2020 noted that ‘[MRH-90] was a developmental platform’. This project has continued to experience schedule slippage, with an additional six months slippage in 2020–21. Another project, UHF SATCOM, is classified as MOTS but includes a significant software development component. Delays in software development have led to 42 months of slippage to the FOC milestone, however there were no additional delays in 2020–21.

2.50 Figure 10, below, indicates that no projects are currently forecasting an FOC date earlier than originally approved. However, a number of projects have experienced both schedule recovery and delay that has offset that recovery. Projects fitting this pattern are Joint Strike Fighter, Overlander Medium/Heavy, Repl Replenishment Ships, and Collins Comms and EW. In the case of Joint Strike Fighter, Overlander Medium/Heavy, and Repl Replenishment Ships, their schedule recovery and schedule delay is equal such that all three projects are currently forecasting the achievement of FOC as originally approved by government. In total, these four projects have contributed 28 months of schedule recovery to the selected Major Projects.

Figure 10: Projects’ original and 30 June 2021 Final Operational Capability forecasts

 

 

Note 1: Future Frigates and Future Subs did not have government approved FOC dates at 30 June 2021. The JORN Upgrade and CMATS FOC forecasts are disclosed in their PDSSs as ‘TBA’.

Source: ANAO analysis of the 2020–21 PDSSs.

In-year schedule performance

2.51 In 2020–21, there was schedule slippage of 73 months in the forecast achievement of FOC across 19 of the Major Projects¹²², as shown in Figure 11, on p.67. In-year project performance, measured by slippage over 12 months, may not reflect the project trend.

2.52 In-year schedule slippage occurred for the following five projects¹²³:

• MRH90 Helicopters — the variance reflects ongoing delays in the delivery of capabilities resulting from technical deficiencies and issues resulting from replacement or re-design.
• MQ-4C Triton — the variance reflects the alignment of the project’s FOC schedule with the aircraft production schedule.
• Light Tactical Fixed Wing — FOC has been delayed while Defence re-evaluates the business case for delivery of this project’s remaining capability.
• CMATS — FOC is disclosed in the PDSS as ‘TBA’ while Thales replans deliverables and the flow on impacts are assessed. The PDSS reported 13 months slippage.
• Battle Comm. Sys. (Land) 2B — FOC has been delayed due to an extension to the project schedule as a result of COVID-19 delays.

Figure 11: In-year (2020–21) schedule changes to achieving Final Operational Capability

 

 

Note: Defence’s PDSSs indicate that 14 of the 19 Major Projects did not record changes to their FOC dates this year. Future Frigates and Future Subs did not have FOC dates approved by government at 30 June 2021. The JORN Upgrade and CMATS projects’ FOC forecasts are disclosed in the PDSS as ‘TBA’. The delays for the JORN Upgrade project are anticipated to be several years and CMATS has disclosed 13 months slippage in 2020‍–‍21.

Source: ANAO analysis of the 2020–21 PDSSs.

2.53 Project delays may indicate unanticipated problems with project progress or optimism in previous forecasting, regardless of whether the delay makes the project later than originally approved by government. All delays should be monitored to ensure that a project remains on track and any issues can be managed.

Schedule performance by year of entry to Major Projects Report

2.54 Table 11, on p.68, shows the accumulated schedule slippage of the Major Projects included in the 2020–21 MPR.¹²⁴ The table shows that over a third of the total schedule slippage across the selected Major Projects (33.8 years or 405 months) comprises slippage from the two oldest projects, approved prior to 2010.

Table 11: Project slippage as at 30 June 2021

Note 1: Future Frigates and Future Subs did not have FOC dates approved by government at 30 June 2021. The JORN Upgrade and CMATS projects FOC forecast is disclosed in the PDSS as ‘TBA’. The delays for the JORN Upgrade project are anticipated to be several years. CMATS has disclosed 13 months delay in 2020–21.

Source: ANAO analysis of the 2020–21 PDSSs.

Capability performance analysis

2.56 Defence defines capability as the power to achieve a desired operational effect in a nominated environment, within a specified time, and to sustain that effect for a designated period.¹²⁵ An operational effect is achieved by combining the nine Fundamental Inputs to Capability — organisation, command and management, personnel, collective training, major systems, facilities and training areas, supplies, support, and industry — and undertaking designated operations.¹²⁶

2.57 In acquiring Defence platforms and systems, a range of documentation (including capability definition, operational concept, function and performance specification, and Test and Evaluation Master Plans) is developed, which establishes the detailed requirements/performance attributes to be achieved.

2.58 The Defence PDSSs report that 13 projects in this year’s MPR will deliver all their key capability requirements without elevated levels of risk to the achievement of requirements.¹²⁷ Defence’s assessment indicates that some elements of the capability required may be ‘under threat’, but the risk is assessed as ‘manageable’. The six project offices experiencing challenges with expected capability delivery (2019–20: five) were Joint Strike Fighter, MRH90 Helicopters, Overlander Medium/Heavy, Light Tactical Fixed Wing¹²⁸, Battlefield Command System, and Battle Comms. Sys. Land (2B). Two of these projects, Light Tactical Fixed Wing and Battlefield Command System, report that they are unable to deliver all the required capability by FOC.

2.59 Table 12, below, summarises the issues reported by Defence in its PDSSs as impacting the achievement of the expected capabilities.

Table 12: Issues impacting expected material capability delivery performance in 2020‍–‍21

Note 1: Amber indicates that the capability is under threat but considered manageable for that portion of the pie chart.

Note 2: Red indicates that the capability is unlikely to be met for that portion in the pie chart.

Source: Defence Project Data Summary Sheets (PDSS).

Capability reporting

2.60 Since the 2009–10 MPR, capability reporting¹²⁹ has been based on Defence’s prediction of the final capability that would be achieved on the basis of deliverables and/or activities completed. This assessment of capability performance (Expected Capability) is measured against the Materiel Release Milestones (MRMs) and Completion Criteria specified in each project’s Materiel Acquisition Agreement (MAA). This is distinct from an assessment of whether milestones will be achieved on schedule. As the ANAO has previously noted, this data involves making certain assumptions in forecasting achievements and is therefore subjective in approach.¹³⁰

2.61 For example, the Light Tactical Fixed Wing project reported a 100 per cent Green capability prediction at its inclusion in the MPR in 2013–14. However, the 2013–14 PDSS also reported major risks relating to capability deficiencies arising from the United States Government divesting from the program, with Australia no longer able to rely on United States Air Force processes. These risks have continued to affect the project, with a mature training system and a number of baseline capability requirements not expected to be delivered until after FOC. These capability issues were reported in Section 4.1 of the PDSS (Materiel Capability Delivery Performance pie chart) for the first time in 2018–19, indicating that the earlier level of confidence in the project’s ability to achieve the required capability may have been overly optimistic. In 2020 the Australian Government approved an operational capability ‘pivot’ for this project, from ‘Battlefield Airlifter’ to ‘Light Tactical Fixed Wing’. This involved re-scoping and re-scheduling activities, resulting in an updated Materiel Acquisition Agreement (MAA). The revised FOC is anticipated to be achieved in 2021–22. Notwithstanding the ‘pivot’ relating to this project, which represents a substantive change in capability delivery, the Materiel Capability Delivery Performance pie chart in this year’s PDSS includes a slim red wedge representing a three per cent reduction in capability delivery.

2.62 Defence does not have a standard methodology for the assessment of capability delivery performance. A combination of methods is used, including an assessment based on the proportion of overall cost for each milestone, or the percentage the milestone represents with respect to the overall capability. Defence’s approach to assessment does not include weighting of the elements of capability, which affects the effectiveness of its reporting.

2.63 Over time, the JCPAA has sought the use of a more robust measure of capability performance.¹³¹ In October 2017, the JCPAA recommended that Defence ‘review the procedure for the development of expected capability estimates for future Major Projects Reports. The outcomes of this review should be provided to the Committee within six months of the tabling of this report. Further, the Committee requests that Defence provide a progress report within three months of the tabling of this report.’¹³²

2.64 Defence made a submission to the JCPAA in March 2018 regarding the JCPAA recommendation, which advised that:

Defence will conduct a schedule baseline validation activity for the Major Projects Report projects to drive greater consistency in schedule reporting.

Once this activity is complete, Defence should be in a better position to investigate a more robust approach to measuring Capability estimates. Utilising the validated baseline data could inform:

• A simple percentage of schedule milestones achieved to measure progress to date. This is a quantitative assessment that relies on the maintenance of a robust project baseline, which is not dissimilar to the approach proposed by ANAO previously;
• CASG working with Force Design to identify how to measure capability, that considers all elements of Fundamental Inputs to Capability, and that is suitable for unclassified publication; and
• Defence is working towards a new whole of organisational reporting system (the Enterprise Resource Planning (ERP) System) which is expected to roll-out in Financial Year 2020–21. CASG will endeavour to incorporate the work conducted with Force Design on measuring capability.¹³³

2.65 In September 2018, the JCPAA noted that ‘Materiel Capability Delivery Performance charts continue to be ambiguous in displaying actual current capability levels.’¹³⁴

2.66 Defence advised the ANAO in November 2018 that partial progress had been made on its schedule baseline validation activity discussed in paragraph 2.63. The ANAO notes that a measurement of schedule milestones will not necessarily reflect a measurement of capability delivered.

2.67 The Deputy Secretary CASG advised the JCPAA in a public hearing on 27 May 2020 that:

I acknowledge the issues of the National Audit Office and would like to work with them, as we indicated in our submission, by perhaps reviewing the report and the way in which we articulate the information.¹³⁵

2.68 As at November 2021, Defence was yet to update the method of capability forecasting in the MPR.

Transfers of project scope

2.69 As part of Second Pass Approval, government directs Defence projects to deliver certain defined capabilities within the scope of the project. During a project, Defence may change the scope to be delivered, which can be approved through a revised government approval. A project’s scope may be expanded or reduced and may include a budget increase or decrease for the project to deliver its revised requirements.

2.70 The 2020–21 MPR Guidelines require information on all scope transfers that have occurred across the current Major Projects to be reported in Section 1.3 of the PDSS. These transfers are described in Table 13, below.

2.71 A variety of transfers were also reported by Defence in Section 2.1 of some PDSSs, either as ‘Real Variation – Transfer’ or ‘Real Variation – Scope’. Explanatory notes relating to Section 2.1 indicated that project deliverables, and associated funding, had been transferred into or out of the relevant project.¹³⁶ These transfers are also described in Table 13.

Table 13: Examples of transfers of scope occurring in the Major Projects as at 30 June 2021

Note 1: The transfer for Joint Strike Fighter was reported in Auditor-General Report No.19 2019–20 2018–19 Major Projects Report, paragraphs 1.38-1.39.

Source: 2020–21 Defence PDSSs.

Part 2. Defence Major Projects Report is available to download in a separate PDF file, or in the complete report PDF which is available to download at Related documents on this page.

.

Due to the complex nature of the material, the Project Data Summary Sheets (PDSS) are available in PDF only. You can view the PDSS section in the complete report PDF which is available to download at Related documents on this page, or download individual PDSS PDF files below:

• New Air Combat Capability
• Future Frigates
• Future Submarines Design Acquisition
• Combat Reconnaissance Vehicles
• Multi-Role Helicopter
• Offshore Patrol Vessel
• Medium Heavy Capability, Field Vehicles, Modules and Trailers
• MQ-4C Triton Remotely Piloted Aircraft System
• Protected Mobility Vehicle – Light (PMV-L)
• Battlefield Airlift – Caribou Replacement
• Short Range Ground Based Air Defence
• Jindalee Operational Radar Network
• Maritime Operational Support Capability
• Civil Military Air Traffic Management System
• Battlefield Command System
• Battlespace Communications System Phase 2B
• Collins Class Communications and Electronic Warfare Improvement Program
• Pacific Patrol Boat Replacement
• Maritime Communications Modernisation
• ANZAC Air Search Radar Replacement
• Indian Ocean Region UHF SATCOM

The JCPAA 2020–21 Major Projects Report Guidelines part is available to download in a separate PDF file, or in the complete report PDF which is available to download at Related documents on this page.