Due to the complexity of material and the multiple sources of information for the 2019–20 Major Projects Report, we are unable to represent the entire document in HTML. You can download the full report in PDF or view selected sections in HTML below. PDF files for individual Project Data Summary Sheets (PDSS) are also available for download.

About the Major Projects Report

1. The Department of Defence’s (Defence) major defence equipment acquisition projects (Major Projects) continue to be the subject of parliamentary and public interest. This is due to their high cost and contribution to national security, the challenges involved in completing them within the specified budget and schedule, and to the required capability, and their contribution to industrial and employment policy objectives.

2. The Major Projects Report (MPR) comprises Defence information and commentary on selected Major Projects, and assurance and analysis of that information by the Australian National Audit Office (ANAO). The objective of the MPR is ‘to improve the accountability and transparency of Defence acquisitions for the benefit of Parliament and other stakeholders.’ ¹

3. The ANAO has reviewed information provided by Defence regarding 25 of its Major Projects in this thirteenth annual report (2018–19: 26). In February 2012, the Parliament’s Joint Committee of Public Accounts and Audit (JCPAA) identified this review as a ‘Priority Assurance Review’, under subsection 19A(5) of the Auditor‐General Act 1997 (the Act), allowing the ANAO full access to the information gathering powers under the Act.

4. Defence’s Capability Acquisition and Sustainment Group (CASG) manages the process of bringing new defence capabilities into service and providing information for ANAO review as part of the MPR process. As at 30 June 2020, CASG was managing 206 active major and minor capital equipment projects worth $130.5 billion², with an in-year budget of $8.7 billion.³ Defence capitalised some $8.8 billion from these projects in 2019–20.⁴

5. The 2020 Defence Strategic Update adjusted some of the strategic policy objectives of the 2016 Defence White Paper and foreshadowed that the Defence Budget will grow over the next ten years to $73.7 billion in 2029–30, with total funding over the decade of $575 billion (including approximately $270 billion of investment in Defence capability).^{5 6}

Major Projects selected for review

6. Major Projects are selected for inclusion in the MPR based on the criteria included in the 2019–20 Major Projects Report Guidelines (the Guidelines), as endorsed by the JCPAA.⁷ They represent a selection of the most significant Major Projects managed by CASG.

7. The total approved budget for the Major Projects included in this report is approximately $78.7 billion, covering 60 per cent of the total budget of active major and minor capital equipment projects of $130.5 billion.⁸ The selected projects are listed in Table 1.

Table 1: 2019–20 MPR projects and approved budgets at 30 June 2020 ^{1, 2}

Note 1: The 2019–20 MPR Guidelines, at Part 4 of this report, include 30 projects. Subsequent to endorsement of the 2019–20 MPR Guidelines in September 2020, five projects were removed from the MPR based on the low-risk nature of the remaining activities to FOC. Once a project is selected for review, it remains within the portfolio of projects under review until the JCPAA endorses its removal, normally once it has met the capability requirements of Defence.

Note 2: SEA 5000 Phase 1 Future Frigates, SEA 1000 Phase 1B Future Submarines Design Acquisition, LAND 400 Phase 2 Combat Reconnaissance Vehicles, AIR 7000 Phase 1B and LAND 200 Tranche 2 Battlefield Command System are included in the MPR for the first time in 2019–‍20.

Note 3: These projects have been the subject of individual performance audits. See Table 7 for more information.

Note 4: SEA 1439 Phase 3 Collins Class Submarine Reliability and Sustainability is a group of 24 activities primarily sustainment in nature. While not an acquisition project, it has been included at the JCPAA’s request.

Source: The Project Data Summary Sheets in Part 3 of this report.

Engagement objective and scope

8. The objective of this assurance engagement and related procedures is to provide the Auditor-General’s independent assurance over the status of the selected Major Projects. The status of the selected Major Projects is reported in the Statement by the Secretary of Defence (see Part 3 of this report) and the Project Data Summary Sheets (PDSSs) prepared by Defence (see Part 3 of this report). Assurance from the ANAO’s review is conveyed in the Independent Assurance Report by the Auditor-General (see Part 3 of this report).

9. The following forecast information found in the PDSSs is excluded from the scope of the ANAO’s review:

• Section 1.2 Current Status—Materiel Capability Delivery Performance, and Section 4.1—Measures of Materiel Capability Delivery Performance;
• Section 1.3 Project Context—Major Risks and Issues, and Section 5—Major Risks and Issues; and
• forecast dates where included in each PDSS.

10. Accordingly, the Auditor-General’s Independent Assurance Report does not provide any assurance in relation to this information. However, material inconsistencies identified in relation to this information are required to be considered in forming the Auditor-General’s conclusion.

11. The exclusions to the scope of the review, noted above, are due to a lack of Defence systems from which to provide complete and accurate evidence⁹ in a sufficiently timely manner to facilitate the review. This has been an area of focus of the JCPAA over a number of years¹⁰, and it is intended that all components of the PDSSs will eventually be included within the scope of the ANAO’s review.

12. Separate to the formal assurance review, the ANAO has undertaken an analysis of key elements of the PDSSs—including cost, schedule, progress towards delivery of required capability, project maturity, and risks and issues. Longitudinal analysis across these key elements of projects has also been undertaken.

13. Defence provides further insights and context in its commentary and analysis contained in Part 2. This commentary and analysis is not included within the scope of the ANAO’s review.

Review methodology

14. The criteria for Defence’s preparation of information for ANAO review are provided by the Guidelines approved by the JCPAA. There is an expectation that Defence has procedures in place designed to ensure that project information and data was recorded in a complete and accurate manner, for all 25 projects.

15. The ANAO has reviewed the PDSSs prepared by Defence as a priority assurance review under subsection 19A(5) of the Auditor-General Act 1997 (the Act), allowing the ANAO full access to the information gathering powers under the Act. The ANAO’s review provides limited assurance and was undertaken in accordance with the applicable Auditing Standards. The ANAO’s review included an assessment of Defence’s systems and controls, including the governance and oversight in place, to ensure appropriate project management. The ANAO also sought representations and confirmations from Defence senior management and industry (through Defence) in relation to the status of the Major Projects in this report.

Report structure

16. The report is organised into four parts:

• Part 1 comprises the ANAO’s review and analysis (pp. 1–70);
• Part 2 comprises Defence’s commentary, analysis and appendices (not included within the scope of the Independent Assurance Report by the Auditor-General) (pp. 71–116);
• Part 3 incorporates the Independent Assurance Report by the Auditor-General, the Statement by the Secretary of Defence, and the PDSSs prepared by Defence as part of the assurance review process (pp. 117–368); and
• Part 4 reproduces the 2018–19 Major Projects Report Guidelines endorsed by the JCPAA, which provide the criteria for the compilation of PDSSs by Defence (pp. 369–396).

Figure 1, below, depicts the four parts of this report.

Figure 1: 2019–20 Report structure

Note: To assist in conducting inter-report analysis, the presentation of data in the PDSSs remains largely consistent and comparable with the 2018–19 MPR.

Project Data Summary Sheets

17. The PDSSs include unclassified information on project performance, prepared by Defence.¹¹ As projects generally appear in the MPR for multiple years, changes to the PDSS from the previous year are depicted in bold orange text in the PDSS.

18. Each PDSS comprises:

• Project Header: including name; capability and acquisition type; Capability Manager; approval dates; total approved and in-year budgets; stage; complexity; and an image;
• Section 1—Project Summary: including description; current status including financial assurance and contingency statement; and context, including background, uniqueness, major risks and issues, and other current sub-projects;
• Section 2—Financial Performance: including budgets and expenditure; variances; and major contracts in place (in addition to quantities delivered as at 30 June 2020);
• Section 3—Schedule Performance: providing information on design development; test and evaluation; and forecasts and achievements against key project milestones, including Initial Materiel Release (IMR), Final Materiel Release (FMR)¹², Initial Operational Capability (IOC) and Final Operational Capability (FOC)¹³;
• Section 4—Materiel Capability Delivery Performance: provides a summary of Defence’s assessment of its expected delivery of key capabilities, the extent to which milestones were achieved (particularly where caveats are placed on the Capability Manager’s declaration of significant milestones), and a description of the constitution of each key milestone;
• Section 5—Major Risks and Issues: outlines the major risks and issues of the project and remedial actions undertaken for each;
• Section 6—Project Maturity: provides a summary of the project’s maturity, as defined by Defence¹⁴, and a comparison against the benchmark score;
• Section 7—Lessons Learned: outlines the key lessons that have been learned at the project level (further information on lessons learned by Defence is included in Defence’s Appendix 6 in Part 2 of this report); and
• Section 8—Project Line Management: details current project management responsibilities within Defence.

The role of the Joint Committee of Public Accounts and Audit

19. The JCPAA has taken an active role in the development and review of the MPR program. Each year, the Committee considers the draft Guidelines, incorporating the selection of projects for review, and provides the Committee’s views in relation to the Guidelines’ content and development, prior to their endorsement. Following endorsement by the Committee, the Guidelines provide the criteria for Defence’s preparation of PDSSs for ANAO review. The main changes to the MPR Guidelines have tended to follow on from the JCPAA’s recommendations.¹⁵

20. The main change to the MPR Guidelines this year was amending the entry criteria to include the Future Subs project.¹⁶ This was in response to the JCPAA’s interest in including major naval shipbuilding projects in the MPR due to their large size and complexity.

Overall outcomes

Statement by the Secretary of Defence

21. The Statement by the Secretary of Defence was signed on 20 November 2020. The Secretary’s statement provides his opinion that the PDSSs for the 25 selected projects ‘comply in all material respects with the Guidelines and reflect the status of the projects as at 30 June 2020’.

22. In addition, the Statement by the Secretary of Defence details significant events occurring post 30 June 2020, which materially impact the projects included in the report, and which should be read in conjunction with the individual PDSSs. These include: Joint Strike Fighter, Future Subs, Combat Recon. Vehicles, P-8A Poseidon, MRH90 Helicopters, Hawkei, Repl Replenishment Ships, Battlefield Command System, Battle Comm. Sys. (Land) 2B, Collins Comms and EW, Pacific Patrol Boat Repl, Night Fighting Equip Repl, Maritime Comms, and ANZAC Air Search Radar Repl.

23. The 2019–20 MPR Guidelines require Defence to report in the Statement by the Secretary of Defence on projects which have been removed from the MPR which still have outstanding caveats, significant remaining materiel capability or milestones to be delivered. Defence has reported on outstanding caveats in the 2019–20 Statement by the Secretary of Defence.

Conclusion by the Auditor-General

24. The Auditor-General has concluded in the Independent Assurance Report for 2019–20 that ‘nothing has come to my attention that causes me to believe that the information in the 25 Project Data Summary Sheets in Part 3 (PDSSs) and the Statement by the Secretary of Defence, excluding the forecast information, has not been prepared in all material respects in accordance with the 2019–20 Major Projects Report Guidelines (the Guidelines), as endorsed by the Joint Committee of Public Accounts and Audit.’

Key observations from the review

25. The ANAO’s review includes Defence’s project management and reporting arrangements contributing to the overall governance of Major Projects. The ANAO has made observations regarding the following areas:

Status of JCPAA Recommendations and Requests

• Following a JCPAA recommendation made in September 2018¹⁷, Defence advised the Committee in May 2020 that Predict! will be mandated as the risk management system.¹⁸ Defence advised the ANAO in October 2020 that there have been no updates to Defence policy to reflect that Predict! is now the mandated risk management tool. An update to Defence’s policy is expected in May 2021 (see paragraphs 1.66 to 1.68).
• Following JCPAA recommendations made in May 2016 and October 2017¹⁹, Defence considered updates to the Project Maturity Score policy, but did not implement changes. Defence advised the ANAO in August 2020 that the Project Maturity Score will no longer be required as an input to the management of project performance. Defence further advised that an alternative to replace the Project Maturity Score in the PDSS will not be implemented. Therefore project maturity will not be reported in future MPRs (see paragraphs 1.71 to 1.72).
• Following JCPAA recommendations made in May 2014 and May 2016²⁰, Defence is yet to implement a system of materiel capability delivery performance/scope reporting, with a robust methodology applicable to materiel acquisition (see paragraphs 2.54 to 2.59).
• Following a JCPAA request made in 2018²¹ ‘on how Defence major project cost variations and the costs of retaining project staff over time might be reported annually in future Major Projects Reports,’ Defence advised that it is not yet in a position to provide the staff cost component of projects and its systems are not capable of calculating the cost of retaining project staff over time (see paragraph 1.51 and paragraphs 1.53 to 1.56).

Status of Auditor-General Report Recommendations

• In July 2020 Defence closed both recommendations from Auditor-General Report No. 31 2018–19 Defence’s Management of its Projects of Concern (see paragraphs 1.15 to 1.16).
• Defence closed the recommendation from Auditor-General Report No. 3 2019–20 Defence’s Quarterly Performance Report on Acquisition and Sustainment in March 2020 (see paragraphs 1.18 to 1.19).

Defence Acquisition Governance

• The importance of capturing Government decisions in internal Defence documentation and ensuring that Materiel Acquisition Agreements are appropriately aligned with these decisions (see paragraphs 1.23 to 1.28).
• Defence’s implementation of the Smart Buyer Framework to support strategic decision making in the acquisition of major projects. The framework was not utilised at the Second Pass government approval stage for projects in the current MPR (see paragraphs 1.29 to 1.30).
• Defence has advised that it is transitioning to a different reporting system for reporting on the status of acquisition projects from August 2020 (see paragraphs 1.31 to 1.33).
• Projects’ use of contingency funds (see paragraph 1.45). One project in the MPR, MRH90 Helicopters, committed contingency funds of $26.3 million in 2019–20 for supportability and performance risks.
• The status of CASG’s Risk Management Reform Program (initiated by Deputy Secretary CASG in 2017) and establishment of the CASG Risk Management Framework (see paragraphs 1.59 to 1.62).
• A number of projects had not fully met the requirements of Defence’s Project Risk Management Manual Version 2.5 (PRMM V2.5) (see paragraph 1.64).
• Defence updated its policy on Lessons Learned in May 2020. Compliance is not expected until May 2021 (see paragraphs 1.73 to 1.75).
• Defence has not defined, in its internal policies and procedures, the terms ‘caveat’ or ‘deficiency’ relating to the declaration of significant capability milestones. The ANAO has observed use of these terms by Defence to represent exceptions to the achievement of significant milestones (see paragraphs 1.76 to 1.80).

ANAO’s analysis of project performance

26. In addition to its limited assurance review, the ANAO has undertaken an analysis of key elements of the Defence PDSSs—relating to cost, schedule, progress towards delivery of required capability, project maturity, risks and issues, and longitudinal analysis across these key elements of projects.

27. Table 2 provides: summary data on Defence’s progress toward delivering the capabilities for the Major Projects covered in this report; and compares current data against that reported in previous editions of the MPR. This section also contains a summary analysis of the three principal components of project performance: cost, schedule and capability.

Table 2: Summary longitudinal analysis¹

Refer to paragraphs 26 to 41 in Part 1 of this report.

Note 1: The data for the 25 Major Projects in the 2019–20 MPR compares the data from projects in the 2018–19 MPR and 2017–18 MPR. The Major Projects included within each MPR are based on entry and exit criteria in the Guidelines, which have been included in Part 4 of this report. The entry and exit of projects should be considered when comparing data across years.

Note 2: Where a project has multiple Second Pass Approvals, the MPR has historically reported budget variations from the initial Second Pass Approval. The figures in this row are consistent with prior year reporting. See Table 3 for a breakdown of the major components of this variance, and Table 7 for all real variations.

Note 3: Where a project has multiple Second Pass Approvals, the budget at Second Pass Approval reported in the Header refers to the total budget as at the final Second Pass Approval. The figures in this row use this methodology.

Note 4: Slippage refers to a delay in the current forecast date compared to the original government approved date of FOC. Slippage can occur due to late delivery, increases in scope or at times can be a deliberate management decision. In 2019–20, these figures have been adjusted to exclude delays to a project’s schedule that do not result in slippage past the original government approved date, and to exclude schedule reductions over the life of the project. However, paragraph 2.40 reports total schedule reductions over the life of the projects.

Note 4a: As shown in Table 4 below and Table 10 on p. 66 of this report, for the fourteen 2019–20 major projects which have experienced slippage, the range is 1 to 108 months of total slippage.

Note 5: The figures represent the total of the reported maturity scores divided by the total benchmark maturity score, in the PDSSs across all projects.

Note 6: The grey section of the table is excluded from the scope of the ANAO’s priority assurance review, due to a lack of systems from which to obtain complete and accurate evidence in a sufficiently timely manner to facilitate the review.

Note 7: The figures represent the combined number of open ‘high’ and ‘extreme’ risks and issues reported in the PDSSs across all projects. Risks and issues may be aggregated at a strategic level.

Note 8: These figures represent the average predicted capability delivery across all of the Major Projects. This method reduces the effect of any individual project’s size on the aggregate figure. Previously, these figures were calculated based on the number of distinct capability measures defined by each project and therefore projects with more capability measures had more of an effect on the aggregate figure.

Note 9: Defence advised in these years that AWD Ships would not deliver one element of capability at FOC (which equated to approximately one per cent). However, across all the Major Projects this percentage rounded to zero per cent.

Cost

28. Cost management is an ongoing process in Defence’s administration of the Major Projects. While all projects reported that they could continue to operate within the total approved budget of $78.7 billion, the MRH90 Helicopters project was required to draw upon contingency funds to complete project activities.

29. The approved budget for Major Projects included in this MPR has increased by $24.2 billion (30.7 per cent) since initial Second Pass Approval. Budget variations greater than $500 million are detailed in Table 3, below.²² However, as the MPR predominantly focusses on the approved capital budget for acquisition, the ongoing costs of Project Offices²³, training, replacement capability, etc., are not reported here.

Table 3: Budget variation over $500m post initial Second Pass Approval by variation type ^{1, 2}

Note 1: For the variations related to all projects and their value, refer to Table 7 of this report. For the breakdown of in-year variation, refer to Table 8 of this report.

Note 2: For projects with multiple Second Pass Approvals, this table shows variations from the initial approval.

Note 3: Defence has advised that ‘project supplementation’ is a unique term used to describe the approvals history of this project as follows: ‘The original amount of $2549.2, was the Government decision to split Phase 3 into Phase 3A and 3B. In 2011, Government approved Second Pass approval of Phase 3A and the ‘Interim Pass’ Government approval for Phase 3B. The decision to grant Phase 3B ‘Interim Pass’ was to allow greater bargaining power for Defence while negotiating Phase 3A. Phase 3B was always going to return to Government for formal Second Pass approval, which occurred in July 2013, once contract negotiations were complete.’

Note 4: Prior to 1 July 2010, projects were periodically supplemented for price indexation, whereas the allocation for price indexation is now provided for on an out-turned basis at Second Pass Approval.

Note 5: Figures do not add precisely due to rounding.

Source: ANAO analysis of the 2019–20 PDSSs.

Schedule

30. Delivering Major Projects on schedule continues to present challenges for Defence, affecting when the capability is made available for operational release and deployment by the Australian Defence Force, as well as the cost of delivery.

31. The total schedule slippage for the 25 selected Major Projects, as at 30 June 2020, was 507 months when compared to the initial schedule.²⁴ This represents a 21 per cent increase since Second Pass Approval. Across MPR projects that have experienced slippage (14 of 23 projects with approved FOC dates²⁵), the average slippage is 36.2 months (3.0 years). Table 4 below includes details of in-year and total schedule slippage by project. The table shows an increase of 68 months of in-year slippage during 2019–20.

32. The total slippage of 507 months in 2019–20 is 144 months lower than the total in 2018–19 of 651 months. This is due to:

• the exclusion of projects which have exited the MPR (LHD Ships, Additional MRTT, ANZAC ASMD 2B, HATS, Battle Comm. Sys. (Land) 2A, and LHD Landing Craft) removing 218 months of slippage from the total reported in 2018–19 (see Table 5);
• a reduction of 6 months of slippage due to the Collins Comms and EW project recovering this slippage in 2019–20;
• the addition of 68 months of in-year slippage described above; and
• the Battlefield Command System project adding 12 months of slippage to the total of 507 months. The slippage occurred prior to the project entering the MPR in 2019–20.

Table 4: Schedule slippage from original planned Final Operational Capability ¹

Note 1: Refer to footnote 24.

Note 2: These projects have been identified by Defence as Projects of Interest (see paragraph 1.21 in Part 1).

Note 3: These projects’ FOC milestones are yet to be approved by Government.

Source: ANAO analysis of the 2019–20 PDSSs.

33. Platform availability has contributed to the slippage experienced within some projects. For example, Maritime Comms and Collins R&S have been impacted by changes to docking schedules of the Anzac Class frigates and Collins Class submarines respectively. Significant delays have also been experienced by those projects with the most developmental content: AWD Ships, MRH90 Helicopters, CMATS, MQ-4C Triton, and Battle Comm. Sys. (Land) 2B.

34. Table 5, below, provides details of total schedule slippage by project, for projects which have exited the MPR. Compared to the 507 months total schedule slippage for the current 23 Major Projects²⁶, the 27 projects which have exited the MPR²⁷ have reported accumulated schedule slippage of 1146 months, as at their respective exit dates. Table 5 indicates that schedule slippage for projects which have exited the MPR was more pronounced in projects with the most developmental content.

Table 5: Schedule slippage for projects which have exited the MPR ¹

Note 1: The Hornet Refurb and Battle Management System (BMS) projects are not included in this table as they did not have FOC milestones.

Note 2: Next Gen Satellite shows slippage in Figure 9, which related to the final capability milestones at the time. By the time it reached FOC, a new final capability milestone had been introduced and slippage was reduced.

Source: PDSSs in Major Projects Reports and ANAO analysis.

35. Additional ANAO analysis (refer to Figure 8, on page 59) has compared project slippage against the Defence classification of projects as Military Off-The-Shelf (MOTS), Australianised MOTS or developmental. These classifications are a general indicator of the difficulty associated with the procurement process.

36. Figure 9 (on page 60) provides analysis of projects either completed, or removed from the MPR review, and shows that a focus on MOTS²⁸ acquisitions has assisted in reducing schedule slippage. Prima facie, the more developmental in nature a project is, the more likely it will result in a greater degree of project slippage. Figure 9 was requested by the JCPAA in May 2014.²⁹

37. Longitudinal analysis indicates that while the reasons for schedule slippage vary, it primarily reflects the underestimation of both the scope and complexity of work, particularly for Australianised MOTS and developmental projects (see paragraphs 2.27 to 2.34).

Capability

38. The third principal component of project performance examined in this report is progress towards the delivery of capability required by government. While the assessment of expected capability delivery by Defence is outside the scope of the Auditor-General’s formal review conclusion, it is included in the analysis to provide an overall perspective of the three principal components of project performance.

39. The Defence PDSSs report that 19 projects in this year’s report will deliver all of their key capability requirements. Defence’s assessment indicates that some elements of the capability required may be ‘under threat’, but the risk is assessed as ‘manageable’. The five project offices experiencing challenges with expected capability delivery (2018–19: five) are Joint Strike Fighter, MRH90 Helicopters, Hawkei, Battlefield Command System and Battlefield Airlifter. One project office (AWD Ships) reports that it is unable to deliver all of the required capability by FOC.

40. Table 6, below, summarises expected capability delivery as at 30 June 2020, as reported by Defence.

Table 6: Capability delivery

Note 1: Defence advised in these years that AWD Ships would not deliver one element of capability at FOC (which equates to approximately one per cent of elements required). However, across all the Major Projects this percentage rounds to zero.

Source: PDSSs in Major Projects Reports and ANAO analysis.

41. In addition to reporting on expected capability delivery, Defence has continued the practice of including in the PDSSs declassified information on contractual remedies for projects, including stop payments and liquidated damages. During 2019–20, Combat Recon. Vehicles, Hawkei and UHF SATCOM had negotiated contractual remedies involving stop payments or liquidated damages. Prior settlements for projects within this report related to MRH90 Helicopters, Hawkei, Pacific Patrol Boat Repl, Maritime Comms, and UHF SATCOM.

Impact of COVID-19 on the Major Projects

42. Sixteen of the 25 projects have reported experiencing an impact as a result of the COVID-19 pandemic in their PDSSs. The pandemic has seen projects reporting impacts to cost and schedule.

Cost

43. Six projects reported an impact on project budget as a result of the COVID-19 pandemic. A further two projects highlighted an impact to the budget as an emerging issue. Of the six projects that have reported impact to the budget, four experienced an underspend, citing varying reasons for this (delay to training and support, overseas suppliers, shipyard closures and international travel restrictions). Two projects experienced an overspend – one due to accelerated payment terms and the other citing FOREX losses. The remaining 19 projects did not report an impact to the project budget.

Schedule

44. Fifteen projects reported an impact on scheduling as a result of the COVID-19 pandemic, relating to:

1. supplier disruption (supplier production and/or shipping delays);
2. workforce limitations – travel (specialists and crew were due to travel both interstate and from other countries to work with/on the projects); and/or
3. contractor delays (scope, delivery and certification delays).

45. Some projects identified multiple impacts across these three areas. Ten projects reported nil impacts to scheduling.

Capability

46. No projects reported an impact relating to capability.

47. Further information on COVID-19 impacts is reported in the Statement by the Secretary of Defence.

1.1 This chapter provides the Australian National Audit Office’s (ANAO) overview of the scope and approach adopted for its limited assurance review of the 25 Project Data Summary Sheets (PDSSs) prepared by the Department of Defence (Defence). The chapter also includes information and commentary on developments in Defence’s acquisition governance processes, based on the ANAO’s review.

Review scope and approach

1.2 In 2012 the Parliament’s Joint Committee of Public Accounts and Audit (JCPAA) identified the ANAO’s review of Defence PDSSs as a priority assurance review, under subsection 19A(5) of the Auditor-General Act 1997 (the Act). This provided the ANAO with full access to the information gathering powers under the Act. The ANAO’s review of the individual project PDSSs, which are reproduced in Part 3 of the Major Projects Report (MPR), was conducted in accordance with the auditing standards set by the Auditor-General under section 24 of the Act through its incorporation of the Australian Standard on Assurance Engagements (ASAE) 3000 Assurance Engagements Other than Audits or Reviews of Historical Financial Information, issued by the Australian Auditing and Assurance Standards Board.

1.3 The following forecast information is excluded from the scope of the ANAO’s review: capability delivery, risks and issues, and forecast dates. These exclusions are due to the lack of Defence systems from which to provide complete and accurate evidence³⁰, in a sufficiently timely manner to complete the review. Accordingly, the Independent Assurance Report by the Auditor-General does not provide any assurance in relation to this information. However, material inconsistencies identified in relation to this information are required to be considered in forming the conclusion.

1.4 The ANAO’s work is appropriate for the purpose of providing an Independent Assurance Report in accordance with the ANAO Auditing Standards. However, the review of individual PDSSs is based on a limited assurance approach and is not as extensive as individual performance and financial statement audits conducted by the ANAO, in terms of the nature and scope of issues covered, and the extent to which evidence is required by the ANAO. Consequently, the level of assurance provided by this review, in relation to the 25 major Defence equipment acquisition projects (Major Projects), is less than that provided by the ANAO’s program of performance and financial statement audits.

1.5 Separately, the ANAO reviews: developments in Defence’s acquisition governance processes (information and commentary on governance issues appears in this chapter); and undertakes analysis of key elements of the PDSSs (information and commentary on systemic issues and longitudinal analysis for the 25 projects reviewed appears in the next chapter).

1.6 The ANAO’s review was conducted in accordance with the ANAO Auditing Standards at a cost to the ANAO of approximately $2.0 million.³¹

Review methodology

1.7 The ANAO’s review of the information presented in the individual Defence PDSSs included:

• examination and assessment of the governance and oversight in place to ensure appropriate project management;
• an assessment of the systems and controls that support project financial management, risk management and project status reporting, within Defence;
• an examination of each PDSS and the documents and information relevant to them;
• a review of relevant processes and procedures used by Defence in the preparation of the PDSSs;
• discussions with persons responsible for the preparation of the PDSSs and management of the projects;
• analysis of project information, for example, cost and schedule variances;
• taking account of industry contractor comments provided on draft PDSS information;
• assessing the assurance by Defence managers attesting to the accuracy and completeness of the PDSSs;
• examination of the representations by the Chief Finance Officer supporting the project financial assurance and contingency statements;
• examination of confirmations, provided by the Capability Managers, relating to each project’s progress toward Initial Materiel Release (IMR), Final Materiel Release (FMR), Initial Operational Capability (IOC) and Final Operational Capability (FOC); and
• examination of the Statement by the Secretary of Defence, including significant events occurring post 30 June, and management representations by the Secretary of Defence.

1.8 The ANAO’s review of PDSSs also focused on project management and reporting arrangements contributing to the overall governance of the Major Projects. The ANAO considered:

• developments in acquisition governance (see paragraphs 1.10 to 1.33, below);
• the financial framework, particularly as it applies to the project financial assurance and contingency statements, and Defence’s advice that project financial reporting during 2019–20 would be prepared on the same basis as project approvals and expenditure represented in the Portfolio Budget Statements and the Defence Annual Report (i.e. on a cash basis) (see Section 2 of the PDSSs);
• schedule management and test and evaluation processes (see Section 3 of the PDSSs);
• materiel capability / scope delivery forecast assessments, including Defence statements of the likelihood of delivering key capabilities, particularly where caveats are placed on the Capability Manager’s declaration of significant milestones (see Section 4 of the PDSSs);
• the changes due to Defence’s reform of the Defence Enterprise Risk Management Framework, and the completeness and accuracy of major risk and issue data (see Section 5 of the PDSSs);
• the project maturity framework along with its related reporting and the systems in place to support the consistent and accurate application and the provision of this data (see Section 6 of the PDSSs); and
• the impact of acquisition issues on sustainment to ensure the PDSS is a complete and accurate representation of the acquisition project.

1.9 This review informed the ANAO’s understanding of the systems and processes supporting the PDSSs for the 2019–20 review period. It also highlighted issues in those systems and processes that warrant attention.

Acquisition governance

1.10 Consistent with previous years, the ANAO considered Defence’s Major Project acquisition governance processes when planning and conducting the review for the 2019–20 MPR. While some of these processes are now established, others continue to mature or require further development to achieve their intended impact.

Defence Independent Assurance Reviews

1.11 The Defence Independent Assurance Review (IAR) process provides the Defence Senior Executive with assurance that projects and products will deliver approved objectives and are prepared to progress to the next stage of activity. These management initiated reviews consider a project’s status while sufficient time remains for corrective action to be implemented.^{32, 33}

1.12 IARs are intended to commence at project initiation and are usually conducted on an annual basis through to FOC. They are an important input to key acquisition and sustainment decision points or milestones as defined in the Capability Life Cycle.³⁴

1.13 Sixteen of the 25 projects included in this report had an Independent Assurance Review conducted during 2019–20³⁵, which formed key evidence for the ANAO’s review.³⁶ Defence advised that there was a pause in the conduct of IARs in mid-2020 due to COVID-19.

Projects of Concern

1.14 The Projects of Concern process is intended to focus the attention of the highest levels of government, Defence and industry on remediating problem projects.³⁷ As at 30 June 2020, one MPR project, MRH90 Helicopters, was a continuing Project of Concern. The project was placed on the list in November 2011 due to contractor performance relating to significant technical issues preventing the achievement of milestones on schedule.³⁸ The project has progressed the materiel capability delivery relating to the cargo hook, mission troop seats and fast roping and rappelling. There remains an ongoing inability to meet materiel capability delivery milestones and performance criteria to do with the Taipan Gun Mount, aero-medical evacuation equipment and the Common Mission Management System.³⁹ Final Operational Capability is scheduled for December 2021.

1.15 Auditor-General Report No.31 2018–19, Defence’s Management of its Projects of Concern, assessed whether the Department of Defence’s Projects of Concern regime was effective in managing the recovery of underperforming projects. It concluded that, while the regime is an appropriate mechanism for escalating troubled projects to the attention of senior managers and ministers, Defence was not able to demonstrate the effectiveness of its regime in managing the recovery of underperforming projects. Moreover, the audit observed that the transparency and rigour of the framework’s application has declined in recent years. The audit recommended that:

• Recommendation no.1: Defence introduce, as part of its formal policy and procedures, a consistent approach to managing entry to, and exit from, its Projects of Interest and Projects of Concern lists. This should reflect Defence’s risk appetite and be made consistent with the new Capability Acquisition and Sustainment Group Risk Model and other, Defence-wide, frameworks for managing risk. To aid transparency, the policy and the list should be made public.
• Recommendation no.2: Defence evaluates its Projects of Concern regime.⁴⁰

1.16 In July 2020 Defence closed both these recommendations, advising that: CASG has developed a consistent approach to entry and exit from the Projects of Interest and Projects of Concern lists and the Projects of Concern list is publicly available; and CASG has evaluated the Projects of Concern regime and has effective assurance mechanisms in place, underpinned by Independent Assurance Reviews.

Quarterly Performance Report

1.17 The Defence Quarterly Performance Report (QPR) aims to provide senior stakeholders within government and Defence with insight into the delivery of capability to the Australian Defence Force.⁴¹ The report is provided to the Minister for Defence and the Minister for Defence Industry on a quarterly basis.⁴²

1.18 In July 2019, the ANAO completed an audit on the effectiveness of the QPR in providing senior stakeholders with accurate and timely information on the status and emerging risks and issues. It found the June 2018 QPR, reviewed by the ANAO, to be largely effective, contained mostly accurate information, and was valued by senior stakeholders.⁴³ The ANAO recommended that Defence improve the QPR as a tool for senior leaders by reporting on:

1. trend performance data for sustainment products; and
2. emerging candidates for the Projects/Products of Concern list and Products/Projects of Interest list that have been recommended by an Independent Assurance Review or which are under active consideration by senior management.⁴⁴

1.19 In the course of its review for the 2018–19 MPR, the ANAO observed that Defence’s June 2019 QPR reported on both improved and deteriorated performance for both acquisition and sustainment products since the previous QPR. This reflected a change in trend reporting consistent with the agreed ANAO recommendation. The ANAO also observed that Defence’s June 2019 QPR reported the emerging candidates for the Projects/Products of Concern list and Products/Projects of Interest list which had been recommended either by an IAR or which were under active consideration. This change was also consistent with the agreed ANAO recommendation.⁴⁵ Defence closed this recommendation in March 2020.

1.20 The ANAO examines QPRs as part of the procedures for its limited assurance review of Defence’s PDSSs.⁴⁶ For the 2019–20 review, the ANAO examined the four QPRs from September 2019 to June 2020.

1.21 The June 2020 QPR identified nine MPR projects as Projects of Interest⁴⁷:

• Joint Strike Fighter—the QPR notes risks to the achievement of IOC in December 2020 due to the impacts of COVID-19, which have included delayed aircraft production, travel restrictions, the postponement of air exercises and the isolation of personnel.⁴⁸
• Future Frigates—due to size, complexity, risk-profile and media interest. The QPR notes that all scheduled activities have been progressing as planned, although some will need to be reassessed in light of COVID-19 restrictions.⁴⁹
• Hawkei—the QPR notes schedule risks due to COVID-19 which may impact Army’s ability to undertake the full range of operational test and evaluation activities, and operator and maintainer training, required to achieve IOC in December 2020.⁵⁰
• Battlefield Airlifter—Final Operational Capability (FOC) was not met in December 2019 and the QPR notes that ‘residual’ activities remain outstanding, including fleet fitment and certification of Mode 5 Identification Friend or Foe and remediation of the Missile Approach Warning System. Air Force expects to review the capability requirement and advise the Government of a revised capability definition and Final Operational Capability date, no later than December 2020.⁵¹
• Repl Replenishment Ships (Maritime Operational Support Capability)—the QPR notes delays to delivery of both ships due to COVID-19, which has slowed construction progress and resulted in international travel restrictions, preventing Australian based contractors from travelling to Spain to complete their activities. There is no forecast delay to FOC.⁵²
• MQ-4C Triton, due to the United States Navy announcing a production funding suspension for its Triton program until 2023. The suspension will have capability, schedule and cost implications, and potential sustainment cost and capability impacts. The project is undergoing a fundamental review.⁵³
• Civil Military Air Traffic Management System (CMATS)—the QPR notes risks to schedule due to execution of design milestones, integration with related projects (procurement of digital radios and surveillance radars),and the ongoing need for Airservices Australia to implement cost saving changes agreed with Defence.⁵⁴
• Battlefield Command System—the QPR notes schedule risks due to vehicle integration issues.⁵⁵
• UHF SATCOM—the QPR notes that software rectification issues require attention.^{56, 57}

1.22 These ongoing issues for the Joint Strike Fighter, Future Frigates, Hawkei, Battlefield Airlifter, Repl Replenishment Ships, MQ-4C Triton, CMATS, Battlefield Command System and UHF SATCOM projects align with the results of the ANAO’s review of the PDSSs. Delays to progress have impacted the delivery schedule of Joint Strike Fighter, Battlefield Airlifter, MQ-4C Triton, CMATS and Battlefield Command System during 2019–20⁵⁸ (see Table 4, on page 15).

Project Directives and Materiel Acquisition Agreements

1.23 Project Directives (previously known as Joint Project Directives) state the terms of government approval, reflecting the approved scope and timeframes for activities, responsibilities and resources allocated, and key risks and issues.⁵⁹ Project Directives have historically been used to inform internal Defence documentation such as Materiel Acquisition Agreements (MAAs) between Capability Acquisition and Sustainment Group (CASG) and the Service Chiefs.^{60, 61} Project Directives had previously been described as a key governance document under the Capability Life Cycle⁶², intended to ensure that all parties in Defence are informed of Government decisions. Defence updated the Capability Life Cycle Manual in June 2020, which no longer refers to Project Directives as a key governance document. Defence has advised the ANAO that Government decisions are recorded in Defence’s Capability Lifecycle Management Tool, which records Government decisions in relation to a project. In some cases, the Joint Force Authority may provide a specific documented directive. The ANAO has previously highlighted the importance of ensuring that Project Directives properly reflect the relevant Government decision, and that MAAs are appropriately aligned with the relevant Project Directive.⁶³

1.24 Of the five new projects entering the 2019–20 MPR, the Future Frigates and MQ-4C Triton projects advised the ANAO that they had direct access to government approved documentation. The Future Subs, Combat Recon. Vehicles and Battlefield Command System projects advised that they did not have access to Government approvals through the Capability Life Cycle Management Tool. In November 2020, Defence advised the ANAO that ‘the internal Cabinet Liaison Services section provides advice to Defence in relation to information pertaining to Government approvals. Where a Project has not been identified as having a need to know, the Project can request access to relevant Cabinet documents via a business case.’

1.25 The Combat Recon. Vehicles achieved Second Pass Approval (Government Approval) in March 2018. Defence proceeded to update the Materiel Acquisition Agreement between CASG and the Capability Manager in May 2019. Key dates listed in the MAA do not align with Government approvals, and Defence has advised the ANAO that it intends to update the MAA to ensure alignment.⁶⁴

1.26 The risk of misalignment or error is reduced if Defence has appropriate access to Government records, such as that previously provided by the Project Directive.

1.27 The ANAO requires access to original approval documents to validate the requirements of projects. At this time, validation based on internal Defence documentation is not always possible.

1.28 First advised by Defence in July 2016⁶⁵, Product Delivery Agreements (PDAs)⁶⁶ were to be developed to replace the existing MAAs and Materiel Sustainment Agreements (MSAs). Defence advised the ANAO that this initiative is still in the concept phase and will not apply until a PDA framework is approved and implemented.

Smart Buyer Framework

1.29 The 2015 First Principles Review recommended the construction of a ‘smart buyer’ framework, with the aim of ‘[ensuring] Defence can make strategic decisions regarding the most appropriate procurement and contracting methodologies’. Defence has begun to conduct Smart Buyer assessments for acquisition projects at different stages of approval. None of the projects currently in the Major Projects portfolio have been approved under the Smart Buyer processes. The ANAO will continue to report on the outcomes of the First Principles Review and the Smart Buyer framework.

1.30 Defence advised the ANAO that no MPR projects have been considered by the Smart Buyer Framework in 2019–20 as the Smart Buyer Framework is not applicable post Government approval. Defence records indicate that two MPR projects have undergone a Smart Buyer workshop during 2019–20: AIR 8000 Ph. 2 (Battlefield Airlifter) and LAND 200 Ph. 2 (Battlefield Command System). In both cases, the workshops were conducted to remediate project issues, not to initiate an approach to a Gate.⁶⁷

Business systems

1.31 Defence continues to review its business systems with the aim of consolidating processes and systems in order to provide a more manageable system environment. During 2019–20, Defence continued to report on the status of acquisition projects in the Monthly Reporting System (MRS), which provides much of the data for the PDSSs. In August 2020, Defence discontinued the Monthly Reporting System with project reporting now occurring via the Monthly Reporting Module (MRM). Defence advised the ANAO that MRM replicates the functionality of MRS while delivering an updated platform and user interface, and that the Project Performance Report Information Platform (PPRIP) delivers a platform for projects to also conduct monthly reviews of their project and enable raising of risks and actions with line management.

1.32 Defence further advised the ANAO in November 2020 that all 25 of the Major Projects in this report are using the PPRIP.

1.33 As the MRM was implemented in the 2020–21 financial year, the ANAO will review Defence’s use of the MRM in the 2020–21 MPR. As the replacement for the MRS, the MRM is a key system of reporting used to support disclosures in Defence’s PDSSs.

Results of the review

1.34 The following sections outline the results of the ANAO’s review, which inform the overall conclusion in the Independent Assurance Report by the Auditor-General for 2019–20.

Financial framework

1.35 The project financial assurance statements were introduced in the 2011–12 Major Projects Report and have been included within the scope of the Independent Assurance Report by the Auditor-General since 2014–15. The contingency statements were introduced for the first time in the 2013–14 report and these describe the use of contingency funding to mitigate project risks. Together, they are aimed at providing greater transparency over projects’ financial status.

1.36 A project’s total approved budget comprises:

• the allocated budget, which covers the project’s approved activities, as indicated in the MAA; and
• the contingency budget, which is set aside for the eventuality of risks occurring and includes unforeseen work that arises within the delivery of the planned scope of work.⁶⁸

1.37 In 2019–20, the ANAO reviewed the financial framework as it applied to managing project budgets and expenditure, including: project financial assurance, contingency, the reporting environment, and reporting cost variations and personnel costs.

Project financial assurance statement

1.38 The project financial assurance statement’s objective is to enhance transparency by providing readers with information on each project’s financial position (in relation to delivering project capability) and whether there is ‘sufficient remaining budget for the project to be completed’.⁶⁹

1.39 As discussed in the 2018–19 MPR (para 1.38), in September and November 2018, due to cost pressures, the Joint Strike Fighter (JSF/F-35) project received government approval to transfer project scope of $1.5 billion to other phases of the program (none of which have been approved by government). There was no corresponding transfer of funds out of the project budget.⁷⁰ Defence has not made any further disclosures of this sort for other projects in this year’s PDSSs. The 2019–20 PDSS for the Joint Strike Fighter (found in Part 3 of this MPR) states that:

The majority of these scope items were no longer needed, as FOC requirements will be met without major upgrades. Beyond Line of Sight Communications (BLOS) was only desirable and will now be delivered as a cost effective common capability rather than Australian unique. In conjunction with the retirement of cost risks within the project, this has remediated the cost issues identified to Government in 2017. These adjustments have also aligned Australian delivery schedules with the global JSF development program. While the approved changes have reduced the capability being delivered by Phase 2A/2B it has not increased or reduced funding, of the capability being delivered, in the broader AIR 6000 program. As the changes have minimal impact on overall delivery schedule of the project, AIR 6000 Phase 2A/B plans for FOC in 2023 remain unchanged.

1.40 The Joint Strike Fighter PDSS reports a risk that project capability may be affected by overall funding or programming issues arising from internal cost growth, forecasting accuracy and external budget constraints. The remedial actions to address this risk reported in the PDSS include:

• ‘Conduct ongoing engagement with the [United States] F-35 Joint Program Office and major project suppliers to facilitate improved cost data to allow the F-35 project to meet budgeting and programming expectations’ — i.e. clarifying potential cost pressures on the project;
• ‘Proactive management of cost risk identification and engagement with the Capability Manager to prioritize requirements to deliver project capability within the approved project budget’ — i.e. actively identify cost risk and engage with senior leaders; and
• ‘Options may be developed for Capability Manager consideration to achieve project affordability by aligning project expenditure with the Defence Integrated Investment Program capacity in any specific year’ — i.e. consider options for scheduling project expenditure to align with Defence’s available funding.

1.41 The Chief Finance Officer’s representation letter to the Secretary of Defence on the 2019–‍20 MPR’s project financial assurance statements was unqualified. The project financial assurance statement is restricted to the current financial contractual obligations of Defence for these projects, including the result of settlement actions and the receipt of any liquidated damages, and current known risks and estimated future expenditure as at 30 June 2020.

Contingency statements and contingency management

1.42 The purpose of the project contingency budget is to estimate the inherent cost, schedule and technical uncertainties of projects’ in-scope work.⁷¹ Defence policy requires project managers to ensure that all decisions in regards to a project’s contingency budget are included in the project’s contingency budget log to ensure ongoing transparency and traceability.⁷²

1.43 PDSSs are required to include a statement regarding the application of contingency funds during the year, if applicable, as well as disclosing the risks mitigated by the application of those contingency funds. Defence’s Project Risk Management Manual (PRMM version 2.5, page 105) requires that contingency be applied for identified risk mitigation activities.

1.44 Contingency provisions for projects are not programmed or funded in cash terms⁷³ and projects are encouraged to meet contingency funding requirements from within their currently programmed cash funding. If this cannot be achieved, a project may propose to access contingency funding from the relevant capital program (Approved Major Capital Investment Program (AMCIP), Facilities and Infrastructure Program (FIP) and ICT Capital Program). If this cannot be achieved, the contingency call will be presented to the Defence Investment Committee, which if agreed will potentially be met by budget offsets across the whole Integrated Investment Program.⁷⁴

1.45 One project in the MPR, MRH90 Helicopters, had contingency funds of $26.3 million committed in 2019–20 for supportability and performance risks. Access to these funds was approved in previous financial years.

1.46 The ANAO’s examination of projects’ contingency logs as at 30 June 2020 highlighted that the clarity of the relationship between contingency allocation and identified risks continues to be an issue. Four projects (Combat Recon. Vehicles, Offshore Patrol Vessel, Repl Replenishment Ships, and Pacific Patrol Boat Repl) did not explicitly align their contingency log with their risk log to ensure the expected cost impact of risks is maintained effectively, as required by PRMM version 2.5.

1.47 The ANAO will continue to monitor non-compliance with PRMM version 2.5 and the release of specific guidance following the implementation of the CASG Risk Management Framework expected to be implemented in stages to 2022 (see paragraphs 1.61–1.63).

Reporting environment

1.48 On 4 April 2018, Defence advised project offices that project financial reporting for 2017–‍18 PDSSs would be prepared on the same basis as project approvals and expenditure represented in the Portfolio Budget Statements and the Defence Annual Report (i.e. on a cash basis).⁷⁵

1.49 Defence obtains cash expenditure data using a management reporting tool called Budget and Output Reporting Information System (BORIS). Prior to the 2017–18 MPR, accrual expenditure data was extracted from the Financial Management Information System known as Resource and Output Management and Accounting Network (ROMAN). Given the change in the extraction method, the ANAO requested that Defence perform reconciliations of the cash expenditure figures from BORIS to ROMAN for each project.

1.50 In the 2019–20 MPR, Defence continued to report the projects’ financial information on a cash basis and therefore continued to perform these reconciliations. This activity concluded in October 2020 and enabled the ANAO to obtain assurance over the cash expenditure. Defence’s Chief Finance Officer has determined that, from the 2020–21 MPR onwards, Defence would report expenditure data on an accrual basis. The 2019–20 MPR will therefore be the last year that these reconciliations need to be performed.

Reporting cost variations since Second Pass Approval and personnel costs

1.51 In May 2018, the JCPAA wrote to the Auditor-General to request that the ANAO report back to it ‘on how Defence major project cost variations and the costs of retaining project staff over time might be reported annually in future Major Projects Reports.’⁷⁶

1.52 A new table was included in the 2017–18 MPR showing all budget variations post initial Second Pass Approval for projects. Refer to Table 7 on page 44.

Project Personnel Numbers and Costs

1.53 In terms of calculating the cost of retaining project staff, Defence advised the ANAO in November 2018 that its IT systems did not provide a direct mapping of personnel to projects. It noted that personnel often work on multiple projects and sustainment activities at any given time.

1.54 The ANAO observed during fieldwork in 2019 that several MPR projects had staff who worked concurrently on other projects, which included shared corporate staff. Some of these projects did not have systems in place to record accurately the proportion of time these shared staff attributed to the project. Moreover, the ANAO observed that MPR projects used different methods to record personnel data.

1.55 In April 2020, the Defence Finance Group (DFG) indicated that it was possible to extract employee expenses (excluding contractors) from Defence’s personnel system, known as the Personnel Management Key Solution (PMKeyS). DFG advised the ANAO that it would need to work with Defence People Group to ensure all relevant Department IDs within PMKeyS for a project have been captured, as well as ensuring that people are properly allocated to the correct Department IDs. In November 2020 Defence advised that it continued to investigate whether PMKeyS could be used as a robust source to track employee costs by projects.

1.56 However, Defence has advised the ANAO that it is still not yet in a position to provide the staff cost component of projects and that its systems are not capable of calculating the cost of retaining project staff over time. Accordingly, Defence has not provided any data on the costs of project staff for projects in the MPR.⁷⁷ The ANAO will continue to monitor Defence’s progress in recording project personnel numbers and costs.

Enterprise Risk Management Framework

1.57 While major risks and issues data in the PDSSs remains excluded from the formal scope of the Auditor-General’s Independent Assurance Report⁷⁸, material inconsistencies identified in relation to this information are required to be detailed in the report. The following information is included to provide an overall perspective of how risks and issues are managed within Defence and the selected Major Projects.

1.58 Risk management has been a focus of the MPR since its inception. The CASG risk management environment consists of multiple policies and varying implementation mechanisms and documentation. There are multiple group-level (i.e. CASG), sub-group (i.e. Divisional) and project-level risk management documents. The primary focus of the ANAO’s examination of risk management is at the project level, in order to provide assurance over the PDSSs.

1.59 At the Group level, the Deputy Secretary CASG issued a directive in May 2017 establishing a CASG Risk Management Reform Program to implement a risk management model that is situated within Defence’s risk management framework. Defence advised the ANAO in November 2020 that it has delivered all three phases of the reform. The third phase, which included the development of risk management policies and toolsets for use by projects, was initially planned to be concluded in June 2019; Defence concluded the contract with its industry partner in May 2020. In addition, Risk Profiles for some CASG Domains remain in draft, and Risk Management Implementation Plans and the Risk Management Annual Program Plan are still being updated.

1.60 In June 2020 the Deputy Secretary CASG issued a directive establishing the CASG Risk Management Framework, which is the key deliverable of the CASG Risk Management Reform Program. This initiative includes:

• the framework—which is the primary policy and operating framework for the management of risk across the group; and
• the Group Risk Management Strategy 2020-22, which provides a structured pathway to implementing the remodelled approach to managing risk across the 2020-22 period.

1.61 Defence also advised on the release of tools to standardise risk practices across CASG. This will include the rollout of Predict! version 6.0 across CASG.⁷⁹ Defence anticipates the roll-out of these tools and risk practices to be complete by February 2022.

1.62 It is not clear whether Defence has achieved the outcomes of all three phases of the CASG Risk Reform as intended in the 2017 Deputy Secretary CASG Directive. That Directive included implementation of the remodelled approach to management of risk, which is now not due to be completed until February 2022.⁸⁰

1.63 The ANAO will continue to monitor implementation of the Risk Management Framework, but will not be able to consider including risks and issues in the scope of the MPR until implementation of the framework is more mature.

1.64 In 2019–20, the ANAO again examined project offices’ risk and issue logs at the Group and Service level, which are predominantly created and maintained utilising spreadsheets and/or Predict! software. Overall, the issues with risk management that the ANAO observed related to:

• variable compliance with corporate guidance. While all 25 MPR projects had a Risk Management Plan, five did not validate the currency of the Risk Management Plan in line with PRMM version 2.5⁸¹;
• the visibility of risks and issues when a project is transitioning to sustainment;
• for one project (Joint Strike Fighter), sustainment and acquisition risks are managed together⁸²;
• the frequency with which risk and issue logs are reviewed to ensure risks and issues are accurate and complete, appropriately managed in a timely manner, and accurately reported to senior management;
• risk management logs and supporting documentation of variable quality, particularly where spreadsheets are being used⁸³; and
• lack of quality control resulting in inconsistent approaches in the recording of issues within Predict!

1.65 The ANAO has previously observed that Defence’s use of spreadsheets as a primary form of record for risk management is a high risk approach. Spreadsheets lack formalised change/version control and reporting, thereby increasing the risk of error. This can make spreadsheets unreliable corporate data handling tools as accidental or deliberate changes can be made to formulae and data, without there being a record of when, by whom, and what change was made. As a result, a significant amount of quality assurance is necessary to obtain confidence that spreadsheets are complete and accurate at 30 June, which is not an efficient approach.

1.66 The JCPAA recommended in September 2018 that Defence plan and report a methodology to the Committee showing how acquisition projects can transition from the use of spreadsheet risk registers to tools with better version control.⁸⁴ In response, Defence advised in May that Predict! will be mandated as the risk management system.⁸⁵ Defence advised the ANAO in October 2020 that there have been no updates to Defence policy to reflect that Predict! is now the mandated risk management tool. An update to Defence’s policy is expected in May 2021.

1.67 Defence advised the ANAO in November 2020 that as at 30 June, 21 out of the 25 MPR projects⁸⁶ have a presence in Predict!, but are not necessarily using it to manage risks and issues. The ANAO’s review of the documentation from CASG’s 25 project offices at 30 June indicates that:

• nine utilise spreadsheets⁸⁷ as their primary risk management tool;
• nine utilise Predict!⁸⁸;
• three (Maritime Comms, Combat Recon. Vehicle and Future Subs) utilise both Microsoft Excel and Predict!;
• two (Joint Strike Fighter and CMATS) utilise a bespoke SharePoint based tool;
• one (Night Fighting Equip Repl) utilises the Project Performance Report (see paragraph 1.31 to 1.32); and
• one (Future Frigates) uses Predict! and Defence’s Capability Lifecycle Management Tool.

1.68 Defence advised in November 2020 that CASG has an approved rollout plan to transition all CASG projects to Predict! over the next few years. The ANAO will report on MPR projects’ use of the mandated Predict! tool in the next MPR.

Project maturity framework

1.69 Project Maturity Scores have been a feature of the Major Projects Report since its inception in 2007–08. The DMO [Defence Materiel Organisation] Project Management Manual 2012 defined a maturity score as:

The quantification, in a simple and communicable manner, of the relative maturity of capital investment projects as they progress through the capability development and acquisition life cycle.⁸⁹

1.70 Maturity scores are a composite indicator, cumulatively constructed through the assessment and summation of seven different attributes. The attributes are: Schedule, Cost, Requirement, Technical Understanding, Technical Difficulty, Commercial, and Operations and Support, which are assessed on a scale of one to 10.⁹⁰ Comparing the maturity score against its expected life cycle gate benchmark provided internal and external stakeholders with a useful indication of a project’s progress.

1.71 In 2016⁹¹ and again in 2017⁹², the JCPAA recommended that Defence update the policy on Project Maturity Scores. In 2018 and 2019 Defence considered updates to the policy, but did not implement changes.⁹³

1.72 Defence advised the ANAO in August 2020 that the Project Maturity Score will no longer be required as an input to the management of project performance. Defence also advised that an alternative to replace the Project Maturity Score currently included in its PDSSs will not be implemented. This component of the PDSS has therefore been removed from the 2020–21 MPR Guidelines.

Lessons Learned

1.73 In May 2020, CASG released a revised version of its Lessons Program Policy. The Policy is enforced by a Defence Joint Directive which directs all Defence ‘Groups and Services, as required, to establish and lead a whole-of-Defence Joint Lessons that provides centralised Lessons management and coordination’.

1.74 Version 2.0 of the CASG Lessons Program Policy states that⁹⁴:

Deputy Secretary CASG expects the CASG leadership to share best practices and lessons. To facilitate this the CASG Lessons Program continually analyses lessons collected from programs, projects, products and governance review outcomes; and ensures they are readily available via the Defence Lessons Repository (DLR).

1.75 Defence advised the ANAO that this policy does not impact the reporting of lessons in the MPR and that compliance with the new policy is not expected until May 2021. Defence also advised that it is establishing a framework for monitoring compliance.

Caveats and deficiencies

1.76 Defence has not defined the terms ‘caveat’ or ‘deficiency’ to the declaration of significant milestones in its internal policies and procedures. The ANAO has observed the use of these terms by Defence to represent exceptions to the achievement of significant milestones declared by Defence such as IMR, IOC, FMR and FOC.

1.77 The 2017–18 MPR noted a ‘reduced trend of Major Projects which had achieved significant milestones with caveats’, consistent with Defence’s advice at the time that it discourages Independent Assurance Reviews recommending caveats at FOC.⁹⁵ Only one project (Growler) achieved a major milestone with caveats in 2017–18.⁹⁶

1.78 In 2018–19, Defence declared major milestones with caveats or deficiencies for four projects (P-8A Poseidon, Growler, Additional MRTT and LHD Ships). In 2019–20, Defence declared a similar number of milestones with caveats to the previous year:

• P-8A Poseidon — as at 30 June the project expected to declare three caveats to the achievement of the Material Release 3 (MR3) milestone, related to Objective Search and Rescue (SAR) store capability (UNIPAC III), spares and permanent installation of one Mobile Tactical Operations centre. MR3 was acknowledged by the Capability Manager in September 2020.
• Overlander Medium/Heavy — Defence declared one caveat to the achievement of the IOC milestone in December 2019, related to certification of the Air Movements Training and Development Unit.
• Hawkei — Defence declared four caveats to the achievement of the IMR milestone in May 2020, related to low risk verification and validation activities, delivery of spares and deficiencies in the delivery of the mission system Complete Equipment Schedule and Repair Parts Scale.
• Additional MRTT⁹⁷ — Defence declared caveats to FOC in January 2020 relating to aircrew staffing across the KC-30A fleet.

1.79 In addition, Defence accepted two ‘concessions’ to the declaration of FOC in December 2019 for the Battle Comm. Sys (Land) 2A project⁹⁸, with two out of six nodal configurations having ‘capability limitations’ due to exceeding weight limits.

1.80 The ANAO will continue to monitor Defence’s declaration of caveats (or exceptions) to the achievement of significant Capability Milestones. The MPR Guidelines provide that in respect to projects which have been removed from the MPR with outstanding caveats, reporting will be provided in the Statement by the Secretary of Defence until their final status is accepted by the Capability Manager.⁹⁹

2.1 Performance information is important in the management and delivery of major Defence equipment acquisition projects (Major Projects). It informs decisions about the allocation of resources, supports advice to government, and enables stakeholders to assess project progress.

2.2 Project performance has been the subject of many of the reviews of the Department of Defence (Defence), and a consistent area of focus of the Parliament’s Joint Committee of Public Accounts and Audit (JCPAA) since the first Major Projects Report (MPR). This chapter progresses previous Australian National Audit Office (ANAO) analysis over project performance.

Project performance analysis by the ANAO

2.3 The major dimensions of projects’ performance examined in this chapter are:

• Cost performance (pp. 40–51) — this includes the percentage of budget expended (Budget Expended), changes in budget since Second Pass Approval, in-year changes to budget, and in-year expenditure.
• Schedule performance (pp. 52–66) — this includes the percentage of time elapsed (Time Elapsed), total schedule slippage, and in-year changes to schedule.
• Capability performance (pp. 67–69) — this includes the key challenges faced by Defence in the delivery of key materiel capabilities.

2.4 The following sections of this chapter provide ANAO analysis relating to the three principal dimensions of project performance noted above, drawing on the Defence PDSSs for the 25 Major Projects. This work includes analysis of in-year information, longitudinal analysis and analysis of the results of project progress for the year-ended 30 June 2020. Figure 2 (below) directly compares cost performance with schedule performance through two metrics, Budget Expended and Time Elapsed.¹⁰⁰

Figure 2: Budget Expended and Time Elapsed

Note: Future Frigates and Future Subs are yet to have their FOC milestones approved by Government.

Source: ANAO analysis of the 2019–20 PDSSs.

2.5 Figure 2 shows that for most projects (20 of 23¹⁰¹), Budget Expended is broadly in line with (within 10 per cent), or lagging, Time Elapsed. This relationship is generally expected in an acquisition environment predominantly based on milestone payments. However, due to the varying complexity, stages and acquisition approaches across the portfolio of projects, further analysis of these simple performance measures is required to provide a better understanding of key variances.

2.6 Where Budget Expended is significantly lagging Time Elapsed, the project schedule may be at risk — i.e. expenditure lags may indicate delays in milestone achievement. In 2019–20, the Budget Expended for three projects lagged Time Elapsed by at least 20 per cent. For two of these three projects, milestones have been delayed, as detailed below:

• Hawkei (Budget Expended 39 per cent, Time Elapsed 62 per cent) — the project’s achievement of milestones has been delayed by reliability issues, design maturity, and production delays. The project completed its Production Reliability Acceptance Test 24 months later than originally contracted.
• Battlefield Airlifter (Budget Expended 63 per cent, Time Elapsed 94 per cent) — the project anticipates delivering some requirements after the achievement of the FOC milestone, including some training equipment, aircraft upgrades, the Structural Substantiation Program, and the Missile Approach Warning System. Defence is revalidating the business case and execution strategy for this acquisition activity during 2020.

2.7 For the third project, Joint Strike Fighter (Budget Expended 39 per cent, Time Elapsed 75 per cent), the expenditure lag reflects the recent transition from the aircraft development stage, where relatively little budget was expended. The project has now entered into main production contracts for aircraft, with in-year expenditure increasing compared to prior years.

2.8 Where Budget Expended leads Time Elapsed, the project budget may be at risk — i.e. expenditure increases may indicate real cost increases. However, for the three projects where Budget Expended leads Time Elapsed by 10 per cent or more, the cause of the variance does not relate to insufficient project funds, as detailed below:

• Collins Comms and EW (Budget Expended 52 per cent, Time Elapsed 42 per cent) — the project’s contract is structured so that more significant milestone payments fall earlier in the life of the project. The project also expects expenditure to reduce as lessons learned on early installations are applied to later installations.
• Night Fighting Equip Repl (Budget Expended 64 per cent, Time Elapsed 54 per cent) — this project’s FOC date anticipates the purchase of a second tranche of equipment, which has not yet been funded. The currently funded tranche of equipment was largely delivered at 30 June 2020, leading the approved expenditure to exceed the approved schedule.
• ANZAC Air Search Radar Repl (Budget Expended 63 per cent, Time Elapsed 43 per cent) — this project spent approximately nine per cent of its budget prior to Second Pass, to conduct a Risk Reduction Program and make early purchases of equipment to ensure the schedule would be met.¹⁰²

2.9 In each case of significant variance between Budget Expended and Time Elapsed, the performance information highlights projects that may require further attention. This is to ensure that unspent funds are returned to the Defence budget for re-allocation in a timely manner, the timing of key deliverables remains in focus, or planning focuses on bringing together all elements in a timely manner, as equipment is delivered.

Cost performance analysis

Budget Expended and Project Maturity

2.10 Figure 3, below, sets out each project’s Budget Expended against Project Maturity¹⁰³ and shows that Budget Expended lags Project Maturity for the majority of projects (21 of 25). This relationship is typical of acquisition projects for two reasons:

• in an acquisition environment predominantly based on milestone payments, projects will typically develop confidence in delivering their scope through design reviews, testing and demonstration, ahead of formal acceptance of milestone achievement or equipment deliveries (and expenditure of budget); and
• more generally, Budget Expended will often lag Project Maturity as the result of Defence’s project maturity framework attributing approximately 50 per cent (35 out of 70 points) of total Project Maturity at Second Pass Approval (the main investment decision by government) prior to any significant expenditure of budget.

2.11 In both cases, the Budget Expended is expected to catch up to Project Maturity over the course of the project’s life, with projects approaching closure expected to show Budget Expended and Project Maturity broadly in line with each other.

2.12 Budget Expended lags Project Maturity with a variance of 20 per cent or more in 12 projects. As expected, the majority of these projects are at a relatively early stage and have expended minimal budget while progressing through design and testing phases, or are waiting on significant amounts of equipment to be delivered. The exception to this is the Night Fighting Equip Repl project, where most Tranche 1 equipment has been delivered, leading to an advanced maturity score, while the evaluation of future Tranche 2 equipment is still in its early stages.

2.13 Where Budget Expended leads Project Maturity by a significant amount, this may indicate that the project is behind in development or achievement of its scope, or that the required scope is not affordable. There are no instances where Budget Expended leads Project Maturity by 20 per cent or more. The largest variance is for UHF SATCOM, where Budget Expended leads Project Maturity by 11 per cent. The project’s maturity score has been affected by delays in software development, while the majority of budget has been expended and the project has funded further development with contingency.

Figure 3: Budget Expended and Project Maturity

 

Source: ANAO analysis of the 2019–20 PDSSs.

Second Pass Approval and 30 June 2020 approved budget

2.14 Figure 4, below, compares each project’s approved budget at initial Second Pass Approval and its approved budget at 30 June 2020. Six projects have variations of $500 million or more. The list below describes the components of these variations:

• Joint Strike Fighter — net increase of $13.9 billion, comprising $10.5 billion for 58 additional aircraft in 2013–14, $3.0 billion for exchange rate variation and $0.4 billion for price indexation.¹⁰⁴
• AWD Ships — net increase of $1.9 billion, comprising $1.2 billion for a Real Cost Increase¹⁰⁵ in July 2015 to complete the project, $1.2 billion for price indexation, offset by a $0.4 billion decrease for exchange rate variation and a $0.1 billion decrease for transfers to facilities projects in 2013–14.
• P-8A Poseidon — net increase of $1.8 billion, comprising $1.3 billion for four additional aircraft in 2015–16 and $0.5 billion for exchange rate variation.
• MRH90 Helicopters — net increase of $2.8 billion, comprising $2.6 billion for 34 additional aircraft in 2005–06 and other minor scope changes, and $0.7 billion for price indexation, offset by a $0.3 billion decrease due to scope transfers for facilities, and a $0.1 billion decrease for exchange rate variation.
• Growler — net increase of $0.8 billion, comprising $0.9 billion for exchange rate variation, $0.3 billion for the Mobile Threat Training Emitter System and weapons, offset in 2015–‍16 by a $0.2 billion decrease for transfers to facilities projects and $0.2 billion for the return to the Defence budget of surplus funds and contingency for reallocation.
• Overlander Medium/Heavy — net increase of $0.8 billion, comprising $0.7 billion ‘project supplementation’ to reduce cost pressures and $0.1 billion exchange rate variation.

Figure 4: Projects’ initial Second Pass Approval and 30 June 2020 approved budget ($m)

Note 1:  indicates that the budget for the project at 30 June 2020 is less than the original budgeted cost.

Note 2: On 22 May 2015, the Minister for Defence and the Minister for Finance announced there would be further delays to the delivery of the Air Warfare Destroyers (AWD) and an additional $1.2 billion would be required to complete the project. The budget increase was incorporated into the approved project budget as at 30 June 2016.

Source: ANAO analysis of the 2019–20 PDSSs. Previous MPRs have reported that budget variances since initial Second Pass Approval have resulted from: increasing the scope of a project via revised Second Pass Approvals, programmatic decisions, Real Cost Increases/Decreases, transfers to/from other projects and budgetary adjustments. Project budgets may also be affected by price indexation¹⁰⁶ and foreign exchange variation.

2.15 The total budget for the 25 projects at 30 June 2020 was $78.7 billion, a net increase of $24.2 billion, when compared to the approved budget at initial Second Pass Approval of $54.5 billion. A summary of budget variations is at Table 3 (page 13) and a more detailed analysis of budget variations is included in Table 7, below.

Budget performance

2.16 The following figures and tables illustrate the budget performance of the 25 selected projects by way of:

• in-year budget variations by project (see Table 8, below); and
• expenditure forecasting performance against actual expenditure for 2019–20 (see Figure 5, on page 51).

In-year budget variance analysis

2.17 Table 8, below, sets out the in-year budget variations for each project. Overall, the approved budget for the projects as at 30 June 2020 increased by $74.8 million, or 0.1 per cent, compared to their approved budget as at 30 June 2019. This was driven by exchange rate variation increases of $85.5 million and net real decreases of $10.6 million.

2.18 Exchange rate variations result from projects’ exposure to foreign currencies and movements in foreign exchange rates against the Australian dollar.¹⁰⁷ Budget adjustments aim to maintain the relative buying power of the project budget. Movements in the US dollar and the Euro are the main influences. Projects with larger movements in foreign exchange in 2019–20 included:

• Joint Strike Fighter — movement of $108.7 million, or 0.7 per cent increase in budget.
• Combat Recon. Vehicles — movement of -$50.7 million, or 0.9 per cent decrease in budget.

2.19 Real Variations¹⁰⁸ primarily reflect changes in the scope of projects, transfers between projects for approved equipment/capability and budgetary adjustments such as administrative savings decisions. In 2019–20, the two projects with more significant Real Variations were:

• Repl Replenishment Ships — variation of $12.0 million reflecting budget transfers covering additional costs expected in Australian fit out activities, engineering and ILS costs associated with CCPs and additional project support costs to cover the period of delay.
• P-8A Poseidon — variation of -$20.3 million reflecting the return of funds to the Integrated Investment Program.

Table 8: In-year (2019–20) budget variations by project

Note 1: The Total Variance and components for this project do not add up due to rounding differences.

Note 2: The difference between the total approved budgets for 2018–19 and 2019–20 is partly due to the projects entering the MPR in 2019–20 (Future Frigates, Future Subs, Combat Recon. Vehicles, MQ-4C Triton, and Battlefield Command System) not contributing to the total budget figure for 2018–19.

Source: ANAO analysis of the 2018–19 and 2019–20 PDSSs.

In-year forecast and actual expenditure

2.20 Accurately forecasting and managing budget expenditure is an important element in the management of a portfolio of projects. Figure 5, below, sets out the expenditure forecasting performance of each project against actual expenditure in 2019–20. In total, actual expenditure for the 25 projects at 30 June 2020 was $5651.9 million. This is compared against an initial Portfolio Budget Statements (PBS) forecast expenditure of $6621.6 million, a mid-year Portfolio Additional Estimates Statements (PAES) forecast of $6171.7 million, and a final forecast of $6110.2 million (Final Plan, approved during May 2020).

2.21 Figure 5 highlights the following in-year variances occurred in the following projects:

• Joint Strike Fighter (expenditure of $1938.4 million compared to $2388.6 million PBS, $1897.6 million PAES and $1884.6 million Final Plan estimates) – the variation reflects a refinement of the in-year budget based on improved forecast expenditure and schedule information from the US-based Joint Program Office.
• Future Frigates (expenditure of $263.6 million compared to $492.3 million PBS, $372.9 million PAES and $375.2 million Final Plan estimates) – the variation is due to delays against the Head Contract.
• Future Submarines (expenditure of $553.1 million compared to $289.3 million PBS, $580.9 million PAES and $579.5 million Final Plan estimates) — the variation is due to Government approval to enter into the Strategic Partnering Agreement and Submarine Design Contract with Naval Group, as well as reprogramming of contract activities.
• P-8A Poseidon (expenditure of $223.5 million compared to $360.3 million PBS, $301.6 million PAES and $299.8 million Final Plan estimates) – the variation is due to early payment for the final aircraft in 2018–19, and a deliberate approach by Air Force to manage acquisition budget requirements in 2019–20 by delaying a range of payments to future financial years.
• Offshore Patrol Vessel (expenditure of $227.2 million compared to $349.2 million PBS, $249.2 million PAES and $248.9 million Final Plan estimates) – the variation is due to the reprogramming of activities as well as delays in contractual milestones and underspend on Project Office costs and government furnished equipment.
• Repl Replenishment Ships (expenditure of $107.1 million compared to $191.8 million PBS, $196.0 million PAES and $195.3 million Final Plan estimates) – the variation is due to the closure of the naval shipyard in Spain and nationwide as a result of the COVID-19 pandemic, as well as Contract Change Proposals and delays in delivery of the Close In Weapon System.

Figure 5: In-year (2019–20) projects’ forecast expenditure performance compared to actual expenditure ($m)

Sources: ANAO analysis of the 2019–20 PDSSs and Defence Portfolio Budget Statements.

Schedule performance analysis

2.22 Defence data continues to show that schedule performance is a key issue in delivering and sustaining equipment.¹⁰⁹ Project schedule slippage can have the effect of introducing or exacerbating a capability gap, or requiring an extension to the planned withdrawal date for those platforms being replaced.¹¹⁰

Time Elapsed and Project Maturity

2.23 Based on the findings of the Defence Procurement Review 2003¹¹¹, in 2005 Defence began to increase the proportion of MOTS acquisitions, which are generally lower risk projects and therefore more likely to meet schedule timelines. Analysis of the available performance information highlights that the selection of MOTS projects assists in reducing risk during project acquisition, where Project Maturity is more advanced at Second Pass Approval than developmental projects. For example, CMATS is a developmental project that has experienced significant schedule slippage; its maturity score at Second Pass Approval was 31 points, below the expected benchmark of 35 points for projects at Second Pass Approval. In contrast, MH-60R Seahawk is a MOTS project that has not experienced any slippage to date; its maturity score at Second Pass Approval was 37 points.

2.24 Figure 6, below, sets out each project’s Time Elapsed against Project Maturity.¹¹² Time Elapsed lags Project Maturity for 16 of 23¹¹³ projects. Similar to the analysis of Budget Expended and Project Maturity, at paragraphs 2.10 to 2.13, this pattern is expected as projects will generally score 50 per cent of their Project Maturity at Second Pass Approval, when Time Elapsed is zero (for the purposes of the ANAO’s analysis in this report). The lag is most pronounced in MOTS and Australianised MOTS acquisitions, including the Offshore Patrol Vessel, Combat Recon. Vehicles, and Night Fighting Equip Repl projects. The exception is MQ-4C Triton, a developmental project where the lag in Time Elapsed against Project Maturity reflects the project’s extensive schedule to FOC, driven by the aircraft production schedules.

2.25 For the 7 projects where Time Elapsed lags Project Maturity by 20 per cent or more, this generally reflects projects at relatively early stages of acquisition processes, including proceeding through design activities, or awaiting significant amounts of their major equipment to be constructed and delivered. There is one significant exception to this:

• Night Fighting Equip Repl, where this project’s FOC date anticipates the purchase of a second tranche of equipment, which has not yet been approved. The currently approved tranche of equipment is MOTS products that were largely delivered at 30 June 2020, leading the maturity of the approved project scope to exceed the approved schedule.

2.26 For the seven projects where Time Elapsed leads Project Maturity, there were no instances where this difference was significant (20 per cent or more). The greatest variance was for the Battlefield Airlifter project, where Time Elapsed leads Project Maturity by 16 per cent. The project anticipates delivery of some requirements after the achievement of the FOC milestone, including some training equipment, aircraft upgrades, the Structural Substantiation Program, and the Missile Approach Warning System. Defence is revalidating the business case and execution strategy for this acquisition activity during 2020.

Figure 6: Time Elapsed and Project Maturity

Note: Future Frigates and Future Subs are yet to have FOC milestones approved by Government.

Source: ANAO analysis of the 2019–20 PDSSs.

Schedule slippage and acquisition type by approval date

2.27 As noted in paragraphs 35 to 37, the ANAO compares project slippage against the Defence classification of projects as Military Off-The-Shelf (MOTS), Australianised MOTS, or developmental, as these classifications are a general indicator of the difficulty associated with the procurement process. Prima facie, the more developmental in nature a project is, the greater the schedule risk and therefore the greater the need for more robust planning by Defence^114,115.

Project classification

2.28 Table 9 below provides information on the classification of all 54 Major Projects included in the MPR since its inception, and the year of approval (generally second pass) for each Major Project. The table indicates that of the 54 Major Projects:

• 13 projects (24 per cent) were developmental;
• 25 projects (46 per cent) were Australianised MOTS (AMOTS); and
• 16 projects (30 per cent) were MOTS.

Table 9: Project year of approval and acquisition type

2.29 Table 9 (above) and Figure 7 (below) indicate that developmental projects became less common after 2005. Based on the findings of the Defence Procurement Review 2003 (Kinnaird Review)¹¹⁶, in 2005 Defence began to increase the proportion of MOTS acquisitions, which are generally lower risk projects and therefore more likely to meet schedule timelines.¹¹⁷ Table 9 and Figure 7 also indicate a more recent trend, where developmental projects have become more common since 2014. Of the 17 Major Projects which have received government approval since 2014:

• 7 projects (41 per cent) were developmental;
• 6 projects (35 per cent) were Australianised MOTS; and
• 4 projects (24 per cent) were MOTS.

Schedule slippage and acquisition type

2.30 The challenge of gaining a full understanding of the complexities of developmental aspects of projects at Second Pass Approval is evident by the extent of slippage over time. Figure 8, below, illustrates the total schedule slippage¹¹⁸ since Second Pass Approval for 23 of the selected projects in this MPR.¹¹⁹ It also depicts the acquisition type and places projects in order of government approval. Figure 9, below, illustrates the total schedule slippage for the 27 projects that have exited the review.¹²⁰

2.31 Figure 9 shows that the inclusion of MOTS acquisitions contributed, prima facie, to a reduction in schedule slippage in the Major Projects portfolio. For projects that have exited the MPR, MOTS projects report an average of 14 months of slippage per project, while Australianised MOTS projects report an average of 39 months and developmental projects report an average of 97 months. Decisions on whether to undertake developmental projects should be considered on a risk basis.¹²¹ In this context, the consideration of risk should be holistic and weigh up the level of capability to be acquired against potential risks relating to cost and schedule.

2.32 Figures 8 and 9 illustrate that older projects have experienced the most slippage. These projects tended to be more developmental (complex) in nature and typically experienced schedule slippage in the past, and have often continued to do so. This demonstrates an ongoing trend of slippage in historically late projects, which is more pronounced in older projects. This trend is also visible, but less prominent, in newer projects.

2.33 Two recent developmental projects, Hawkei and ANZAC Air Search Radar Repl, are yet to experience slippage to their FOC dates. However, these projects have experienced slippage to design reviews, test programs, or material release milestones. In the case of Hawkei, there was 24 months slippage to the Production Reliability Acceptance Test, leading to 17 months slippage to Initial Materiel Release, which was declared in May 2020 with four caveats.¹²² In contrast, recent MOTS projects, Pacific Patrol Boat Repl and Night Fighting Equip Repl, have adhered more closely to their design and materiel release schedules with only minor variances. This indicates that although developmental projects currently in the MPR are not reporting significantly more slippage to FOC than MOTS projects, developmental projects still carry a higher level of technical risk.¹²³

2.34 While it is not possible to predict the full extent of slippage a project will experience, Figure 9 analysis has been provided to highlight changes since the Kinnaird Review. Fifteen post Kinnaird and 12 pre Kinnaird projects have exited the MPR. Total slippage of the 15 post Kinnaird projects is 24.8 years. Total slippage of the 12 pre Kinnaird projects is 70.7 years. One of the 15 post Kinnaird projects was a developmental acquisition and four of the 12 pre Kinnaird acquisitions were developmental.

Schedule performance

2.35 The figures and tables that follow illustrate:

• the original and 30 June 2020 forecasts for achieving FOC;
• in-year schedule changes to achieving FOC; and
• total schedule slippage across the Major Projects.

Original and 30 June 2020 Final Operational Capability forecasts

2.36 Figure 10, below, presents information on the selected projects’ original and 30 June 2020 forecasts for achieving FOC. The total schedule slippage¹²⁴ for the 25 Major Projects to date is 507 months compared to the initial prediction when approved by government. This represents a 21 per cent increase on the approved schedule. Of the 25 projects in the 2019–20 report, 14 have experienced schedule slippage and two do not have FOC dates approved by Government. A further three projects have experienced delays from earlier FOC forecasts, but have not slipped past the original FOC date approved by Government (Joint Strike Fighter, Overlander Medium/Heavy, and Repl Replenishment Ships).

2.37 Total schedule slippage across the Major Projects was 507 months in 2019–20. This is 144 months lower than the figure of 651 months reported for 2018–19 in the MPR. The difference is mainly due to the exit of projects with significant slippage — including ANZAC ASMD 2B, LHD Landing Craft, Battle Comm. Sys. (Land) 2A, and LHD Ships — which reduced the total accumulated slippage by 218 months. This was offset by in-year slippage for MQ-4C Triton (associated with changes in the planned aircraft delivery schedule) and Battlefield Airlifter (with Defence revalidating the business case for the delivery of this project’s remaining scope). These two projects, combined, added 55 of the 68 months schedule slippage experienced in 2019–20. Additionally, Battlefield Command System added 12 months of slippage to the total of 507 months; the slippage occurred prior to its inclusion in the MPR in 2019–20.

2.38 The reasons for schedule slippage often require a deep understanding of project technical elements and a realistic assessment of the capacity of the private sector to deliver in the expected timeframe. A project office’s ability to gain access to the platform for upgrading can also result in schedule delay (for example, Maritime Comms and Collins R&S).¹²⁵

2.39 A closer examination of the reasons for schedule slippage demonstrates the importance of initial assessments of project complexity. Past experience indicates that a key factor is whether a project is MOTS, Australianised MOTS or developmental¹²⁶, as discussed at paragraphs 2.30–2.34. One project, MRH90 Helicopters¹²⁷, was originally misclassified as MOTS. The project was reclassified by Defence to Australianised MOTS (i.e. more developmental) subsequent to Second Pass Approval.¹²⁸ This project has experienced extended schedule slippage. Another project, UHF SATCOM, is still classified as MOTS but includes the development of significant amounts of software. Delays in software development have led to 42 months of slippage to the FOC milestone.

2.40 Figure 10 indicates that no projects are currently forecasting an FOC date earlier than originally approved. However, a number of projects have experienced both schedule recovery and schedule delay that has offset that schedule recovery. Projects fitting this pattern are Joint Strike Fighter, AWD Ships, Growler, Overlander Medium/Heavy, Repl Replenishment Ships, Collins Comms and EW, and Collins R&S. In the case of Joint Strike Fighter, Overlander Medium/Heavy, and Repl Replenishment Ships, their schedule recovery and schedule delay is equal such that all three projects are currently forecasting the achievement of FOC as originally approved by Government. In total, these seven projects have contributed 36 months of schedule recovery to the Major Projects. Previously, the ANAO’s analysis (for example in Table 2 and Figure 8) has excluded this effect so as to report on the total slippage experienced by the Major Projects. This year the analysis has been adjusted to exclude delays to a project’s recovered schedule that do not result in slippage past the original government approved date.

Figure 10: Projects’ original and 30 June 2020 FOC forecasts

Source: ANAO analysis of the 2019–20 PDSSs.

In-year schedule performance

2.41 In 2019–20, there was schedule slippage of 68 months in the forecast achievement of FOC across the 23 Major Projects¹²⁹, as shown in Figure 11 below. In-year project performance, measured by slippage over the last 12 months, may not reflect the project trend.

2.42 In-year schedule slippage occurred for the following 5 projects¹³⁰ (the explanation provided, drawn from the 2019–20 PDSSs, may also include the reasons for prior slippage):

• Battlefield Airlifter — FOC has been delayed while Defence re-evaluates the business case for delivery of this project’s remaining capability.
• MQ-4C Triton— the variance reflects the alignment of the project’s FOC schedule with the aircraft production schedule.
• CMATS — the variance reflects the incorporation of additional system automation requirements.
• Battlefield Command System — the variance reflects the time taken to establish platform integration contracts, availability of Government Furnished Materiel, delays to materiel and data from interdependent projects, and contractor performance.
• Maritime Comms — the variance reflects alignment of the project schedule with the maintenance schedule for the Anzac Class frigates.

2.43 Further, the Joint Strike Fighter project saw a delay of two months from the forecast reported in 2018–19. This has not been included as slippage in the number above or in Figure 11 below, as this delay did not see the project slip past the FOC date originally approved by Government. Rather, this project was previously forecasting an earlier date of achievement compared to the original approval.

2.44 Additionally, one project shows recovery of previously reported slippage:

• Collins Comms and EW — this project’s schedule is aligned with the docking schedule for the Collins Class submarines.

Figure 11: In-year (2019–20) schedule slippage and recovery

Note: Defence’s PDSSs indicate that 16 of the 25 Major Projects did not record changes to their FOC dates this year. The Joint Strike Fighter project changed its FOC date but without slipping past the date originally approved by Government, and has not been included in this figure. Future Frigates and Future Subs do not yet have FOC dates approved by Government.

Source: ANAO analysis of the 2019–20 PDSSs.

2.45 Project delays may indicate unanticipated problems with project progress or optimism in previous forecasting, regardless of whether the delay makes the project later than originally approved by Government. For example, in 2018–19 the Repl Replenishment Ships project delayed its FOC forecast by seven months based on integrated planning producing a more mature understanding of FOC activities, though the later FOC forecast remained within the window originally approved by Government. All delays should be monitored to ensure that a project remains on track and any issues can be managed.

Schedule performance by year of entry to MPR

2.46 Table 10, below, shows the accumulated schedule slippage of the Major Projects included in the 2019–20 MPR.¹³¹ The table shows that over half the total schedule slippage across the Major Projects covered in the 2019–20 MPR (42.3 years or 507 months) comprises slippage from the four oldest projects, approved prior to 2010.

2.47 Table 10 also shows that 21 per cent (108 of 507 months) of the total schedule slippage across the 2019–20 Major Projects is attributed to the sole remaining project approved prior to the Kinnaird reforms, Collins R&S.

Table 10: Project slippage

Note: The Future Frigates and Future Subs projects are not included as they do not yet have FOC dates approved by Government.

Source: ANAO analysis of the 2019–20 PDSSs.

Capability performance analysis

2.48 Defence defines capability as the power to achieve a desired operational effect in a nominated environment, within a specified time, and to sustain that effect for a designated period.¹³² An operational effect is achieved by combining the nine Fundamental Inputs to Capability — organisation, command and management, personnel, collective training, major systems, facilities and training areas, supplies, support, and industry¹³³ — and undertaking designated operations.

2.49 In acquiring Defence platforms and systems, a range of documentation (including capability definition, operational concept, function and performance specification, and Test and Evaluation Master Plans) is developed, which establishes the detailed requirements/performance attributes to be achieved.

2.50 The Defence PDSSs report that 19 projects in this year’s MPR will deliver all of their key capability requirements. Defence’s assessment indicates that some elements of the capability required may be ‘under threat’, but the risk is assessed as ‘manageable’. The five project offices experiencing challenges with expected capability delivery (2018–19: five) are Joint Strike Fighter, MRH90 Helicopters, Hawkei, Battlefield Command System and Battlefield Airlifter. One project office (AWD Ships) reports that it is unable to deliver all of the required capability by FOC.

2.51 Since the 2009–10 MPR, capability reporting¹³⁴ has been based on Defence’s prediction of the final capability that would be achieved on the basis of deliverables and/or activities completed. This assessment of capability performance (Expected Capability) is measured against the Materiel Release Milestones (MRMs) and Completion Criteria specified in each project’s Materiel Acquisition Agreement (MAA). This is distinct from an assessment of whether milestones will be achieved on schedule. As the ANAO has previously noted, this data involves making certain assumptions in forecasting achievements and is therefore subjective in approach.¹³⁵

2.52 For example, the Battlefield Airlifter project reported a 100 per cent Green capability prediction at its inclusion in the MPR in 2013–14. However, the 2013–14 PDSS also reported major risks relating to capability deficiencies arising from the US Government divesting from the program, with Australia no longer able to rely on the US Air Force processes. These risks have continued to affect the project, with a mature training system and a number of baseline capability requirements now not expected to be delivered until after FOC. These capability issues were reported in the PDSS Pie Chart for the first time in 2018–19, indicating that the earlier level of confidence in the ability to achieve the required capability may have been overly optimistic.¹³⁶

2.53 The inherent subjectivity of the capability prediction is also apparent through a comparison of a platform’s predicted hours in service (sometimes referred to as ‘rate of effort’) and the actual hours achieved. For the 2019–20 financial year, Defence predicted that the MH-60R Seahawk helicopters would achieve 7200 flying hours.¹³⁷ In October 2020 Defence reported an estimated actual achievement of 4900 flying hours¹³⁸, indicating that the platform flew for only 68 per cent of the time expected by Defence. The PDSS for the MH-60R Seahawk acquisition project reports that 100 per cent of capability is expected to be achieved. Similar discrepancies between the prediction of capability to be delivered by the acquisition project, and the actual performance of platforms in service, can be seen for other Major Projects, including the Joint Strike Fighter project (100 per cent of capability expected to be delivered with 1 per cent at risk but considered manageable, 68 per cent of planned flying hours achieved in 2019–20) and the Growler project (100 per cent of capability expected to be delivered, 70 per cent of planned flying hours achieved in 2019–20). This comparison further demonstrates that Defence’s capability forecasts may be optimistic.

2.54 Over time, the JCPAA has sought the use of a more robust measure of capability performance.¹³⁹

2.55 In October 2017, the JCPAA recommended that Defence ‘review the procedure for the development of expected capability estimates for future Major Projects Reports. The outcomes of this review should be provided to the Committee within six months of the tabling of this report. Further, the Committee requests that Defence provide a progress report within three months of the tabling of this report.’¹⁴⁰

2.56 Defence made a submission to the Committee in March 2018 regarding this recommendation, which advised that:

Defence will conduct a schedule baseline validation activity for the Major Projects Report projects to drive greater consistency in schedule reporting.

Once this activity is complete, Defence should be in a better position to investigate a more robust approach to measuring Capability estimates. Utilising the validated baseline data could inform:

• A simple percentage of schedule milestones achieved to measure progress to date. This is a quantitative assessment that relies on the maintenance of a robust project baseline, which is not dissimilar to the approach proposed by ANAO previously;
• CASG working with Force Design to identify how to measure capability, that considers all elements of Fundamental Inputs to Capability, and that is suitable for unclassified publication; and
• Defence is working towards a new whole of organisational reporting system (the Enterprise Resource Planning (ERP) System) which is expected to roll-out in Financial Year 2020-21. CASG will endeavour to incorporate the work conducted with Force Design on measuring capability.¹⁴¹

2.57 In September 2018, the JCPAA noted that ‘Materiel Capability Delivery Performance charts continue to be ambiguous in displaying actual current capability levels.’¹⁴²

2.58 Defence advised the ANAO in November 2018 that partial progress had been made on its schedule baseline validation activity discussed in paragraph 2.56. The ANAO notes that a measurement of schedule milestones will not necessarily reflect a measurement of capability delivered.

2.59 The Deputy Secretary CASG advised the JCPAA in a public hearing on 27 May 2020 that:

I acknowledge the issues of the National Audit Office and would like to work with them, as we indicated in our submission, by perhaps reviewing the report and the way in which we articulate the information.¹⁴³

Part 2. Defence Major Projects Report is available to download in a separate PDF file, or in the complete report PDF which is available to download at Related documents on this page.

.

Due to the complex nature of the material, the Project Data Summary Sheets (PDSS) are available in PDF only. You can view the PDSS section in the complete report PDF which is available to download at Related documents on this page, or download individual PDSS PDF files below:

• New Air Combat Capability
• Air Warfare Destroyer Build
• Future Frigates
• Future Submarines Design Acquisition
• Combat Reconnaissance Vehicles
• Maritime Patrol and Response Aircraft System
• Multi-Role Helicopter
• Offshore Patrol Vessel
• EA-18G Growler Airborne Electronic Attack Capability
• Medium Heavy Capability, Field Vehicles, Modules and Trailers
• Future Naval Aviation Combat System Helicopter
• Protected Mobility Vehicle – Light (PMV-L)
• Battlefield Airlift – Caribou Replacement
• MQ-4C Triton Remotely Piloted Aircraft System
• Maritime Operational Support Capability
• Civil Military Air Traffic Management System
• Battlefield Command System
• Battlespace Communications Systems 2B
• Collins Class Communications and Electronic Warfare Improvement Program
• Pacific Patrol Boat Replacement
• Night Fighting Equipment Replacement
• Collins Class Submarine Reliability and Sustainability
• Maritime Communications Modernisation
• ANZAC Air Search Radar Replacement
• Indian Ocean Region UHF SATCOM

The JCPAA 2019–20 Major Projects Report Guidelines part is available to download in a separate PDF file, or in the complete report PDF which is available to download at Related documents on this page.