Due to the complexity of material and the multiple sources of information for the 2018–19 Major Projects Report, we are unable to represent the entire document in HTML. You can download the full report in PDF or view selected sections in HTML below. PDF files for individual Project Data Summary Sheets (PDSS) are also available for download.

.

About the Major Projects Report

1. Major Defence equipment acquisition projects (Major Projects) continue to be the subject of parliamentary and public interest. This is due to their high cost and contribution to national security, and the challenges involved in completing them within the specified budget and schedule, and to the required capability.

2. The Australian National Audit Office (ANAO) has reviewed 26 of Defence’s Major Projects in this twelfth annual report (2017–18: 26). The Major Projects Report (MPR) reviews overall issues, risks, challenges and complexities affecting Major Projects and also reviews the status of each of the selected Major Projects, in terms of cost, schedule and forecast capability.¹ The objective of the report is ‘to improve the accountability and transparency of Defence acquisitions for the benefit of Parliament and other stakeholders.’²

3. The Capability Acquisition and Sustainment Group (CASG) within the Department of Defence (Defence), manages the process of bringing new capabilities into service.³ As at 30 June 2019, Defence was managing 205 active major and minor capital equipment projects worth $132 billion, with an in-year budget of $8.6 billion.⁴ Defence capitalised some $8.6 billion from these projects in 2018–19.⁵

4. The February 2016 Defence White Paper established the Australian Government’s priorities for future capability investment for the next 20 years and provided for additional spending of over $29 billion across the next decade. The 2019–20 Defence Portfolio Budget Statements indicated that the Defence budget would grow to approximately $200 billion over the coming decade, for investing in Defence capability.⁶ The Government commenced its $89 billion investment in Australia’s future shipbuilding industry in April 2017⁷, and on 29 June 2018 announced Second Pass Approval⁸ of the $35 billion Future Frigate program.⁹ Further, on 11 February 2019, the Government announced the signing of the Strategic Partnership Agreement¹⁰ for the $50 billion Future Submarine program.¹¹

Major Projects selected for review

5. Major Projects are selected for review based on the criteria included in the 2018–19 Major Projects Report Guidelines (the Guidelines), as endorsed by the Joint Committee of Public Accounts and Audit (JCPAA).¹² They represent a selection of the most significant Major Projects managed by Defence.

6. The total approved budget for the Major Projects included in this report is approximately $64.1 billion, covering 49 per cent of the total budget of active major and minor capital equipment projects of $132 billion.¹³ The selected projects are listed in Table 1.

Table 1: 2018–19 MPR projects and approved budgets at 30 June 2019 ^{1, 2}

Note 1: Once a project is selected for review, it remains within the portfolio of projects under review until the JCPAA endorses its removal, normally once it has met the capability requirements of Defence.

Note 2: SEA 1180 Phase 1 Offshore Patrol Vessel, SEA 1439 Phase 5B2 Collins Class Communications and Electronic Warfare Improvement Program, LAND 53 Phase 1BR Night Fighting Equipment Replacement and SEA 1448 Phase 4B ANZAC Air Search Radar Replacement are included in the MPR for the first time in 2018–‍19.

Note 3: These projects have been the subject of individual performance audits. See Table 8 for more information.

Note 4: SEA 1439 Phase 3 Collins Class Submarine Reliability and Sustainability is a group of 24 activities primarily sustainment in nature. While not an acquisition project, it has been included at the JCPAA’s request.

Source: The Project Data Summary Sheets in Part 3 of this report.

7. In September 2019, the JCPAA endorsed project selection for the 2019–20 MPR, including the entry of five new projects.¹⁴ The ANAO advised the JCPAA that it proposed to consider the exit of seven projects from the 2018–19 MPR that were expected to reach FOC by the end of 2019 and/or to confirm that only low risk deliverables were remaining.¹⁵ Six of these projects are now considered suitable to exit on the basis that they:

• have achieved FOC (SEA 1448 Phase 2B ANZAC ASMD 2B);
• will achieve FOC in December 2019 (JP 2072 Phase 2A Battle Comm. Sys. (Land) 2A); or
• have provided a pre-FOC risk assessment supporting the timely declaration of FOC (JP 2048 Phase 4A/4B LHD Ships, AIR 7403 Phase 3 Additional MRTT, JP 9000 Phase 7 HATS, and JP 2048 Phase 3 LHD Landing Craft).

8. In the case of one project, AIR 8000 Phase 2 Battlefield Airlifter, the project’s FOC has been delayed and it is not considered to be suitable for exit yet (see paragraph 2.54).

Report objective and scope

9. The objective of this report is to provide the Auditor-General’s independent assurance over the status of the selected Major Projects. The status of the selected Major Projects is reported in the Statement by the Secretary of Defence (see Part 3 of this report) and the Project Data Summary Sheets (PDSSs) prepared by Defence (see Part 3 of this report). Assurance from the ANAO’s review is conveyed in the Independent Assurance Report by the Auditor-General (see Part 3 of this report).

10. The following forecast information found in the PDSSs is excluded from the scope of the ANAO’s review:

• Section 1.2 Current Status—Materiel Capability Delivery Performance and Section 4.1 Measures of Materiel Capability Delivery Performance;
• Section 1.3 Project Context—Major Risks and Issues and Section 5 – Major Risks and Issues; and
• forecast dates where included in each PDSS.

Accordingly, the Independent Assurance Report by the Auditor-General does not provide any assurance in relation to this information. However, material inconsistencies identified in relation to this information are required to be considered in forming the conclusion.

11. The exclusions to the scope of the review, noted above, are due to a lack of Defence systems from which to provide complete and accurate evidence¹⁶ in a sufficiently timely manner to facilitate the review. This has been an area of focus of the JCPAA over a number of years¹⁷, and it is intended that all components of the PDSSs will eventually be included within the scope of the ANAO’s review.

12. Separate to the formal review, the ANAO has undertaken an analysis of key elements of the PDSSs — including cost, schedule, progress towards delivery of required capability, project maturity, and risks and issues. Longitudinal analysis across these key elements of projects has also been undertaken.

13. Defence provides further insights and context in its commentary and analysis. This commentary and analysis is not included within the scope of the ANAO’s review.

Review methodology

14. The ANAO has reviewed the PDSSs prepared by Defence as a priority assurance review under subsection 19A(5) of the Auditor-General Act 1997. The criteria to conduct the review are provided by the Guidelines approved by the JCPAA, and include whether Defence has procedures in place designed to ensure that project information and data was recorded in a complete and accurate manner, for all 26 projects.

15. The review included an assessment of Defence’s systems and controls, including the governance and oversight in place, to ensure appropriate project management. The ANAO also sought representations and confirmations from Defence senior management and industry in relation to the status of the Major Projects in this report.

Report structure

16. The report is organised into four parts:

• Part 1 comprises the ANAO’s review and analysis (pp. 1–66);
• Part 2 comprises Defence’s commentary, analysis and appendices (not included within the scope of the Independent Assurance Report by the Auditor-General) (pp. 67–122);
• Part 3 incorporates the Independent Assurance Report by the Auditor-General, the Statement by the Secretary of Defence, and the PDSSs prepared by Defence as part of the assurance review process (pp. 123–398); and
• Part 4 reproduces the 2018–19 Major Projects Report Guidelines endorsed by the JCPAA, which provide the criteria for the compilation of the PDSSs by Defence and the ANAO’s review (pp. 399–426).

Figure 1, below, depicts the four parts of this report.

Figure 1: 2018–19 Report structure

 

 

Note: To assist in conducting inter-report analysis, the presentation of data in the PDSSs remains largely consistent and comparable with the 2017–18 MPR.

Project Data Summary Sheets

17. The PDSSs include unclassified information on project performance, prepared by Defence.¹⁸ As projects appear in the MPR for multiple years, changes to the PDSS from the previous year are depicted in bold orange text in the PDSS.

18. Each PDSS comprises:

• Project Header: including name; capability and acquisition type; Capability Manager; approval dates; total approved and in-year budgets; stage; complexity; and an image;
• Section 1—Project Summary: including description; current status including financial assurance and contingency statement; and context, including background, uniqueness, major risks and issues, and other current sub-projects;
• Section 2—Financial Performance: including budgets and expenditure; variances; and major contracts in place (in addition to quantities delivered as at 30 June 2019);
• Section 3—Schedule Performance: providing information on design development; test and evaluation; and forecasts and achievements against key project milestones, including Initial Materiel Release (IMR), Final Materiel Release (FMR)¹⁹, Initial Operational Capability (IOC) and Final Operational Capability (FOC)²⁰;
• Section 4—Materiel Capability Delivery Performance: provides a summary of Defence’s assessment of its expected delivery of key capabilities, the extent to which milestones were achieved (particularly where caveats are placed on the Capability Manager’s declaration of significant milestones), and a description of the constitution of each key milestone;
• Section 5—Major Risks and Issues: outlines the major risks and issues of the project and remedial actions undertaken for each;
• Section 6—Project Maturity: provides a summary of the project’s maturity, as defined by Defence²¹, and a comparison against the benchmark score;
• Section 7—Lessons Learned: outlines the key lessons that have been learned at the project level (further information on lessons learned by Defence are included in Defence’s Appendix 2 in Part 2 of this report); and
• Section 8—Project Line Management: details current project management responsibilities within Defence.

Overall outcomes

Statement by the Secretary of Defence

19. The Statement by the Secretary of Defence was signed on 10 December 2019. The Acting Secretary’s statement provides her opinion that the PDSSs for the 26 selected projects ‘comply in all material respects with the Guidelines and reflect the status of the projects as at 30 June 2019’.

20. In addition, the Statement by the Secretary of Defence details significant events occurring post 30 June 2019, which materially impact the projects included in the report, and which should be read in conjunction with the individual PDSSs. These include: Joint Strike Fighter, P-8A Poseidon, MRH90 Helicopters, Offshore Patrol Vessel, Growler, LHD Ships, Hawkei, Battlefield Airlifter, Repl Replenishment Ships, CMATS, Additional MRTT, Collins Comms and EW, Night Fighting Equip Repl, Maritime Comms, ANZAC Air Search Radar Repl, UHF SATCOM, and LHD Landing Craft.

21. The 2018–19 MPR Guidelines require Defence to report in the Statement by the Secretary of Defence on projects which have been removed from the MPR which still have outstanding caveats. Defence has not reported any outstanding caveats in the 2018–19 Statement by the Secretary of Defence.

Conclusion by the Auditor-General

22. The Auditor-General has concluded in the Independent Assurance Report for 2018–19 that ‘nothing has come to my attention that causes me to believe that the information in the 26 Project Data Summary Sheets in Part 3 (PDSSs) and the Statement by the Secretary of Defence, excluding the forecast information, has not been prepared in all material respects in accordance with the 2018–19 Major Projects Report Guidelines (the Guidelines), as endorsed by the Joint Committee of Public Accounts and Audit.’

23. Additionally, in 2018–19, a number of observations were made in the course of the ANAO’s review, as summarised below:

• Defence is unable to provide point-in-time data on project personnel numbers and costs. See further explanation in paragraphs 1.53 to 1.55;
• instances of a lack of oversight, non-compliance with corporate guidance and the use of spreadsheets in the management of risks and issues (Section 5 of the PDSS). See further explanation in paragraphs 1.56 to 1.62²²;
• outdated policy guidance for the project maturity framework (Section 6 of the PDSS). See further explanation in paragraphs 1.63 to 1.67²³; and
• an increase in the number of MPR projects which have achieved significant milestones with caveats. See further explanation in paragraphs 1.68 to 1.73.

ANAO’s analysis of project performance

24. As discussed, the ANAO has undertaken an analysis of key elements of the Defence PDSSs — cost, schedule, progress towards delivery of required capability, project maturity, risks and issues, and longitudinal analysis across these key elements of projects. Table 2 provides: summary data on Defence’s progress toward delivering the capabilities for the Major Projects covered in this report; and compares current data against that reported in previous editions of the MPR. This section also contains a summary analysis of the three principal components of project performance: cost, schedule and capability.

Table 2:  Summary longitudinal analysis¹

Refer to paragraphs 24 to 39 in Part 1 of this report.

Note 1: The data for the 26 Major Projects in the 2018–19 MPR compares the data from projects in the 2017–18 MPR and 2016–17 MPR. The Major Projects included within each MPR are based on entry and exit criteria in the Guidelines, which have been included in Part 4 of this report. The entry and exit of projects should be considered when comparing data across years.

Note 2: Where a project has multiple Second Pass Approvals (see footnote 8), the MPR has historically reported budget variations from the initial Second Pass Approval. The figures in this row are consistent with prior year reporting. See Table 3 for a breakdown of the major components of this variance, and Table 8 for all real variations.

Note 3: In the 2017–18 and 2018–19 PDSSs, where a project has multiple Second Pass Approvals, the budget at Second Pass Approval reported in the Header refers to the total budget as at the final Second Pass Approval. The figures in this row use this methodology.

Note 4: Slippage refers to the difference between the original government approved date and the current forecast date. Slippage can occur due to late delivery, increases in scope or at times can be a deliberate management decision. These figures exclude schedule reductions over the life of the project. However, paragraph 2.43 reports total schedule reductions over the life of the projects.

Note 5: Based on the 25 repeat projects from the 2015–16 MPR plus one new project (CMATS) that had slippage in 2016–17, 23 repeat projects from the 2016–17 MPR, and 22 repeat projects from the 2017–18 MPR respectively.

Note 6: The figures represent the total of the reported maturity scores divided by the total benchmark maturity score, in the PDSSs across all projects.

Note 7: The grey section of the table is excluded from the scope of the ANAO’s priority assurance review, due to a lack of systems from which to obtain complete and accurate evidence in a sufficiently timely manner to facilitate the review.

Note 8: The figures represent the combined number of open high and extreme risks and issues reported in the PDSSs across all projects. Risks and issues may be aggregated at a strategic level.

Note 9: Defence advised in this year that Joint Strike Fighter would not deliver one element of capability at FOC (which equated to approximately one per cent). However, across all the Major Projects this percentage rounded to zero per cent.

Note 10: Defence advised in this year that AWD Ships would not deliver one element of capability at FOC (which equated to approximately one per cent). However, across all the Major Projects this percentage rounded to zero per cent.

Cost

25. Cost management is an ongoing process in Defence’s administration of the Major Projects. While all projects reported that they could continue to operate within the total approved budget of $64.1 billion, MRH90 Helicopters, Repl Replenishment Ships, and Battle Comm. Sys (Land) 2B were required to draw upon contingency funds to complete project activities.

26. The approved budget for Major Projects included in this MPR has increased by $24.4 billion (38.0 per cent) since initial Second Pass Approval. Budget variations greater than $500 million are detailed in Table 3, below.²⁴ However, as the MPR predominantly focusses on the approved capital budget for acquisition, the ongoing costs of Project Offices²⁵, training, replacement capability, etc., are not reported here.

Table 3: Budget variation over $500m post initial Second Pass Approval by variation type ^{1, 2}

Note 1: For the variations related to all projects and values refer to Table 8 of this report. For the breakdown of in-year variation, refer to Table 9 of this report.

Note 2: For projects with multiple Second Pass Approvals, this table shows variations from the initial approval.

Note 3: Defence has advised that ‘project supplementation’ is a unique term used to describe the approvals history of this project as follows: ‘The original amount of $2549.2, was the Government decision to split Phase 3 into Phase 3A and 3B. In 2011, Government approved Second Pass approval of Phase 3A and the ‘Interim Pass’ Government approval for Phase 3B. The decision to grant Phase 3B ‘Interim Pass’ was to allow greater bargaining power for Defence while negotiating Phase 3A. Phase 3B was always going to return to Government for formal Second Pass approval, which occurred in July 2013, once contract negotiations were complete.’

Note 4: Prior to 1 July 2010, projects were periodically supplemented for price indexation, whereas the allocation for price indexation is now provided for on an out-turned basis at Second Pass Approval.

Source: ANAO analysis of the 2018–19 PDSSs.

Schedule

27. Delivering Major Projects on schedule continues to present challenges for Defence, affecting when the capability is made available for operational release and deployment by the Australian Defence Force, as well as the cost of delivery.

28. The total schedule slippage for the 26 selected Major Projects, as at 30 June 2019, is 691 months when compared to the initial schedule.²⁶ This represents a 27 per cent increase since Second Pass Approval. Across MPR projects that have experienced slippage (21 of 26 projects), the average slippage is 32.9 months (2.7 years). Table 4 below includes details of in-year and total schedule slippage by project. The table shows an increase of 108 months of in-year slippage during 2018–19.

29. The total slippage of 691 months in 2018–19 is 110 months lower than the total in 2017–18 of 801 months. This is due to:

• the removal of completed projects (Collins RCS, Hw Torpedo, ANZAC ASMD 2A and Battle Management System) removing 254 months of slippage from the total reported in 2017–18 (see Table 5);
• the addition of 108 months of in-year slippage described above; and
• the Collins Comms and EW project adding 36 months of slippage to the total of 691 months; the slippage occurred in 2016–17 but the project was reported in the MPR for the first time in 2018–19.

Table 4: Schedule slippage from original planned Final Operational Capability ¹

Note 1: Refer to footnote 25.

Note 2: These projects have been identified by Defence as Projects of Interest (see paragraph 1.23 in Part 1).

Note 3: Slippage for this project is reflective of movement from a forecast of May 2022 in 2017–18, to a forecast of December 2022 in 2018–19. While the forecast of December 2022 remains within the window for FOC achievement approved by Government, the previous forecast of May 2022 was approved by this project’s line management. See paragraph 2.45.

Source: ANAO analysis of the 2018–19 PDSSs.

30. Platform availability has contributed to the slippage experienced within some projects. For example, Maritime Comms and Collins R&S have been impacted by changes to docking schedules of the Anzac Class frigates and Collins Class submarines respectively. Significant delays have also been experienced by those projects with the most developmental content: AWD Ships, MRH90 Helicopters, CMATS and Battle Comm. Sys. (Land) 2B. Additionally, delays to operational test and evaluation activities have led to delays to the LHD Ships and LHD Landing Craft projects.

31. Table 5, below, provides details of total schedule slippage by project, for projects which have exited the MPR. Compared to the 691 months total schedule slippage for the current 26 Major Projects, the 22 projects which have exited the MPR have reported accumulated schedule slippage of 953 months, as at their respective exit dates. Table 5 indicates that schedule slippage for projects which have exited the MPR was more pronounced in projects with the most developmental content.

Table 5: Schedule slippage for projects which have exited the MPR ¹

Note 1: The Hornet Refurb and Battle Management System (BMS) projects are not included in this table as they did not have FOC milestones.

Note 2: Next Gen Satellite shows slippage in Figure 8, which related to the final capability milestones at the time. By the time it reached FOC, a new final capability milestone had been introduced and slippage was reduced.

Source: PDSSs in published Major Projects Reports and ANAO analysis.

32. Additional ANAO analysis (refer to Figure 7, on page 54) has compared project slippage against the Defence classification of projects as Military Off-The-Shelf (MOTS), Australianised MOTS or developmental. These classifications are a general indicator of the difficulty associated with the procurement process.

33. Figure 8 (on page 55) provides analysis of projects either completed, or removed from the MPR review, and shows that a focus on MOTS²⁷ acquisitions has assisted in reducing schedule slippage. Prima facie, the more developmental in nature a project is, the more likely it will result in a greater degree of project slippage. Figure 8 was requested by the JCPAA in May 2014.²⁸

34. Longitudinal analysis indicates that while the reasons for schedule slippage vary, it primarily reflects the underestimation of both the scope and complexity of work, particularly for Australianised MOTS and developmental projects (see paragraphs 2.31 to 2.37).

Capability

35. The third principal component of project performance examined in this report is progress towards the delivery of capability required by government. While the assessment of expected capability delivery by Defence is outside the scope of the Auditor-General’s formal review conclusion, it is included in the analysis to provide an overall perspective of the three principal components of project performance.

36. The Defence PDSSs report that 20 projects in this year’s report will deliver all of their key capability requirements. Defence’s assessment indicates that some elements of the capability required may be ‘under threat’, but the risk is assessed as ‘manageable’. The five project offices experiencing challenges with expected capability delivery (2017–18: three) are Joint Strike Fighter, MRH90 Helicopters, Hawkei, Battlefield Airlifter and LHD Landing Craft. One project office (AWD Ships) reports that it is unable to deliver all of the required capability by FOC.

37. Defence’s presentation of capability delivery performance in the PDSSs is a forecast and therefore has an element of uncertainty. In 2015–16, the ANAO developed an additional measure of the status of current capability delivery progress to assist the Parliament — Capability Delivery Progress — which is a tally of the capability delivered as at 30 June 2019, as reported by Defence. Table 6 below provides a worked example of the ANAO’s methodology, utilising the performance information provided in the relevant PDSS.

Table 6: Capability Delivery Progress assessment — Additional MRTT (multi-role tanker transport)

Source: PDSSs in published Major Projects Reports and ANAO analysis.

38. Table 7 below, summarises expected capability delivery as at 30 June 2019 — as reported by Defence and using the ANAO’s Capability Delivery Progress measure.

39. In addition to reporting on expected capability delivery, Defence has continued the practice of including in the PDSSs declassified information on settlement actions for projects, including stop payments and liquidated damages. During 2018–19, Hawkei, Pacific Patrol Boat Repl, and UHF SATCOM had negotiated contractual settlements involving stop payments or liquidated damages. Prior settlements for projects within this report related to MRH90 Helicopters, LHD Ships and Maritime Comms.

1.1 This chapter provides an overview of the scope and approach adopted by the Australian National Audit Office (ANAO) for the review of the 26 Project Data Summary Sheets (PDSSs) prepared by the Department of Defence (Defence). This chapter also provides the results of the Major Projects Report (MPR) review.

Review scope and approach

1.2 In 2012 the Joint Committee of Public Accounts and Audit (JCPAA) identified the review of the PDSSs as a priority assurance review, under subsection 19A(5) of the Auditor-General Act 1997 (the Act). This provided the ANAO with full access to the information gathering powers under the Act. The ANAO’s review of the individual project PDSSs, which are reproduced in Part 3 of this report, was conducted in accordance with the auditing standards set by the Auditor-General under section 24 of the Auditor-General Act 1997 through its incorporation of the Australian Standard on Assurance Engagements (ASAE) 3000 Assurance Engagements Other than Audits or Reviews of Historical Financial Information, issued by the Australian Auditing and Assurance Standards Board.

1.3 The following forecast information is excluded from the scope of the ANAO’s review: capability delivery, risks and issues, and forecast dates. These exclusions are due to the lack of Defence systems from which to provide complete and accurate evidence²⁹, in a sufficiently timely manner to complete the review. Accordingly, the Independent Assurance Report by the Auditor-General does not provide any assurance in relation to this information. However, material inconsistencies identified in relation to this information, are required to be considered in forming the conclusion.

1.4 The ANAO’s work is appropriate for the purpose of providing an Independent Assurance Report in accordance with the above auditing standard. However, the review of individual PDSSs is not as extensive as individual performance and financial statement audits conducted by the ANAO, in terms of the nature and scope of issues covered, and the extent to which evidence is required by the ANAO. Consequently, the level of assurance provided by this review, in relation to the 26 major Defence equipment acquisition projects (Major Projects), is less than that provided by the ANAO’s program of audits.

1.5 Separately, the ANAO undertakes analysis of key elements of the PDSSs and examines systemic issues and provides longitudinal analysis for the 26 projects reviewed.

1.6 The review was conducted in accordance with the ANAO Auditing Standards at a cost to the ANAO of approximately $2.2 million.³⁰

Review methodology

1.7 The ANAO’s review of the information presented in the individual PDSSs included:

• examination and assessment of the governance and oversight in place to ensure appropriate project management;
• an assessment of the systems and controls that support project financial management, risk management and project status reporting, within Defence;
• an examination of each PDSS and the documents and information relevant to them;
• a review of relevant processes and procedures used by Defence in the preparation of the PDSSs;
• discussions with persons responsible for the preparation of the PDSSs and management of the projects;
• analysis of project information, for example, cost and schedule variances;
• taking account of industry contractor comments provided on draft PDSS information;
• assessing the assurance by Defence managers attesting to the accuracy and completeness of the PDSSs;
• examination of the representations by the Chief Finance Officer supporting the project financial assurance and contingency statements;
• examination of confirmations, provided by the Capability Managers, relating to each project’s progress toward Initial Materiel Release (IMR), Final Materiel Release (FMR), Initial Operational Capability (IOC) and Final Operational Capability (FOC); and
• examination of the Statement by the Secretary of Defence, including significant events occurring post 30 June, and management representations by the Secretary of Defence.

1.8 The ANAO’s review of PDSSs also focused on project management and reporting arrangements contributing to the overall governance of the Major Projects. The ANAO considered:

• developments in acquisition governance (see paragraphs 1.10 to 1.32, below);
• the financial framework, particularly as it applies to the project financial assurance and contingency statements, and Defence’s advice that project financial reporting during 2018–19 would be prepared on the same basis as project approvals and expenditure represented in the Portfolio Budget Statements and the Defence Annual Report (i.e. on a cash basis) (Section 2 of the PDSSs);
• schedule management and test and evaluation processes (Section 3 of the PDSSs);
• capability assessments, including Defence statements of the likelihood of delivering key capabilities, particularly where caveats are placed on the Capability Manager’s declaration of significant milestones (Section 4 of the PDSSs);
• the ongoing reform process for Defence’s Enterprise Risk Management Framework, and the completeness and accuracy of major risk and issue data (Section 5 of the PDSSs);
• the project maturity framework along with its related reporting and the systems in place to support the consistent and accurate application and the provision of this data (Section 6 of the PDSSs); and

40. the impact of acquisition issues on sustainment to ensure the PDSS is a complete and accurate representation of the acquisition project.

1.9 This review informed the ANAO’s understanding of the systems and processes supporting the PDSSs for the 2018–19 review period. It also highlighted issues in those systems and processes that warrant attention.

Acquisition governance

1.10 Consistent with previous years, the ANAO considered Defence’s acquisition governance processes when planning and conducting review for the 2018–19 MPR. While some of these processes are now established, others continue to mature or require further development to achieve their intended impact.

Defence Independent Assurance Reviews

1.11 The Defence Independent Assurance Review process provides the Defence Senior Executive with assurance that projects and products will deliver approved objectives and are prepared to progress to the next stage of activity. Reviews allow early identification of problem projects and products, facilitating their timely recovery.^{31, 32}

1.12 Formerly called Gate Reviews, Independent Assurance Reviews are intended to be conducted at key acquisition and sustainment ‘gates’ in the Capability Life Cycle.³³

1.13 Twenty of the 26 projects included in this report had an Independent Assurance Review conducted during 2018–19³⁴, which formed key corroborative evidence for the ANAO’s review.

Projects of Concern

1.14 The Projects of Concern process is intended to focus the attention of the highest levels of government, Defence and industry on remediating problem projects. The process has continued to play a role across the portfolio of MPR projects.³⁵ As at 30 June 2019, one MPR project, MRH90 Helicopters, was a continuing Project of Concern. The project was placed on the list in November 2011 due to contractor performance relating to significant technical issues preventing the achievement of milestones on schedule.³⁶

1.15 Auditor-General Report No.31 2018–19, Defence’s Management of its Projects of Concern, assessed whether the Department of Defence’s Projects of Concern regime was effective in managing the recovery of underperforming projects. It concluded that, while the regime is an appropriate mechanism for escalating troubled projects to the attention of senior managers and ministers, Defence was not able to demonstrate the effectiveness of its regime in managing recovery of underperforming projects. Moreover, the audit observed that the transparency and rigour of the framework’s application has declined in recent years. The audit recommended that:

• Recommendation no.1: Defence introduce, as part of its formal policy and procedures, a consistent approach to managing entry to, and exit from, its Projects of Interest and Projects of Concern lists. This should reflect Defence’s risk appetite and be made consistent with the new Capability Acquisition and Sustainment Group Risk Model and other, Defence-wide, frameworks for managing risk. To aid transparency, the policy and the list should be made public.
• Recommendation no.2: Defence evaluates its Projects of Concern regime.³⁷

Defence has agreed to implement these recommendations. It has advised that the Project Risk Management System (PRMS) being developed as part of the risk reform should provide a consistent approach for entry into and exit from the Projects of Interest and Projects of Concern Lists. Defence advised that it intends for this recommendation to be implemented by March 2020 but notes that this will depend on progress in implementing the CASG Risk reform (see paragraphs 1.56 to 1.62)

1.16 In relation to Recommendation 2, Defence has advised that CASG’s Project Management Branch will analyse Projects of Concern to understand the root cause of project performance. Defence has also advised that lessons will be published, communicated and used to treat systemic issues in project practice and performance, including the training and skilling of project management. The CASG Project Management Branch will assess the contribution of the Projects of Concern regime to managing the recovery of underperforming projects.

1.17 The ANAO will continue to monitor Defence’s implementation of the recommendations and report on progress in the next MPR.

Quarterly Performance Report

1.18 The Defence Quarterly Performance Report (QPR) aims to provide senior stakeholders within government and Defence with insight into the delivery of major capability to the Australian Defence Force.³⁸ The report is provided to the Minister for Defence and the Minister for Defence Industry on a quarterly basis.³⁹

1.19 In July 2019, the ANAO completed an audit on the effectiveness of the QPR in providing senior stakeholders with accurate and timely information on the status and emerging risks and issues. It found the June 2018 QPR, reviewed by the ANAO, to be largely effective, contained mostly accurate information, and was valued by senior stakeholders.⁴⁰ The ANAO recommended that Defence improve the QPR as a tool for senior leaders by reporting on:

1. trend performance data for sustainment products; and
2. emerging candidates for the Projects/Products of Concern list and Products/Projects of Interest list that have been recommended by an Independent Assurance Review or which are under active consideration by senior management.⁴¹

1.20 In the course of its review for the 2018–19 MPR, the ANAO observed that Defence’s June 2019 QPR reported on both improved and deteriorated performance for both acquisition and sustainment products since the previous QPR.⁴² This reflects a change in trend reporting consistent with the agreed ANAO recommendation.

1.21 The ANAO also observed as part of its review that Defence’s June 2019 QPR reported the emerging candidates for the Projects/Products of Concern list and Products/Projects of Interest list which had been recommended either by an IAR or which were under active consideration. This change was also consistent with the agreed ANAO recommendation.⁴³

1.22 The ANAO examines QPRs as part of the procedures for its limited assurance review of Defence’s PDSSs.⁴⁴ For the 2018–19 review, the ANAO examined the five QPRs from September 2018 to September 2019.⁴⁵

1.23 The June 2019 QPR identified six MPR projects as Projects of Interest⁴⁶:

• Joint Strike Fighter, noting risks related to affordability and IOC and FOC deliverables⁴⁷;
• LHD Ships, due to ongoing operational testing, a large number of outstanding requirements, defects and deficiencies, and an immature support system⁴⁸;
• Hawkei, due to risks to capability and schedule that relate to ongoing reliability issues, design maturity, and production delays caused by the voluntary administration of the engine manufacturer (Steyr Motors)⁴⁹;
• Repl Replenishment Ships (Maritime Operational Support Capability), due to a change in delivery strategy with the prime contractor, which caused significant schedule changes, and deficiencies in the Integrated Logistics Support system⁵⁰;
• Civil Military Air Traffic Management System (CMATS), noting risks to schedule due to execution of design milestones and poor scope definition, and ongoing contract negotiations⁵¹; and
• UHF SATCOM, due to delays in contract negotiation, software development and US Government certification. In the March 2019 QPR, the entire JP2008 program was identified as a Program of Interest, which is inclusive of UHF SATCOM.⁵²

1.24 These ongoing issues for the Joint Strike Fighter, LHD Ships, Hawkei, Repl Replenishment Ships, CMATS, and UHF SATCOM projects align with the results of the ANAO’s review of the PDSSs. Delays to progress have impacted the delivery schedule of Hawkei, Repl Replenishment Ships, and UHF SATCOM during 2018–19⁵³ (see Table 4, on page 15).

Project Directives and Materiel Acquisition Agreements

1.25 Project Directives (previously known as Joint Project Directives) state the terms of government approval, reflecting the approved scope and timeframes for activities, responsibilities and resources allocated, and key risks and issues.⁵⁴ Project Directives are used to inform internal Defence documentation such as Materiel Acquisition Agreements (MAAs) between Capability Acquisition and Sustainment Group (CASG) and the Service Chiefs.^{55, 56} Project Directives are a key governance document under the Capability Life Cycle⁵⁷, intended to ensure that all parties in Defence are informed of Government decisions. The ANAO has previously highlighted the importance of ensuring that Project Directives properly reflect the relevant Government decision, and that MAAs are appropriately aligned with the relevant Project Directive.⁵⁸

1.26 In some cases Project Directives have been finalised after the MAAs they are intended to inform and, as a result, care is required to ensure that Project Directives properly reflect the relevant government decision, and that MAAs are appropriately aligned with the relevant Project Directive. The mechanism for providing the directive is via the Capability Lifecycle (CLC) Management Tool, which records the Government decision in relation to a project. In some cases, the Joint Force Authority may provide a specific documented directive. For all four new projects entering the 2018–19 MPR, it was not clear whether the projects had their Project Directive signed prior to the MAA. These projects either did not have a specific documented Project Directive, or were unable to access the CLC Management Tool at the time of the ANAO site visits. Project Directives are a requirement of the Interim CLC Manual⁵⁹, and their production increases the likelihood of complying with government decisions.

1.27 The ANAO requires access to original approval documents to validate the requirements of projects. At this time, validation based on internal Defence documentation is not always possible.

1.28 The ANAO will continue to take Project Directives into account as part of its review. The extent to which they can be relied upon will be dependent on the completeness and accuracy of Project Directives, in relation to recording the detail of government approvals.

1.29 Product Delivery Agreements (PDAs)⁶⁰ were being developed to replace the existing MAAs and Materiel Sustainment Agreements (MSAs). Defence has advised the ANAO that this initiative has not progressed.

Business systems

1.30 Defence continues to review its business systems with the aim of consolidating processes and systems in order to provide a more manageable system environment. As reported to the JCPAA on 31 March 2017, Defence stated that there was a ‘need to get a single unified system of accountability and reporting inside the organisation’.⁶¹ During 2018–19, and at the time of this report, the Monthly Reporting System (MRS), which provides much of the data for the PDSSs, remains in place. Defence has advised that an update to the MRS, which may be a new system, will commence pilot testing in April 2020.

1.31 In October 2018, Defence advised that it had concluded its trial of the Project Performance Review (PPR) in November 2017. At that time, PPR was a spreadsheet; now, the PPR is a bespoke Information Communications Technology (ICT) platform which draws project performance data from Defence systems. It is intended for use by project managers to inform discussions with Project Directors and Branch Heads. PPR has not been developed to replace the MRS.

1.32 In January 2018, Defence initiated a plan to implement the PPR across CASG. Defence has now commenced Phase 2 of PPR Release, and its ICT platform is currently being used by 60 projects. Defence has advised that six MPR projects are using PPR.⁶² The information in these documents is largely consistent with the projects’ PDSSs. The ANAO observed inconsistencies relating to:

• project approval dates;
• the number of High- or Extreme-rated project risks; and
• project maturity scores.

Results of the review

1.33 The following sections outline the results of the ANAO’s review, which inform the overall conclusion in the Independent Assurance Report by the Auditor-General for 2018–19.

Financial framework

1.34 The project financial assurance statements were introduced in the 2011–12 Major Projects Report and have been included within the scope of the Independent Assurance Report by the Auditor-General since 2014–15. The contingency statements were introduced for the first time in the 2013–14 report and these describe the use of contingency funding to mitigate project risks. Together, they are aimed at providing greater transparency over projects’ financial status.

1.35 A project’s total approved budget comprises:

• the allocated budget, which covers the project’s approved activities, as indicated in the MAA; and
• the contingency budget, which is set aside for the eventuality of risks occurring and includes unforeseen work that arises within the delivery of the planned scope of work.⁶³

1.36 In 2018–19, the ANAO reviewed the financial framework as it applied to managing project budgets and expenditure, including: project financial assurance, contingency, the reporting environment, and reporting cost variations and personnel costs.

Project financial assurance statement

1.37 The project financial assurance statement’s objective is to enhance transparency by providing readers with information on each project’s financial position (in relation to delivering project capability) and whether there is ‘sufficient remaining budget for the project to be completed’.⁶⁴

1.38 In September and November 2018, due to cost pressures, the Joint Strike Fighter project received government approval to transfer project scope of $1.5 billion to other phases of the Joint Strike Fighter program (none of which have been approved by government). There was no corresponding transfer of funds out of the project budget.⁶⁵ The PDSS states that ‘there is sufficient budget, including contingency, remaining for the project to deliver the revised scope’.

1.39 The Joint Strike Fighter PDSS reports a risk that project capability may be affected by overall funding or programming issues arising from internal cost growth, forecasting accuracy and external budget constraints. The remedial actions to address this risk reported in the PDSS include:

• ‘Conduct ongoing engagement with the F-35 Joint Program Office and major project suppliers to facilitate improved cost data to allow the F-35 project to meet budgeting and programming expectations’ — i.e. clarifying potential cost pressures on the project;
• ‘Proactive management of cost risk identification and engagement with the Capability Manager to prioritize requirements to deliver project capability within the approved project budget’ — i.e. actively identify cost risk and engage with senior leaders; and
• ‘Options may be developed for Capability Manager consideration to achieve project affordability by aligning project expenditure with the Defence Integrated Investment Program capacity in any specific year’ — i.e. consider options for scheduling project expenditure to align with Defence’s available funding.

1.40 The Chief Finance Officer’s representation letter to the Secretary of Defence on the 2018–‍19 MPR’s project financial assurance statements was unqualified. The project financial assurance statement is restricted to the current financial contractual obligations of Defence for these projects, including the result of settlement actions and the receipt of any liquidated damages, and current known risks and estimated future expenditure as at 30 June 2019.

Contingency statements and contingency management

1.41 The purpose of the project contingency budget is to estimate the inherent cost, schedule and technical uncertainties of projects’ in-scope work.⁶⁶ Defence policy requires project managers to ensure that all decisions in regards to a project’s contingency budget are included in the project’s contingency budget log to ensure ongoing transparency and traceability.⁶⁷

1.42 PDSSs are required to include a statement regarding the application of contingency funds during the year, if applicable, as well as disclosing the risks mitigated by the application of those contingency funds. Defence’s Project Risk Management Manual (PRMM version 2.4, page 110)⁶⁸ requires that contingency be applied for identified risk mitigation activities which have been assessed as being cost effective and representing value for money.

1.43 Contingency provisions for projects are not programmed or funded in cash terms.⁶⁹ As such, projects are encouraged to meet contingency funding requirements from within their currently programmed cash funding. If this cannot be achieved, a project may then propose to access contingency funding from the relevant capital program (Approved Major Capital Investment Program (AMCIP), Facilities and Infrastructure Program (FIP) and ICT Capital Program). If this cannot be achieved, the contingency call will be presented to the Defence Investment Committee, which if agreed will potentially be met by budget offsets across the whole Integrated Investment Program.⁷⁰

1.44 Three projects in the MPR had contingency funds applied in 2018–19: MRH90 Helicopters (supportability and performance risks), Repl Replenishment Ships (implementation of capability requirements such as spares and identification of friend or foe, and delivery from Spain to Australia), and Battle Comm. Sys. (Land) 2B (Interface Control Integration).

1.45 In 2017–18, the remaining scope of two projects was transferred into Collins R&S (SEA 1114 Phase 3 and SEA 1439 Phase 5B1). In 2018–19, the associated budgets were transferred to Collins R&S. A contingency allocation was transferred along with the budget of SEA 1439 Phase 5B1. Previous to this, Collins R&S did not have a contingency allocation. As a result, all 26 projects in the 2018–19 MPR now have a contingency allocation.

1.46 The ANAO’s examination of projects’ contingency logs as at 30 June 2019 highlighted that:

• the clarity of the relationship between contingency allocation and identified risks continues to be an issue. Five projects (LHD Ships, Repl Replenishment Ships, CMATS, ANZAC ASMD 2B, and Pacific Patrol Boat Repl) did not explicitly align their contingency log with their risk log, by including risk identification numbers as required by PRMM version 2.4; and
• there were two project offices (CMATS and ANZAC Air Search Radar Repl) that did not meet all the requirements of PRMM version 2.4 in terms of keeping a record of review of contingency logs. However, the ANAO observed that the information required could be located in other documents.

1.47 Non-compliance with PRMM version 2.4 has resulted in inconsistent approaches between projects in the management of contingency logs, with some projects advising that they will not remedy these non-compliances until the outcomes of the risk management reform within CASG are known (see para 1.59).

Reporting environment

1.48 On 4 April 2018, Defence advised project offices that project financial reporting for 2017–‍18 PDSSs would be prepared on the same basis as project approvals and expenditure represented in the Portfolio Budget Statements and the Defence Annual Report (i.e. on a cash basis).⁷¹

1.49 Defence obtains cash expenditure data using a management reporting tool called BORIS. Prior to the 2017–18 MPR, accrual expenditure data was extracted from the Financial Management Information System known as ROMAN. Given the change in the extraction method, the ANAO requested that Defence perform reconciliations of the cash expenditure figures from BORIS to ROMAN for each project.

1.50 In the 2018–19 MPR, Defence continued to report the projects’ financial information on a cash basis and therefore continued to perform these reconciliations. This activity concluded in October 2019 and enabled the ANAO to obtain assurance over the cash expenditure. The Defence Chief Finance Officer has determined that, from the 2020–21 MPR onwards, Defence would report expenditure data on an accrual basis. The 2019–20 MPR should, therefore, be the last year that these reconciliations need to be performed.

Reporting cost variations since Second Pass Approval and personnel costs

1.51 In May 2018, the JCPAA wrote to the Auditor-General to request that the ANAO report back to it ‘on how Defence major project cost variations and the costs of retaining project staff over time might be reported annually in future Major Projects Reports.’⁷²

1.52 A new table was included in the 2017–18 MPR showing all budget variations post initial Second Pass Approval for projects. Refer to Table 8 on page 42.

Project Personnel Numbers and Costs

1.53 In terms of calculating the cost of retaining project staff, Defence advised the ANAO in November 2018 that its current IT systems do not provide a direct mapping of personnel to projects. It noted that personnel often work on multiple projects and sustainment activities at any given time.

1.54 The ANAO observed during fieldwork in 2019 that several MPR projects had staff who worked concurrently on other projects, which included shared corporate staff. Some of these projects did not have systems in place to record accurately the proportion of time these shared staff attributed to the project. Moreover, the ANAO observed that MPR projects used different methods to record personnel data.

1.55 Defence has advised the ANAO that it is not yet in a position to provide the staff cost component of projects and its systems are not capable of calculating the cost of retaining project staff over time. Accordingly, Defence has not provided any data on the costs of project staff for projects in the MPR. The ANAO will continue to monitor Defence’s progress in recording project personnel numbers and costs in future reports.

Enterprise Risk Management Framework

1.56 While major risks and issues data in the PDSSs remains excluded from the formal scope of the Auditor-General’s Independent Assurance Report⁷³, material inconsistencies identified in relation to this information are required to be detailed in the report. The following information is included to provide an overall perspective of how risks and issues are managed within Defence and the selected Major Projects.

1.57 Risk management has been a focus of the MPR since its inception. The CASG risk management environment consists of multiple policies and varying implementation mechanisms and documentation. There are multiple group-level (i.e. CASG), sub-group (i.e. Divisional) and project-level risk management documents. The primary focus of the ANAO’s examination of risk management is at the project level, in order to provide assurance over the PDSSs.

1.58 At the Group level, Deputy Secretary CASG issued a directive in May 2017 establishing a CASG Risk Management Reform Program to implement a risk management model that is situated within Defence’s risk management framework. To date, Defence has delivered the first two phases of the reform (establishment of the CASG Risk Management Group Model, and the associated support activities such as training and consultation). The third phase of the reform — which includes the development of risk management policies and toolsets for use by projects — was initially planned to be concluded in June 2019. Defence has advised that it has experienced delays in completing certain deliverables, and that the contract for the completion of the Risk Reform Program has been extended to February 2020. Defence expects that, once the Reform has concluded, implementation will take a number of annual cycles to reach maturity.⁷⁴

1.59 The ANAO has observed that some projects chose not to review risk and issues management procedures until this stage has been completed, as noted at paragraph 1.47. The ANAO will continue to monitor the implementation of the reform as part of future reviews, but will not be able to consider including risks and issues in the scope of the MPR until the reform is sufficiently progressed.

1.60 In 2018–19, the ANAO again examined project offices’ risk and issue logs at the Group and Service level, which are predominantly created and maintained utilising spreadsheets and/or Predict! software.⁷⁵ Overall, the issues with risk management that the ANAO observed related to:

• variable compliance with corporate guidance, for example all projects had a Risk Management Plan, however; 10 out of 26 Major Projects did not validate the currency of the Risk Management Plan in line with PRMM version 2.4⁷⁶;
• the visibility of risks and issues when a project is transitioning to sustainment;
• for one project (Joint Strike Fighter), sustainment and acquisition risks are managed together, despite Defence risk management policy for acquisition and sustainment providing inconsistent guidance⁷⁷;
• the frequency with which risk and issue logs are reviewed to ensure risks and issues are appropriately managed in a timely manner, and accurately reported to senior management;
• risk management logs and supporting documentation of variable quality, particularly where spreadsheets are being used⁷⁸; and
• lack of quality control resulting in inconsistent approaches in the recording of issues within Predict!

1.61 The ANAO has previously observed that Defence’s use of spreadsheets as a primary form of record for risk management is a high risk approach. Spreadsheets lack formalised change/version control and reporting, thereby increasing the risk of error. This can make spreadsheets unreliable corporate data handling tools as accidental or deliberate changes can be made to formulae and data, without there being a record of when, by whom, and what change was made. As a result, a significant amount of quality assurance is necessary to obtain confidence that spreadsheets are complete and accurate at 30 June, which is not an efficient approach. The ANAO’s review of CASG’s 26 project offices indicates that 15 utilise spreadsheets⁷⁹ as their primary risk management tool, five utilise Predict!⁸⁰, one (Maritime Comms) utilises both Microsoft Excel and Predict!, two (Joint Strike Fighter and CMATS) utilise a bespoke SharePoint based tool, one (MH-60R Seahawk) utilises Microsoft Word, one (Night Fighting Equip Repl) utilises the Project Performance Review (see paragraph 1.31), and one (HATS) does not currently manage any risks given the delivery of all primary project elements. Defence has advised that a risk management system will not be mandated until the outcomes of the CASG risk reform are known (see paragraph 1.58).

1.62 The JCPAA recommended in September 2018 that Defence plan and report a methodology to the Committee showing how acquisition projects can transition from the use of spreadsheet risk registers to tools with better version control.⁸¹ In response, Defence advised that the Risk Reform Program was developing a revised methodology for managing project risk and intended to commence prioritised transition of projects into the remodelled risk management approach from the first quarter of 2019. However, the ANAO observed in the course of its 2019 site visits that MPR projects had not received project specific guidance. Defence advised the ANAO in October 2019 that it is looking to mandate a standardised ICT tool for the management of risk across projects. The decision on this tool is expected in November 2019. The 2019–20 MPR will report on MPR projects’ use of the mandated tool.

Project maturity framework

1.63 Project Maturity Scores have been a feature of the Major Projects Report since its inception in 2007–08. The DMO Project Management Manual 2012 defined a maturity score as:

The quantification, in a simple and communicable manner, of the relative maturity of capital investment projects as they progress through the capability development and acquisition life cycle.⁸²

1.64 Maturity scores are a composite indicator, cumulatively constructed through the assessment and summation of seven different attributes. The attributes are: Schedule, Cost, Requirement, Technical Understanding, Technical Difficulty, Commercial, and Operations and Support, which are assessed on a scale of one to 10.⁸³ Comparing the maturity score against its expected life cycle gate benchmark provides internal and external stakeholders with a useful indication of a project’s progress.

1.65 The ANAO has previously identified that the policy guidance underpinning the attribution of maturity scores would benefit from a review for internal consistency and the relationship to Defence’s contemporary business. For example, allocating approximately 50 per cent of the maturity score at Second Pass Approval, regardless of acquisition type, is often inconsistent with the proportion of project budget expended, and the remaining work required to deliver the project. Further, the existing project maturity score model does not always reflect a project’s progress during the often protracted build phase, particularly for developmental projects. During this phase it can be expected that maximum expenditure will occur, and that many risks will be realised, some of which will only emerge as test and evaluation activities are pursued through to acceptance into operational service.

1.66 In 2016⁸⁴ and again in 2017⁸⁵, the JCPAA recommended that Defence update the policy on Project Maturity Scores. At the JCPAA hearing held on 23 March 2018, Defence undertook to update the framework by mid-2018 with a two-stage process: first to remediate inconsistencies in the policy and accommodate Interim Capability Life Cycle terminology; then to undertake a more substantial amendment of the policy.⁸⁶ In September 2018, the JCPAA sought a written update from Defence outlining progress towards updating the policy.⁸⁷ Also in September 2018, Defence advised the ANAO that the maturity score process was being re-considered within the CASG risk reform context. In December 2018, Defence reported this to the JCPAA in its response to the 2018 recommendation.

1.67 In October 2019, Defence advised the ANAO that a draft Project Maturity Score policy has been developed (renamed the ‘Project Progress Score’) and will be trialled and evaluated in late 2019, for implementation in the 2020–21 MPR. Defence further advised that while the progress score has been decoupled from the risk reform work, it will be informed by existing project risk polices and enhanced by improvements arising out of the risk reform framework. The ANAO was provided the draft Project Progress Score policy in November 2019. The draft indicates that the Project Progress Score will be calculated based on 10 project attributes, leading to a total score out of 100, instead of the seven attributes and total score out of 70 under the current Project Maturity Score policy. The draft policy also notes that ‘The [Project Progress Score] is not designed as an assessment of project risk’.

Caveats and deficiencies

1.68 Defence has not defined the terms ‘caveat’ or ‘deficiency’ to the declaration of significant milestones in its internal policies and procedures. The ANAO has observed use of these terms by Defence to represent exceptions to the achievement of significant milestones declared by Defence such as IMR, IOC, FMR and FOC.

1.69 The 2017–18 MPR noted a ‘reduced trend of Major Projects which had achieved significant milestones with caveats’ and Defence’s advice that it discourages Independent Assurance Reviews recommending caveats at FOC.⁸⁸ Only one project (Growler) achieved a major milestone with caveats in 2017–18.⁸⁹

1.70 In 2018–19, Defence declared more milestones with caveats than in 2017–18, as follows:

• P-8A Poseidon — Defence declared two caveats to the achievement of the Operational Capability 2 (OC2) milestone in February 2019, related to deficiencies of spares (Fly Away Kits) and Operational Flight Trainer (pilot simulator) qualification; and
• Growler — Defence declared one caveat to the achievement of the IOC milestone in February 2019, related to in-country aircrew training not yet possible due to delays in delivery of the Mobile Threat Training Emitter System.

1.71 The Chief of Air Force acknowledged achievement of FMR for the Additional MRTT project in October 2019, with an ‘accepted deficiency’ relating to the non-delivery of a minor piece of support equipment.

1.72 In addition, the Chief of Navy declared FOC for the LHD Ships project in November 2019, with seven ‘notable deficiencies’. Key deficiencies referenced in the PDSS relate to technical issues and defects, primarily affecting the propulsion pods, and Integrated Logistic Support. Remediation is underway through the Transition and Remediation Program, and the prime contract has been extended to allow for closure of the outstanding contractual requirements.⁹⁰

1.73 The ANAO will continue to monitor Defence’s declaration and resolution of caveats (or exceptions) to the achievement of significant Capability Milestones in future reviews. This will include projects which have been removed from the MPR with outstanding caveats which are required to be reported by Defence in the Statement by the Secretary of Defence until their final status is accepted by the Capability Manager.⁹¹

2.1 Performance information is important in the management and delivery of major Defence equipment acquisition projects (Major Projects). It informs decisions about the allocation of resources, supports advice to government, and enables stakeholders to assess project progress.

2.2 Project performance has been the subject of many of the reviews of the Department of Defence (Defence), and a consistent area of focus of the Joint Committee of Public Accounts and Audit (JCPAA) since the first Major Projects Report (MPR). This chapter progresses previous Australian National Audit Office (ANAO) analysis over project performance.

Project performance analysis by the ANAO

2.3 The major dimensions of projects’ performance are:

• Cost performance (pp. 37–49) — this includes the percentage of budget expended (Budget Expended), changes in budget since Second Pass Approval, in-year changes to budget, and in-year expenditure;
• Schedule performance (pp. 50–61) — this includes the percentage of time elapsed (Time Elapsed), total schedule slippage, and in-year changes to schedule; and
• Capability performance (pp. 62–66) — this includes the percentage of key materiel capabilities delivered (Capability Delivery Progress).

2.4 The ANAO has previously utilised Defence’s prediction of expected final capability, as reported in Section 4.1 of each Project Data Summary Sheet (PDSS). In 2015–16, the ANAO derived an indicator for ‘Capability Delivery Progress’, which aims to show the current capability delivered, in terms of capability elements included within the agreed Materiel Acquisition Agreements (MAAs). These performance indicators are measured in percentage terms, to enable comparisons between projects of differing scope, and to provide a view across the selected projects of progress and performance.

2.5 The following sections of this chapter provide analysis relating to the three principal dimensions of project performance noted above. This includes in-year information, longitudinal analysis and the results of project progress for the year-ended 30 June 2019. Figure 2 directly compares cost performance with schedule performance through two metrics, Budget Expended and Time Elapsed.⁹²

Figure 2: Budget Expended and Time Elapsed

 

 

Source: ANAO analysis of the 2018–19 PDSSs.

2.6 Figure 2 shows that for nearly all projects (25 of 26), Budget Expended is broadly in line with (within 10 per cent), or lagging, Time Elapsed. This relationship is generally expected in an acquisition environment predominantly based on milestone payments. However, due to the varying complexity, stages and acquisition approaches across the portfolio of projects, further analysis of these simple performance measures is required to provide a better understanding of key variances.

2.7 Where Budget Expended is significantly lagging Time Elapsed, the project schedule may be at risk — i.e. expenditure lags may indicate delays in milestone achievement. In 2018–19, the Budget Expended for three projects lagged Time Elapsed by at least 20 per cent. For two of these three projects, milestones have been delayed, as detailed below:

• Hawkei (Budget Expended 28 per cent, Time Elapsed 49 per cent) — the project’s achievement of milestones has been delayed by reliability issues, design maturity, and production delays. The project is expected to complete its Production Reliability Acceptance Test 21 months later than originally contracted; and
• Battlefield Airlifter (Budget Expended 58 per cent, Time Elapsed 93 per cent) — the project has not signed contracts for the acquisition of training devices, and some other equipment is also outstanding. Delivery of this equipment has been significantly delayed; as at 30 June 2019, the project planned to deliver these items after the achievement of the FOC milestone.

2.8 For the third project, Joint Strike Fighter (Budget Expended 28 per cent, Time Elapsed 69 per cent), the expenditure lag reflects the recent transition from the aircraft development stage, where relatively little budget was expended. The project is now entering into main production contracts for aircraft, with in-year expenditure increasing compared to prior years.

2.9 Where Budget Expended leads Time Elapsed, the project budget may be at risk — i.e. expenditure increases may indicate real cost increases. However, for the one project where Budget Expended leads Time Elapsed by 10 per cent or more, the cause of the variance relates to work being performed prior to Second Pass Approval. For ANZAC Air Search Radar Repl (Budget Expended 46 per cent, Time Elapsed 29 per cent), this project spent approximately nine per cent of its budget prior to Second Pass, to conduct a Risk Reduction Program and make early purchases of equipment to ensure the schedule would be met.⁹³

2.10 In each case of significant variance between Budget Expended and Time Elapsed, the performance information highlights projects that may require further attention. This is to ensure that unspent funds are returned to the Defence budget for re-allocation in a timely manner, the timing of key deliverables remains in focus, or planning focuses on bringing together all elements in a timely manner, as equipment is delivered.

Cost performance analysis

Budget Expended and Project Maturity

2.11 Figure 3, below, sets out each project’s Budget Expended against Project Maturity⁹⁴ and shows that Budget Expended lags Project Maturity for the majority of projects (22 of 26). This relationship is typical of acquisition projects for two reasons:

• in an acquisition environment predominantly based on milestone payments, projects will typically develop confidence in delivering their scope through design reviews, testing and demonstration, ahead of formal acceptance of milestone achievement or equipment deliveries (and expenditure of budget); and
• more generally, Budget Expended will often lag Project Maturity as the result of Defence’s project maturity framework attributing approximately 50 per cent (35 out of 70 points) of total Project Maturity at Second Pass Approval (the main investment decision by government)⁹⁵ prior to any significant expenditure of budget.

2.12 In both cases, the Budget Expended is expected to catch up to Project Maturity over the course of the project’s life, with projects approaching closure expected to show Budget Expended and Project Maturity broadly in line with each other.

2.13 Budget Expended lags Project Maturity with a variance of 20 per cent or more in 12 projects. As expected, the majority of these projects are at a relatively early stage and have expended minimal budget while progressing through design and testing phases, or are waiting on significant amounts of equipment to be delivered. The exceptions to this are projects that have delivered the majority of their major equipment, leading to an advanced maturity score, while the budget expended is lagging as items such as training equipment or weapons are yet to be delivered and paid for. Projects fitting this pattern are Battlefield Airlifter (all aircraft have been delivered while some training devices and other equipment are outstanding), and Night Fighting Equip Repl (most Tranche 1 equipment has been delivered while the evaluation of future Tranche 2 equipment is still in its early stages).

2.14 Where Budget Expended leads Project Maturity by a significant amount, this may indicate that the project is behind in development or achievement of its scope, or that the required scope is not affordable. There are no instances where Budget Expended leads Project Maturity by 20 per cent or more. The largest variance is for UHF SATCOM, where Budget Expended leads Project Maturity by 11 per cent. The project’s maturity score has been affected by delays in software development, while the majority of budget has been expended and the project has funded further development with contingency.

Figure 3: Budget Expended and Project Maturity

 

 

Source: ANAO analysis of the 2018–19 PDSSs.

Second Pass Approval and 30 June 2019 approved budget

2.15 Figure 4, below, compares each project’s approved budget at initial Second Pass Approval and its approved budget at 30 June 2019.

2.16 The total budget for the 26 projects at 30 June 2019 was $64.1 billion, a net increase of $24.4 billion, when compared to the approved budget at initial Second Pass Approval of $39.7 billion.

2.17 Figure 4 indicates all budget variations from initial Second Pass Approval. Six projects have variations of $500 million or more. The list below describes the components of these variations:

• Joint Strike Fighter — increase of $13.8 billion, comprising $10.5 billion for 58 additional aircraft in 2013–14, $2.9 billion for exchange rate variation and $0.4 billion for price indexation⁹⁶;
• AWD Ships — increase of $1.9 billion, comprising $1.2 billion for a Real Cost Increase⁹⁷ in July 2015 to complete the project, $1.2 billion for price indexation, offset by a $0.4 billion decrease for exchange rate variation and a $0.1 billion decrease for transfers to facilities projects in 2013–14;
• P-8A Poseidon — increase of $1.8 billion, comprising $1.3 billion for four additional aircraft in 2015–16 and $0.5 billion for exchange rate variation;
• MRH90 Helicopters — increase of $2.8 billion, comprising $2.6 billion for 34 additional aircraft in 2005–06 and other minor scope changes, and $0.7 billion for price indexation, offset by a $0.3 billion decrease due to scope transfers for facilities, and a $0.1 billion decrease for exchange rate variation;
• Growler — increase of $0.8 billion, comprising $0.9 billion for exchange rate variation, $0.3 billion for the Mobile Threat Training Emitter System and weapons, offset in 2015–‍16 by a $0.2 billion decrease for transfers to facilities projects and $0.2 billion for the return to the Defence budget of surplus funds and contingency for reallocation; and
• Overlander Medium/Heavy — increase of $0.8 billion, comprising $0.7 billion ‘project supplementation’ to reduce cost pressures and $0.1 billion exchange rate variation.

Figure 4: Projects’ initial Second Pass Approval and 30 June 2019 approved budget ($m)

 

 

Note 1: 

indicates that the budget for the project at 30 June 2019 is less than the original budgeted cost.

 

Note 2: On 22 May 2015, the Minister for Defence and the Minister for Finance announced there would be further delays to the delivery of the Air Warfare Destroyers and an additional $1.2 billion would be required to complete the project. The budget increase was incorporated into the approved project budget as at 30 June 2016.

Source: ANAO analysis of the 2018–19 PDSSs.

2.18 Budget variances since initial Second Pass Approval may result from: increasing the scope of a project via revised Second Pass Approvals, programmatic decisions, Real Cost Increases/Decreases, transfers to/from other projects and budgetary adjustments. A summary of budget variations is at Table 3 (page 13) and a more detailed analysis of this variance is included in Table 8, below.

Budget performance

2.19 The following figures and tables illustrate the budget performance of the 26 selected projects by way of:

• in-year budget variations by project (see Table 9, below); and
• expenditure forecasting performance against actual expenditure for 2018–19 (see Figure 5, on page 49).

In-year budget variance analysis

2.20 Table 9, below, sets out the in-year budget variations for each project. Overall, the approved budget for the projects as at 30 June 2019 increased by $1225.1 million, or 2.1 per cent, compared to their approved budget as at 30 June 2018. This was driven by exchange rate variation increases of $1221.6 million and net real increases of $3.7 million.

2.21 Exchange rate variations result from projects’ exposure to foreign currencies and movements in foreign exchange rates against the Australian dollar.⁹⁸ Budget adjustments aim to maintain the relative buying power of the project budget. Movements in the US dollar and the Euro are the main influences. Projects with larger movements in foreign exchange in 2018–19 included:

• Joint Strike Fighter — movement of $1018.6 million, or 6.6 per cent increase in budget; and
• MH-60R Seahawk — movement of -$217.8 million, or 6.3 per cent decrease in budget.

2.22 Real Variations⁹⁹ primarily reflect changes in the scope of projects, transfers between projects for approved equipment/capability and budgetary adjustments such as administrative savings decisions. In 2018–19, the two projects with more significant Real Variations were:

• Collins R&S — variation of $33.7 million reflecting budget transfers associated with a transfer of scope from two related projects; and
• Overlander Medium/Heavy — variation of -$30.0 million reflecting the return of funds to the Integrated Investment Program.

Table 9: In-year (2018–19) budget variations by project

Note 1: The Total Variance and components for this project do not add up due to rounding differences.

Note 2: The difference between the total approved budgets for 2017–18 and 2018–19 is partly due to the projects entering the MPR in 2018–19 (Offshore Patrol Vessel, Collins Comms and EW, Night Fighting Equip Repl, and ANZAC Air Search Radar Repl) not contributing to the total budget figure for 2017–18.

Source: ANAO analysis of the 2017–18 and 2018–19 PDSSs.

In-year forecast and actual expenditure

2.23 Accurately forecasting and managing budget expenditure is an important element in the management of a portfolio of projects. Figure 5, below, sets out the expenditure forecasting performance of each project against actual expenditure in 2018–19. In total, actual expenditure for the 26 projects at 30 June 2019 was $4831.4 million. This is compared against an initial Portfolio Budget Statements (PBS) forecast expenditure of $5809.4 million, a mid-year Portfolio Additional Estimates Statements (PAES) forecast of $5382.3 million, and a final forecast of $5173.0 million (Final Plan, approved during May 2019).

2.24 Figure 5 highlights that notable in-year underspends occurred in the following projects:

• AWD Ships (expenditure of $198.9 million compared to $375.9 million PBS, $226.2 million PAES and $226.6 million Final Plan estimates) — the variance is due to deferring planned expenditure to subsequent years and Defence Finance Group deferring payment of invoices incurred during 2018–19;
• P-8A Poseidon (expenditure of $472.4 million compared to $592.3 million PBS, $408.9 million PAES and $472.6 million Final Plan estimates) — the variance is due to having paid some planned 2018–19 expenditure in 2017–18 and deferring some planned 2018–19 expenditure to subsequent years; and
• Hawkei (expenditure of $89.3 million compared to $395.6 million PBS, $202.8 million PAES and $117.5 million Final Plan estimates) — the variance is due to ongoing vehicle reliability issues, design maturity, and delays in the delivery of engine components delaying payment of milestones, as well as Defence Finance Group deferring payment of invoices incurred during 2018–19.

2.25 Figure 5 also highlights that a notable in-year overspend occurred in the following project:

• Joint Strike Fighter (expenditure of $1942.0 million compared to $1821.1 million PBS, $1933.3 million PAES and $1977.6 million Final Plan estimates) — the variance is due to foreign exchange updates.

Figure 5: In-year (2018–19) projects’ forecast expenditure performance compared to actual expenditure ($m)

 

 

Sources: ANAO analysis of the 2018–19 PDSSs and Defence Portfolio Budget Statements.

Schedule performance analysis

2.26 Defence data continues to show that schedule performance is a key issue in delivering and sustaining equipment.¹⁰⁰ Project schedule slippage can have the effect of introducing or exacerbating a capability gap, or requiring an extension to the planned withdrawal date for those platforms being replaced.¹⁰¹

Time Elapsed and Project Maturity

2.27 Based on the findings of the Defence Procurement Review 2003¹⁰², in 2005 Defence began to increase the proportion of MOTS acquisitions, which are generally lower risk projects and therefore more likely to meet schedule timelines. Analysis of the available performance information highlights that the selection of MOTS projects assists in reducing risk during project acquisition, where Project Maturity is more advanced at Second Pass Approval than developmental projects. For example, CMATS is a developmental project that has experienced significant schedule slippage; its maturity score at Second Pass Approval was 31 points, below the expected benchmark of 35 points for projects at Second Pass Approval. In contrast, MH-60R Seahawk is a MOTS project that has not experienced any slippage to date; its maturity score at Second Pass Approval was 37 points.

2.28 Figure 6, below, sets out each project’s Time Elapsed against Project Maturity.¹⁰³ Time Elapsed lags Project Maturity for 19 of 26 projects. Similar to the analysis of Budget Expended and Project Maturity, at paragraphs 2.11 to 2.14, this pattern is expected as projects will generally score 50 per cent of their Project Maturity at Second Pass Approval, when Time Elapsed is zero (for the purposes of the ANAO’s analysis in this report). The lag is most pronounced in MOTS and Australianised MOTS acquisitions, including Offshore Patrol Vessel, Collins Comms and EW, Pacific Patrol Boat Repl, and Night Fighting Equip Repl. The exception is ANZAC Air Search Radar Repl, a developmental project where the lag in Time Elapsed against Project Maturity reflects the project’s extensive schedule to FOC, required to upgrade all eight ships in the Anzac class.

2.29 For the 10 projects where Time Elapsed lags Project Maturity by 20 per cent or more, this generally reflects projects at relatively early stages of acquisition processes, including proceeding through design activities, or awaiting significant amounts of their major equipment to be constructed and delivered. There are two significant exceptions to this:

• MH-60R Seahawk, where the majority of equipment has been delivered but the project needs to test and integrate a number of ADF Mission System Options and modify Navy ships to operate with the helicopter; and
• HATS, where all helicopters and training devices have been delivered, but more time is required to prove the full capability of the equipment in operational service.

2.30 For the seven projects where Time Elapsed leads Project Maturity, there were no instances where this difference was significant (20 per cent or more). The greatest variance was for Battlefield Airlifter, where Time Elapsed leads Project Maturity by 15 per cent. At 30 June 2019 this project was forecasting to achieve FOC in December 2019, despite major contracts for the acquisition of training devices not yet being signed and some other equipment also not yet acquired.

Figure 6: Time Elapsed and Project Maturity

 

 

Source: ANAO analysis of the 2018–19 PDSSs.

Schedule slippage and acquisition type by approval date

2.31 Figure 7, below, illustrates the total schedule slippage¹⁰⁴ since Second Pass Approval for the 26 selected projects. It also depicts the acquisition type and places projects in order of government approval. Figure 8 illustrates the total schedule slippage for the 22 projects that have exited the review.

2.32 Following implementation of the recommendations of the Defence Procurement Review 2003¹⁰⁵, in 2005 Defence began focusing on MOTS and Australianised MOTS acquisitions.¹⁰⁶ Figure 8 shows that the inclusion of MOTS acquisitions contributed, prima facie, to a reduction in schedule slippage in the Major Projects portfolio. For projects that have exited the MPR, MOTS projects report an average of 11 months of slippage per project, while Australianised MOTS projects report an average of 45 months and developmental projects report an average of 105 months. Decisions on whether to undertake developmental projects should be considered on a risk basis.¹⁰⁷ In this context, the consideration of risk should be holistic and weigh up the level of capability to be acquired against potential risks relating to cost and schedule.

2.33 The 2015 First Principles Review recommended the construction of a ‘smart buyer’ framework, with the aim of ‘[ensuring] Defence can make strategic decisions regarding the most appropriate procurement and contracting methodologies’.¹⁰⁸ Defence has begun to conduct Smart Buyer assessments for acquisition projects at different stages of approval. None of the projects currently in the Major Projects portfolio have been approved under the Smart Buyer processes. The ANAO will continue to report on the outcomes of the First Principles Review and the Smart Buyer framework in subsequent years.

2.34 The First Principles Review also identified technical risk as the major cause of post Second Pass Approval schedule slippage, and observed that schedule slippage causes cost escalation.¹⁰⁹ The challenge of gaining a full understanding of the complexities of developmental aspects of projects at Second Pass Approval is evident by the extent of slippage over time.

2.35 Figures 7 and 8 illustrate that older projects have experienced the most slippage. These projects tended to be more developmental (complex) in nature and typically experienced schedule slippage in the past, and have often continued to do so. This demonstrates an ongoing trend of slippage in historically late projects, which is more pronounced in older projects. This trend is also visible, but less prominent, in newer projects.

2.36 The more recent developmental projects, Hawkei and ANZAC Air Search Radar Repl, are yet to experience slippage to their FOC dates. However, these projects have experienced slippage to design reviews, test programs, or material release milestones; in the case of Hawkei, 21 months slippage to the Production Reliability Acceptance Test, leading to 17 months slippage to Initial Materiel Release. In contrast, recent MOTS projects, Pacific Patrol Boat Repl and Night Fighting Equip Repl, have adhered more closely to their design and materiel release schedules with only minor variances. This indicates that although developmental projects currently in the MPR are not reporting significantly more slippage to FOC than MOTS projects, developmental projects still appear to carry a higher level of technical risk.

2.37 While it is not possible to predict the full extent of slippage a project will experience, Figure 8 analysis has been provided to highlight changes since the Kinnaird Review. Nine post Kinnaird and 12 pre Kinnaird projects have exited the MPR. Total slippage of the nine post Kinnaird projects is 7.1 years. Total slippage of the 12 pre Kinnaird projects is 72.3 years. Six of the nine post Kinnaird projects were MOTS acquisitions and 11 out of the 12 pre Kinnaird acquisitions were Australianised MOTS or developmental.

Schedule performance

2.38 The figures and tables that follow illustrate:

• the original and 30 June 2019 forecasts for achieving FOC;
• in-year schedule changes to achieving FOC; and
• total schedule slippage across the Major Projects.

Original and 30 June 2019 Final Operational Capability forecasts

2.39 Figure 9, below, presents information on the selected projects’ original and 30 June 2019 forecasts for achieving FOC. The total schedule slippage¹¹⁰ for the 26 Major Projects to date is 691 months compared to the initial prediction when approved by government. This represents a 27 per cent increase on the approved schedule. Of the 26 projects in the 2018–19 report, 21 have experienced schedule slippage.

2.40 Total schedule slippage across the Major Projects was 691 months in 2018–19. This is 110 months lower than the figure of 801 months reported in the 2017–18 report. The difference is mainly due to the exit of projects with significant slippage — Collins RCS, Hw Torpedo, ANZAC ASMD 2A and BMS — which reduced the total accumulated slippage by 254 months. This was offset by in-year slippage for Battle Comm. Sys. (Land) 2B (significant technical issues requiring a change in technical approach), ANZAC ASMD 2B (administrative delays to declaring FOC), UHF SATCOM (further delays in software development), and LHD Landing Craft (rescheduling of heavy load carriage sea trials). These projects, combined, added 68 months of the 108 months schedule slippage in 2018–19. Additionally, Collins Comms and EW added 36 months of slippage to the total of 691 months; the slippage occurred in 2016–17 but the project was reported in the MPR for the first time in 2018–19.

2.41 The reasons for schedule slippage often require a deep understanding of project technical elements and a realistic assessment of the capacity of the private sector to deliver in the expected timeframe. A project office’s ability to gain access to the platform for upgrading can also result in schedule delay (for example, Maritime Comms and Collins R&S).¹¹¹

2.42 A closer examination of the reasons for schedule slippage demonstrates the importance of initial assessments of project complexity. A key factor is whether a project is MOTS, Australianised MOTS or developmental.¹¹² One project, MRH90 Helicopters¹¹³, was originally misclassified as MOTS. The project was reclassified by Defence to Australianised MOTS (i.e. more developmental) subsequent to Second Pass Approval.¹¹⁴ This project has experienced extended schedule slippage. Another project, UHF SATCOM, is still classified as MOTS but includes the development of significant amounts of software. Delays in software development have led to 42 months of slippage to the FOC milestone.

2.43 Figure 9 further indicates that one project (Joint Strike Fighter) is currently forecasting an FOC date earlier than originally approved. However, Joint Strike Fighter has previously forecast an earlier date than its 30 June 2019 forecast, and has experienced slippage from that previous forecast, partially offsetting its schedule recovery. Other projects with schedule recovery offset by slippage are AWD Ships, Growler, Overlander Medium/Heavy, Additional MRTT, ANZAC ASMD 2B, HATS, Collins R&S, and LHD Landing Craft. In total, these projects have contributed 35 months of schedule recovery to the Major Projects; however, ANAO analysis (for example, in Table 2 and Figure 11) excludes this effect to portray the complete amount of slippage experienced by the Major Projects.

Figure 9: Projects’ original and 30 June 2019 FOC forecasts

 

 

Note 1: 

indicates that the forecast FOC date for the project at 30 June 2019 is earlier than the original FOC date.

 

Source: ANAO analysis of the 2018–19 PDSSs.

In-year schedule performance

2.44 In 2018–19, there was schedule slippage of 108 months in the forecast achievement of FOC across the 26 Major Projects, as shown in Figure 10, below. In-year project performance, measured by slippage over the last 12 months, may not reflect the project trend.

2.45 In-year schedule slippage occurred for the following 13 projects¹¹⁵ (the explanation provided, drawn from the 2018–19 PDSSs, may also include the reasons for prior slippage):

• AWD Ships — the variance reflects rescheduling of the FOC milestone until after completion of Combat System Ship Qualification Trials;
• P-8A Poseidon — the variance reflects minor rescheduling of the FOC milestone;
• Growler — the variance reflects minor rescheduling of the FOC milestone;
• Overlander Medium/Heavy — the variance reflects a more accurate forecast of the duration of activities leading up to FOC;
• Repl Replenishment Ships — the project’s understanding of FOC requirements has matured and more time will be required to achieve them;
• Battle Comm. Sys. (Land) 2B — delays in interfacing projects required a change in this project’s technical solution. More time is required to conduct design, testing and evaluation of the new solution;
• Additional MRTT — minor schedule savings were predicted in 2017–18, but were not realised;
• ANZAC ASMD 2B — administrative processes delayed the declaration of FOC;
• HATS — minor delays to align with the completion time frame agreed to in the project’s MAA;
• Maritime Comms — delay to FOC to allow for Navy to conduct processes following completion of FMR;
• Battle Comm. Sys. (Land) 2A — administrative processes are still required to declare FOC;
• UHF SATCOM — the project has experienced further delay to FOC due to contractor delays and additional security certification requirements by the US Government; and
• LHD Landing Craft — final operational test and evaluation trials were rescheduled for Quarter 3, 2019.

Figure 10: In-year (2018–19) schedule changes to achieving FOC

 

 

Note: Defence’s PDSSs indicate that 13 of the 26 Major Projects Report projects did not record changes to their Final Operational Capability dates this year.

Source: ANAO analysis of the 2018–19 PDSSs.

Schedule performance by year of entry to MPR

2.46 Figure 11, below, shows the accumulated schedule slippage of the Major Projects included in the MPR reports from 2008–09 to 2018–19.¹¹⁶ Table 10 provides the details of the specific projects included in the analysis. The figure shows that over half of the total schedule slippage across the Major Projects covered in the 2018–19 report (57.6 years or 691 months) is made up of the slippage from the oldest projects, added in 2008–09 and 2009–10.

Figure 11: Schedule slippage by year of entry to the MPR in 2018–19 (in years)

 

 

Note: The total schedule slippage in 2018–19 across the 26 projects is 691 months.

Source: ANAO analysis of the PDSSs in Major Projects Reports.

Table 10: Year of entry for projects included in Figure 11 analysis

Source: ANAO analysis of the PDSSs in published Major Projects Reports.

2.47 Table 11 shows that 16 per cent (112 of 691 months) of the total schedule slippage across the 2018–19 Major Projects is attributed to the sole remaining project approved prior to the Kinnaird reforms, Collins R&S.

Table 11: Project slippage

Note 1: These figures do not add horizontally due to the exclusion of schedule reductions over the life of the project. Refer to footnote 26.

Note 2: Collins R&S was approved prior to the implementation of the Kinnaird reforms. It accounts for 16% of the total schedule slippage.

Source: ANAO analysis of the 2018–19 PDSSs.

Capability performance analysis

2.48 Defence defines capability as the power to achieve a desired operational effect in a nominated environment, within a specified time, and to sustain that effect for a designated period.¹¹⁷ An operational effect is achieved by combining the nine Fundamental Inputs to Capability — organisation, command and management, personnel, collective training, major systems, facilities and training areas, supplies, support, and industry¹¹⁸ — and undertaking designated operations.

2.49 In acquiring Defence platforms and systems, a range of documentation (including capability definition, operational concept, function and performance specification, and Test and Evaluation Master Plans) is developed, which establishes the detailed requirements/performance attributes to be achieved.

2.50 Since the 2009–10 MPR, capability reporting¹¹⁹ has been based on Defence’s prediction of the final capability that would be achieved on the basis of deliverables and/or activities completed. This assessment of capability performance (Expected Capability) is measured against the Materiel Release Milestones (MRMs) and Completion Criteria specified in each project’s Materiel Acquisition Agreement (MAA). This is distinct from an assessment of whether milestones will be achieved on schedule. As the ANAO has previously noted, this data involves ‘making certain assumptions in forecasting achievements and is therefore subjective in approach …’.¹²⁰

2.51 For example, for the LHD Landing Craft project, in the 2015–16 MPR Defence predicted and reported that 99 per cent of elements of capability had a ‘high level of confidence of delivery’, with the capability to transport heavy loads (of up to 65 tonnes) still requiring trials prior to declaration of capability achievement; this capability was assessed as ‘under threat, considered manageable’.

2.52 These trials have been delayed on multiple occasions since 30 June 2016, as reported to the JCPAA by Defence at public hearings.¹²¹ The 2018–19 PDSS continues to report a one per cent Amber rating corresponding to the outstanding trials. This indicates that the subjective 2015–16 forecast of the capability achievement may not have been accurate, with the capability proving harder to achieve than expected.

2.53 The Chief of Navy declared FOC for the LHD Landing Craft project in November 2019 on the basis of trials conducted in July 2019 demonstrating heavy load carriage of up to 62 tonnes, and a desktop analysis extrapolating the results of the trials to 65 tonnes.¹²²

2.54 A further example is the Battlefield Airlifter project which reported a 100 per cent Green capability prediction at its inclusion in the MPR in 2013–14. However, the 2013–14 PDSS also reported major risks relating to capability deficiencies arising from the US Government divesting from the program, with Australia no longer able to rely on the US Air Force processes. These risks have continued to affect the project, with a mature training system and a number of baseline capability requirements now not expected to be delivered until after FOC. These capability issues were reported in the PDSS Pie Chart for the first time in 2018–19, indicating that the earlier confidence in the ability to achieve the required capability may have been overly optimistic.¹²³

2.55 Over time, the JCPAA has sought the use of a more robust measure of capability performance.¹²⁴

2.56 In October 2017, the JCPAA recommended ‘that the Department of Defence review the procedure for the development of expected capability estimates for future Major Projects Reports. The outcomes of this review should be provided to the Committee within six months of the tabling of this report. Further, the Committee requests that Defence provide a progress report within three months of the tabling of this report.’¹²⁵

2.57 Defence made a submission to the Committee in March 2018 regarding this recommendation.

Defence will conduct a schedule baseline validation activity for the Major Projects Report projects to drive greater consistency in schedule reporting.

Once this activity is complete, Defence should be in a better position to investigate a more robust approach to measuring Capability estimates. Utilising the validated baseline data could inform:

• A simple percentage of schedule milestones achieved to measure progress to date. This is a quantitative assessment that relies on the maintenance of a robust project baseline, which is not dissimilar to the approach proposed by ANAO previously;
• CASG working with Force Design to identify how to measure capability, that considers all elements of Fundamental Inputs to Capability, and that is suitable for unclassified publication; and
• Defence is working towards a new whole of organisational reporting system (the Enterprise Resource Planning (ERP) System) which is expected to roll-out in Financial Year 2020-21. CASG will endeavour to incorporate the work conducted with Force Design on measuring capability.¹²⁶

2.58 In September 2018, the JCPAA noted that ‘Materiel Capability Delivery Performance charts continue to be ambiguous in displaying actual current capability levels.’¹²⁷

2.59 Defence advised the ANAO in November 2018 that partial progress had been made on its schedule baseline validation activity discussed with the JCPAA. The ANAO notes that a measurement of schedule milestones will not necessarily reflect a measurement of capability delivered.

Modified method of capability reporting

2.60 In light of the above, in 2015–16 the ANAO developed a measure of key materiel capabilities delivered (Capability Delivery Progress). This presents a current assessment of the capability delivered, which differs from Defence’s prediction of final capability. The information used in forming the ANAO’s assessment is primarily based on Section 4.2 of the PDSS, which sets out the capability elements required to achieve Initial Materiel Release and Final Materiel Release, combined with other information in the PDSS reporting the delivery of equipment/achievement of these requirements toward FOC.

2.61 Noting that a system of capability reporting with a robust methodology applicable to materiel acquisition does not exist within Defence, the information presented below is a more meaningful reflection of current project progress than an end-state prediction.

Capability Delivery Progress and Project Maturity

2.62 Figure 12, below, sets out each project’s Capability Delivery Progress against Project Maturity.¹²⁸ It shows that Capability Delivery Progress lags Project Maturity for the majority of projects (19 of 26). This relationship is expected as projects will typically develop confidence in the ability to deliver their scope and capability through testing and demonstration of capability components (for example, design reviews and acceptance tests) prior to delivery of the majority of equipment.

2.63 Figure 12 also shows that Capability Delivery Progress lags Project Maturity by 20 per cent or more in 12 projects, and for 10 of these, Capability Delivery Progress lags by 50 per cent or more.

Figure 12: Project snapshot — Capability Delivery Progress and Project Maturity

 

 

Source: ANAO analysis of the 2018–19 PDSSs.

2.64 As noted in paragraph 2.11, Defence’s project maturity framework attributes approximately 50 per cent of total project maturity at Second Pass Approval.¹²⁹ As a result, Defence’s project maturity framework is not appropriately structured to assign project maturity progress throughout the project life cycle, particularly within the acquisition phase, which is predominantly the longest and most expensive component.

2.65 Figure 12 also highlights a continuing issue with the level of specification of capability elements. For the projects that show little or no Capability Delivery Progress, this can be attributed to Defence’s high level description of requirements in the capability elements. This indicates that it would be worthwhile for Defence to undertake additional work to track project progress. In respect of the four projects that show no capability delivery at 30 June 2019, progress was as follows:

• Offshore Patrol Vessel — this project had completed almost all major design review activities;
• Repl Replenishment Ships — the first ship had been launched and was being outfitted prior to acceptance, while 70 per cent of blocks had been erected for the second ship;
• CMATS — this project was in early stages of procurement, and was progressing through early design processes; and
• Maritime Comms — this project had progressed through design reviews and commenced ship installations.

2.66 Further, Figure 12 indicates that:

• one project, ANZAC ASMD 2B, has delivered all of the required capability;
• three projects, HATS, Battle Comm. Sys. (Land) 2A, and LHD Landing Craft, have delivered essentially all of their capability with only minor items of capability or administrative processes remaining prior to declaration of FOC;
• 18 projects are still to deliver part of their capability; and
• one project, AWD Ships, will not deliver all of the required capability.

Part 2. Defence Major Projects Report is available to download in a separate PDF file, or in the complete report PDF which is available to download at Related documents on this page.

.

Due to the complex nature of the material, the Project Data Summary Sheets (PDSS) are available in PDF only. You can view the PDSS section in the complete report PDF which is available to download at Related documents on this page, or download individual PDSS PDF files below:

• New Air Combat Capability
• Air Warfare Destroyer Build
• Maritime Patrol and Response Aircraft System
• Multi_Role Helicopter
• Offshore Patrol Vessel
• EA_18G Growler Airborne Electronic Attack Capability
• Medium Heavy Capability, Field Vehicles, Modules and Trailers
• Future Naval Aviation Combat System Helicopter
• Amphibious Ships (LHD)
• Protected Mobility Vehicle – Light
• Battlefield Airlift – Caribou Replacement
• Maritime Operational Support Capability
• Civil Military Air Traffic Management System
• Battlespace Communications Systems 2B
• Additional KC_30A Multi_role Tanker Transport
• ANZAC Anti_Ship Missile Defence 2B
• Collins Class Communications and Electronic Warfare Improvement Program
• Pacific Patrol Boat Replacement
• Helicopter Aircrew Training System
• Collins Class Submarine Reliability and Sustainability
• Night Fighting Equipment Replacement
• Maritime Communications Modernisation
• Battlespace Communications System 2A
• ANZAC Air Search Radar Replacement
• Indian Ocean Region UHF SATCOM
• Amphibious Watercraft Replacement

The JCPAA 2018–19 Major Projects Report Guidelines part is available to download in a separate PDF file, or in the complete report PDF which is available to download at Related documents on this page.