Foreword

Defence’s Major Projects have continued to be the subject of considerable parliamentary and public interest throughout 2014–15, and are expected to remain so into the foreseeable future. With acquisition expenditure expected to nearly double over the period to 2018–19, and government investment in Defence acquisitions covering major land, sea and air platforms, there is also strong justification to maintain a high level of transparency and accountability over this activity.

Following the delisting of the Defence Materiel Organisation (DMO) from 1 July 2015, and consistent with the recommendations of the Government’s First Principles Review: Creating One Defence, the functions of the DMO were merged back into Defence under the new Capability Acquisition and Sustainment Group. The ongoing reporting of the status of Major Projects under the new Capability Acquisition and Sustainment Group is the subject of this, the eighth Major Projects Report.

The former DMO oversaw the introduction of a Military-Off-The-Shelf focused acquisition strategy for Major Projects, following the Defence Procurement Review 2003 (Kinnaird Review). This has resulted in an improvement in schedule performance over time, with current analysis showing that 73 per cent of the total schedule slippage across the Major Projects relates to projects approved prior to DMO’s demerger from Defence in 2005. It is important that Defence continue to pursue these improvements in project delivery.

For the first time, the 2014–15 Major Projects Report’s scope includes the project financial assurance statement, which provides readers with an articulation of each project’s financial position in relation to delivering project capability. The independent conclusion now provides assurance over these statements within each Project Data Summary Sheet.

The ongoing support of the Joint Committee of Public Accounts and Audit (JCPAA) remains important to the development of the Major Projects Report, providing guidance and insight through their considerations. Each year the JCPAA endorses the Guidelines for the review, and provides direction and recommendations to assist the development of future Major Projects Reports.

As previously, this year’s review continued the strong working relationship between the ANAO and Defence departmental staff. The three Defence service chiefs and the Chief Information Officer, and industry, also provided valuable input to the review.

Introduction

1. Due to the high cost and impact on the economy, contribution to national security and the challenges involved in completing them within budget, on time and to the required level of capability, major Defence equipment acquisition projects (Major Projects) have continued to be the subject of considerable parliamentary and public interest throughout 2014–15. In this respect, the proposed Defence White Paper is expected to set out the Government’s priorities for future capability investment over the next 20 years. This is expected to include investment of over $89 billion in the acquisition of new submarines, frigates, offshore patrol vessels and other specialist naval vessels, following the Government’s commitment to a ‘continuous build’ of surface warships in Australia.¹ Additionally, the land vehicle fleet will be replaced, new and improved air capabilities will be delivered, and investments will be made in supporting infrastructure, personnel and information technology systems.²

2. The proposed Defence White Paper is to be released during the implementation of the recommendations from the Government’s First Principles Review: Creating One Defence (First Principles Review). Notably, as an outcome of this process, the former Defence Materiel Organisation (DMO) was delisted from 1 July 2015, and its functions merged back into the Department of Defence (Defence), under the new Capability Acquisition and Sustainment Group.³ Previously, the DMO had provided support to the Australian Defence Force’s (ADF) operations through the acquisition and sustainment of ADF capabilities⁴, expending some $5.3 billion on major and minor capital acquisition projects in 2014–15.⁵

3. While acquisition alone does not generate new capability for the ADF, the DMO performed a significant role in Defence acquisition. As such, the DMO was the focus of the Australian National Audit Office’s (ANAO’s) review work on Major Projects, including for the majority of this, the 2014–15 Major Projects Report (2014–15 MPR).

4. Following prescription as a Financial Management and Accountability Act 1997 agency in 2005, the DMO oversaw the introduction of a Military-Off-The-Shelf focused acquisition strategy, the analysis of which is depicted in Figures 8 and 9. The DMO also introduced a performance reporting regime in 2004, the Maturity Score Framework (discussed further at paragraphs 1.48 to 1.58), the data from which has been included within the analysis in the MPRs, to assist users in assessing the progress of projects over time. The DMO was responsible for acquisition risk management and financial management frameworks; assisted in the development of Materiel Acquisition (and Sustainment) Agreements, which progressed the level of governance in Defence acquisition; and introduced programs of professionalisation for project managers throughout Defence and Defence Industry.

5. The newly formed group will now manage the process of bringing new capabilities into service, including the Fundamental Inputs to Capability⁶, for example, the provision of personnel, training and command. The ANAO will continue to review Defence acquisition in the 2015–16 MPR, as the group assumes its acquisition responsibilities, and while progress on the implementation of the First Principles Review recommendations is ongoing.⁷

The 2014–15 Major Projects Report

6. This eighth report covers 25 of Defence’s Major Projects (2013–14: 30; 2012–13 and 2011–12: 29), and builds on the earlier work to improve the transparency of, and accountability for, the status of Major Projects. The Major Projects review is supported by the commitment of the Joint Committee of Public Accounts and Audit (JCPAA), ‘…to maximise transparency and accountability in the Defence acquisition process for Major Projects that have been managed by DMO and will continue to be managed by the Department of Defence in future.’⁸

7. The benefits of this report have been noted by a variety of stakeholders, including Ministers, Parliamentary Committee members, industry and the media.

8. The ANAO’s review of Major Projects is completed in conjunction with the regular program of performance and financial statement audits conducted within the Defence portfolio. While by its nature, the report is not as in depth as a performance audit, it provides an opportunity to analyse data across a consistent range of projects over time, and complements the ANAO’s other Defence auditing and assurance functions.

2014–15 Major Projects selected for review

9. Projects are selected based on criteria included in the 2014–15 Major Projects Report Guidelines (the Guidelines), as endorsed by the JCPAA⁹, and provide a selection of the most significant Major Projects managed by Defence. The total approved budget for the Major Projects included in the report is approximately $60.5 billion, covering nearly 63 per cent of the budget within the Approved Major Capital Investment Program of $96.1 billion.¹⁰ The projects and their approved budgets are listed in Table 1, below.

Table 1: 2014–15 MPR projects and approved budgets at 30 June 2015

Note 1: P-8A Poseidon and Maritime Comms are included in the MPR program for the first time in 2014–15.

Note 2: Once a project is selected for review, it remains within the portfolio of projects under review until the JCPAA endorses its removal, normally once it has met the capability requirements of the Australian Defence Force.

Source: See the Project Data Summary Sheets in Part 3 of this report.

Report objective and structure

10. The objective of this report is to provide the Auditor-General’s independent assurance over the status of the selected Major Projects, as reflected in the Statement by the Secretary of Defence, and the Project Data Summary Sheets (PDSSs), prepared by Defence. Assurance from the ANAO’s review of the preparation of the PDSSs is conveyed in the Independent Review Report by the Auditor-General, prepared pursuant to the endorsed Guidelines, and included in Part 3 of this report (pp. 141–144).

11. For the first time in 2014–15, the review’s scope includes the project financial assurance statement within each PDSS, which was first introduced in the 2011–12 review. The Independent Review Report by the Auditor-General now provides assurance over these statements within each PDSS.

12. Excluded from the scope of the ANAO’s review is PDSS data on the identification of Risks and Issues, the Measures of Materiel Capability Delivery Performance, and ‘forecasts’ of future dates and the achievement of future outcomes. Accordingly, the Independent Review Report by the Auditor-General does not provide any assurance in relation to this information.

13. It is intended that all components of the PDSSs will be included within the ANAO’s scope of review, once Defence and ANAO work programs can facilitate the review and conclusion over all of the components of the PDSSs. However, the exclusions to the scope of the review noted above, are due to the lack of a system or systems from which to provide complete and accurate evidence¹¹, in a sufficiently timely manner to facilitate the review. This has been an area of focus of the JCPAA over a number of years¹², and will continue to be a part of the Defence and ANAO work program into the future.

14. The ANAO’s analysis on the three key elements of the PDSSs—cost, schedule and the progress towards delivery of required capability, and in particular, longitudinal analysis across these key elements of projects over time, are contained in Part 1 (pp. 1–80). The ANAO’s analysis over other elements of the PDSSs, for example, project maturity and elements excluded from the formal scope of the review, is also included in this part, to provide readers with a balanced perspective over all key acquisition elements.

15. Further insights and context by Defence on issues highlighted during the year are contained in Part 2 (pp. 81–138)—although not included within the scope of the review by the ANAO.

16. Part 4 includes the Guidelines endorsed by the JCPAA (pp. 451–479), which provide the criteria for the compilation of the PDSSs by Defence and the ANAO’s review. Figure 1, below, depicts the key parts of this report.

Figure 1: 2014–15 Report structure

Refer to paragraphs 10 to 16 in Part 1 of this report.

Note: To assist in conducting inter-report analysis, the presentation of data remains largely consistent and comparable with the 2013–14 MPR.

17. For each Major Project, a corresponding PDSS includes unclassified information on project performance, prepared by Defence in accordance with the Guidelines. Additionally, as projects appear in the MPR for multiple years, changes to the PDSS from the previous year are depicted in bold purple text.

18. Each PDSS comprises:

• Project Header: including name; capability and acquisition type; approval dates; total approved and in-year budgets; stage; complexity; and image;
• Section 1—Project Summary: including description; current status, financial assurance and contingency statement; context, including background, uniqueness, major risks and issues; and other current sub-projects;
• Section 2—Financial Performance: including budgets and expenditure; variances; and major contracts in place (in addition to quantities delivered as at 30 June 2015);
• Section 3—Schedule Performance: providing information on design development; test and evaluation; and forecasts and achievements against key project milestones including Initial Materiel Release (IMR), Final Materiel Release (FMR)¹³, Initial Operational Capability (IOC) and Final Operational Capability (FOC)¹⁴;
• Section 4—Materiel Capability Delivery Performance: provides a summary of Defence’s assessment of its progress on delivering key capabilities and whether the milestones were achieved¹⁵;
• Section 5—Major Risks and Issues: outlines the major risks and issues of the project and remedial actions undertaken for each;
• Section 6—Project Maturity: provides a summary of the project maturity as defined by Defence and a comparison against the benchmark score;
• Section 7—Lessons Learned: outlines the key lessons that have been learned at the project level (further information on lessons learned by Defence are included in Defence’s Appendix 3); and
• Section 8—Project Line Management: details current project management responsibilities within Defence.

The role of the Joint Committee of Public Accounts and Audit

19. Influential in establishing the MPR, the JCPAA has taken an active role in the development and review of the MPR program. Each year, the Committee considers the draft Guidelines, incorporating the selection of projects for review, and provides the Committee’s views in relation to the Guidelines’ content and development, prior to their endorsement. Following endorsement by the Committee, the Guidelines provide the criteria for Defence’s preparation of the PDSSs and the ANAO’s review.¹⁶

20. The main changes to the 2014–15 MPR Guidelines follow on from the recommendations made by the JCPAA in Report 442, Review of the 2012–13 Defence Materiel Organisation Major Projects Report, May 2014. Recommendations 2 and 7 from this report requested changes to the PDSS template to increase analysis and disclosure in relation to published estimates figures and 30 June actual expenditure¹⁷, and additional detail supporting the capability performance information. Implementation of these changes became effective in 2014–15.

21. Following the tabling of the 2013–14 MPR, the Committee published Report 448, Review of the 2013–14 Defence Materiel Organisation Major Projects Report, in May 2015. The JCPAA’s recommendation is set out below¹⁸:

22. As noted previously, the Committee’s recommendations contribute to the development of the MPR each year and the ANAO was advised that the formal response by Defence to this recommendation was provided to the Committee in late November 2015. At the time of preparing this report actioning of the recommendation is summarised below:

• Recommendation 1—to progress the enhancement of Defence sustainment reporting, an in-camera briefing was provided by Defence in November 2015. This provided the JCPAA with the opportunity to seek answers to (classified) sustainment matters directly from the relevant Capability Managers, expanding on information already provided in publicly available documents including the Defence Capability Plan, Portfolio Budget Statements, the Defence Annual Report and the MPR. To support this process, the JCPAA requested a sustainment brief from the ANAO, which was provided on 7 October 2015, and discussed at a private briefing of the JCPAA on 22 October 2015.

23. While Defence sustainment¹⁹ projects are generally outside the scope of the review, the Collins R&S project (which is defined as a sustainment project by Defence) has been included within the review at the request of the JCPAA since 2009–10. In addition, while ARH Tiger Helicopters and Collins RCS have been transferred to ‘sustainment’ by Defence, they remain included within the 2014–15 review, following the endorsement of the 2014–15 Guidelines.

24. In 2014–15, the ANAO also completed two performance audits on the former DMO’s outcomes from its Programme 1.2: Management of Capability Sustainment. The performance audit examining the management of the disposal of specialist military equipment was tabled in February 2015²⁰, and the performance audit examining the contribution made by Materiel Sustainment Agreements to the effective sustainment of specialist military equipment, was tabled in April 2015.²¹

Overall outcomes

25. This eighth report continues to review four of the nine Defence Major Projects which were initially introduced in the 2007–08 MPR²², and has continued to introduce new projects up to the originally agreed maximum of 30 projects. In 2014–15, 25 projects are reviewed. Maintaining a stable portfolio of projects over time has facilitated transparency and accountability for performance relating to cost, schedule and progress towards delivering the key capabilities of Major Projects, and provides opportunities for further longitudinal and other analysis into the future.

The 2014–15 Major Projects review (Chapter 1)

26. Under section 19A(5) of the Auditor-General Act 1997, the ANAO has reviewed the PDSS data as a priority assurance review and presents the Independent Review Report by the Auditor-General. As part of this review, the ANAO assessed the progress of Defence in addressing previously raised issues in relation to the administration of Major Projects.

27. In 2014–15, issues were noted within the following areas of project management:

• budget and project management, with acknowledgement by AWD Ships that the project did not have sufficient funds to complete delivery of the approved capability and required $1.2 billion in additional funding.²³ See further explanation in paragraph 1.24;
• price indexation and budget allocations, and inconsistency in the determination and recording of contingency funds (Section 1 of the PDSS). See further explanation in paragraphs 1.32 to 1.35;
• variability in the interpretation of project progress towards delivering required capability (Section 4 of the PDSS). See further explanation in paragraph 2.60;
• inconsistency in the recording and reporting of major risks and issues by project offices, and reporting within the mandated Predict! and Excel risk management systems²⁴ (Section 5 of the PDSS). See further explanation in paragraphs 1.40 to 1.44; and
• inconsistency in the application of the project maturity framework²⁵ (although improved from 2013–14), which is weighted towards pre Second Pass Approval processes, reducing the ability to adequately indicate progress during the acquisition phase (Section 6 of the PDSS). See further explanation in paragraphs 1.48 to 1.58.

28. In 2014–15, the results of the ANAO’s priority assurance review of the 25 PDSSs, was that nothing has come to the attention of the ANAO that causes us to believe that the information and data in the PDSSs, within the scope of our review, has not been prepared, in all material respects, in accordance with the Guidelines.

Analysis of projects’ performance (Chapter 2)

29. The data reviewed in the PDSSs covers the three major dimensions of project performance: cost, schedule, and progress towards delivering the planned capability. Table 2 below, provides summary data on the Defence approved budget, schedule performance and progress toward delivering capabilities for the Major Projects covered in this report, and compares data against that reported in previous MPR editions.

30. A significant contributor to the reduction of slippage shown in the 2014–15 MPR (1 115 months to 768 months), is the removal of a number of projects which had not reached Final Operational Capability (FOC) by 30 June 2014. These projects are Hornet Upgrade (39 months), FFG Upgrade (132 months), HF Modernisation (147 months) and SM-2 Missile (26 months), totalling 344 months of the total net decrease shown of 347 months.

31. Of the above projects, Hornet Upgrade achieved FOC for Phase 2.3 in October 2014. The HACTS/HIP component, which will upgrade aircraft simulators, was transferred out of the project, and is currently due to achieve FOC in January 2017. In addition, SM-2 Missile, which was reported in the 2013–14 MPR with a forecast FOC date of February 2015, achieved FOC in June 2015. The FFG Upgrade and HF Modernisation projects have not achieved FOC as yet, with current forecast dates of March 2016 and December 2016 respectively.

Table 2: Summary longitudinal analysis

Refer to paragraphs 29 to 44 in Part 1 of this report.

Note 1: The data for the 25 Major Projects in the 2014–15 MPR compares the data from projects in the 2013–14 MPR and 2012–13 MPR.

Note 2: Slippage refers to the difference between the original government approved date and the current forecast date. These figures exclude schedule reductions over the life of the project.

Note 3: Based on the 29 projects from the 2011–12 MPR, 26 projects from the 2012–13 MPR and 23 projects from the 2013–14 MPR respectively.

Note 4: The grey section of the table is excluded from the scope of the ANAO’s priority assurance review. See further explanation in paragraph 12 in Part 1 of this report.

Cost

32. Within the review period, all projects except for AWD Ships, reported that they could continue to operate within the total approved budget of $60.5 billion.²⁶ The joint Ministerial announcement on 22 May 2015, by the then Minister for Defence and Minister for Finance, stated that the AWD Ships project would require additional approved funding of $1.2 billion to deliver the required capability (approved July 2015).²⁷

33. The total budget for Major Projects included in this MPR has increased by $18.5 billion (44.0 per cent) since Second Pass Approval. Refer to Table 3, below.

Table 3: Budget variation post Second Pass Approval by Variation type

Note: Variations greater than $500 million are depicted in this table. For the breakdown of in-year variation, refer to Table 6 of this report.

Source: ANAO analysis of the 2014–15 PDSSs.

Schedule

34. Maintaining Major Projects on schedule remains an ongoing challenge for Defence²⁸; in turn affecting when the capability is made available for operational release and deployment by the ADF, and increasing the cost to delivery.²⁹ In the 2014–15 MPR, the total schedule slippage for the 25 Major Projects as at 30 June 2015 is 768 months (2013–14: 1 115 months) when compared to the initial schedule first approved by government. This represents a 28 per cent (2013–14: 36 per cent) increase on the originally approved schedule. Refer to Table 4, below for details of in-year and total schedule slippage by project, for projects in the 2014–15 MPR.

Table 4: Schedule slippage from original planned Final Operational Capability

Source: ANAO analysis of the 2014–15 PDSSs.

35. While it should be noted that platform availability can contribute to the slippage within some projects, significant delays relate to those projects with the most developmental content. For example, AWD Ships, Wedgetail, MRH90 Helicopters and ANZAC ASMD 2B.

36. Additional ANAO analysis (refer to Figure 8) presents project slippage as reported in each of the eight reports against the Defence classification of projects as Military-Off-The-Shelf (MOTS), Australianised MOTS or developmental.³⁰ These classifications are a general indicator of the difficulty associated with the procurement process. This figure highlights, prima facie, that the more developmental in nature a project is, the more likely it will result in project slippage, as well as demonstrating one of the advantages of selecting MOTS acquisitions.³¹ First included in the 2013–14 report, Figure 9 provides analysis of projects either completed, or removed from the review. While initial suggestions by Defence were that analysis in Figure 8 did not allow for future slippage in MOTS projects, Figure 9 shows that MOTS acquisitions assist in reducing schedule slippage.

37. It should also be noted that the impact of a MOTS focused acquisition strategy has received significant attention since the inception of the 2007–08 review and Figure 9 shows the benefit of collating data over a consistent portfolio of projects across time, to allow the longitudinal analysis first sought by the JCPAA. Further development and results in this area will be covered in the 2015–16 report.

38. The reasons for schedule slippage vary but primarily reflect the underestimation of both the scope and complexity of work, particularly for Australianised MOTS and developmental projects (see paragraphs 2.29 to 2.31 in Part 2).

Capability

39. The third major aspect of project performance examined in this report is progress towards the delivery of capability required by government and specified by the Australian Defence Force. Assessment of expected capability delivery by Defence is outside the scope of the Auditor-General’s formal review conclusion, but is included in the analysis to provide an overall perspective of the three major components of project performance.

40. The Defence PDSSs reflect that the 25 projects in this year’s report will deliver all of their key capability requirements, recognising that some elements of the capability required may be under threat, but are considered manageable (assessed as either green or amber). This is consistent with the 2013–14 presentation, and reflects only one project office currently having significant challenges (2013–14: five). Refer to Table 5, below.

Table 5: Longitudinal Expected Capability Delivery

Note 1: 155mm Howitzer was removed from the MPR in 2014–15. However, the Course Correcting Fuze element of the project has been transferred to another project and will be delivered by LAND 17 Phase 1C1 Lightweight Howitzer.

Note 2: FFG Upgrade was removed from the Major Projects Report in 2014–15 prior to achieving Final Operational Capability.

Source: PDSSs in published Major Projects Reports.

41. However, the results of analysis by the ANAO show that some project managers may have taken different perspectives in assessing future achievements in relation to delivering final capability. For example, the ARH Tiger Helicopters project, faces significant capability risks and issues in relation to delivering the required Rate of Effort (flying hours), and technological obsolescence caused by delays in delivery, which impact future use. The expected impact of these risks and issues has not translated into Defence’s assessment of future capability performance, although it could reasonably be assumed to have a long term capability effect.

42. Similarly, the initial results of testing for the LHD Landing Craft project, highlight issues of significance to be addressed prior to project conclusion, not disclosed as impacting expected capability delivery. This is reflective of reporting against arrangements, post government approval, where responsibilities have been allocated within the acquisition and capability delivery framework, to the Chief of Navy post FMR (see also paragraph 2.60).

43. This year, as reported by Defence, the delivery of only three per cent (2013–14: four per cent) of the key capabilities is considered to be under threat but manageable, which as noted above, may be overly optimistic. The project with some elements under threat but considered manageable is Joint Strike Fighter. Further details are outlined at paragraph 2.65.

44. In addition to expected capability delivery, Defence has continued the practice of including declassified information on settlement actions for projects, in the interests of providing greater transparency to readers of this report. As well as prior settlements for projects included in this report³², in 2014–15, five project offices reported receiving goods and services as a result of liquidated damages. These are LHD Ships³³, ARH Tiger Helicopters³⁴, Air to Air Refuel³⁵, Maritime Comms³⁶ and Battle Comm. Sys.³⁷

Developments in acquisition governance (Chapter 3)

45. Consistent with previous years, developments in acquisition governance processes are covered in the ANAO’s review, with general indications of positive impacts from some of the more recent initiatives. As might be anticipated, while some initiatives continue to mature, others require further progress prior to achieving their intended impact. These developments broadly relate to the following key acquisition governance areas.

Gate Review Boards

46. First introduced in 2008, the Gate Review (acquisition) process³⁸ was designed to provide the Senior Executive with assurance that all identified risks for a project are manageable, and that costs and schedule are likely to be under control prior to a project passing various stages of its life cycle. The Gate Review process continues to evolve, with additional types of Gate Reviews introduced in 2013–14, as fewer acquisition projects will be eligible for (acquisition) Gate Reviews, and more projects are transferred into sustainment. These include Project Initiation Gate Reviews, Acquisition and Support Concept Gate Reviews (pre First Pass), and Sustainment Gate Reviews. The introduction of Sustainment Gate Reviews in particular, provides continuing oversight of sustainment capabilities, which was previously not available.

Projects of Concern

47. First established in 2008, the Projects of Concern process was implemented to focus the attention of the highest levels of government, Defence and industry on remediating problem projects. The process has continued to play an important, although limited role, across the portfolio of MPR projects. As at 30 June 2015, two MPR projects were continuing projects of concern: AWD Ships and MRH90 Helicopters. The Air to Air Refuel project was removed from the Projects of Concern list in March 2015.

Quarterly Project Performance Report

48. The Quarterly Project Performance Report (QPPR) was implemented in 2014 to replace the Early Indicators and Warnings system (first announced in May 2011), to identify problems with projects in the formative stages of the life cycle. The QPPR aims to provide senior stakeholders within government and Defence with a clear and timely understanding of key risks facing projects, and their progress against cost, schedule and capability performance. In 2014–15, the QPPR identified three MPR projects for continuing underperformance: Battlefield Airlifter, Overlander Light and UHF SATCOM. The Overlander Light project was removed from the list of underperforming projects in the April–June 2015 QPPR. The ongoing issues highlighted for Battlefield Airlifter and UHF SATCOM align with the results of the ANAO’s review, where delays to progress have impacted the delivery schedule of the two projects.

Joint Project Directives

49. There has been a longstanding issue for Defence in maintaining complete and accurate records of government approvals for Major Projects³⁹, which led to the development of Joint Project Directives (JPDs). The implementation of JPDs is intended to provide an appropriately declassified means of promulgating, accurately and completely, the details of project approvals by government.

50. The introduction of a requirement for JPDs⁴⁰ in 2009–10, for all projects approved by government from March 2010, is maturing and expected to have a greater influence over the portfolio of MPR projects in future years. To date, all of the eleven MPR projects, approved from 1 March 2010, have completed a JPD.⁴¹ The ANAO will continue to take JPDs into account in its review program in future years, where these have been prepared. However, this will be dependent on the completeness and accuracy of JPDs, in relation to recording the detail of government approvals.

Business systems rationalisation

51. Defence’s business systems⁴² rationalisation is aimed at consolidating processes and systems in order to provide a more manageable system environment. The new Capability Acquisition and Sustainment Group’s Business Information Management Branch advised that it is making progress in this area through the development of an Integrated Project Management System, which was being progressively built into the then DMO corporate data warehouse in 2014–15.

Project management and skills development

52. Project management and skills development within Defence and the Defence Industry is a key challenge for the Government and industry alike. Over the last decade, more than $300 million has been provided by government to assist with professionalising Defence staff and up-skilling participants within the Defence Industry. While it is widely believed that Defence activities have increased professional competencies held by Defence staff, the measurement of the impact within industry has been limited. The ANAO will continue to review project management and skills development programs in the 2015–16 MPR.

53. Consistent with previous years, the ANAO’s detailed assessment of these governance initiatives is contained in Chapter 3 of Part 1.

Introduction

1.1 This chapter provides an overview of the 2014–15 Major Projects Report (2014–15 MPR) review’s scope and approach, implemented by the Australian National Audit Office (ANAO) for the review of the Project Data Summary Sheets, and the subsequent results of the MPR review.

1.2 During the 2014–15 review period the Government released the First Principles Review: Creating One Defence (First Principles Review), a major government review of the Australian Defence Organisation. As a result, the Defence Materiel Organisation (DMO), the focus of the majority of this review, was delisted from 1 July 2015, and merged back into the Department of Defence (Defence).

1.3 The re-merger of the DMO, a large project management organisation, into the Capability Acquisition and Sustainment Group of Defence, is complex, and expected to take time to fully complete. Consequently, broader managerial changes are in the process of being developed and implemented within Defence, at the time of this report. The ANAO has referred to the effects of the change process where appropriate, however, the development and implementation of changes within Defence following the recommendations of the First Principles Review are not expected to be completed until 2017–18.

The impact of the First Principles Review on materiel acquisition

1.4 The First Principles Review was released on 1 April 2015. The First Principles Review is the thirty-sixth substantive government review of Defence since the 1973 Tange Review, Australian Defence, Report on the reorganistion of the Defence group of departments.⁴³ The First Principles Review’s approach to reforming Defence includes addressing ‘waste, inefficiency and rework’⁴⁴ by looking holistically at Defence’s business structures, materiel acquisition and sustainment capability, and the efficiency and effectiveness of practices within the department.

1.5 The review made 76 recommendations, and supports a business model which is comprised of three identifiable key features:

• a stronger and more strategic centre within the department;
• an end-to-end approach for capability development with Capability Managers assigned clear authority and accountability; and
• enablers that are integrated and customer-centric.⁴⁵

1.6 The First Principles Review has recommended that the DMO’s functions, with part of the Capability Development Group’s previous responsibilities⁴⁶ be remerged into the Department of Defence in the newly created group, led by a single Deputy Secretary, who reports directly to the Secretary of Defence. The appointment of the permanent Deputy Secretary of the Capability Acquisition and Sustainment Group, occurred on 31 August 2015. Defence has provided advice on impacts to their organisation as a result of the First Principles Review in Part 2.

Review scope and approach

1.7 In 2012 the Joint Committee of Public Accounts and Audit (JCPAA) identified the review of the Project Data Summary Sheets (PDSSs) as a priority assurance review under section 19A(5) of the Auditor-General Act 1997, providing the ANAO full access to the information gathering powers under the Act. The ANAO’s review of the individual project PDSSs, which are contained in Part 3 of this report, was conducted in accordance with the Australian Standard on Assurance Engagements (ASAE) 3000 Assurance Engagements Other than Audits or Reviews of Historical Financial Information issued by the Australian Auditing and Assurance Standards Board. In addition, and for the first time in 2014–15, review of the project financial assurance statement, included within each PDSS, has been included in the scope of the Independent Review Report by the Auditor-General. See paragraph 1.23 below, for more detail.

1.8 Excluded from the scope of the ANAO’s review is PDSS data on the identification of Risks and Issues, the Measures of Materiel Capability Delivery Performance, and ‘forecasts’ of future dates and the achievement of future outcomes, due to the lack of a system or systems from which to provide complete and accurate evidence⁴⁷, in a sufficiently timely manner to complete the review. Accordingly, the Independent Review Report by the Auditor-General does not provide any assurance in relation to this information.

1.9 While the ANAO’s work is appropriate for the purpose of providing an Independent Review Report in accordance with ASAE 3000, the review of individual PDSSs is not as extensive as individual performance and financial statements audits conducted by the ANAO, in terms of the nature and scope of issues covered, and the extent to which evidence is required by the ANAO. Consequently, the level of assurance provided by this review in relation to the 25 Major Projects is less than that provided by our program of audits.

1.10 However, the Major Projects Report examines systemic issues and provides longitudinal analysis for the 25 projects reviewed, and may also reflect on, or have implications for, general project management practices, including overall performance, or financial matters.

1.11 The review was conducted in accordance with the ANAO Auditing Standards at a cost to the ANAO of approximately $2.2 million.

Areas of review focus

1.12 The ANAO’s review of the information presented in the individual PDSSs included:

• examination of each PDSS and the documents and information relevant to them;
• review of relevant processes and procedures used in the preparation of the PDSSs;
• assessment of the systems and controls that support project financial management, risk management, and project status reporting;
• interviews with persons responsible for the preparation of the PDSSs and those responsible for the management of the 25 projects;
• taking account of industry contractor comments provided to the ANAO and Defence regarding draft PDSS information;
• assessing the assurance by Defence managers attesting to the accuracy and completeness of the PDSSs;
• examination of the representations by the Chief Finance Officer of Defence supporting the project financial assurance and contingency statements, and the independent third-party review of the project financial assurance statements;
• examination of representations, provided by the Capability Managers, relating to each project’s progress toward Initial Materiel Release (IMR) and Final Materiel Release (FMR), and Initial Operational Capability (IOC) and Final Operational Capability (FOC); and
• examination of the ‘Statement by the Secretary of Defence’, including significant events occurring post 30 June, and any supporting management representations.

1.13 The ANAO’s processes and procedures to provide independent assurance over the PDSSs also focused on reviewing project management and reporting arrangements in place that contribute to the overall governance of Major Projects. These included:

• the financial framework, particularly as it applies to the project financial assurance and contingency statements and managing project budgets in the out-turned budget environment (Section 2 of the PDSSs);
• schedule management and test and evaluation processes (Section 3 of the PDSSs);
• the capability assessment framework, as it relates to Defence’s evaluation of the likelihood of delivering key capabilities (Section 4 of the PDSSs);
• ongoing review of the implementation of the Enterprise Risk Management Framework and major risk and issue data (Section 5 of the PDSSs);
• the project maturity framework and reporting and the systems in place to support the provision of this data (Section 6 of the PDSSs); and
• developments in the areas of acquisition governance (Chapter 3 in Part 1).

1.14 This review informed the ANAO’s understanding of the systems and processes supporting the PDSSs for the 2014–15 review period, and highlighted issues in those systems and processes that could be beneficially addressed in the longer term.

Results of the review

1.15 The following sections outline the results of the ANAO’s review, which contribute to the overall conclusion in the Independent Review Report by the Auditor-General for 2014–15.

Financial framework

1.16 The project financial assurance statement was introduced in the 2011–12 Major Projects Report and the contingency statements were introduced for the first time in the 2013–14 report. Together, they are aimed at providing greater transparency of projects’ financial status in the out-turned environment, following the move to out-turned budgeting in 2010, and highlight the use of contingency funding to mitigate projects risks.

1.17 In 2013–14, the ANAO reviewed the financial framework, as it applied to managing project budgets and expenditure in the out-turned budget environment, although the project financial assurance statements were formally excluded from the scope of the review and the Independent Review Report by the Auditor-General. The review indicated that all project offices expected to deliver all required capabilities within the allocated budget. However, for AWD Ships, the CEO DMO indicated that the project may have insufficient approved funds to complete the program.⁴⁸

1.18 In addition, in 2013–14, a number of project offices added additional disclosures to their PDSSs, and in particular, AWD Ships, LHD Ships and ANZAC ASMD Phase 2B, recognised that available funding for price indexation was a key concern. Prior to 1 July 2010, projects were periodically supplemented for price indexation, whereas the allocation for price indexation is now provided for on an out-turned basis at Second Pass Approval.⁴⁹ This change in supplementation policy has meant that price Indexation has emerged as a risk for some projects, which is required to be managed individually, by each project office.

1.19 In effect, projects which slip past original delivery dates must now access contingency funding where pre-calculated indexation is insufficient. Previously, the separation of yearly indexation funding from other budget components allowed for greater transparency in reporting and fewer risks for project offices to manage.⁵⁰

1.20 A project’s total approved budget comprises:

• the programmed budget, which covers the project’s approved activities, for both contractual and departmental aspects of each project; and
• the contingency budget, which is established to provide adequate budget to cover the inherent cost, schedule and technical risks involved in managing complex acquisitions.⁵¹

1.21 The then DMO’s management of financial risk was based on a portfolio management approach, within the responsibilities of the Chief Finance Officer of DMO (refer to paragraph 1.12 in Part 2 of the 2011–12 Major Projects Report). It is expected that Defence will review this approach as part of the implementation of the First Principles Review, and the ANAO will consider any proposed or implemented changes in 2015–16.

1.22 In 2014–15, the ANAO reviewed the financial framework as it applied to managing project budgets and expenditure, including contingency, in the out-turned budget environment, and the project financial assurance and contingency statements.

Project financial assurance statement

1.23 The project financial assurance statement was added to the PDSSs to provide readers with an articulation of a project’s financial position in relation to delivering project capability and to provide transparency in regard to whether there is ‘sufficient remaining budget for the project to be completed’.⁵²

1.24 In 2014–15, while most projects again continued to operate within their total approved budget, the AWD Ships, LHD Ships and ANZAC ASMD 2B project offices continued to recognise that available funding may be insufficient as contracted indices escalation may be greater than the approved project budget. The AWD Ships⁵³ project also included reference in its project financial assurance statement to the announcement by the Ministers for Finance and Defence on 22 May 2015, that the project would require an additional $1.2 billion to be completed, and that this would be funded at the expense of other Defence acquisitions.⁵⁴ The 2014–15 Statement by the Secretary of Defence disclosed the approval of this real cost increase by government in July 2015. In addition, Battlefield Airlifter disclosed a cost risk for contracts yet to be executed.

1.25 As noted previously, for the first time in 2014–15, the ANAO has included the project financial assurance statements within the scope of the Independent Review Report by the Auditor-General, reflecting the increased level of confidence sought by stakeholders in the accuracy and completeness of the related disclosures and the work completed by the ANAO and the then DMO in this area, since the introduction of the statement.

1.26 Defence has continued to support the project financial assurance statements with an independent third-party review, which considered the following in selecting the review sample: remaining budget, Projects of Concern listing, complexity, diversity across divisions and past history. This independent review remains critical to the ongoing validation of the project financial assurance statements.

1.27 In reality, until systems and processes are improved, from end-to-end in the project acquisition life cycle, the ANAO’s assurance in this area will remain one of the higher risk elements of the review. For example, while Joint Project Directives⁵⁵ mature and until they can be relied upon to completely and accurately reflect the approval outcome of government, the ANAO will require access to original approval documents to validate the requirements of projects. At this time, validation by internal Defence documentation is not always possible.

1.28 In addition, the current status of performance information, including for contingency management, project maturity scores, and capability delivery (excluded from the scope of this review) are being impacted by inconsistent application and supporting systems, and lack of management review. All of the above are discussed in more detail elsewhere in this report, and directly relate to the validation of expected project delivery, within budget.

1.29 Projects selected for the 2014–15 third-party review, in support of the financial assurance statement assurance process included:

• detailed review—ANZAC ASMD 2B and ANZAC ASMD 2A; and
• compliance review—P-8A Poseidon, MRH90 Helicopters and Maritime Comms.

1.30 Observations from the review included that there were inconsistent approaches to the weighting of risks and assigning of contingency against risks, with only two of the five project offices being assessed as compliant with the Defence Materiel Instruction for project financial assurance statements.⁵⁶ However, it was determined that the three project offices in question had sufficient contingency despite this non-compliance and that price escalation risk was manageable for ANZAC ASMD 2B. The ANAO’s review also noted a number of issues in terms of compliance with Project Risk Management Manual version 2.4 across project offices, which have been reported to management.

1.31 In conclusion, while for the 2014–15 Major Projects Report, the Chief Finance Officer’s representation letter to the Secretary on the project financial assurance statements was unqualified (except for the known injection of $1.2 billion required for AWD Ships and the affordability risk for Battlefield Airlifter), the project financial assurance statement is restricted to the current financial contractual obligations of Defence for these projects, including the result of settlement actions and the receipt of any liquidated damages; and current known risks and estimated future expenditure as at 30 June 2015.

Contingency statements and contingency management

1.32 As noted above, the 2013–14 Guidelines introduced the requirement for a ‘contingency statement’ within each PDSS. PDSSs are now required to include a statement as to whether contingency funds have been applied during the year, as well as disclosing the risks mitigated by the application of those contingency funds. The five project offices which had contingency funds applied in 2014–15 were AWD Ships (indexation funding shortfall and Counter Measure Lockers for storing explosive ordnance), MRH90 Helicopters (technical and integration risks), ARH Tiger Helicopters (for discounts on upgrades to Ground Mission Equipment received as liquidated damages), Air to Air Refuel (risks related to the modification program and spares) and Battlefield Airlifter (divestiture and contracting risk).⁵⁷

1.33 The examination of the contingency statements as at 30 June 2015 also highlighted that:

• where project offices had contingency funds applied, the purpose was within the approved scope of the project;
• the clarity of the relationship between contingency application and identified risks varied. Of the 24 project offices that have a formal contingency allocation⁵⁸, two did not explicitly align their contingency log with their risk log, and of the 22 project offices that could demonstrate alignment, 10 did not always meet all the requirements of Project Risk Management Manual (PRMM) version 2.4; and
• the method for applying contingency varied, with only four project offices using the ‘expected costs’ of the risk treatment (as required by PRMM version 2.4), seven for which no application of contingency was necessary (as there were no high/extreme risks or no cost implications), and the remaining 13 using either a proportionate allocation of the likelihood of the risk eventuating (the method outlined in PRMM version 2.2), an alternate method, or having no application of contingency against risk.

1.34 Although the ANAO found that all project offices tracked their contingency budget in some form, the methods of recording the balance of contingency budgets and application of contingency funds differed between projects. For example, project offices varied in whether they maintained an up to date record of reviews of their contingency log, and its adequacy, or included the expected cost for all high/extreme risks in the risk log with corresponding entries in the contingency log, or risk identifiers and descriptors for allocations of their contingency budget, all of which are requirements outlined in PRMM version 2.4.

1.35 The purpose of the project contingency budget is identified as being ‘to provide adequate budget to cover the inherent risk of the in-scope work of the project’.⁵⁹ Defence policy requires project offices to maintain a contingency budget log to identify and track components of the contingency budget. However, the lack of oversight of compliance with this policy has resulted in inconsistent approaches taken to contingency allocation. For example, in 2014–15, the ANAO observed that half of the project offices were unable to demonstrate clear links in compliance with PRMM version 2.4 for the contingency allocation to individual risks.

Enterprise Risk Management Framework

1.36 In 2013–14, the ANAO’s review concluded that while the standards of risk management arrangements applying to Major Projects have continued to improve, the inherently uncertain nature of risks and issues meant that PDSS data could not be considered complete due to unknown risks and issues that may emerge in the future. In 2014–15, major risks and issues data in the PDSSs continues to remain out of scope of the Independent Review Report by the Auditor-General under arrangements for the priority assurance review, but is included in the analysis to provide an overall perspective of how risks and issues are managed within Defence.

1.37 The ANAO monitors developments in risk management at the enterprise and project level, in order to maintain its understanding of the new Capability Acquisition and Sustainment Group’s risk management systems and processes. Organisationally, the development of the former DMO’s Enterprise Risk Management Framework, was identified by the ANAO in 2008–09, as a challenging but necessary step for the then DMO in achieving its goal of improving project management.⁶⁰ In 2014–15, the ANAO again monitored the progress of the Enterprise Risk Management Framework and its associated policies and guidance, which continues to mature.

1.38 Finalised in July 2014, Defence conducted an internal audit on Risk Management and the Enterprise Risk Management Framework in Defence. With a broader scope than the ANAO’s examination of project level risk management, the findings of the audit were as follows:

• risk management in Defence is inadequately mandated and implemented and also has deficient senior ownership;
• risk management in Defence is inadequately integrated with other Defence processes⁶¹;
• the enterprise risk deep dive process is incomplete and the enterprise risks are not widely communicated or fully understood;
• the application of risk management in Defence is inconsistent, lacks quality and fails to cascade through the organisation; and
• many Defence risk managers are inadequately and inconsistently skilled.⁶²

Defence advised in August 2015 that work on the Enterprise Risk Management Framework continues in the new Capability Acquisition and Sustainment Group.

1.39 The Interdependent Mission Management System (IMMS) was developed in response to a recommendation from the 2011 Plan to Reform Support Ship Repair and Management Practices (Rizzo Report) to ‘establish an integrated risk management system’.⁶³ The Rizzo Report stated that in relation to risk management, ‘Navy and DMO need to improve coordination and integrate their interdependent activities more effectively’.⁶⁴ The IMMS is intended to provide joint visibility of risks at the enterprise level, and facilitate greater accountability in relation to risk management. Following the Navy Reform Board’s May 2014 endorsement of the use of IMMS to manage interdependent risks in Navy, in January 2015, a Materiel Sustainment Agreement⁶⁵ was signed between Navy and the then DMO, providing the governance framework for interdependent risk management. Currently, IMMS is in the pilot stage with the Guided Missile Frigate (FFG) System Program Office, with final testing forecast for completion in October 2015.⁶⁶

1.40 In 2014–15, the ANAO again examined project offices’ risk and issue logs, which are created and maintained utilising the new group’s mandated Excel or Predict! software.⁶⁷ The ANAO’s review indicates that the majority of project offices maintained risk and issue logs appropriately, but not consistently. The ANAO has continued to note inconsistencies in the practices of various project offices, including:

• where Excel and Predict! logs are maintained concurrently for the same project, risks are not always consistently recorded; and
• where Excel is used, there is variability in the information content of the log and in some cases, evidence that it has not received appropriate management attention.

1.41 While some project offices will experience greater challenges with risks and issues administration, considering project complexity, scale and timing, it is important that Defence provide systems and processes to ensure risks are appropriately managed and reviewed across the life of the project. Large complex projects, which are more likely to have a higher number of risks and issues, face the challenge of putting internal systems in place to meet organisational requirements. Particularly in the case of higher cost developmental projects, which also tend to exist in an environment of often considerable scrutiny and political risk.

1.42 In this context, the Joint Strike Fighter project is an example where the project has developed a hierarchical view of risks, due to the number of risks in existence. For example, the project has 51 ‘high’ rated risks (pre-mitigation). By necessity, these are summarised in the Joint Strike Fighter PDSS at a strategic level. However, consideration of the Joint Strike Fighter’s risks is made more difficult given that the project has not finalised its Risk Management Plan, which has remained in draft since 2014.

1.43 Smaller projects, while typically having fewer risks and issues, also have fewer and less experienced staff, and are less likely to have experienced dedicated risk managers as part of their staffing profile. The Battlefield Airlifter project is an example where the ANAO was provided with both Predict! and Excel risk logs, which were inconsistent.

1.44 Overall, the issues with risk management that the ANAO observed related to:

• variable compliance with corporate guidance to ensure complete, timely and accurate representation of project risks and issues;
• the currency of Risk Management Plans and the frequency of their update to capture changes in policy and practice⁶⁸;
• the frequency with which risk and issue registers are reviewed to ensure risks and issues are appropriately managed and reported to senior management; and
• risk management logs and supporting documentation of variable quality, particularly when Excel is used.

1.45 The Standardisation Office is the corporate area responsible for the development, amendment and publishing of corporate risk management policy within the Capability Acquisition and Sustainment Group. Gate Reviews held by the Independent Project Performance Office also have a degree of oversight over project risk management processes.⁶⁹ In 2014–15, both areas confirmed that they provide guidance and advice only. Neither have the mandate or resources for systematic compliance monitoring of risk management.

1.46 Since 2007–08, risks and issues have been a consistent focus of review, although excluded from the formal scope of the Independent Review Report by the Auditor-General, due to issues with completeness, supporting systems and executive oversight. Throughout this time the ANAO has identified deficiencies and worked with project offices to improve compliance with the risk management policies applicable in the then DMO. It is clear that increased scrutiny and accountability of project performance is required to identify shortcomings in corporate performance to support project offices manage their risks, and deficiencies in local project risk management performance.

1.47 To achieve greater consistency in the approach to risk management and in response to the release of a Commonwealth Risk Management Policy on 1 July 2014, the Capability Acquisition and Sustainment Group is developing a single Risk Management Manual, which is expected to be finalised at the end of 2015.⁷⁰

Project maturity framework

1.48 Initially introduced as Project Risk Scores in 2004, and later renamed Project Maturity Scores in 2005, they have been a feature of the Major Projects Report since inception in 2007–08. The DMO Project Management Manual 2012, defines a maturity score as:

The quantification, in a simple and communicable manner, of the relative maturity of capital investment projects as they progress through the capability development and acquisition life cycle.⁷¹

1.49 Maturity scores are a composite indicator, constructed through the assessment and summation of seven different attributes, which cumulatively form a project ‘maturity score’. The attributes are: Schedule, Cost, Requirement, Technical Understanding, Technical Difficulty, Commercial, and Operations and Support, which are assessed on a scale of one to 10.⁷² Project Maturity is a composite performance indicator available for all Major Projects, for decision making, and to assess their overall status.⁷³

1.50 Historically, while the DMO had raised some doubts about the effectiveness of their maturity score framework, they agreed to retain maturity scores following a JCPAA recommendation that ‘DMO maintain the ability to publish maturity scores in future Major Projects Reports until these are no longer required by the guidelines endorsed by the JCPAA’.⁷⁴ The Committee viewed the retention of maturity scores as important in relation to providing a measure of capability delivered for each project, until a measure equal to or better than current arrangements is available. Recently, the decision to maintain maturity scores, while seeking to develop an improved measure, was again reaffirmed by Defence to the ANAO in the context of this 2014–15 review.

1.51 In 2014–15, Defence also indicated that the organisation is relying less on project maturity scores and are instead moving towards other project management tools, such as the Materiel Implementation Risk Assessment (MIRA). The MIRA is used during the First Pass Approval stage for projects and is designed to assist project offices in submitting details of their top five risks in the acquisition business case for cabinet submission. The DMO Project Risk Management Manual 2013 defines MIRA as:

A summary of the most significant risks (as recorded in the project risk register) that will impact on DMO’s ability to deliver the Materiel System (Mission and Support System) outcomes on time, within budget, and to the required scope and quality.⁷⁵

1.52 As the MIRA outlines a project’s key risks at only one point in time, government First Pass Approval, the ANAO notes that for reporting purposes, the MIRA does not provide the same level of oversight on a project’s delivered capability as maturity scores. During the course of the review, the ANAO reviewed the MIRA for new projects, to ensure that the risks disclosed in the MIRA were included in the project risk registers. The results of which were consistent with general alignment with current PDSS disclosures, with any differences due to the passage of time, increased project knowledge, and risk management efforts.

1.53 However, comparing the maturity score against its expected life cycle gate benchmark provides internal and external stakeholders with an indication of a project’s progress. This may trigger further management attention or provide confidence that progress against the appropriate maturity score benchmark is satisfactory.

1.54 While the ANAO has previously raised inconsistency in the application of project maturity scores as an issue, and as maintained in this review, the ANAO noted that project offices were more consistently assigning maturity scores than in previous years. While some subjectivity remains, in the context of a framework that relies upon the application of professional judgement, across a diverse range of project circumstances, with the detailed guidance available, assigning a maturity score is a repeatable process, and is appropriate for external review or audit.

1.55 As previously noted by the ANAO, the guidance underpinning the attribution of maturity scores would benefit from a review for internal consistency and relationship to the Defence’s contemporary business. For example, allocating approximately 50 per cent of the maturity score at Second Pass Approval, regardless of acquisition type, is often inconsistent with the proportion of project budget expended, and the remaining work required in order to deliver the project.

1.56 Further, the existing project maturity score model does not always effectively reflect a project’s progress during the often protracted build phase, particularly for developmental projects. During this phase it can be expected that maximum expenditure will occur, and risks realised, some of which will only emerge as test and evaluation activities are pursued through to acceptance into operational service.

1.57 Finally, while the guidance underpinning maturity scores was due for review in September 2012⁷⁶, this review is not yet finalised. The ANAO was advised that while work had occurred to review the guidance, the release of the First Principles Review meant that the guidance would require further consideration.

1.58 The ANAO will continue to review the framework and attribution of maturity scores in subsequent reviews.

Efficiency of the Major Projects Report process

1.59 As in previous years, project offices prepared indicative PDSSs and supporting evidence prior to 30 June 2015 to support the initial stage of ANAO fieldwork for the 2014–15 review. In general, project offices provided high quality PDSSs, with a few exceptions.

1.60 However, while more recent requirements of the review, such as the provision of the contingency statement, are still maturing, the provision of core project information on cost, schedule and the progress towards delivery of required capability, remains a largely manual process. Additionally, a significant amount of project information is not centrally maintained, requiring extensive contact with individual project offices to obtain evidence to assure the information in the PDSSs. Further, inconsistent application of Defence policies for project management increases these difficulties.⁷⁷

1.61 The ANAO will continue to assess the application of policy and maintain its focus on the efficiency and consistency of the production and assurance of the PDSSs.

Review conclusion

1.62 The Independent Review Report by the Auditor-General takes into account the overall governance of Major Projects, the results of the ANAO’s examination of the then DMO’s project management and reporting arrangements, and the results of the ANAO’s substantive procedures to gain assurance in relation to key information reported in PDSSs. In 2014–15, the results of the ANAO’s priority assurance review of the 25 PDSSs, was that nothing has come to the attention of the ANAO that causes us to believe that the information and data in the PDSSs, within the scope of our review, has not been prepared, in all material respects, in accordance with the Guidelines.

Introduction

2.1 Performance information is important in the management and delivery of major Defence equipment acquisition projects (Major Projects) as it informs decisions about the allocation of resources, supports advice to government on project progress and performance, and allows for the Parliament and the public to assess the progress of projects. Project performance information has been the subject of many of the reviews of Defence and a consistent area of focus of the Joint Committee of Public Accounts and Audit (JCPAA) over the time of the Major Projects Report (MPR). This chapter progresses previous Australian National Audit Office (ANAO) analysis of project performance, and utilises data collected over the life of the MPR, to provide updated analysis across the portfolio of MPR projects.

Project performance analysis by the ANAO

2.2 The ANAO utilises three key performance indicators to lead analysis over the three major dimensions of projects’ progress and performance against Defence’s Project Maturity scores.⁷⁸ These indicators are the:

• percentage of budget expended (Budget Expended)—which measures the total expenditure as a percentage of the total current budget;
• percentage of time elapsed (Time Elapsed)—which measures the percentage of time elapsed from original approval to the forecast Final Operational Capability (FOC)⁷⁹; and
• percentage of key materiel capabilities expected to be delivered (Expected Capability)—which is Defence’s assessment of the likelihood of delivering the required level of capability.

These are measured in percentage terms, to enable comparisons between projects, and to provide a portfolio view across project progress and performance.

2.3 Following this analysis, each section then provides additional analysis of each of the three major dimensions, with key in-year information, longitudinal analysis compiled since the inception of the review, and results of project progress for the year-ended 30 June 2015.

2.4 Prior to disaggregation and analysis against Project Maturity scores, Figure 2 below, provides an overview of the three major dimensions of project performance, and sets out Budget Expended, Time Elapsed⁸⁰ and Expected Capability.⁸¹

Figure 2: Budget Expended, Time Elapsed and Expected Capability

Note: The Expected Capability for Wedgetail has been assessed against the Supplies section of the Materiel Acquisition Agreement, which lists the equipment to be delivered.

Source: The ANAO’s analysis of Budget Expended and Time Elapsed of the 2014–15 PDSSs. Refer to paragraphs 2.4 to 2.9 in Part 1 of this report. Expected Capability is as presented in the PDSSs.

2.5 The figure shows that for most projects (19 of 25), Budget Expended is broadly in line with, or lagging, Time Elapsed.⁸² This relationship is generally expected in an acquisition environment predominantly based on milestone payments. However, due to the varying complexities, stages and acquisition approaches across the portfolio of projects, further analysis of these simple performance measures provides an overall picture of key variances.

2.6 Where Budget Expended is lagging Time Elapsed the project schedule may be at risk or milestones not met. For example, projects where the Budget Expended is approximately 20 per cent less than the Time Elapsed, include:

• Joint Strike Fighter (Budget Expended five per cent, Time Elapsed 40 per cent)—a large scope increase ($10.5 billion) for the purchase of additional aircraft was approved in April 2014 with the project yet to enter main production contracts;
• Battle Comm. Sys. (Budget Expended 72 per cent, Time Elapsed 100 per cent)—the project achieved FOC in April 2015, with some items still awaiting delivery, and final contract acceptance payments dependant on these deliveries; and
• LHD Landing Craft (Budget Expended 64 per cent, Time Elapsed 85 per cent)—expenditure for delivery and acceptance of the final batch of landing craft is forecast for January 2016, which is only one month prior to the scheduled Final Materiel Release (FMR) and FOC date of February 2016, placing additional pressure on schedule.

2.7 Where Budget Expended leads Time Elapsed the project budget may be at risk. Projects where Budget Expended leads Time Elapsed (from government approval to FOC) by more than 10 per cent, include:

• AWD Ships (Budget Expended 76 per cent, Time Elapsed 58 per cent)—contracted indexation rates have been higher than the out-turning applied in the project budget while productivity has been lower than expected. This has resulted in the need for an additional $1.2 billion to complete the project and an extension of the project schedule⁸³;
• Growler (Budget Expended 47 per cent, Time Elapsed 23 per cent)—most of the expenditure on equipment is in line with aircraft production in advance of training;
• MH-60R Seahawk (Budget Expended 47 per cent, Time Elapsed 32 per cent)—the project has taken delivery of 12 aircraft to June 2015, with further aircraft in production and all aircraft forecast to be delivered by August 2016. The variance is also exacerbated by the long time between final aircraft delivery and FOC, with this time being used to implement Australian unique modifications and modify navy vessels to operate with the MH-60R;
• Collins RCS (Budget Expended 97 per cent, Time Elapsed 78 per cent) and Reliability and Sustainability (Budget Expended 86 per cent, Time Elapsed 68 per cent) projects—most of the materiel has been acquired and expenditure undertaken. In addition, originally planned installation dates have been extended, based on submarine availability, reducing the proportion of total Time Elapsed; and
• UHF SATCOM (Budget Expended 80 per cent, Time Elapsed 68 per cent)—while significant milestones have been achieved, recent problems associated with software development have delayed FMR, and the project anticipates these will affect FOC.

2.8 In each case, reviewing the performance information highlights projects where reasonable explanations for variances or anomalies exist, however, there are situations where the review of performance information shows projects which require additional Defence or government attention.

2.9 Defence expects that all 25 projects in this year’s report will deliver all of their key capability requirements, recognising that some elements of the capability required for some projects may be under threat, but are considered manageable (assessed as either green or amber). While this is consistent with the 2013–14 Defence assessment, it may be considered overly optimistic and is not further analysed here.

Cost performance analysis

Budget Expended and Project Maturity

2.10 Figure 3, below, sets out each project’s Budget Expended against Project Maturity⁸⁴ and shows that Budget Expended lags Project Maturity for the majority of the projects (19 of 25). This relationship is to be expected as greater numbers of Military-Off-The-Shelf (MOTS) projects are included in the sample of Major Projects, as MOTS products are in-service, and have generally had significant development and testing prior to selection for acquisition.

2.11 Where Budget Expended lags Project Maturity with an approximate differential of 20 per cent or more, the project is classified as either MOTS or Australianised MOTS, except Joint Strike Fighter, which is expected to be MOTS by the time of aircraft delivery. Unlike the relationship between Budget Expended and Time Elapsed, and consistent with prior years, there are no instances where Budget Expended leads Project Maturity by greater than 20 per cent.

2.12 These variances, in part, are the result of Defence’s project maturity framework attributing approximately 50 per cent of the total project maturity at Second Pass Approval. This reduces the value of project maturity assessments within the acquisition phase, which is predominantly the longest and most expensive component of Major Project acquisition.

2.13 Again, analysis of the performance information available highlights useful information for Defence and government, i.e. the selection of MOTS projects significantly reduces risk during project acquisition, where Project Maturity is far more advanced at approval than developmental projects.

2.14 The Defence Materiel Organisation’s (DMO’s) focus on MOTS acquisitions has seen reduced slippage across projects, but may not provide acceptable solutions in all cases. The balance across the portfolio, of MOTS versus developmental acquisitions, should reflect the risk profile of government and Defence, to allow appropriate mitigation strategies to be employed.

Figure 3: Budget Expended and Project Maturity

Note: ANZAC ASMD 2B’s Project Maturity is based on the progress of the lead ship, not on the current eight ship program.

Source: The ANAO’s analysis of the 2014–15 PDSSs. Analysis for the 2014–15 Major Projects Report continues to highlight inconsistencies within the application of project maturity, reducing the level of reliability of maturity assessments for key decision makers and other stakeholders; however, improved focus and review by project offices was noted by the ANAO during 2014–15 fieldwork. Refer to paragraphs 2.10 to 2.14 in Part 1 of this report.

Second Pass Approval and 30 June 2015 approved budget

2.15 For the 25 Major Projects, Figure 4, below, compares each project’s approved budget at Second Pass Approval (the main investment decision by government) and their approved budget at 30 June 2015.

2.16 The total approved budget for the 25 projects at 30 June 2015 was $60.5 billion, a net increase of $18.5 billion, when compared to their budget at Second Pass Approval of $42.0 billion (analysis of this variance is included in Figure 5).⁸⁵

2.17 Figure 4 indicates relative budget variations from Second Pass Approval of 150 per cent or greater for the following projects:

• Joint Strike Fighter—increase of $12.4 billion, ($10.5 billion for 58 additional aircraft in 2013–14, cumulative exchange rate variations of $1.6 billion and cumulative price variations of $0.3 billion – total increase of 451.7 per cent);
• MRH90 Helicopters—increase of $2.8 billion, ($2.3 billion for 34 additional aircraft in 2005–06 and other minor scope changes, cumulative exchange rate variations of -$0.2 billion and cumulative price variations of $0.7 billion – total increase of 291.5 per cent);
• Bushmaster Vehicles—increase of $955.5 million, ($832.2 million for 715 additional vehicles in 2007–08 (437 vehicles), 2011–12 (70 vehicles) and 2012–13 (208 vehicles) and other minor scope changes, cumulative exchange rate variations of -$1.3 million, and cumulative price variations of $124.6 million – total increase of 323.9 per cent);
• ANZAC ASMD 2B—increase of $429.8 million, ($363.4 million for replacing the Very Short Air Defence system with the CEA Technologies Phased Array Radar system in 2005–06, approval of the rollout of upgrades for ships 2–8 in 2011–12 and other minor scope changes, cumulative exchange rate variations of -$9.7 million, and cumulative price variations of $76.1 million – total increase of 172.7 per cent); and
• Collins R&S—increase of $339.7 million, ($271.2 million for additional scope in 2001–02, cumulative exchange rate variations of -$5.9 million, and cumulative price variations of $74.4 million – total increase of 471.8 per cent).

2.18 Figure 4 also highlights projects which have movements in excess of $500 million where the movement as a percentage of the total approved project budget is less than 150 per cent. In these instances the movements are predominantly related to price variation and exchange rate movements but also include various minor scope variations. These projects include:

• AWD Ships—$683.7 million⁸⁶;
• Wedgetail—$623.7 million; and
• Growler—$863.3 million.

Figure 4: Projects’ Second Pass Approval and 30 June 2015 approved budget ($m)

Note 1: [vertical white-filled rectangle] indicates that the budget for the project at 30 June 2015 is less than the original budgeted cost. However, for Overlander Light this reflects a transfer of $2.2 billion to Overlander Medium/Heavy on separation of the original project into two phases in December 2011.

Note 2: The Second Pass Approval amount for the Overlander Medium/Heavy project includes an amount of $0.7 billion for general program supplementation, which was provided as part of the revised Second Pass Approval in July 2013.

Note 3: On 22 May 2015, the then Minister for Defence and the Minister for Finance announced there would be further delays to the delivery of the Air Warfare Destroyers and a further $1.2 billion would be required to complete the project. This budget increase had not been incorporated into the approved project budget at 30 June 2015. This budget increase will be analysed in the 2015–16 Major Projects Report.

Source: The ANAO’s analysis of the 2014–15 PDSSs. Refer to paragraphs 2.15 to 2.18 in Part 1 of this report.

Budget performance

2.19 The following figures and tables illustrate the budget variations for the 25 projects in the 2014–15 report across three differing methodologies:

• the three main factors contributing to budget variations in each of the last seven years, presented longitudinally from 2008–09 to 2014–15;
• the in-year budget variations by project; and
• the expenditure forecasting performance against the actual expenditure for the 2014–15 financial year.

Longitudinal and in-year budget variance analysis

2.20 As noted in paragraph 1.18, from 1 July 2010 price indexation is now provided as part of a project’s budget at Second Pass Approval. This is defined as the budget being ‘out-turned’.⁸⁷

2.21 Figure 5, below, examines the three main factors contributing to budget variations in each of the last seven years, and highlights the in-year impact of variations for the 25 projects in the 2014–15 report. The 2014–15 net life to date budget variation of $18.5 billion for the 25 Major Projects, when compared to their Second Pass Approval budget, comprises price variation increases of $5.1 billion, foreign exchange rate movement increases of $1.5 billion and real variation increases of $11.9 billion.

2.22 Exchange rate variations in project budgets are a result of projects’ exposure to foreign currencies and movement in foreign exchange rates against the Australian dollar.⁸⁸ The exposure of specific projects to currency exchange variations is established through the initial government procurement decision and contractual agreement. The US dollar and the Euro are the main influences, although other currencies also have an impact. This year the Australian dollar generally weakened against the US dollar, while against the Euro, similar to last year, it varied significantly across the year but had stabilised to a similar level to where it began by the end of the financial year.

2.23 The large in-year exchange variation of $2.8 billion is due to adjustments in December 2014, February 2015 and May 2015 totalling $0.2 billion, $2.1 billion and $0.5 billion respectively.

2.24 Real variations in project budgets primarily reflect changes in the scope of projects, transfers between projects for approved equipment/capability, and budgetary adjustments such as administrative savings decisions. The in-year variation of $0.2 billion reflects an increase in the total approved budget for the Growler project for the Mobile Threat Training Emitter System, air-to-air and anti-radiation weapons, partially offset by a small scope reduction in the Battle Comm. Sys. project, for work being transferred to the LAND 75 Phase 4 Battle Management System project (not included in the 2014–15 Major Projects Report).

Figure 5: Longitudinal budget variations for 2008–09 to 2014–15 ($m)

Note: The significant In-year Real Variation recorded in 2013–14 reflects total approved budget increases of $10.5 billion for the Joint Strike Fighter project for additional aircraft and $0.7 billion for the Overlander Medium/Heavy project for general program supplementation which was provided as part of the revised Second Pass Approval in July 2013.

Sources: The ANAO’s analysis of the 2014–15 PDSSs and Project Budget Approval Histories. Refer to paragraphs 2.20 to 2.24 in Part 1 of this report.⁸⁹

2.25 Table 6, below, sets out the 2014–15 in-year budget variations for each of the 25 Major Projects. Overall, the 30 June 2015 approved budget for the 25 projects in the 2014–15 report increased by $2 958.2 million in 2014–15 or 5.5 per cent, compared to their 30 June 2014 approved budget. As discussed above, this was driven by net real increases of $192.3 million (mostly for scope increases), and exchange variation increases of $2 765.9 million.

2.26 In 2014–15, only two projects had real variations, being Growler, which received $200.6 million for the Mobile Threat Training Emitter System, air-to-air and anti-radiation weapons, and Battle Comm. Sys., which had a scope reduction of $8.3 million for work that was transferred to the LAND 75 Phase 4 Battle Management System project.

2.27 Projects with larger movements in foreign exchange, as a result of the depreciating Australian dollar, include:

• Joint Strike Fighter—$1 725.6 million, or 12.8 per cent increase in budget;
• P-8A Poseidon—$472.8 million, or 11.9 per cent increase in budget;
• Growler—$294.2 million, or 9.6 per cent increase in budget; and
• MH-60R Seahawk—$211.6 million, or 6.6 per cent increase in budget.

Table 6: In-year (2014–15) budget variations by project

Source: The ANAO’s analysis of the 2013–14 and 2014–15 PDSSs. Refer to paragraphs 2.25 to 2.27 in Part 1 of this report.

In-year forecast and actual expenditure

2.28 Accurately forecasting and managing budget expenditure is an important element in the effective management of a portfolio of projects and understandably receives close attention within Defence and by key stakeholders, including the JCPAA. Figure 6 sets out the expenditure forecasting performance of each project against the actual expenditure for the 2014–15 financial year. In total, the actual expenditure for the 25 projects at 30 June 2015 was $4.8 billion, against an initial (PBS) forecast expenditure of $4.3 billion, a mid-year (PAES) forecast of $4.2 billion, and a final forecast of $5.0 billion (Final Plan). The main factors contributing to the variances were changes to delivery and payment schedules, and foreign exchange fluctuations.

2.29 As part of its review of the 2012–13 report, the JCPAA recommended that expanded information regarding projects’ budget estimates and actual expenditure be included in the review. The review now includes projects’ PBS, PAES and Final Plan estimates, projects’ actual expenditure, and explanations for the variances between each of these amounts.⁹⁰ This information is set out below in Figure 6, in Table 2.2 of Part 2 of this report and in Sections 2.2A and 2.2B of the individual Project Data Summary Sheets (PDSSs).

2.30 The ANAO’s analysis at Figure 6 highlights that notable in-year underspends occurred in the following projects:

• Joint Strike Fighter (expenditure of $233.2 million compared to $237.9 million PBS, $277.9 million PAES and $296.5 million Final Plan estimates)—the variance is primarily due to the unpredictability of the United States Government F-35 Joint Program Office invoicing;
• LHD Ships (expenditure of $81.3 million compared to $142.6 million PBS, $85.6 million PAES and $86.6 million Final Plan estimates)—the variance is primarily due to the delay in production and testing milestones after ship acceptance. Expected reductions were reflected in the updated estimate figures; and
• Battlefield Airlifter (expenditure of $158.5 million compared to $313.8 million PBS, $255.4 million PAES and $271.5 million Final Plan estimates)—the variance is due to schedule slippage, delays to deliveries of equipment and a Foreign Military Sales (FMS) case payment of $54.0 million that was paid early, in the 2013–14 financial year rather than the 2014–15 financial year, which was not known at the time of PBS compilation.

2.31 Notable in-year overspends occurred in the following projects:

• AWD Ships (expenditure of $734.2 million compared to $615.6 million PBS, $759.3 million PAES and $763.2 million Final Plan estimates)—the variance is primarily due to increases associated with shipbuilding activities and cost over-runs, which was reflected in the updated PAES and Final Plan estimates. The project underspent against the PAES and Final Plan estimates, primarily due to further schedule slippage for Ship 1;
• P-8A Poseidon (expenditure of $531.5 million compared to $324.0 million PBS, $450.8 million PAES and $516.4 million Final Plan estimates)—the variance is primarily due to payments being brought forward into the 2014–15 financial year from the 2015–16 financial year and the associated foreign exchange impacts. The increase in payments was largely reflected in the updated estimates figures;
• Growler (expenditure of $1 241.9 million compared to $797.4 million PBS, $728.5 million PAES and $1 202.5 million Final Plan estimates)—the variance is primarily due to payments being brought forward into the 2014–15 financial year from the 2015–16 financial year for aircraft production and Airborne Electronic Attack kits and the associated foreign exchange impacts. The increase in payments was largely reflected in the updated Final Plan estimate figure; and
• MH-60R Seahawk (expenditure of $685.5 million compared to a $504.7 million PBS, $511.7 million PAES and $670.8 million Final Plan estimates)—the variance is primarily due to the acceleration of FMS payments for the MH-60R Seahawk helicopters and the associated foreign exchange impacts. The increase in payments was largely reflected in the updated Final Plan estimate figure.

2.32 The overall project expenditure in the portfolio of projects was comparable to the final plan budget, however, it should be noted that the then DMO budget management was on a portfolio basis, limiting the performance information value of project by project analysis. For example, individual project underspends in one project are offset by bringing forward payments in other projects for cash management purposes.

Figure 6: In-year (2014–15) projects’ forecast expenditure performance compared to actual expenditure ($m)

Sources: The ANAO’s analysis of the 2014–15 PDSSs and Defence Portfolio Budget Statements. Refer to paragraphs 2.28 to 2.32 in Part 1 of this report.

Schedule performance analysis

2.33 Defence continues to acknowledge that schedule performance is the key issue in delivering and sustaining equipment for the Australian Defence Force.⁹¹ Further, in 2013, the then Minister for Defence, Senator the Hon. David Johnston stated:

We need to be able to do things much more time effectively, because time is money and this is one of the things we see as the principal causes of problems inside Defence. We want to know when things are beginning to lag, schedule is starting to slip… being forewarned we can give the project appropriate attention.⁹²

2.34 In addition, project slippage could effectively introduce or exacerbate an existing capability gap or require extension to the planned withdrawal date for those platforms being replaced. For example, the withdrawal dates for the Sea King and Black Hawk helicopter fleets included consideration of the introduction of replacement capability.

Time Elapsed and Project Maturity

2.35 Figure 7, below, sets out each projects’ Time Elapsed against Project Maturity⁹³, and shows that Time Elapsed lags Project Maturity for 19 of 25 projects with no exceptions greater than 20 per cent. Projects where Time Elapsed lags Project Maturity with an approximate differential of 20 per cent, similar to Figure 3, and as noted in paragraph 2.11, are either MOTS or Australianised MOTS, except Joint Strike Fighter, which is expected to be MOTS by the time of aircraft delivery. Again, analysis of the performance information available highlights useful information for Defence and government, i.e. the selection of MOTS projects significantly reduces risk during project acquisition.

2.36 Time Elapsed lagging Project Maturity by 20 per cent or more reflects that projects are at an early stage of the capability acquisition life cycle and awaiting significant amounts of their major equipment to be constructed and delivered.

Figure 7: Time Elapsed and Project Maturity

Note: ANZAC ASMD 2B’s Project Maturity is based on the progress of the lead ship, not on the current eight ship program.

Source: The ANAO’s analysis of the 2014–15 PDSSs. Analysis for the 2014–15 Major Projects Report continues to highlight inconsistencies within the application of project maturity, reducing the level of reliability of maturity assessments for key decision makers and other stakeholders; however, improved focus and review by project offices was noted by the ANAO during 2014–15 fieldwork. Refer to paragraphs 2.35 to 2.36 in Part 1 of this report.

Schedule slippage and acquisition type by approval date

2.37 Figure 8, below, illustrates the total schedule slippage since Second Pass Approval for projects in the 2014–15 report, demonstrating how this key performance indicator has altered over time. It also depicts the acquisition type and places projects in order of government approval, allowing for more detailed analysis. In addition, Figure 9 illustrates the total schedule slippage for projects which have exited the review to allow further analysis.

2.38 Figures 8 and 9 continue to show that a greater focus on MOTS and Australianised MOTS acquisitions is, prima facie, reducing the slippage in the Major Projects profile. However it is not always possible to acquire the necessary capability in this manner and should be considered with government and Defence’s preferred risk profile and resources in mind.

2.39 The data in Figures 8 and 9, illustrates that older projects, which achieved Second Pass Approval prior to 2005, generally experienced the most slippage. These projects tend to be more developmental (complex) in nature and typically experienced schedule slippage in the past, and have continued to do so. This shows an ongoing trend of slippage in historically late projects.

Figure 8: Current Major Projects—Total slippage post Second Pass Approval and acquisition type by approval date (years)

Note 1: The order of the projects is from latest to earliest approved. All project slippage relates to Final Operational Capability (FOC) dates.

Note 2: Bushmaster Vehicles has an FOC date for each Production Period (discrete order). The FOC used for this year’s Major Projects Report analysis is Production Period Five.

Source: The ANAO’s analysis of the PDSSs in published Major Projects Reports. Refer to paragraphs 2.37 to 2.40 in Part 1 of this report.

2.40 The challenge of gaining a full understanding of the complexities of developmental aspects of projects at Second Pass Approval is evident by the extent of slippage over time. The 2008 Audit of the Defence Budget (Pappas Review) identified technical risk as the largest source of post Second Pass Approval schedule slippage for ‘post Kinnaird’ projects⁹⁴, and also observed that schedule slippage causes cost escalation.⁹⁵

2.41 In JCPAA Report 442, the Committee flagged its intention to continue to monitor activity in this area, and the ANAO will continue to compile data for further longitudinal analysis in the future.⁹⁶

Figure 9: Exited Major Projects—Total slippage post Second Pass Approval and acquisition type by approval date (years)

Note 1: The order of the projects is from latest to earliest approved. All project slippage relates to FOC dates.

Note 2: Does not include the AIR 5376 Phase 3.2 Hornet Refurb project which exited in 2012 as this project did not introduce a new capability and so did not have an FOC date.

Source: The ANAO’s analysis of the PDSSs in published Major Projects Reports. Refer to paragraphs 2.37 to 2.40 in Part 1 of this report.

Schedule performance

2.42 The subsequent figures and tables illustrate the following for the 25 projects in the 2014–15 report:

• the original and 30 June 2015 forecasts for achieving FOC;
• in-year schedule changes to achieving FOC;
• total schedule slippage across the Major Projects; and
• total slippage according to a project’s Second Pass Approval date.

Original and 30 June 2015 Final Operational Capability forecasts

2.43 Figure 10, below, presents information on the projects’ original and 30 June 2015 forecasts for achieving FOC. The total schedule slippage for the 25 Major Projects to date is 768 months when compared to the initial prediction when first approved by government. This slippage represents a 28 per cent increase on the expected schedule since the main investment decision.⁹⁷ Of the 25 projects in the 2014–15 report, 17 have experienced schedule slippage.

2.44 The total schedule slippage across the 2014–15 Major Projects of 768 months, is 347 months lower than the figure of 1 115 months reported in the 2013–14 report. As noted earlier, the difference is mainly due to projects with large amounts of accumulated schedule slippage exiting the review at the end of 2013–14, partially offset by in-year schedule slippage.

2.45 The reasons for schedule slippage often include underestimation of the difficulties associated with technical factors such as design problems, industry capacity and capability, difficulties in system integration to achieve the required capability, or emergent work associated with upgrades.⁹⁸ In other cases, a project office’s ability to gain access to the platform for upgrading can delay the schedule (for example, the two Collins submarine projects and Hw Torpedo).⁹⁹

2.46 A closer examination of the reasons for schedule slippage demonstrates the importance of initial assessments of the purchase type, i.e. MOTS, Australianised MOTS or developmental.¹⁰⁰ Two projects, MRH90 Helicopters¹⁰¹ and ARH Tiger Helicopters¹⁰², were misclassified as MOTS when the projects were both actually Australianised MOTS (i.e. more developmental), which has resulted in extended schedule slippage.

Figure 10: Projects’ original and 30 June 2015 FOC forecasts

Note 1: indicates that the forecast FOC date for the project at 30 June 2015 is earlier than the original FOC date.

Note 2: Bushmaster Vehicles has an FOC date for each Production Period (discrete order). The FOC used for this year’s Major Projects Report analysis is Production Period Five.

Source: The ANAO’s analysis of the 2014–15 PDSSs. Refer to paragraphs 2.43 to 2.46 in Part 1 of this report.

In-year schedule performance

2.47 In 2014–15, there was a total schedule slippage of 41 months in the forecast achievement of FOC for the 25 Major Projects. In-year project performance, measured by slippage over the last 12 months, may not reflect the project trend, however, Figure 11 below, shows recovery of previously anticipated slippage for three projects:

• Wedgetail—achieved FOC in May 2015, one month ahead of the revised forecast schedule;
• Overlander Light—currently expects to achieve FOC in October 2016, three months later than originally planned, but five months ahead of the 2013–14 forecast schedule; and
• Collins R&S—changes in the Full Cycle Docking Schedule have resulted in the project now predicting the achievement of FOC in May 2022, four months ahead of the 2013–14 forecast schedule of September 2022.

2.48 In-year schedule slippage involved the following eight projects¹⁰³ (the explanation provided, drawn from the 2014–15 PDSSs, may also include the reasons for prior slippage):

• Joint Strike Fighter—minor delay resulting from the reassessment of the projected schedule;
• AWD Ships—delays reflected in the Comprehensive Cost Review¹⁰⁴, which indicated further delays in the delivery of the three ships by 15, 12 and 12 months respectively¹⁰⁵, this impact has flowed through to the subsequent major milestones;
• Overlander Medium/Heavy—there has been minor delays of two months this year, however the project still expects to achieve FOC in March 2023, nine months ahead of the reapproved schedule;
• Air to Air Refuel—delays resulting from issues around the Aerial Refuelling Boom System, in particular the fleet modification program to upgrade all aircraft to the final boom configuration and service release of the boom capability;
• Battlefield Airlifter—delays resulting from aircraft production setbacks, the acquisition of mature training system devices and delays to approvals for construction of facilities;
• Hw Torpedo—slippage resulting from changes to the Full Cycle Docking schedule affecting the installation schedule based on submarine availability;
• Battle Comm. Sys.—minor delay resulting from the certification of the achievement of FOC; and
• LHD Landing Craft—delays resulting from the supply of supporting products, such as training and spare parts being delivered to the contracted schedule rather than earlier than contracted as reported in the 2013–14 report.

Figure 11: In-year (2014–15) schedule changes to achieving FOC

Note: Defence’s PDSSs indicate that 14 of the 25 Major Projects Report projects did not record changes to their Final Operational Capability dates this year.

Source: The ANAO’s analysis of the 2014–15 PDSSs. Refer to paragraphs 2.47 to 2.48 in Part 1 of this report.

Longitudinal schedule performance

2.49 Figure 12, below, shows the schedule slippage of the Major Projects included in the 2007–08 to 2014–15 reports (Table 7 provides the details of the projects included in the analysis). The figure shows that the total cumulative schedule slippage for the:

• four projects in 2007–08 is 11.3 years or 135 months over seven years to 2014–15;
• eight projects in 2008–09 is 18.0 years or 216 months over six years to 2014–15;
• 13 projects in 2009–10 is 24.2 years or 290 months over five years to 2014–15;
• 17 projects in 2010–11 is 23.0 years or 276 months over four years to 2014–15;
• 18 projects in 2011–12 is 20.5 years or 246 months over three years to 2014–15;
• 19 projects in 2012–13 is 15.0 years or 180 months over two years to 2014–15; and
• 25 projects in 2013–14 is 3.4 years or 41 months over one year to 2014–15.

2.50 The figure also shows that 34.6 per cent (22.2 years or 266 months) of the total schedule slippage across the Major Projects covered in the 2014–15 report (64.0 years or 768 months) is made up of the slippage from the four remaining projects reported in the 2007–08 report.

Figure 12: Longitudinal schedule slippage across years (in years)

Note 1: The total schedule slippage in 2014–15 across the 25 projects is 768 months. P-8A Poseidon and Maritime Comms, which are new to this year’s Major Projects Report, have not experienced slippage against Final Operational Capability (FOC) according to Defence’s PDSSs.

Note 2: Bushmaster Vehicles has an FOC date for each Production Period (discrete order). The FOC used for this year’s Major Projects Report analysis is Production Period Five.

Source: The ANAO’s analysis of the PDSSs in published Major Projects Reports.

Table 7: Projects included in Figure 12 analysis by Major Projects Report

Source: The ANAO’s analysis of the PDSSs in published Major Projects Reports.

2.51 Further disaggregation according to a project’s Second Pass Approval date in Table 8, below, shows that 73 per cent (2013–14: 80 per cent) of the total schedule slippage across the 2014–15 Major Projects is made up of projects approved prior to the DMO’s demerger from the Department of Defence, in July 2005. This is a positive indicator of the benefits that the then DMO, as a specialist acquisition organisation, was able to bring to complex Defence procurement and the impact on schedule performance (and cost) during the transition to higher levels of MOTS acquisitions following the Defence Procurement Review 2003 (Kinnaird Review).¹⁰⁶

Table 8: Project slippage by Project approval

Note 1: These figures do not add precisely due to the exclusion of schedule reductions over the life of the project.

Source: The ANAO’s analysis of the 2014–15 PDSSs. Refer to paragraph 2.51 in Part 1 of this report.

Capability performance analysis

2.52 The Australian Defence Force defines capability as the power to achieve a desired operational effect in a nominated environment, within a specified time, and to sustain that effect for a designated period.¹⁰⁷ An operational effect is achieved by combining the eight Fundamental Inputs to Capability: personnel; organisation; collective training; major systems; supplies; facilities and training areas; support; and command and management.¹⁰⁸

2.53 In acquiring Defence platforms and systems, a range of documentation (including capability definition, operational concept, function and performance specification, and test concept documents) is developed and sets out the detailed requirements/performance attributes to be achieved. In the case of an aircraft, for example, this would include elements such as its range and speed, handling characteristics and self protection abilities. Depending on the nature of this information, it can be classified for national security reasons and therefore cannot be made public.

2.54 Assessment of capability performance (Expected Capability) is measured against the Materiel Release Milestones (MRMs) and Completion Criteria specified in each project’s Materiel Acquisition Agreement (MAA). The exception is Wedgetail, where the assessment is against the Supplies section of the MAA which lists the equipment to be delivered, consistent with prior year reports. However, these measures do not encompass the full delivery of capability, as they were designed to reflect the subset of the then DMO’s undertakings in relation to project acquisition.

2.55 In general, MRMs and Completion Criteria (previously Measures of Operational Effectiveness) were designed to define, at a strategic level, the key elements required to achieve IMR and FMR.¹⁰⁹ They focus on the achievement of technical, regulatory or operational requirements to be delivered by the then DMO. Where key requirements are not achieved, this could have a significant effect on a system’s likely suitability for acceptance into operational service.

2.56 The assessment against MRMs is based on a forecast of future events. In the JCPAA’s Report 442, the Committee recommended that the ANAO and the DMO consult to apply a more objective method of assessing capability performance and distinguish capability achieved, capability yet to be achieved, capability unlikely to be achieved and capability exceeded.¹¹⁰

2.57 However, while the then DMO advised there was no system that universally tracks the progress of inputs to capability¹¹¹, it has included information regarding the key elements which constitute IMR and FMR, as stipulated in the MAA for each project, in the PDSSs for the 2014–15 report. These changes have assisted in providing greater clarity regarding the proportion of capability delivered at these two key milestones.

2.58 The ANAO continues to observe that there is considerable diversity across the projects in the number, level of specification, articulation and focus of MRMs and Completion Criteria in project MAAs. For example, AWD Ships (total budget $7.9 billion) has four MRMs and Completion Criteria, while ANZAC ASMD 2A (total budget $0.4 billion) reports on 15 MRMs and Completion Criteria. To enable appropriate accountability and performance reporting, the number and level of specificity of performance measures should encompass all key components of acquisition, not just the finalisation of each platform (and supplies) as is the case in AWD Ships.

2.59 The MRMs and Completion Criteria are assessed using traffic light indicators by the project managers, and reported monthly within the Australian Defence Organisation. The June 2015 traffic light assessment is represented in the PDSSs as Measures of Materiel Capability Delivery Performance. Due to national security considerations, only the overall status from each project office’s assessment of the likelihood of delivering the required materiel capability is disclosed in the 2014–15 Major Projects Report and earlier reports.

2.60 However, these results at times contrast with other available information and highlight the difficulties in estimating future capability. For example, the PDSS does not disclose the capability impacts related to technological obsolescence and Rate of Effort (flying hours) in the case of the ARH Tiger Helicopters project, which are caused by delays in delivery and the availability/quality of the helicopters. In addition, the PDSS for the LHD Landing Craft project does not disclose the integration issues for introduction into service accepted by Chief of Navy, which are due to risks that were not mitigated through the acquisition phase of the project.

Expected Capability and Project Maturity

2.61 Figure 13, below, sets out Defence’s assessment of the likelihood of delivering all of the key capabilities (Expected Capability) and Project Maturity.¹¹² Defence expects that all 25 projects in this year’s report will deliver all of their key capability requirements, recognising that some elements of the capability required for some projects may be under threat but considered manageable (assessed as either green or amber).

2.62 The complexity and type of acquisition (MOTS, Australianised MOTS or developmental) are significant factors affecting the Expected Capability; the risk is appreciably higher for more developmental projects. Defence’s increased expectations of delivery of key capabilities is in line with the increase in MOTS or Australianised MOTS projects selected for acquisition in recent years. Notwithstanding, the following analysis on pages 61 to 63, highlights a prediction of 100 per cent capability delivery and an ongoing trend of improvement, in contrast to the issues noted at paragraph 2.60. To that end, in some circumstances, Defence’s assessment of expected capability delivery is overly optimistic, which impacts the following analysis.

2.63 In addition, the project’s current stage within the capability development and acquisition life cycle has an impact on the maturity score provided by Defence, with all variances greater than 20 per cent reflecting projects that are either at an earlier stage of the life cycle or have experienced schedule delays, which impacts on the ability of the project to gain maturity. Defence’s assessment of Expected Capability should become better informed as a project matures along its capability development and acquisition life cycle.

Figure 13: Project snapshot—Expected Capability and Project Maturity

Note: ANZAC ASMD 2B’s Project Maturity is based on the progress of the lead ship, not on the current eight ship program.

Source: 2014–15 PDSSs. Analysis for the 2014–15 Major Projects Report continues to highlight inconsistencies within the application of project maturity, reducing the level of reliability of maturity assessments for key decision makers and other stakeholders; however, improved focus and review by project offices was noted by the ANAO during 2014–15 fieldwork. Refer to paragraphs 2.61 to 2.63 in Part 1 of this report.

Capability performance

2.64 Figures 14 and 15, below, present Defence’s assessment of the percentage of materiel capability delivery that:

• has a high level of confidence will be met (green);
• are under threat but still considered manageable (amber); and
• at this stage are unlikely to be met (red).

In-year capability performance

2.65 Figure 14, below, highlights the assessment that there is one project facing challenges in delivering elements of their system’s planned capability.¹¹³ This shows an improvement from 2013–14, where five projects had challenges in this area. The project currently highlighted is:

• Joint Strike Fighter—where there are a number of risks to achieving some of the materiel capabilities required for Initial Operational Capability (IOC) and FOC including final aircraft software builds meeting IOC and FOC requirements and establishing the sustainment of the aircraft.

Figure 14: Expected Capability at 30 June 2015

Note 1: The Expected Capability assessment for Wedgetail has been against the Supplies section of the MAA, which lists the equipment to be delivered.

Note 2: Expected Capability is as per paragraph 2.2 in Part 1 of this report.

Source: 2014–15 PDSSs. Refer to paragraph 2.65 in Part 1 of this report.

Longitudinal capability performance

2.66 A multi-year comparison of Expected Capability should be considered with caution as a project’s Measures of Materiel Capability Delivery (MRMs and Completion Criteria) can change from year to year as Materiel Acquisition Agreements are updated. Therefore, any comparison of an individual or a group of projects’ Measures of Materiel Capability Delivery may not involve comparing ‘like with like’. Changes in anticipated capability outcomes should be read in conjunction with the information in the PDSSs.

2.67 Figure 15, below, examines the assessment of Expected Capability for the same projects from 2008 to 2015 as reported in the PDSSs using traffic light indicators as defined above.¹¹⁴

2.68 As reflected in the portfolio of projects in the 2014–15 report, the assessment is comparable to the 2013–14 report, that it has a ‘high level of confidence’ in delivering 97 per cent (2013–14: 96 per cent) of the key capabilities associated with the Major Projects in this report. While the delivery of the remaining three per cent (2013–14: four per cent) of the key capabilities is considered to be ‘under threat’, the risk is still considered to be ‘manageable’.

Figure 15: Expected Capability across the 2007–08 to 2014–15 Major Projects Reports

Note 1: * The ANAO did not examine the accuracy of the recording of this data in previous Major Projects Reports.

Note 2: ^ Super Hornet did not have Measures of Operational Effectiveness, but does have Measures of Materiel Capability Delivery Performance in the latest MAA.

Note 3: # Hornet Refurb was removed from the PDSSs in 2011–12.

Note 4: ∆ Joint Strike Fighter is excluded from this analysis due to a lack of data.

Note 5: Ω C-17 Heavy Airlift was removed from the PDSSs in 2012–13.

Note 6: ∞ Super Hornet was removed from the PDSSs in 2013–14.

Note 7: √ Armidales was removed from the PDSSs in 2013–14.

Note 8: ∑ C-RAM was removed from the PDSSs in 2013–14.

Note 9: † HF Modernisation, FFG Upgrade and Hornet Upgrade were removed from the PDSSs in 2014–15.

Note 10: Ɏ Next Gen Satellite and Stand Off Weapon were both removed from the PDSSs in 2014–15.

Note 11: Ⱡ 155mm Howitzer and SM-2 Missile were both removed from the PDSSs in 2014–15.

Note 12: The Expected Capability assessment for Wedgetail has been against the Supplies section of the MAA, which lists the equipment to be delivered.

Source: PDSSs in published Major Projects Reports. Refer to paragraphs 2.66 to 2.68 in Part 1 of this report.

Introduction

3.1 Major Defence equipment acquisition projects (Major Projects) are large, complex, high-cost procurement activities, characterised by risk and long timeframes between concept and acceptance into service. Combined with varying degrees of required interoperability, these characteristics pose significant challenges to the governance of projects and highlight the importance of applying a robust governance and performance information framework. Such frameworks have the capacity to support project offices during acquisition and provide for the consistent assessment of a project’s progress and the transparency and accountability across Major Projects as a whole.

3.2 As noted earlier, the former Defence Materiel Organisation (DMO) was delisted from 1 July 2015, and its functions merged back into the Department of Defence, as part of the implementation of the recommendations from the First Principles Review: Creating One Defence (First Principles Review).¹¹⁵ The newly formed Capability Acquisition and Sustainment Group will now manage the process of bringing new capabilities into service including Fundamental Inputs to Capability, for example the provision of personnel, training and command.¹¹⁶ While progress on the implementation of the First Principles Review recommendations is ongoing¹¹⁷, the Australian National Audit Office (ANAO) will continue to review the governance over Defence acquisition, including Defence’s review of prior DMO policy and its application within the new Defence structure.

3.3 In addition to the key acquisition governance aspects discussed in Chapter 1 of Part 1 (financial assurance, contingency management, enterprise risk management and project maturity), the ANAO’s review included assessing specific developments in: Gate Review Boards; Projects of Concern; Quarterly Project Performance Reporting; Joint Project Directives; business systems rationalisation; and project management and skills development, to gain a greater understanding of the Capability Acquisition and Sustainment Group’s business and to assist in planning the approach to the review.

Additional areas of review focus—Acquisition Governance

3.4 Part of the ANAO’s planning for the Major Projects Report (MPR) each year focuses on reviewing the processes, including developments, that contribute to the overall governance of Major Projects. This informs the ANAO’s assessment of the governance initiatives supporting the PDSSs for the review period, and highlights areas in those frameworks that could also be the subject of the ANAO’s future focus.

3.5 These include:

• the Gate Review process, which is intended to provide the Defence Senior Executive with assurance that projects and products will deliver approved objectives and focus on preparedness to progress to the next stage of activity;
• the Projects of Concern process, which is designed to address project issues, which are of concern to the Senior Executive and government, relating to cost, schedule and capability;
• the Quarterly Project Performance Report, which is designed to identify emerging issues and risks facing projects, for remediation;
• the introduction of a requirement for Joint Project Directives¹¹⁸ in 2009–10, for all projects approved by government from 1 March 2010, to capture the requirements from government approval;
• business systems¹¹⁹ rationalisation, which is aimed at consolidating processes and systems in order to provide a more manageable system environment; and
• the project skills professionalisation and development program within Defence and industry support, which is directed at enhancing the skill sets available to manage Defence’s Major Projects.

Gate Review Boards

3.6 The Gate Review (acquisition) process¹²⁰ was designed to provide assurance that identified risks for a project were manageable, and that costs and schedule were under control, prior to a project passing the various stages of its life cycle. In practice, Gate Reviews (acquisition) involve a periodic assessment of a project in advance of key ‘gates’¹²¹ by a Gate Review (acquisition) Board, appointed by Defence¹²², to provide the Defence Senior Executive with assurance that project offices will deliver on government approved objectives, and focusing on the project office’s preparedness to progress to the next stage of activity, ability to deliver the required capability, and the continued validity of projects’ business cases.¹²³

3.7 Gate Reviews were first held in 2008, during the period of the Mortimer Review, Going to the Next Level: the report of the Defence Procurement and Sustainment Review,¹²⁴ which believed that the Gate Review process should be expanded. The Mortimer Review recommended establishment of an Independent Project Performance Office (IPPO)¹²⁵ to oversee project reviews. IPPO also facilitates the Projects of Concern process (discussed further at paragraphs 3.13 to 3.20), and is implementing the Quarterly Project Performance Report (discussed further at paragraphs 3.21 to 3.24).¹²⁶

3.8 When first introduced, Defence’s Gate Reviews (acquisition) covered four specified gates (First Pass consideration; Second Pass consideration; Contract Solicitation; and Contract Negotiation).¹²⁷ Gate Reviews can also be undertaken in response to government or management concerns, or to enlist senior management assistance to solve significant issues.¹²⁸ On completion of reviews, Gate Review Boards report on the health and outlook of a project, and its readiness to proceed to the next stage of activity.¹²⁹

3.9 In addition, in 2013–14, three new Gate Reviews were introduced for the Project Initiation; Acquisition and Support Concept (pre First Pass); and Sustainment stages¹³⁰, with a Sustainment Gate Review conducted on three of the projects in the 2014–15 report.¹³¹ Previously, project offices that had undergone a Gate Review in the lead up to Final Materiel Release (FMR) did not undergo further Gate Reviews, despite residual issues.¹³² However, Sustainment Gate Reviews are able to provide continuing oversight by bringing a valuable perspective to the consideration of key issues and risks in reviewing the health and outlook of sustainment capabilities.¹³³

3.10 The increased scope of the Gate Review Program is consistent with the continued confidence of the Senior Executive that Gate Reviews are assisting project offices to manage.¹³⁴ IPPO undertakes systematic surveys of the views of officers involved in Gate Reviews, and the 2014–15 results confirmed to IPPO that the program is generally viewed to provide benefit to the project/s.¹³⁵ However, as identified in the ANAO Report No.52 2011–12, Gate Reviews for Defence Capital Acquisition Projects, tabled in June 2012, an objective assessment of the program’s success is difficult to measure due to the qualitative nature of Gate Reviews and the lengthy project lifecycles.¹³⁶

3.11 During 2014–15 the ANAO accessed the Gate Reviews (acquisition) and their outcomes where available, to gain additional information on projects’ progress and assurance that the information in this year’s PDSSs was complete and accurate. Of the Gate Reviews (acquisition) conducted, 17 were on 16 of the projects in the 2014–15 report¹³⁷ (2013–14: 22 MPR projects were subjected to Gate Reviews). However, Defence has now formalised the practice where projects deemed ‘low risk’ as a result of preliminary IPPO analysis, are not subject to a Gate Review. For example, in 2014–15, the Collins R&S project did not proceed to a Gate Review.

3.12 The First Principles Review report released on 1 April 2015 recommended that IPPO be significantly enhanced and strengthened to provide a robust and disciplined contestability function.¹³⁸ The ANAO appreciates that the Gate Review process is continuing to evolve and will monitor its progress in 2015–16.

Projects of Concern

3.13 The Projects of Concern list was established in 2008 to focus the attention of the highest levels of government, Defence and industry on remediating problem projects. Projects identified as having significant issues that are beyond the normal capacity of a project team to manage, may be recommended by the Defence Senior Executive for inclusion on the list. Defence then makes a recommendation to the Minister for Defence for final determination.¹³⁹

3.14 Projects are removed from the Projects of Concern list once government is satisfied that remediation activity has been completed successfully, or government has decided to cancel the project.¹⁴⁰ Project of Concern summits¹⁴¹, implemented in February 2011 to further increase focus from government, Defence and industry, to remediate Projects of Concern, have recommenced with the most recent summit held in July 2015. Project of Concern reports continue to be provided to the Minister for Defence on a monthly basis.

3.15 The IPPO, who is responsible for overseeing and administering the remediation process for all Projects of Concern, has advised the ANAO that updated policy and procedures for Projects of Concern remain in draft.¹⁴²

3.16 During 2014–15, three MPR projects were continuing Projects of Concern, these being AWD Ships, MRH90 Helicopters and Air to Air Refuel. However, in March 2015, Defence announced that the Air to Air Refuel project was to be removed from the list following formal acceptance of the Aerial Refuelling Boom System capability.¹⁴³ In June 2014, it was announced¹⁴⁴ that the AWD Ships project was being placed on the list to better address the increasing commercial, schedule and cost risks identified in ANAO Report No.22 2013–14, Air Warfare Destroyer Program, tabled in March 2014, and following the earlier announcement of a government review.¹⁴⁵ The project was added as a Project of Concern to ‘ensure that this important project is delivered successfully’.¹⁴⁶ As at June 2015, the AWD Reform program was underway.¹⁴⁷

3.17 In May 2013, the then Government announced that the MRH90 Helicopters project would be considered for removal from the Projects of Concern list by the end of 2013, following the signing of a Deed of Variation (Deed 2), to the original contract, to address commercial, technical and schedule issues.¹⁴⁸ As at June 2015, the MRH90 Helicopters project is still listed as a Project of Concern¹⁴⁹ as remediation actions are yet to be completed.¹⁵⁰

3.18 As noted by the Australian Strategic Policy Institute in August 2013:

The introduction in 2008 of the ‘Projects of Concern’ management process has seen some projects that faced serious challenges remediated (and in a couple of instances terminated).¹⁵¹

3.19 Since the Projects of Concern process was established, 10 MPR projects have been on the list and eight have been removed due to government decisions that the projects have been remediated.¹⁵² The eight projects considered remediated are: Wedgetail; Overlander Medium/Heavy; ARH Tiger Helicopters; Air to Air Refuel; FFG Upgrade; ANZAC ASMD 2B; HF Modernisation; and Stand Off Weapon.

3.20 While specific concerns may have been addressed for some ‘remediated’ projects, it should be noted that residual or emergent risks and issues exist post FMR and may even impact the achievement of Final Operational Capability (FOC). For example, while the ARH Tiger Helicopters project was removed from the Projects of Concern list in April 2008¹⁵³, it only achieved FMR with significant caveats in March 2014¹⁵⁴, and has yet to achieve FOC (currently 79 months behind schedule). Similarly, and as noted previously, FFG Upgrade and HF Modernisation have still not reached FOC. More robust and objective capability assessments by Defence across the life of Major Projects, from Second Pass Approval to Final Operational Capability, would better support the Projects of Concern governance process.

Quarterly Project Performance Report

3.21 In May 2011, the then Government announced that it would implement an Early Indicators and Warnings (EI&W) system designed to help identify and correct potential problems in the formative stages of the project life cycle.¹⁵⁵ In December 2013, the then DMO advised that while the EI&W reporting system had provided benefits across Defence, it was no longer providing the required outcomes in its current form.¹⁵⁶ Consequently, the Quarterly Project Performance Report (QPPR) was implemented in 2014.

3.22 The QPPR aims to provide senior stakeholders within government and Defence with a clear and timely understanding of emerging issues and risks in the delivery of capability to end-users, with the focus on highlighting underperforming projects.¹⁵⁷ Defence has advised that analysis and data for the QPPR are sourced from existing internal reporting systems including Projects of Concern, the Monthly Reporting System and other assessment tools such as Gate Review Boards. The report is submitted to the Minister for Defence on a quarterly basis.

3.23 During 2014–15, three projects from the MPR were identified as key underperforming projects, for internal management attention within Defence:

• Battlefield Airlifter, due to uncertainties arising from the US Air Force leaving the C-27J program (a risk known at Second Pass Approval), which has also impacted on the amount of effort and resources required to deliver project outcomes;
• Overlander Light, due to delays caused by the definition and achievement of the Ambulance and Command Post module requirements, however, by June 2015 this project was removed as an underperforming project; and
• UHF SATCOM, due to ongoing issues with the modification of Commercial-Of-The-Shelf software (now considered developmental), which has caused a further slip of 38 months to FMR from May 2015 to July 2018.

3.24 The ANAO appreciates that the QPPR process is still relatively immature and will monitor its progress in 2015–16.

Joint Project Directives

3.25 Joint Project Directives (JPDs) were introduced by Defence for all projects approved by government from 1 March 2010 to identify the scope and limits of acquisition projects and the party responsible for the major components of the work.¹⁵⁸ They are also intended to be an authoritative document which enables any other necessary project documentation, including Materiel Acquisition Agreements (MAAs)¹⁵⁹, Capability Definition Documents and acquisition strategies to be based on a reliable source. Therefore, JPDs and MAAs are intended to aid sound governance and accountability in Defence acquisition projects.¹⁶⁰

3.26 During the 2014–15 review, the ANAO continued to examine a range of governance documents which define project deliverables, with the Capability Acquisition and Sustainment Group currently providing JPDs for all of the eleven MPR projects approved from 1 March 2010.¹⁶¹ An area of focus is the completeness and accuracy of the JPDs, with some difficulties associated with recording the full detail of government approvals.

3.27 The ANAO will continue to take JPDs into account in its review program in future years, where these have been prepared. With another three post 2010 projects to be included in the 2015–16 report, better alignment with government approved requirements is expected and an initial internal audit would assist in determining that the predefined impacts of JPDs are being met.

Business systems rationalisation

3.28 In previous Major Projects Reports, the ANAO reported that the control environment of each project differed, due to the large range of corporate and project management Information Technology (IT) applications being employed by the different project offices. During the 2014–15 review, the same observations apply across the 25 Major Projects. This has again resulted in an inconsistency between the information produced by each of the project’s IT systems (i.e. risk management, financial management, and document management systems) and highlights an issue for Defence in ensuring reliable and consistent information to properly inform project management and decision making. However, the Capability Acquisition and Sustainment Group’s Business Information Management Branch advised that progress is being made in this area through the development of an Integrated Project Management System (IPMS) which was being progressively built into the then DMO corporate data warehouse in 2014–15. The ANAO was advised that the IPMS interfaces with current IT systems to provide reports to management drawn from the Monthly Reporting System, Project Performance Reporting System, and Predict! data.

3.29 In its Report 436, Review of the 2011–12 Defence Materiel Organisation Major Projects Report, the Joint Committee of Public Accounts and Audit (JCPAA) expressed its ongoing concern with the consistency of information in the then DMO’s business systems, and recommended that ‘the DMO develop a business systems improvement plan which prioritises projects, assigns completion dates and allocates senior level ownership for implementation. A progress update on achievements against the plan should be included in the 2012–13 Major Projects Report’.¹⁶²

3.30 The new Capability Acquisition and Sustainment Group has since advised of the development of an Information Management (IM) Investment Plan to prioritise investment in business systems and other IT applications and the development of an overarching IM Framework and Strategic Plan, including an IM Governance Model. However, the ANAO was advised that the group’s work on the IM Investment Plan, and IM Framework and Strategic Plan was suspended with the implementation of the First Principles Review. The Chief Information Officer Group has taken the lead in developing an IM framework, with the Capability Acquisition and Sustainment Group providing input.

3.31 In 2012–13 the then DMO advised that there were 540 applications in use, and that the organisation was identifying and confirming where duplicate licenses and other inefficiencies existed, as well as researching software and systems, in order to achieve greater visibility over Information and Communications Technology expenditure. As a result of this process, the total number of applications in use as at June 2014 was reduced to approximately 300. During 2014–15, post remerging into Defence, the Capability Acquisition and Sustainment Group advised that 3 393 separate applications were identified as in use, with 1 108 recommended for removal.¹⁶³

3.32 As part of a program to rationalise its information and communications technology systems, the single records management system (Objective) was rolled out in 2013, which centralised the then DMO’s electronic document management.¹⁶⁴ Guidance and training is being offered to all employees, along with regular Communities of Practice on records management, in order to ensure expected National Archives and Defence records management obligations are met. The group advised in 2014–15 that staff familiarity and use of Objective is improving.

Project management and skills development

Within Defence

3.33 Key challenges for both Defence and the Australian Defence Industry include improving the project management, scheduling, logistics, procurement and engineering services provided to the Australian Government, within current and future workforce constraints. Previously, the then DMO’s professionalisation initiative was a central strategy in meeting those challenges into the future¹⁶⁵, as well as to attract and retain a skilled workforce.

3.34 To assist with the professionalisation of staff, the then DMO’s Directorate of Professionalisation Policy was responsible for the development of certification programs that focused on developing specific competencies and gaining professional qualifications. This supported the core business of acquisition and sustainment across the Procurement, Financial Management, Project and Sustainment Management, Materiel Engineering and Materiel Logistics streams.

3.35 In November 2012, the then DMO began the transition of personnel and responsibilities to the Defence Learning Branch as part of a movement to a shared services arrangement. With the delisting of the DMO from 1 July 2015 and the merging of the DMO’s functions back into the Department of Defence, greater emphasis has been placed on the One Defence system, with the First Principles Review noting:

It is critical for the development of the One Defence system that the shared services model is fully implemented. This will mean standardising services, removing duplication of functions, professionalising the workforce and ensuring there are single, clear lines of ownership and accountability… We recommend that the service delivery reform program, including full integration of the current Defence Materiel Organisation corporate functions, be completed.¹⁶⁶

3.36 Additionally, in regard to a plan to deliver a professional workforce, the First Principles Review noted:

While the Navy, Army and Air Force have detailed workforce plans, Defence does not have a strategic workforce plan for its enabling workforce. Without it, Defence struggles to identify skills gaps across the agency and place the right people with the right skills in the right roles at the right time to deliver Defence’s mission.

A strategic workforce plan is a necessary prerequisite for building a highly professional workforce with the necessary skills and balance of public servants and Australian Defence Force personnel.¹⁶⁷

3.37 Based on this, the First Principles Review recommended:

… Defence build a strategic workforce plan for the enabling functions, and incorporate workforce plans for each job family in order to drive recruitment, learning and development, performance and talent management.¹⁶⁸

3.38 The new Capability Acquisition and Sustainment Group’s Directorate of Professionalisation Policy advised that the administrative function for several courses are yet to transition to the Defence Learning Branch, with the Directorate providing a supporting role to ensure the group’s professionalisation requirements continue to be met. The Directorate also advised that the transition of responsibility will continue under the First Principles Review’s renewed focus, noting that First Principles Review implementation plans are still being established.

3.39 The Directorate advised in 2014–15 that approximately 1 077 Capability Acquisition and Sustainment Group staff have either been certified or are enrolled in a certification program with a professional body compared with 1 050 staff in 2013–14.¹⁶⁹ The ANAO will continue to monitor the transition of responsibility from the group’s Directorate of Professionalisation Policy to the Defence Learning Branch during 2015–16.

Industry support

3.40 To assist with the development of participants within the Defence Industry, the Skilling Australia’s Defence Industry (SADI) program was established by the Australian Government in 2005. The aim of the SADI program is to up-skill existing employees, improve the quality and quantity of skills training, and provide funding support to the Defence industry for training activities where there is an identified skills shortage in technical, trade and professional skill sets.¹⁷⁰

3.41 The SADI program provides funding support to companies and industry associations for training and skilling activities where that training is linked to a Defence capability. The Capability Acquisition and Sustainment Group advised that since 2005, the program has funded more than 30 000 training places. Also, approximately 200 industry participants have been provided with funding support in trade, technical and other professional skill sets.¹⁷¹ To meet the needs of the Australian Defence Force, SADI aims to invest $215 million¹⁷² within the period 2005–06 to 2015–16, with more than $5.6 million¹⁷³ allocated in 2015–16 under one round of the SADI program, to boost the skills of workers in the Defence industry.¹⁷⁴

3.42 For 2015–16, administration and delivery of the SADI Grant Program has been transferred to the Department of Industry, Innovation and Science, who will undertake this on behalf of Defence. This change in responsibility has required the transfer of documentation, policies and procedures along with relevant training. The Department of Industry, Innovation and Science will continue to report to Defence on the status of the program.

3.43 Additionally, the 2015–16 SADI Program Guidelines have been updated and now include reworked merit criteria (eligibility requirements remain unchanged) which include five equally weighted criteria to prioritise funding allocations¹⁷⁵: support to Defence capability; meeting programme objectives; value for money; strategically targeted skill sets; and the extent to which applicants are Australian small and medium enterprises.

3.44 The SADI program is currently due to close following the 2015–16 round. The ANAO will continue to monitor the program during this time, noting that Defence is expected to release a Defence Industry Policy Statement in late 2015, with an ANAO performance audit currently underway.

3.45 Other government approved programs aimed at up-skilling participants within the Defence industry include the Industry Skilling Program Enhancement (ISPE) package and the Priority Industry Capability Innovation Program (PIC IP). ISPE, which was announced in 2008, provides up to $60.8 million.¹⁷⁶ In 2012–13, the then DMO advised that there was no funding programmed past 30 June 2014. Since then, the group has advised that some initiatives under this program have been extended until the new Defence Capability Plan, Defence White Paper and Defence Industry Policy Statement have been prepared and released (expected in early 2016). This program incorporates a number of initiatives aimed at enhancing work and future career pathways in the sector by implementing programs at schools and offering Defence engineering internships.¹⁷⁷

3.46 PIC IP, which was implemented in 2011–12, provides up to $45 million over seven years¹⁷⁸, and is aimed at providing Australian Defence companies direct support in the form of repayable and matched grants (the recipient matches the funds provided by the Australian Government on a dollar for dollar basis).¹⁷⁹ The program particularly focuses on assisting small to medium sized enterprises to pursue innovative Defence industry projects.

3.47 The Capability Acquisition and Sustainment Group has since advised that the PIC IP will cease in 2017 with no further funding rounds to be offered under this program, and the PIC Development Fund (PIC DF) is planned to be instituted in its place. It is currently anticipated that PIC DF will be funded from the Defence Capability Plan and will be administered by the new group, with additional oversight responsibility to be confirmed. However, this transition is still in development.

3.48 In addition, on 5 September 2011, the then Minister for Defence Materiel underlined the need for the replacement or upgrade of up to 85 per cent of its military equipment over the next 15 years, and asked Skills Australia to work with Defence and the Defence industry to develop a plan to ensure Australian industry has the skills to meet this challenge.¹⁸⁰ Skills Australia noted the following in their September 2012 report, Building Australia’s Defence Supply Capabilities – Main Report for the Defence Industry Workforce Strategy:

…the current skilling programs offered by the DMO suffer from a lack of longitudinal evaluation processes or strategies in place to assess the effectiveness of these programs. The lack of these mechanisms for either SADI or ISPE means that the DMO and Government are not in a position to determine when success is achieved in these programs and whether they have provided value for money.¹⁸¹

3.49 Skills Australia made three recommendations regarding this issue aimed at undertaking performance evaluations of SADI and ISPE, developing key performance indicators for ISPE, and incorporating performance measures and reporting requirements into SADI funding agreements.¹⁸² In response, the then DMO advised that they have considered outcome focused reporting for new and upcoming programs however, they advised that it would be onerous to re-establish reporting requirements for existing programs in the current climate.

3.50 In June 2014, the Productivity Commission released the Trade & Assistance Review 2012–13 which also commented on Defence industry assistance programs, noting that:

… defence industry assistance programs do not appear to have been reviewed (externally and using an economic framework) to the same extent as many other industry assistance programs… All defence programs with material industry assistance objectives, explicit or not, should be separately reported to Parliament, and from time to time reviewed to ensure relevance to strategic and government priorities.¹⁸³

3.51 The Capability Acquisition and Sustainment Group advised that reviews of industry assistance programs will likely be encapsulated in the new Defence Capability Plan, Defence White Paper and Defence Industry Policy Statement (expected in early 2016). The ANAO will continue to monitor the SADI and other industry assistance programs in 2015–16. Further information about the programs can be found in the upcoming ANAO performance audit on Defence Industry Support and Skill Development Initiatives, due to be tabled in January 2016.¹⁸⁴

Secretary of Defence Foreword

I am pleased to present the 2014-15 Major Projects Report (MPR), which reports on 25 major Defence equipment acquisition projects (major projects) managed by the Defence Materiel Organisation¹⁸⁵ (DMO) – now the Capability Acquisition and Sustainment Group (CASG).

Since last year, seven major projects have been removed following achievement of Final Operational Capability (FOC) or based on a post-Final Materiel Release (FMR) risk assessment of the timely achievement of FOC by the Capability Manager within the Department of Defence (Defence). These projects are the F/A-18 Hornet Upgrade; Guided Missile Frigate Upgrade; Next Generation Satellite Communications Capability; High Frequency Modernisation; SM-1 Missile Replacement; Artillery Replacement; and, Follow-on Stand Off Weapon. Two new major projects included this year are Maritime Communications Modernisation and Maritime Patrol and Response Aircraft (Boeing P-8A Poseidon).

The 25 major projects reported this year have a combined budget of more than $60 billion. The MPR is an authoritative source of quantitative data that has been assured by the Australian National Audit Office (ANAO) and increases Defence’s transparency to the Parliament on project performance.

As demonstrated in this report, major Defence equipment acquisition project financial and capability delivery performances are well ahead of our allies and other sectors. While I note that schedule performance has improved over time, more rigour in schedule estimation and other measures will need to be implemented to improve on-time delivery.

The format of this MPR remains largely the same as last year. Chapter 1 provides an overview of the performance of the DMO (now CASG) and discusses key issues, such as finance, schedule and risk management. Chapter 2 provides a detailed analysis of the schedule, cost and delivery of the materiel elements of capability for the 25 major projects.

In 2015-16, Defence will continue to implement reform across all elements of its business operations. With the implementation of the First Principles Review of Defence underway, I look forward to greater opportunities for improvements to manage and deliver major projects.

The Chiefs of Navy, Army, Air Force, the Chief Information Officer and the major contractors for each project have reviewed the relevant project data and their views have been considered in finalising this report.

Defence and the ANAO continue to have a strong professional working relationship and have further improved the MPR consistent with guidance provided by Parliament’s Joint Committee of Public Accounts and Audit.

I thank the outgoing Auditor-General, Mr Ian McPhee, and the incoming Auditor-General, Mr Grant Hehir, and their staff for their contribution to the overall report. I also commend the various Defence staff in bringing this report together.

I look forward to continued collaboration between Defence and the ANAO on future Reports.

Dennis Richardson
Secretary
Department of Defence
18 December 2015

Executive Summary

In 2014-15, the Defence Materiel Organisation (DMO)¹⁸⁶ managed over $11 billion¹⁸⁷ in expenditure for the Department of Defence (Defence)¹⁸⁸ across the acquisition and sustainment programs and other management services. As at 30 June 2015, the DMO managed 181 major Defence equipment acquisition projects (major projects) with an average value of $530.5 million.

The 2014-15 Defence Major Projects Report (MPR) examines 25 of the largest and most technically challenging of these. The key aspects of this MPR are:

• as at 30 June 2015, all projects are delivering capability within the approved budget. Government is considering a real cost increase for the Air Warfare Destroyer Build (AWD) project;
• analysis has identified that average schedule slippage to Final Materiel Release (FMR), the point at which the DMO has delivered all of the approved materiel requirements, is 14 per cent for the 25 projects in this year’s sample compared with 11 per cent for the 30 projects in last year’s sample (noting that 23 projects are common to both years);
• the removal of seven projects reported in the 2013-14 MPR (F/A-18 Hornet Upgrade; Guided Missile Frigate Upgrade; Next Generation SATCOM Capability; High Frequency Modernisation; SM-1 Missile Replacement; Artillery Replacement and Follow-on Stand Off Weapon); and
• the inclusion of two projects (Maritime Communications Modernisation and Maritime Patrol and Response Aircraft System (Boeing P-8A Poseidon)) bringing the total number of reported projects to 25.
• The DMO’s budget performance bettered that of large commercial projects, and compared favourably with counterpart agencies in other countries. To put this cost performance in perspective:
• the United States Government Accountability Office’s 2015 Assessment of Selected Weapons Programs analysed 78 programs and found an average increase in total acquisition costs of 46.8 per cent against first full cost estimates (noting the trend towards developmental projects in the US Department of Defense); and
• the United Kingdom Ministry of Defence’s 2014 Major Projects Report found an overall cost increase of 11 per cent since approval across its 11 projects.
• Similar to project delivery organisations around the world, schedule performance remains an ongoing challenge. As noted above, the current average slip to FMR is 14 per cent. The average slip to Final Operational Capability (FOC) - where the Capability Manager declares that the whole capability has been reported - is 28 per cent, an improvement from 2013-14 MPR’s 35.8 per cent. Most of the in-year FMR schedule slip experienced by the projects was caused by factors such as technical problems, platform availability and contractual issues (further detail is in Chapter 2). Defence continues to implement changes to deliver improvements in schedule estimation and management (further detail is in Chapter 1). However, the benchmarks below are useful in setting major project schedule performance in context:
• the United States Government Accountability Office’s 2015 Assessment of Selected Weapons Programs (78 programs) found an average schedule delay of 29 months, or 36.5 per cent, against Initial Operational Capability (IOC); and
• the United Kingdom Ministry of Defence’s 2014 Major Projects Report found overall slippage of 22 per cent across its 11 current projects.

For 2014-15, the 25 major projects included in the MPR reported that 234 of the 240 (or 97 per cent) capability elements had a high level of confidence in being delivered. While risks exist, the remaining three per cent of capability elements are still considered achievable. For the third year in a row, no capability elements have been identified as being unlikely to be met.

Reforms stemming from the 2003 Kinnaird review, such as the two-pass Government approval process, have driven measurable improvements in major project delivery performance. This is reflected in the improved schedule performance of post-Kinnaird projects reported in the MPR. Average schedule slippage against FOC for these major projects is 13 per cent¹⁸⁹, compared with 60 per cent for major projects approved prior to the Kinnaird reforms. The outcomes from the First Principles Review will provide further opportunities for improvement in managing schedule performance and other areas of project delivery.

Overview

1.1 On 1 July 2015, the Defence Materiel Organisation (DMO) transitioned to the newly established Capability Acquisition and Sustainment Group (CASG) within the Department of Defence as recommended by the Defence First Principles Review. CASG continues to have the former DMO’s objective to provide the materiel equipment and sustainment elements of capability for the ADF in an effective, efficient, economical and safe manner.

1.2 As at 30 June 2015, the DMO was managing 181 major acquisition projects in support of the Australian Defence Force (ADF) with an average value of $530.5 million. In capability acquisition, the DMO was primarily responsible for managing the acquisition of the materiel systems (the mission and support systems) component of capability.¹⁹⁰

1.3 During 2014-15, 18 projects were closed, having met the required capability, on average, five per cent under the approved budget.

First Principles Review: Creating One Defence (FPR)

1.4 A recommendation of the FPR was the formation of CASG under the leadership of a Deputy Secretary reporting directly to the Secretary. Under the FPR, CASG will take on the responsibility for developing and delivering integrated project plans that will encompass all Fundamental Inputs to Capability (FIC) (including personnel, training, supplies, facilities, training areas, logistics, support, command and management).

1.5 The planned FPR disbandment of Capability Development Group (CDG) will also see accountability for requirements setting and management transferred to the Vice Chief of the Defence Force and the Service Chiefs. Strategic, financial and technical contestability will be located within the Strategy Policy and Intelligence Group, and strengthened in the process.

1.6 In addition to its new role, CASG will continue to extend across the totality of the acquisition lifecycle, from contributing to the early stages of project development before formal Government approval, to delivering the materiel elements of major projects as approved by Government, sustaining and upgrading them once in service and eventually managing their disposal at the end of their service life.

1.7 This work will continue to be performed in an environment of significant complexity. In addition to recent independent studies, the Joint Committee of Public Accounts and Audit (JCPAA), in their review of the 2013-14 MPR, noted the range of complexities that affect the completion of major projects; these include:

• managing induced schedule delays as a result of budgetary constraints;
• employing and maintaining an appropriately skilled workforce where the skills required are in high demand by other industries;
• acquiring new equipment presenting multiple integration challenges;
• contractors overestimating the technical maturity of proposed equipment solutions;
• contractors underestimating the level of effort and complexity required to deliver new equipment;
• unavailability of in‐service equipment (due to operational requirements) limiting the ability of projects to install and test new or upgraded equipment in accordance with the original planned project schedule, and;
• complying with increasingly demanding certification and regulatory requirements, and ensuring access to intellectual property to enable continued further enhancement and improvement of systems.

1.8 To manage the complexity outlined above, Defence is investing in the development of an operational framework for CASG. The framework comprehensively explains how the organisation operates and the roles and responsibilities within it. The framework will detail the life cycle management processes that provide project and engineering discipline to manage complex materiel procurement from initiation to disposal. In addition, Defence will reinforce accountability at all levels and bring together information upon which good management decisions can be made.

1.9 All of these changes resulting from the FPR are expected to improve overall project performance.

Achievements

1.10 An overview of cost and schedule performance is shown in Chapter 2. Examples of significant achievements in 2014-15 are:

• Joint Strike Fighter: on 1 October 2014, Australia’s first F-35A Lightning II Joint Strike Fighter made its inaugural flight, marking another significant milestone for the Australian F-35A program.¹⁹¹ This milestone follows the official rollout of Australia’s first two F-35A on 24 July 2014.¹⁹²
• AWD Ships: the first Air Warfare Destroyer, NUSHIP HOBART, was launched in Osborne, South Australia, on 23 May 2015.¹⁹³
• Wedgetail: the Project achieved Final Operational Capability (FOC) in May 2015.¹⁹⁴ The Wedgetail airborne early warning and control capability provided more than 1,220 hours directing air strikes in the coalition operations in the Middle East since October 2014.¹⁹⁵
• MH-60R Seahawk: the first of two Navy MH-60R Seahawk ‘Romeo’ flight simulators was commissioned into service on 27 March 2015.¹⁹⁶
• LHD Ships: the first of two amphibious ships, HMAS Canberra, was commissioned into service on 28 November 2014.¹⁹⁷
• Air to Air Refuel: KC-30A tanker planes have flown more than 2,200 hours refuelling coalition aircraft in Middle East operations since October 2014.¹⁹⁸ The first air to air refuelling boom contact was made, on 10 June 2015, by an Air Force crew of the KC-30A Multi-Role Tanker Transport (MRTT) aircraft.¹⁹⁹
• Battle Comm. Sys.: The project achieved Final Materiel Release (FMR) on 26 March 2015 and FOC certification on 8 April 2015 (more details can be found in the PDSS).

Projects of Concern

1.11 Projects of Concern (PoC) are those projects identified as having very significant technical, cost and/or schedule difficulties. The primary objective of the PoC regime is to assist with the implementation of an agreed remediation plan. Projects listed as PoC receive a higher level of oversight and management and undertake increased reporting to Government. Since 2008, 23 projects, with a total value of $30.4 billion, have been managed as PoC. There are six active PoC (listed in Table 1.1) with a total value of $12.8 billion as at 30 June 2015. In 2014-15, the Air to Air Refuelling project was removed after successful remediation of technical issues, and the Australian Defence Satellite Communications Terrestrial Enhancement (JP 2008 Phase 3F), was added as a PoC.

Table 1.1 – List of Projects of Concern as at 30 June 2015

Challenges

1.12 In 2014-15, the DMO delivered the materiel elements of capability as described in the Materiel Acquisition Agreements set between the DMO, the CDG and the relevant Capability Manager at project approval. The average schedule slip to FMR is 14 per cent for the 25 projects included in this MPR versus 11 per cent for the 30 projects included in the 2013-14 MPR. Note that seven projects have been removed from, and two new projects added to this MPR. The main factors driving the schedule slip against the FMR milestone are: platform availability; industry delays in resolving technical design issues; unforeseen technical problems and other contractual issues.

Major Acquisition Program – Financial Management

1.13 The Chief Finance Officer within Defence provides financial assurance in relation to individual projects included in the 2014-15 MPR delivering the remaining intended scope within approved major project budgets.

1.14 When considering and approving budgets, the Government takes account of the estimated impact of inflation over the life of a project. This forecasting of future inflation is known as ‘out-turning’. From 1 July 2010, all major Defence equipment acquisition projects have been managed using out-turned budgets. At the time of project approval, project managers estimate the impact of indices tendered (or estimated) for the life of the project. These estimates are built into the project budget as part of the out-turning process.

1.15 Financial assurance statements have been validated by an external independent company for a project sample of five²⁰¹ of the 25 projects reported in this MPR.

Table 1.2 – List of 2014-15 MPR Projects by Total Approved Budget²⁰²

Major Defence Equipment Acquisition Project Performance

1.16 Based on internal and external analysis, the DMO steadily improved its effectiveness in delivering materiel capability between 2000 and 2015. In the last five financial years, 98 major projects were completed after successfully introducing the required capabilities. The materiel capability for these 98 major projects was delivered on average five per cent under budget, resulting in over half a billion dollars in savings.

1.17 As in previous years, the cost performance of five per cent under budget continues to out perform equivalent performance in the private sector, as well as counterpart agencies in other countries. To put this cost performance in perspective:

• the United States Government Accountability Office’s 2015 Assessment of Selected Weapons Programs analysed 78 programs and found an average increase in total acquisition costs of 46.8 per cent against first full cost estimates (noting the trend towards developmental projects in the United States Department of Defense); and
• the United Kingdom Ministry of Defence’s 2014 Major Projects Report found an overall cost increase of 11 per cent since approval across 11 major projects.

1.18 The current average schedule slip to FOC of post-Kinnaird²⁰⁵ projects of approximately 13 per cent compared with 60 per cent for pre-Kinnaird projects indicates consistent project performance improvements have been realised and continue over time. Even so, as is the case for project delivery organisations around the world, schedule performance remains a challenge. The information below puts current major project schedule performance in perspective:

• the United States Government Accountability Office’s 2015 Assessment of Selected Weapons Programs found an average schedule delay of 29 months, or 36.5 per cent, against Initial Operational Capability (IOC); and
• the United Kingdom Ministry of Defence’s 2014 Major Projects Report exhibits average slippage of 23 per cent across 11 current approved projects.

1.19 Table 1.3 displays the average FOC slip as published in the MPR since 2007-08. In the first year of publication, the average FOC slip was represented in months. Since then it has been expressed as an average percentage slip.

1.20 Having overcome technical challenges, the exit of the Guided Missile Frigate Upgrade and High Frequency Modernisation projects from the MPR project list has significantly improved average FOC slip in 2014-15. Chapter 2 examines the factors that affect FOC in more detail.

Table 1.3 – Historical Representation of Final Operational Capability

1.21 Despite the complex delivery environment, the DMO’s performance in delivering project scope continued to progress positively. Figure 1.1 tracks reporting against the capability Measures of Effectiveness for every MPR project since the report’s inception and highlights a clear trend of improvement. Materiel capability performance of 2014-15 MPR projects as at 30 June 2015 was:

• 97 per cent of Measures of Effectiveness are likely to be met (green);
• three per cent of Measures of Effectiveness are under threat but manageable (amber); and
• zero per cent of Measures of Effectiveness are considered unlikely to be met (red).

Figure 1.1 – Materiel Capability Performance as at 30 June 2015

Continuous Improvement

Negotiation Cell

1.22 In 2013-14, the Defence Negotiation Cell established a standing offer panel for negotiation services with 20 nominated highly-performing individuals from 17 organisations. In 2014-15, the number of nominated individuals increased to 22. These individuals are supporting various project and sustainment teams to develop their commercial expertise and to achieve the best outcomes from their dealings with industry. This is aimed at securing agreements that deliver more value, leading to better contract performance and ultimately to improved outcomes for Defence and industry. Further, in 2015, Defence established the Evaluation and Negotiation Centre of Excellence aimed at significantly improving the conduct of evaluations for complex procurements and commercial negotiations.

Acquisition and Sustainment Planning Framework

1.23 The 2008 Mortimer Review recommended improvements to Defence’s acquisition strategy development process during the requirements phase of the capability systems life cycle. To support this improvement, Defence has developed Acquisition and Support Implementation Strategy (ASIS) policy and guidance that place greater emphasis on implementation of the support system to ensure whole of life issues are better addressed early in the project life cycle.

1.24 Following extensive stakeholder consultation, the ASIS policy and guidance was formally released in July 2015 and is supported by targeted executive and project practitioner training. Importantly, the ASIS policy framework has been developed to encompass the FPR recommended end-to-end capability development and management processes from needs definition through to disposal, including consideration of all FIC. As Defence works to streamline and improve its capability life cycle processes, the ASIS policy and guidance will form a central pillar of this improvement activity.

Schedule Management

1.25 Accurate schedule estimation is essential for predictable schedule performance. To this end, Defence is undertaking various initiatives to improve schedule estimation across major Defence equipment acquisition projects, including:

• developing a Tier 1 Defence Cost and Schedule Estimation Manual. Chapter 1 of the Manual has been released. The remainder of the Manual is under review until first quarter of 2016;
• integrating the Schedule Compliance and Risk Assessment Method (SCRAM) with the Integrated Baseline Review (IBR) process to improve early identification of schedule risks, and piloting the new integrated process on selected high risk projects to refine and prove the process; and
• releasing a Project Controls Manual in August 2014 that updates and consolidates Work Breakdown Structure, scheduling and earned value management processes into a framework to improve consistency of application.

Schedule Compliance Risk Assessment Method

1.26 In 2014-15, SCRAM reviews continued to be performed on major projects to identify risk, issues and technical debt that drive schedule slippage and to provide recommendations for schedule achievement. Notable among these were Project SEA 4000 – Air Warfare Destroyer Build and AIR 6000 – New Air Combat Capability – Joint Strike Fighter (JSF) Mission Systems and Autonomic Logistic Information System (ALIS) Software development.

1.27 For the first time a SCRAM assessment was conducted on the JSF ALIS development using scientific Software Parametric Analysis techniques. The ALIS is being developed using an Agile software development method called ‘Scrum’. SCRAM modelling was based on ‘Story Point’ work unit sizing instead of the traditional Software Lines of Code (SLOC), which is not representative of work effort in Commercial-Off-The-Shelf (COTS) based systems such as ALIS. The initial modelling trial results appear valid based on comparison with contractor actual performance outcomes.

1.28 Based on systemic issues identified during multiple SCRAM Reviews, Defence is now trialling integration of SCRAM with the IBR process. The IBR process is used to formally establish a Project’s Performance Management (cost/schedule/scope) Baseline to track performance during project execution. The revised approach incorporating SCRAM reviews includes the feasibility of a practical method to identify and eliminate potential schedule pressures prior to contract award. SCRAM integrated IBR Trials will continue in 2015-16.

1.29 Work is also continuing on the development of a standard Technical Implementation Risk Assessment (TIRA) methodology. The intention is for TIRAs to provide a framework and guidance for the conduct of early Defence technical risk assessments prior to the formation of Performance Management Baseline at IBR. Hence, TIRAs would also be an input for Materiel Implementation Risk Assessments, which are developed by Defence’s Project Managers as part of the Project Approvals process.

1.30 Finally, work has commenced on development of a SCRAM extension for manufacturing and production which is not covered in the current SCRAM Process Reference and Assessment Models. This has been introduced following an increasing demand for SCRAM reviews in production related projects such as AWD shipbuilding and ASMD follow-on production programs.

Information and Communications Technology (ICT) Services

1.31 As the Defence ICT capability owner, the Chief Information Officer Group (CIOG) delivers the ICT infrastructure and services to CASG that are necessary to enable materiel capability to the war fighter as well as the corporate and other functions of CASG. In 2014-15, the DMO positioned its Information Management function to ensure close alignment with CIOG service delivery changes and the wider Defence service delivery reform currently underway.

1.32 The DMO (and now CASG) has supported the Program Definition Phase of a major new program—a Defence-wide Enterprise Resource Planning (ERP) system being managed by CIOG.

1.33 The Defence ERP will consolidate Finance, Logistics, Procurement, Engineering and Maintenance and Estate applications into a single ERP solution.

1.34 Significant benefits of this integrated model include increased availability and accuracy of information to support decision making, improved availability of Defence assets and improved information and operation in communications interrupted environments.

1.35 The implementation of a single ERP-enabled operating model will provide an opportunity to streamline and unify business processes throughout Defence.

Professionalisation

1.36 Defence’s performance is critically dependent on the skills, knowledge, expertise, competency, capability and professionalism of its workforce. The professionalisation of the CASG workforce remains central in ensuring committed people with the right skills are in appropriate roles to deliver capability.

1.37 Defence continues to refine education and skilling pathways for its staff. During 2014-15, the DMO facilitated 22 courses in the project management discipline; 17 in the logistics domain; 25 in the commercial and business domain; 22 in the procurement and contracting domain; and 10 engineering courses. The DMO also facilitated and managed 16 leadership programs for 333 candidates to enhance the leadership and management skills of future DMO (now CASG) and Defence senior executives; as well as sponsoring the Executive Masters in Business Administration Programs in Complex Program Leadership and Strategic Procurement conducted by the Queensland Institute of Technology for 19 DMO participants.

1.38 In 2014-15, the DMO sponsored 660 individuals under its Professional Body Sponsorship and Certification Program. A further 417 personnel have undertaken Australian Qualification Framework vocational training packages in the materiel logistics/sustainment management disciplines. The DMO has continued this program as a key strategy in DMO’s initiatives to professionalise the workforce.

Projects Exited the MPR

1.39 Table 1.4 lists all the projects that have exited the MPR since its inception. The Expenditure to Date is as at 30 June 2015.

Table 1.4 – List of Projects Exited from MPR

Introduction

2.1 This chapter provides a performance overview of the 25 projects included in the 2014-15 MPR. It includes detailed analysis of the three key variables of cost, schedule and materiel systems delivery against the Government’s approved budget, schedule and scope. The analysis commences at an aggregate level, discussing performance of all 25 MPR projects collectively, before addressing some project specifics.

2.2 The MPR only covers 25 of the 181 major projects as at 30 June 2015 (14 per cent of the Approved Major Capital Investment Program and 61 per cent by value), so caution must be applied when extrapolating any analysis to the entirety of Defence’s acquisition effort. This is because the projects in the MPR are not necessarily representative of all projects: the 25 projects are the largest by budget, at the time of inclusion and, in general, involve higher levels of complexity than other Defence projects with relatively smaller budgets.

Project Performance

2.3 Table 2.1 gives a summary of life-to-date budget approvals and Table 2.2 gives an overview of the in-year budget movements and cost performance. Table 2.3 shows schedule performance for the 25 projects in this year’s MPR and Table 2.4 presents a summary of the key characteristics of each project in terms of maturity and level of development required. Defence’s analysis indicates that, while projects have been managed within approved budgets, schedule performance, as identified in previous MPRs, continues to be the key issue.

Table 2.1 – Project Total Budget Status

Table 2.2 – Project In–Year Budget Status

Table 2.3 – Project Schedule Status

Table 2.4 – Project Characteristics

Budget Performance

2.4 Project budgets, against which cost performance is measured, are subject to variations arising from exchange rate variations, Government approval of changes in scope and transfers to Defence Groups. Following the move to out-turned project budgets in July 2010, there are no further variations to project budgets as a result of inflationary effects.

2.5 Table 2.1 includes a summary of the project budget variations from date of Government approval to 30 June 2015. Two variations in scope occurred during 2014-15:

• Growler (December 2014): budget increase of $200.6 million for inclusion of Growler Enabling Capabilities; and
• Battle Comm. Sys. (June 2015): budget decrease of $8.3 million transferred from LAND 75 Phase 3.4 to support design work now conducted as part of LAND 75 Phase 4 work package A.

2.6 Figure 2.1 provides a comparison of expenditure as at 30 June 2015 with the total approved budget at that date. While this provides an indicator of project progress, the percentage of budget spent is dependent on the nature of the project and the level of early investment that may be required for project start-up and non-recurring engineering effort.

2.7 Importantly, as at 30 June 2015 no project has exceeded its total approved budget and the net variation to the original Government approvals is negative 1.1 per cent, reflecting accumulated real savings to date of $417.2 million across the 25 projects.

Figure 2.1: Comparison of Overall Project Budget and Expenditure as at 30 June 2015 (in $m)

2.8 The relationship between project expenditure and project progress is not necessarily linear. The profile of expenditure against total approved budget is determined by several factors including the level of development required for the acquisition. For example, a project with a low level of development acquiring a Commercial or Military Off-The-Shelf (COTS or MOTS) product will generally have an expenditure pattern closely matched to the production of the materiel elements. In comparison, a highly developmental project usually requires a degree of initial ‘seed capital’ on commencement with expenditure declining during the development phase and increasing as the project shifts into the build/integration phase.

2.9 Another factor is the evolution of the project and its performance to date. A project may be well advanced in years but show a low level of expenditure against the total budget. This may result from poor contractual performance culminating in withholding of payments against specific milestones. This is, in effect, a deferral of payments that will be re-instated upon contractor achievement of milestones.

2.10 Table 2.2 provides a summary of the in-year spend against budget for 2014-15. Figure 2.2 compares in-year expenditure as at 30 June 2015 with budget for 2014-15. Most projects have expended close to their given budget with minor variations attributed primarily to exchange rates or rounding issues.

• Within Table 2.2, two projects had significant budget overachievements in percentage terms:
  • Battle Comm. Sys (Land) – 17.6 per cent overachievement ($2.9 million) is due to accelerated support contract initial payment and delivery of attrition spares for Combat Net radio equipment.
  • Battle Comm. Sys. – 11.7 per cent overachievement ($2.5 million) due to a Contract Change Proposal for the Track Management System.
• Five projects had significant budget underachievements in percentage terms:
  • Joint Strike Fighter – 21.3 per cent underspend ($63.3 million) is primarily due to United States F-35 Joint Program Office contracted timeframes and unpredictability of expenditure forecasts.
  • Battlefield Airlifter – 41.6 per cent underspend ($113.0 million) driven primarily by delay in billing by the United States and delayed spares and Ground Support Equipment deliveries, productions milestones for aircraft five through 10, training mobilisation and acquisition of mature training system devices. Delays in acceptance of aircraft three and four resulted in non-achievement of technical data milestones and delays in Ground Support Equipment and Tools procurement also contributed.
  • Collins RCS – 21.4 per cent underspend ($0.3 million) is primarily due to delays in programmed work by Australian industry.
  • UHF SATCOM – 75 per cent underspend ($3.9 million) is due to delays with the prime contract milestones (‘Test Readiness Review’ and ‘Spares, Support and Test Equipment’), which have additionally indirectly impacted Outsourced Service Providers’ ability to meet agreed milestone dates.
  • Collins R&S – 19.7 per cent underspend ($2.7 million) is primarily due to delays and cost savings by Australian Industry.

2.11 Figure 2.2 shows three projects (P-8A Poseidon, Growler and MH-60R Seahawk) had significant overachievements in dollar terms. All were due to early Foreign Military Sales case payments – i.e. payments occurred ahead of schedule.

Figure 2.2: Comparison of In-year Project Budget and Expenditure as at 30 June 2015 (in $m)

Contingency Management

2.12 In keeping with standard commercial practice, budgets for major Defence capital investment programs are approved by Government with a contingency provision that varies between projects depending on the complexity and risk of the acquisition. This allows Project Managers to proactively manage risk, and, when necessary, treat risk events that have materialised into issues. Contingency funding provides a limited financial margin for Project Managers against inherent uncertainties, risks and unexpected events that may arise during the course of a project. It is especially important in Defence projects that typically have greater inherent risk, longer timeframes and are generally more complex than private sector projects.

2.13 Previous reports have included the aggregate amount of contingency that has been spent over the life of the relevant MPR projects. Since the 2013-14 MPR reporting period, each Project Data Summary Sheet now includes advice on whether contingency has been applied to the project during the financial year. The application of contingency is assured by the ANAO. Consistent with this, the aggregate amount of contingency reported over the life of the relevant MPR projects in paragraph 2.14 is the aggregate amount of applied contingency rather than contingency actually spent. The term ‘applied contingency’ is the amount of contingency that a project has allocated against identified risks. It is a combination of both the amount of contingency that has already been spent and the amount that may need to be spent to manage or retire risk should it materialise.

2.14 Across the life of the 25 projects in this year’s MPR (that is, from November 1998 to June 2015), the aggregate amount of applied contingency is approximately $1.8 billion. This represents three per cent of the 25 projects combined project approval value ($60.5 billion) or 26 per cent of the combined contingency budget. The areas where risk has been retired using contingency include:

• systems development;
• systems integration;
• logistics and support;
• schedule constraints; and
• project resourcing.

2.15 Defence has established a framework which provides full accountability and traceability of all management decisions related to the use of the project contingency budget.

Schedule Performance

2.16 In 2010, Defence introduced Initial Materiel Release and Final Materiel Release (FMR) as more appropriate milestones for measuring materiel acquisition performance in contributing to the Capability Managers’ coordination of all capability elements to achieve Initial and Final Operational Capability (IOC and FOC) milestones. Schedule analysis presented in this and previous MPRs was based on achievement, or expected achievement, of FOC.

2.17 Figure 2.3 represents the schedule performance for each of the projects covered in this year’s MPR. The chart shows the original project approval date, the originally approved FMR estimate and the forecast FMR as at 30 June 2015. The chart also highlights the time it takes to deliver complex Defence acquisition projects.

2.18 Two projects achieved FMR during the 2014-15 financial year: Wedgetail and Battle Comm. Sys. Another three are scheduled to achieve FMR during 2015-16: LHD Ships; Air to Air Refuel; and LHD Landing Craft.

Figure 2.3: Schedule from Approval to Current FMR Estimate

2.19 The period between FMR and FOC indicates the time required by Capability Managers to bring together all the other Fundamental Inputs to Capability. This usually occurs after the final materiel elements of capability have been delivered.²¹⁵ Figure 2.4 provides an indication of the estimated time required to complete this work.

2.20 Two projects in the MPR – Wedgetail and Battle Comms. Sys. – achieved FOC during 2014-15; another three are scheduled to achieve FOC during 2015-16. The three projects are: ARH Tiger Helicopters; Air to Air Refuel; and LHD Landing Craft. The declaration of FOC marks the formal transition of capability from the acquisition to the sustainment phase of the capability life cycle at that time.

Figure 2.4: Current FMR and FOC Estimates

In-year Final Materiel Release Schedule Variance

2.21 Figure 2.5 illustrates the in-year schedule variance for FMR as a percentage of the total time expected to reach the FMR milestone. In the 2014-15 MPR, 14 projects re-assessed their estimated FMR date.

2.22 The largest individual in-year variance to forecast FMR dates is 63 per cent for UHF SATCOM reflecting an anticipated delay of 38 months. This is because the Network Control System component of the project is no longer a COTS solution and is now deemed as developmental software design. Due to some design faults, ViaSat has taken on elements of work previously contracted to their sub-contractors and are continuing the software design in-house.

2.23 Two projects are forecasting to be ahead of their FMR estimates from the previous years: Overlander Light – nine per cent and Collins R&S improved its FMR forecast by one per cent (more details are available in the Project Data Summary Sheets for each project).

Figure 2.5: In-year FMR Schedule Variance, by Project Type

Total Final Materiel Release Schedule Variance

2.24 Figure 2.6 shows the variance to forecast FMR dates, as a percentage of originally estimated duration, for each of the 25 projects in the MPR. At 30 June 2015, the average FMR variance for all of the MPR projects was 14 per cent.

2.25 Two projects are forecasting early achievement of FMR: Overlander Medium/Heavy and Collins R&S.

2.26 Seven projects reported an FMR schedule variation of 20 per cent or greater across the life of the project. The largest individual variance to forecast FMR dates for the life of the project is 87 per cent for UHF SATCOM reflecting a delay of 52 months due to the Prime Contractor’s decision to develop the software design in-house which was previously contracted out to sub-contractors. Battle Comm. Sys. has a variance of 56 per cent. The delay was due to the need to better align with the Army Brigade Rotation Cycle. Government approved a revised FMR baseline of March 2015 and the project subsequently met this, achieving FMR on 26 March 2015.

2.27 Hw Torpedo has an FMR variance of 40 per cent, across the life of the project, because the FMR date was set before the Full Cycle Docking program had reached maturity in terms of the length of dockings and impact of emergent work and other capability upgrades. As a result, the Heavy Weight Torpedo installation schedule has been delayed. Air to Air Refuel has a variance of 33 per cent due to a combination of impacts from supporting operations and contractor performance.

Figure 2.6: Total Schedule Variance for FMR, by Project Type

In-year Final Operational Capability Schedule Variance

2.28 Figure 2.7 shows in-year schedule variance to forecast FOC dates, as a percentage of the 2014-15 MPR project duration estimate. In the 2014-15 MPR, 11 projects re-assessed their forecast FOC date. Key points to note are as follows:

• The largest in-year FOC schedule variance was recorded by the Battlefield Airlifter project, which had in-year movement of 13 per cent. The variation was due to delays in aircraft production, to acquisition of Mature Training System devices and construction approval of facilities at RAAF Base Amberley, Queensland.
• Three projects: Wedgetail (one month); Overlander Light (five months); and Collins R&S (four months) have gained some schedule ahead of their FOC baselines.

Figure 2.7: In-year FOC Schedule Variance, by Project Type

Note: Joint Strike Fighter is currently developmental in nature but should ultimately become MOTS when it enters production line delivery.

Total Final Operational Capability Schedule Variance

2.29 The average variance to forecast FOC dates, as a percentage of originally estimated duration, for the 25 projects in MPR 2014-15, was 28 per cent as at 30 June 2015.

2.30 Figure 2.8 charts the schedule variance as a percentage of the originally estimated project duration from Government approval to FOC. The projects are grouped in the categories of MOTS, Australianised MOTS (AMOTS) and Developmental. The chart shows that, generally, MOTS projects are more likely to be delivered on time while AMOTS and Developmental acquisitions are more prone to underestimating technical complexity and systems integration effort.

2.31 The number of projects reporting an FOC schedule variation of 50 per cent or greater has decreased from 12 in last year’s MPR to nine. Delays to FOC over the life of these projects are as follows:

• Wedgetail has faced difficulties integrating the phased array radar and other mission critical elements into an operational system. However, the capability was successfully deployed during recent operations in the Middle East and achieved FOC in May 2015.
• MRH90 Helicopters delays are due to a number of technical and reliability issues.
• ARH Tiger Helicopter delay to FOC resulted from a reduction in the flying rate of effort experienced by the ARH fleet stemming from less than expected maturity of the contractor’s Tiger helicopter program at the time of acquisition.
• Air to Air Refuel delays are mainly due to development and improvements to the avionics and the air refuelling boom systems required to meet full capability. As with Wedgetail, the Air to Air Refuel capability has successfully supported the coalition’s recent operations in the Middle East.
• The two Collins projects – RCS and R&S – have been negatively impacted by limited platform availability due to operational requirements arising from the implementation of the Coles Review and unscheduled maintenance requirements having a detrimental impact on the Full Cycle Docking program.
• Deferral of FOC for the Anzac Anti-Ship Missile Defence program (Phases 2A and 2B) resulted from a Government approved change of project delivery strategy and scope, including the decision to substitute the Very Short Range Air Defence System option with a phased array radar capability. This is a significant capability advantage over the originally approved scope and allowed leading edge Australian technology to be proven in one ship before committing to the upgrade of the remaining seven ships.
• Battle Comm. Sys. delayed the FOC date to align installation of equipment with Capability Manager priorities. In December 2011, Government agreed to align the LAND 75 Phase 3.4 FOC with the Army’s Brigade rotation cycle circa December 2013. The approval was linked to a Basis of Provisioning change sought by the Army.

Figure 2.8: Schedule Variance for FOC since Government Approval, by Project Type

Note: Joint Strike Fighter is currently developmental in nature but should ultimately become MOTS when it enters production line delivery.

Effect of Kinnaird Reforms on Schedule

2.32 Implementation of the 2003 Kinnaird reforms has delivered a marked improvement in schedule performance for Defence capability acquisition projects. As shown in Table 2.5, average schedule variance to FOC for pre-Kinnaird MPR projects is 60 per cent, while the average variance for post-Kinnaird projects is 13 per cent. This is indicative of a general shift to more MOTS rather than developmental projects, as well as the age and stage of the post-Kinnaird projects.

Table 2.5 – Pre-Kinnaird and Post-Kinnaird Schedule Variance

Schedule Variance Attribution

2.33 Schedule slippage for developmental and AMOTS projects is attributable to the higher levels of technical complexity and system integration risk. The higher the technical challenge, the higher the inherent risk to the schedule. Developmental projects are new and often at the leading edge of available technology. As such, it is not uncommon for projects, as they progress, to encounter unforeseen technical difficulties requiring significant modification that results in delay. Similarly, Australianisation can also encounter unforeseen technical difficulties and have an unanticipated impact on existing features of the baseline MOTS product. Rectifications of such issues often require extensive, time consuming remediation work.

2.34 Further analysis of 2014-15 MPR data has revealed additional drivers of schedule delays as shown in Table 2.6. Of the seven projects that underestimated technical complexity, five could also identify some overestimation of industry capability and performance, and another two were also affected by approved changes to scope (ANZAC ASMD Phases 2A/2B attained Government approval to acquire the more capable phased array radar based solution over the original very short range air defence system solution, though without an increase to schedule). In September 2012, Government announced a re-baselining of the AWD Ships construction schedule²¹⁶, and limited platform availability was the primary driver for delays of Collins class submarine projects.

2.35 Some projects also have a greater delay to FOC as a result of Capability Manager decisions to adjust schedule in order to align with expected capability outcomes or operational imperatives.

Table 2.6 – Attribution of Schedule Variance Factors

Materiel Scope Performance

2.36 Materiel Scope Performance measures represent the key materiel capability performance attributes of a project delivered by Defence and industry, which, if not satisfied, could have a significant detrimental effect on the eventual suitability of equipment for operational service. The Materiel Scope Performance measures for each project, as defined in the Materiel Acquisition Agreement, are identified from the project approval documentation, including the Operational Concept Document and the Function and Performance Specification, which detail the capital equipment assets to be delivered.

2.37 For security classification reasons the MPR does not identify the individual measures for each of the projects. However, each PDSS has a percentage breakdown on how the project is tracking against its particular suite of capability elements.

2.38 The subjective ‘traffic light’ assessment of each element is indicative of:

• Green: a high level of confidence that the capability outcome sought will be met;
• Amber: the capability outcome being under threat but still considered manageable and able to be met; and
• Red: at this stage, the capability outcome is unlikely to be fully met.

2.39 Materiel Capability Performance measures indicate the status of the materiel element of capability against the FMR milestone’s definition as at 30 June 2015 and are not necessarily indicative of each project’s ultimate ability to deliver the final intended scope. For the 2014-15 MPR, there are 240 individual Materiel Capability elements across the 25 projects with 234 green (or 97 per cent), six amber (or three per cent) and zero red.

Figure 2.9: Materiel Capability Performance measures for the 2014-15 Report

Figure 2.10: Trends for Materiel Capability Performance measures

2.40 Figure 2.9 indicates that a high percentage (97 per cent) of Materiel Capability Performance measures are currently reported as green.

2.41 The three per cent reported as amber is attributed to Joint Strike Fighter. The project assesses that Phase 2A/B (Combined Stages 1 and 2) will deliver its materiel requirements, noting there are a number of risks to achieving some of the materiel capabilities required to deliver IOC and FOC. These risks include: integration of the F-35A into Defence’s systems; final software builds meeting required functionality by IOC and FOC; establishing the sustainment capability; and, re-establishing the training systems. Mitigation strategies are in place for all the risks identified (more details are in the project’s PDSS).

2.42 Figure 2.10 shows previous MPRs’ Capability Performance Measures, as at 30 June for the respective years. The number of performance measures has increased from 138 across 22 projects in 2009-10 to 240 across the 25 projects in 2014-15. The increase in average number of Capability Performance Measures per project from 6.3 in 2009-10 to 9.6 in 2014-15, provides increased transparency of scope performance. Analysis of these measures indicates that Defence’s performance has improved over the last seven years, with the majority of current performance measures likely to be achieved.

Appendix 1: Categorising Acquisitions

The DMO categorises its acquisition projects to enable it to differentiate between the complexities of business undertakings, focus management attention, provide a basis for professionalising its workforce and facilitate strategic workforce planning. Projects are graded into one of four Acquisition Categories (ACATs):

• ACAT I – These are major capital equipment acquisitions that are normally the ADF’s most strategically significant. They are characterised by extensive project and schedule management complexity and very high levels of technical difficulty, operating, support and commercial arrangements;
• ACAT II – These are major capital equipment acquisitions that are strategically significant. They are characterised by significant project and schedule management and high levels of technical difficulty, operating, support arrangements and commercial arrangements;
• ACAT III – These are major or minor capital equipment acquisitions that have a moderate strategic significance to the ADF. They are characterised by the application of traditional project and schedule management techniques and moderate levels of technical difficulty operating, support arrangements and commercial arrangements; and
• ACAT IV – These are major or minor capital equipment acquisitions that have a lower level of strategic significance to the ADF. They are characterised by traditional project and schedule management requirements and lower levels of technical difficulty, operating, support and commercial arrangements.

As the complexity of a project will vary over its life cycle, Defence reviews project acquisition categories at defined milestones between entry into the Defence Capability Plan and project completion.

The ACAT framework provides a recognised, consistent and repeatable methodology for categorising projects and aligning Project Managers’ certified experience and competencies to the complexity and scale of projects under management. Project Managers are assigned to acquisition projects on the basis that their Certified Professional Project Manager status is consistent with the project’s ACAT level.

The ACAT level of a project is assessed against six project attributes:

• acquisition cost: the approved budget for the project;
• project management complexity: the complexity of project management necessary for its execution;
• schedule complexity: the inherent complexity brought about by delivery pressures on the project;
• technical difficulty: the complexities associated with technical undertakings such as design and development, assembly, integration, test and acceptance;
• operation and support: the complexity associated with readying the organisation and environment in which the system will be operated, supported and sustained; and
• commercial: the readiness and capability of industry to develop, produce and support the required capability, and the complexity of the commercial arrangements being managed.

Appendix 2: Project Maturity Scores – Monitoring Progress

The DMO’s Project Maturity Score quantifies the maturity of a project by way of a score based on the Project Managers’ judgement at defined milestones in its capability development and acquisition phases. This score is then compared against an ideal or benchmark score for that milestone. A project’s maturity is assessed on 16 milestones across its lifecycle and for each of these milestones the ideal or benchmark condition is represented by a benchmark score as shown in Figure 1.

Figure 1: Benchmark Maturity Scores

The Project Maturity Score comprises a matrix of seven attributes:

• Schedule;
• Cost;
• Requirement;
• Technical Understanding;
• Technical Difficulty;
• Commercial; and
• Operations and Support.

The Project Manager assesses the level of maturity that a project reaches at a particular milestone for each of these attributes on a scale of 1 to 10. Score assessment is made by selecting the most appropriate description that fits the question under the attributes columns. Project Maturity Scores provide a means of communicating in a simple fashion an indicative ‘as is’ versus a ’should be’ condition to inform decision making for each project. The scores are not precise and are not intended to enable exact comparisons across projects. Following is a description of the Project Maturity Score Attributes.²¹⁷

Appendix 3: Lessons Learned

The Joint Committee of Public Accounts and Audit recommended in Report 422: Review of the 2009-10 Defence Materiel Organisation Major Projects Report, that the DMO include a Lessons Learned section in the MPR for projects that have met the exit criteria to report on their lessons learned at both the project level and the whole of organisation level.

Table 1 lists their lessons at the project level, against a whole of organisation level category.

Table 1: Lessons Learned – Project Level

Lessons Learned – Whole of Organisation Level

The 2014-15 MPR builds on the lessons learned (at the organisational level) reported in last year’s report. Set out below is a summary of progress against the categories of lessons learned.

Military Off-The-Shelf Equipment

The lessons learned in relation to Military-Off-The-Shelf equipment are incorporated in DMO policies.

Schedule Management

The lessons learned in relation to Schedule Management are incorporated in Chapter 1, Schedule Management, 1.26-1.30, page 95.

Requirements Management

To address issues associated with understanding the level of maturity of requirements and progression through the various systems engineering reviews, the standard procedure covering Function and Performance Specification (FPS) development was amended to provide further guidance on the maturity levels for the FPS when it is progressing through the capability development process. Additionally, policy, guidance handbook and a standard procedure was promulgated to assist with assessing the suitability and risks for system reviews conducted in acquisition.

To improve the quality of objective evidence, or data, relating to acquiring and sustaining materiel systems, emphasis has been applied in the areas of configuration management and Verification and Validation (V&V). Improvements in the V&V policy, handbook and a standard procedure were issued to provide a more robust approach in utilising risk-based methodology to ensure that the necessary objective evidence is obtained. To reinforce the data quality of materiel systems an overarching configuration management policy has been promulgated to ensure reliable materiel system status throughout acquisition and sustainment.

Appendix 4: Glossary