Abbreviations

Note: A full list of the Major Projects and their abbreviations are included in Part 1, at page 7, of this report.

Auditor-General’s Foreword

The Defence Materiel Organisation (DMO) supports the capacity and capability of the Australian Defence Force by undertaking important acquisition and sustainment activities. This important work is the subject of this, the seventh Major Projects Report (MPR), which continues the annual review and analysis by the Australian National Audit Office (ANAO) on the progress of selected major Defence equipment acquisition projects (Major Projects), managed by the DMO.

This report builds on the earlier work by the DMO and the ANAO to improve the transparency of, and accountability for, the status of Major Projects for the benefit of the Parliament, the Government and other stakeholders.

The management of Major Projects is complex and, for this reason, it is a major challenge for the DMO to deliver the required capability on schedule and within budget. Consistent with previous years, schedule slippage remains a key focus for the DMO, particularly for projects regarded as developmental. However, given the ongoing interest in all aspects of the delivery of Major Projects, the ANAO will continue to monitor delivery in terms of cost, schedule and capability.

In such a complex environment, the ongoing support of the Joint Committee of Public Accounts and Audit (JCPAA) has been important to the development of the MPR, providing guidance and insights from their considerations. Each year the JCPAA endorses the Guidelines for the review, and provides direction and recommendations to assist the development of future MPRs.

As previously, this year’s review continued the strong working relationship between the ANAO and the DMO. Other key stakeholders within the Department of Defence, and in particular the Capability Managers, and industry stakeholders, also provided valuable input to assist with the review.

Ian McPhee

8 December 2014

Introduction

1. Major Defence equipment acquisition projects (Major Projects) remain the subject of considerable parliamentary and public interest, in view of their high cost and impact on the economy, contribution to national security and the challenges involved in completing them within budget, on time and to the required level of capability.

2. The proposed 2015¹ Defence White Paper is expected to consider the Australian Defence Force’s (ADF’s) priorities for future capability investment for the Commonwealth. These considerations will concern the acquisition of new submarines and frigates, as well as the replacement of the land vehicle fleet, and are foreshadowed to involve some of the most significant investment decisions in the Commonwealth’s history.² In addition, the Government has commissioned the Defence First Principles Review which is aimed at delivering ‘a more commercially astute and focused materiel acquisition and sustainment capability’.³ The findings of this review are expected to inform the development of the White Paper.

3. The Defence Materiel Organisation (DMO) provides support to ADF operations through the acquisition and sustainment of ADF capabilities⁴ and expended some $4.1 billion on major and minor capital acquisition projects in 2013–14.⁵

4. However, acquisitions by the DMO alone do not generate new capability for the ADF until they have been successfully introduced into service. The overarching responsibility for the introduction into service of Major Projects, for example, provision of personnel, training and command, normally resides within other areas of the Australian Defence organisation.⁶ However, while the DMO’s acquisition role is only part of the introduction into service of new capability, it is a significant one.

The 2013–14 Major Projects Report

5. This seventh Major Projects Report (MPR) covers 30 of the DMO’s Major Projects and is the first report covering this many projects (2012–13 and 2011–12: 29, and 2010–11: 28). This report builds on the earlier work to improve the transparency of, and accountability for, the status of Major Projects, and is supported by the commitment of the Joint Committee of Public Accounts and Audit (JCPAA), ‘…to maximise transparency and accountability in the Defence acquisition process for major projects managed by DMO.’⁷

6. The Australian National Audit Office‘s (ANAO’s) review of Major Projects in the MPR is completed in conjunction with the regular program of performance and financial statement audits conducted within the Defence portfolio. While by its nature, the report is not as in depth as a performance audit, it provides an opportunity to analyse data across a consistent range of projects over time. The benefits of this analysis have been noted by a variety of stakeholders, including Ministers, Parliamentary Committee members, industry and the media.

The 2013–14 Major Projects

7. Projects included in the MPR are selected based on criteria included in the 2013–14 Major Projects Report Guidelines (the Guidelines), as endorsed by the JCPAA. These criteria provide a selection of the most significant Major Projects managed by the DMO, on behalf of the ADF. The total approved budget for the Major Projects included in the 2013–14 MPR is approximately $59.4 billion, covering nearly 63 per cent of the budget within the Approved Major Capital Investment Program of $94.7 billion.⁸

8. The projects and their approved budgets are listed in Table 1, below.

Table 1: MPR projects and approved budgets at 30 June 2014

Source: See the Project Data Summary Sheets in Part 3, from page 175, of this report.

Note 1: Overlander Medium/Heavy, Growler, Battlefield Airlifter and LHD Landing Craft are included in the MPR program for the first time in 2013–14.

Report objective and structure

9. The objective of this report is to provide the Auditor-General’s independent assurance over the status of selected Major Projects, as reflected in the Statement by the Chief Executive Officer (CEO) DMO, and the Project Data Summary Sheets (PDSSs), prepared by the DMO. Assurance from the ANAO’s review of the preparation of the PDSSs is conveyed in the Auditor-General’s Independent Review Report, prepared pursuant to the endorsed Guidelines, and included in Part 3 of this report (pp. 165–542) along with the aforementioned Statement by the CEO DMO and PDSSs.

10. Excluded from the scope of the ANAO’s review is PDSS data on projects’ budget adequacy, the identification of Risks and Issues, the Measures of Materiel Capability Delivery Performance, and ‘forecasts’ of future dates and the achievement of future outcomes. By its nature, this information relates to future events and depends on circumstances that have not yet occurred or may not occur, or may be impacted by events that have occurred but have not yet been identified. Accordingly, the conclusion of this review does not provide any assurance in relation to this information.

11. The ANAO’s analysis on the three key elements of the PDSSs—cost, schedule and the progress towards delivery of required capability, in particular, longitudinal analysis across these key elements of projects over time, are contained in Part 1 (pp. 1–98). The ANAO’s analysis on other elements of the PDSSs, for example, project maturity and elements excluded from the scope of the formal review, are also included in this part, to provide readers with a balanced perspective over all key acquisition elements.

12. Further insights and context by the DMO on issues highlighted during the year are contained in Part 2 (pp. 99–164)—although not included within the scope of the review by the ANAO.

13. Part 4 includes the Guidelines endorsed by the JCPAA (pp. 543–572).

14. Figure 1, below, depicts the key parts of this report. To assist in conducting inter-report analysis, the presentation of data remains largely consistent and comparable with the 2012–13 MPR.

Figure 1: Report structure

 

 

Refer to paragraphs 9 to 13 in Part 1, at page 8, of this report.

15. For each Major Project, a corresponding PDSS includes detailed information on project performance: the approved budgeted cost and expenditure; schedule progress; the DMO’s assessment of progress toward delivering those aspects of key capabilities for which the DMO is responsible; major risks and issues; maturity scores; and lessons learned. This information has been prepared by the DMO in accordance with the Guidelines as endorsed by the JCPAA. Additionally, as projects appear in the MPR for multiple years, changes to the PDSS from the previous year are depicted in bold purple text.

16. Each PDSS comprises:

• Project Header—including name; capability and acquisition type; approval dates; total approved and in-year budgets; stage; complexity; and image;
• Section 1—Project Summary: including description; current status, including a financial assurance and contingency statement; context, including background, unique features and major risks and issues; and other current sub-projects;
• Section 2—Financial Performance: including the project’s budget and expenditure, as well as variations to the budget; in-year variances between budgeted and actual expenditure; and major contracts in place (in addition to quantities delivered as at 30 June 2014);
• Section 3—Schedule Performance: provides information on the design development; test and evaluation process; and forecasts and achievements against key project milestones including Initial Materiel Release (IMR), Final Materiel Release (FMR), Initial Operational Capability (IOC) and Final Operational Capability (FOC)^{9 10};
• Section 4—Project Cost and Schedule Status: represents the project’s cost and schedule status in a graphical format as at 30 June 2014;
• Section 5—Materiel Capability Delivery Performance: provides a summary of the DMO’s assessment of its progress on delivering key capabilities;
• Section 6—Major Risks and Issues: outlines the major risks and issues of the project;
• Section 7—Project Maturity: provides a summary of the project maturity as defined by the DMO and a comparison against the benchmark score;
• Section 8—Lessons Learned: outlines the key lessons that have been learned at the project level (further information on lessons learned by the DMO are included in the DMO’s Appendix 3); and
• Section 9—Project Line Management: details current project management responsibilities within the DMO.

17. Consistent with the Guidelines, information of a classified nature has been excluded from the PDSSs.

The role of the Joint Committee of Public Accounts and Audit

18. Influential in establishing the MPR, the JCPAA has taken an active role in the development and review of the MPR program. Each year, the Committee considers the draft Guidelines, incorporating the selection of projects for review, and provides the Committee’s views in relation to the Guidelines’ content and development, prior to their endorsement. Following endorsement by the Committee, the Guidelines provide the basis for the DMO’s preparation of the PDSSs and the criteria for the ANAO’s review.¹¹

19. The main change to the 2013–14 Guidelines was the inclusion of a statement within each PDSS in relation to whether a project has, or has not, applied contingency funds during the year. The DMO applies contingency to known and emergent risks as a part of standard risk management processes¹², however, the application of contingency does not necessarily result in expenditure, unless risks materialise and require action to be taken. Although the amount of contingency funds applied, or whether the applied contingency funds were expended does not have to be disclosed, this amendment will provide readers with the opportunity to identify, at a minimum, projects that have had contingency funds applied during the financial year.

20. Subsequent to the tabling of the MPR in Parliament each year, the JCPAA considers the inclusion of the report in its schedule of review. In March 2014, the JCPAA held a public hearing into the results of the 2012–13 review and in May 2014 the Committee published Report 442, Review of the 2012–13 Defence Materiel Organisation Major Projects Report. The JCPAA’s recommendations are set out below¹³:

21. The Committee’s recommendations contribute to the development of the MPR each year and the formal response by the Department of Defence to the above-mentioned recommendations was provided to the Committee in late November 2014, for its consideration. However, while at the time of preparing this report the Committee’s views on the Department’s response was not known, the ANAO and the DMO’s actioning of recommendations in the 2013–14 MPR is summarised below:

• Recommendation 1—Actioned: In-year budget to actual variance analysis is included in the 2013–14 MPR at paragraphs 2.29 to 2.31 in Part 1, and Table 2.2 and paragraph 2.10 in Part 2;
• Recommendation 2—Actioned: A revised Section 2.2 is included in the 2014–15 MPR Guidelines, which were endorsed in September 2014;
• Recommendation 3—Actioned: Improved consistency of project names is addressed at Table 1 in Part 1 and throughout the MPR;
• Recommendation 4—See further explanation at paragraphs 22 and 23, below;
• Recommendation 5—Actioned: A longitudinal representation of total slippage post Second Pass Approval and acquisition type by approval date is included at Figures 8 and 9 in Part 1;
• Recommendation 6—Actioned: Statements and graphs relating to capability are appropriately caveated in Sections 1.2 and 5.1 of each PDSS in Part 3;
• Recommendation 7—Actioned (partially): High level enhancements to the clarity of capability performance information is outlined at paragraph 2.65 in Part 1 and included in the 2014–15 MPR Guidelines;
• Recommendation 8—Actioned: Project maturity score analysis is included at paragraphs 2.12 to 2.14 in Part 1, and the PDSSs in Part 3;
• Recommendation 9—Actioned: The DMO has included information on recently exited Major Projects at Table 1.3 in Part 2; and
• Recommendation 10—Actioned: Information on exited Major Projects will be included within the MPR from 2014–15.

22. While Defence sustainment projects are generally outside of the scope of the MPR, the Collins R&S project (which is defined as a sustainment project by the DMO) has been included in the MPR at the request of the JCPAA since 2009–10. In addition, while ARH Tiger Helicopters and Collins RCS have been transferred to ‘sustainment’, they will be included within the 2014–15 MPR following the endorsement of the 2014–15 Guidelines.

23. In September 2014 the JCPAA also requested that the ANAO develop an options paper on sustainment reporting, and review other international works in this area. The ANAO will consult with the DMO in preparing a response to the JCPAA on this issue. The options developed will need to recognise that assessments in relation to readiness and availability of major Defence capabilities are classified.

24. Additionally, a performance audit to examine the contribution made by Materiel Sustainment Agreements to the effective sustainment of specialist military equipment, is expected to be tabled in 2015.

Overall outcomes

25. This seventh MPR continues the review of seven of the nine DMO Major Projects which were initially introduced in the 2007–08 MPR and has continued to introduce new projects up to the originally agreed maximum of 30 projects for review.¹⁴ The MPR maintains the transparency and accountability for performance relating to cost, schedule and progress towards delivering the key capabilities of Major Projects, and provides opportunities for further longitudinal and other analysis into the future.

The 2013–14 Major Projects review (Chapter 1)

26. Under section 19A(5) of the Auditor-General Act 1997, the ANAO has reviewed the PDSS data as contained in this volume as a priority assurance review and presents the Auditor-General’s Independent Review Report, at page 167.

27. As noted in paragraph 19, at page 11, the 2013–14 Guidelines required the introduction of a ‘contingency statement’ in Section 1.2 Current Status (Cost Performance) within each PDSS. Project offices are now required to indicate whether they have had contingency funds applied during the year, as well as disclosing which risks were mitigated by the application of those contingency funds. The six projects which applied contingency funds in 2013–14 were Joint Strike Fighter¹⁵, AWD Ships¹⁶, MRH90 Helicopters¹⁷, LHD Ships, FFG Upgrade and Additional Chinook.

28. Additionally, the ANAO continued to assess the progress of the DMO in addressing previously raised issues in relation to the administration of Major Projects. In 2013–14 issues were again noted within the following areas of project management, including:

• continued concerns of project offices in relation to price indexation and budget allocations, and inconsistency in the recording and application of contingency funds, (Section 1 of the PDSS);
• inconsistency in the application of the project maturity framework¹⁸, reducing its level of reliability of the maturity assessment, (Section 7 of the PDSS);
• inconsistency in the recording and reporting of major risks and issues by project offices, and in the terminology and reporting within the mandated Predict! and Excel risk management systems¹⁹, (Section 6 of the PDSS); and
• inconsistency in the application of the capability assessment framework²⁰, (Section 5 of the PDSS).

29. The Auditor-General’s Independent Review Report takes into account the overall governance of Major Projects, the results of our examination of the DMO’s project management and reporting arrangements, and the results of our substantive procedures to gain assurance in relation to key information reported in PDSSs. In 2013–14, the results of the ANAO’s priority assurance review of the 30 PDSSs was that nothing has come to the attention of the ANAO that causes us to believe that the information and data in the PDSSs, within the scope of our review, has not been prepared, in all material respects, in accordance with the Guidelines.

Analysis of projects’ performance (Chapter 2)

30. The data reviewed in the PDSSs covers the three major dimensions of project performance: cost, schedule, and progress towards delivering the planned capability.

31. Table 2, below, provides summary data on the DMO approved budget, schedule performance and progress toward delivering capabilities for the Major Projects covered in this report, and compares data against that reported in previous MPR editions.

Table 2: Summary longitudinal analysis

Refer to paragraphs 31 to 45 in Part 1, at pages 17 to 22, of this report.

Note 1: The data for the 30 Major Projects in the 2013–14 MPR compares the data from projects in the 2012–13 MPR and 2011–12 MPR. A comparison of the data across years should be interpreted in this context, i.e. once a project is removed from the MPR, data is removed from the total slippage calculation for all years, but remains within in-year calculations where relevant.

Note 2: Slippage refers to the difference between the original government approved date and the current forecast date. These figures exclude schedule reductions over the life of the project.

Note 3: Based on the 27 projects from the 2010–11 MPR, 29 projects from the 2011–12 MPR and 26 projects from the 2012–13 MPR respectively.

Note 4: The grey section of the table is excluded from the scope of the ANAO’s priority assurance review. See further explanation in paragraph 10, at page 8, of this report.

Cost

32. Within the review period, all projects continued to operate within the total approved budget of $59.4 billion.²¹

33. The total budget for Major Projects included in this MPR has increased by $16.8 billion (37.9 per cent) since Second Pass Approval. Refer to Table 3, below.

Table 3: Budget variation over $500 million post Second Pass Approval by Variation type

Source: The ANAO’s analysis of the 2013–14 PDSSs. Refer to paragraph 33, above.

Note: For the breakdown of in-year variation, refer to Table 7, at page 54, of this report.

34. Overlander Vehicles, provided with Second Pass Approval in August 2007, was separated into two phases, Phase 3A Lightweight and Light Capability and Phase 3B Medium and Heavy Capability, in December 2011. Phase 3A was reapproved for Second Pass at that time; however Phase 3B did not receive Second Pass Approval until July 2013 and was reflected as a scope decrease of $2.2 billion in the 2012–13 MPR. This timing difference has now been readjusted in this report.

Schedule

35. Maintaining Major Projects on schedule remains an ongoing challenge for the DMO²²; in turn affecting when the capability is made available for operational release and deployment by the ADF.

36. In the 2013–14 MPR, the total schedule slippage for the 30 Major Projects as at 30 June 2014 is 1 115 months (2012–13: 957 months) when compared to the initial schedule first approved by government. This represents a 36 per cent (2012–13: 36 per cent) increase on the originally approved schedule. Refer to Table 4, below.

Table 4: Schedule slippage from original planned FOC

Source: The ANAO’s analysis of the 2013–14 PDSSs. Refer to paragraph 36, above.

37. While it should be noted that platform availability contributes to the slippage within the ‘Collins’ projects, the other most significant slippage delays relate to those projects with the most developmental content.

38. Disaggregation according to a project’s Second Pass Approval shows that 80 per cent (2012–13: 87 per cent) of the total schedule slippage across the Major Projects covered in the 2013–14 MPR is made up of projects approved prior to the DMO’s demerger from the Department of Defence, in July 2005. This is a positive indicator of the benefits that the DMO, as a specialist acquisition and sustainment organisation, is able to bring to complex Defence procurement. It also demonstrates the impact on schedule performance during the transition to higher levels of Military-Off-The-Shelf (MOTS) acquisitions following the Defence Procurement Review 2003 (Kinnaird Review).²³

39. Additional ANAO analysis (refer to Figure 8, at page 62) presents project slippage as reported in each of the seven MPRs against the DMO classification of projects as MOTS, Australianised MOTS or developmental.²⁴ These classifications are a general indicator of the difficulty associated with the procurement process. This figure highlights, prima facie, that the more developmental in nature a project is, the more likely it will result in project slippage, as well as demonstrating the advantages of selecting MOTS acquisitions.²⁵ For the first time, Figure 9 (at page 63) begins the analysis of completed projects which have been removed from the MPR.

40. The reasons for schedule slippage vary but primarily reflect the underestimation of both the scope and complexity of work, particularly for Australianised MOTS and developmental projects (see paragraphs 2.29 to 2.31 in Part 2).

Capability

41. The third major aspect of project performance examined by this report is progress towards the delivery of capability required by government and specified by the ADF. Assessment of expected capability delivery by the DMO is outside the scope of the Auditor-General’s formal review conclusion, but is included in the analysis to provide an overall perspective of the three major components of project performance.

42. The DMO expects that the 30 projects in this year’s MPR will deliver all of their key capability requirements, recognising that some elements of the capability required may be under threat, but considered manageable (assessed as either green or amber). This is consistent with the 2012–13 assessment, and represents five project offices currently having challenges (2012–13: six).

43. This year, as reported by the DMO, the delivery of four per cent (2012–13: five per cent) of the key capabilities is considered to be under threat but is considered manageable. The projects considered to have some elements under threat, but considered manageable are: Joint Strike Fighter, Wedgetail, MRH90 Helicopters, Air to Air Refuel and FFG Upgrade. Further details are outlined at paragraph 2.77, at page 78. Refer also to Table 5, below.

Table 5: Longitudinal Expected Capability Delivery

Source: The ANAO’s analysis of the PDSSs in published MPRs.

Note 1: All projects had some component/s with high confidence of delivery.

Note 2: Removal of the moving target capability from the project scope in 2010–11, combined with resolution of fuze production issues and successful completion of Operational Test and Evaluation in 2012–13, has led to this project achieving its remaining scope.

Note 3: The Course Correcting Fuze has been cleared for production in the United States and is no longer considered ‘under threat’ of delivery.

44. As shown in Table 5, above, the majority of projects for which the DMO’s expected capability delivery is below 100 per cent, are developmental or Australianised MOTS (AMOTS). The only exception is 155mm Howitzer, which was awaiting clearance of the Course Correcting Fuze for production, which has since occurred.²⁶

45. In addition, the DMO has continued the practice of including declassified information on settlement actions for projects in the interests of providing greater transparency to readers of this report. In 2013–14, a renegotiation with Airbus Defence and Space regarding delivery of Air to Air Refuel contracted requirements resulted in the signing of a Deed of Settlement.²⁷ Prior settlements for projects within this report include Wedgetail²⁸, MRH90 Helicopters²⁹ and ARH Tiger Helicopters.³⁰

Developments in acquisition governance (Chapter 3)

46. Consistent with previous years, developments in acquisition governance processes are covered in the ANAO’s review, with general indications of positive impacts from some of the more recent initiatives. As might be anticipated, while some initiatives continue to mature, others require further progress prior to achieving their intended impact. These developments broadly relate to the following key acquisition governance areas.

Gate Review Boards

47. First introduced in 2008, the Gate Review process³¹ was designed to provide the CEO DMO with assurance that all identified risks for a project are manageable, and that costs and schedule are likely to be under control prior to a project passing various stages of its life cycle. However, while the Gate Review process is continuing to evolve, the DMO intend to review their application within acquisition and expand into sustainment in future years.

Projects of Concern

48. First established in 2008, the Projects of Concern process was implemented to address project issues of concern to the DMO and government, relating to cost, schedule and capability. The process has continued to play an important, although limited role, across the portfolio of MPR projects. The Projects of Concern within the MPR at 30 June 2014 are AWD Ships, MRH90 Helicopters and Air to Air Refuel.

Early Indicators and Warnings

49. In May 2011, the then Government announced the impending implementation of the Early Indicators and Warnings (EI&W) system to identify problems with projects in the formative stages of the life cycle. However, following the June 2013 review by the Defence Capability and Investment Committee, the CEO DMO proposed a new reporting system in December 2013, the Quarterly Project Performance Report, which is expected to replace EI&W. Further analysis of this new system will be undertaken in future MPRs.

Joint Project Directives

50. The introduction of a requirement for Joint Project Directives (JPDs)³² in 2009–10, for all projects approved by government from March 2010 is maturing and expected to have a greater influence over the portfolio of MPR projects in future years. To date only eight MPR projects have completed a JPD.³³ It is expected that JPDs will provide a sound basis for ensuring government requirements are delivered and further review of the JPD process will be undertaken in future MPRs.

Business systems rationalisation

51. The DMO’s business systems³⁴ rationalisation is aimed at consolidating processes and systems in order to provide a more manageable system environment. Although progress to date has been limited, the DMO advises it has taken some steps in this area through the development of an Integrated Project Management System which is expected to improve reporting practices across the DMO and will be tested on selected project offices in 2014–15.

Project management and skills development

52. Project management and skills development within the DMO and the Defence Industry is a key challenge for the Government and industry alike. Over the last decade, more than $300 million has been provided by government alone to assist with professionalising DMO staff and up-skilling participants within the Defence Industry. While DMO activities have increased professional competencies held by the DMO staff, the measurement of the impact within industry has been limited. The ANAO will continue to review project management and skills development programs in the 2014–15 MPR.

53. Consistent with previous years, the ANAO’s detailed assessment of these governance initiatives is contained in Chapter 3 of Part 1.

Introduction

1.1 This chapter provides an overview of the 2013–14 review scope and approach adopted by the Australian National Audit Office (ANAO) in consideration of the Project Data Summary Sheets (PDSSs) and the subsequent results of the review.

1.2 Previous reviews have highlighted issues which impact on the Defence Materiel Organisation’s (DMO’s) administration of major Defence equipment acquisition projects (Major Projects), and their related frameworks. These issues were reconsidered, where appropriate, in order to assess the DMO’s progress in addressing them during 2013–14. Key frameworks considered further in this chapter include:

• the financial framework as it applies to the management of project budgets and expenditure, in an out-turned budget environment;
• the project maturity framework and systems in place to support the provision of maturity data in the PDSS;
• the enterprise risk management framework as it applies to major risk and issue data and its maturity; and
• the capability assessment framework, as it relates to the DMO’s evaluation of the probability of delivering required capabilities.

1.3 This chapter also makes reference to areas of focus raised by the Joint Committee of Public Accounts and Audit (JCPAA) for consideration in the development of this and future Major Projects Reports (MPRs). As noted in paragraph 20, at page 11, the JCPAA examined the 2012–13 MPR in March 2014, publishing Report 442, Review of the 2012–13 Defence Materiel Organisation Major Projects Report in May.

1.4 Chapter 2 of Part 1 provides consideration of the DMO’s project performance, based on the information provided in the PDSSs.

1.5 Chapter 3 of Part 1 includes developments in the DMO’s acquisition governance arrangements, which were also taken into consideration during the review process.

Review scope and approach

1.6 In 2012 the JCPAA identified the review of the DMO’s PDSSs as a priority assurance review, is that it provides the ANAO full access to the information gathering powers under the Auditor-General Act 1997.

1.7 The ANAO’s review of the individual project PDSSs, which are contained in Part 3, were conducted in accordance with the Australian Standard on Assurance Engagements (ASAE) 3000 Assurance Engagements Other than Audits or Reviews of Historical Financial Information issued by the Australian Auditing and Assurance Standards Board.

1.8 Excluded from the scope of the ANAO’s review is PDSS data on projects’ budget adequacy, the identification of Risks and Issues, the Measures of Materiel Capability Delivery Performance, and ‘forecasts’ of future dates and the achievement of future outcomes. By its nature, this information relates to future events and depends on circumstances that have not yet occurred or may not occur, or may be impacted by events that have occurred but have not yet been identified. Accordingly, the conclusion of this review does not provide any assurance in relation to this information.

1.9 While our work is appropriate for the purpose of providing an Independent Review Report in accordance with ASAE 3000, our review of individual PDSSs is not as extensive as individual audits conducted by the ANAO, in terms of the nature and scope of issues covered, and the extent to which evidence is required by the ANAO. Consequently, the level of assurance provided by this review in relation to the 30 Major Projects is less than that provided by our program of audits.

1.10 However, the MPR is well positioned to examine systemic issues and provide longitudinal analysis for the 30 projects reviewed, and may also reflect on, or have implications for, general project management practices in the DMO, or more broadly within other areas of the Australian Defence organisation.

Areas of review focus

1.11 The ANAO’s review of the information presented in the individual PDSSs included:

• examination of each PDSS and the documents and information relevant to them;
• a review of relevant processes and procedures used by the DMO in the preparation of the PDSSs;
• an assessment of the systems and controls that support project financial management, risk management, and project status reporting, within the Australian Defence organisation;
• interviews with persons responsible for the preparation of the PDSSs and those responsible for the management of the 30 projects;
• taking account of industry contractor comments provided to the ANAO and the DMO on draft PDSS information;
• assessing the assurance by the DMO managers attesting to the accuracy and completeness of the PDSSs;
• examination of the representations by the Chief Finance Officer (CFO) DMO supporting the project financial assurance and contingency statements, and the independent third-party review of the project financial assurance statements;
• examination of confirmations, provided by the Secretary of the Department of Defence and Chief of the Defence Force, from the Capability Managers, relating to each project’s progress toward Initial Materiel Release (IMR) and Final Materiel Release (FMR), and Initial Operational Capability (IOC) and Final Operational Capability (FOC); and
• examination of the ‘Statement by the Chief Executive Officer (CEO) DMO’, including significant events occurring post 30 June, and management representations by the CEO DMO.

1.12 The ANAO’s processes and procedures to provide independent assurance over the PDSSs also focused on reviewing the DMO’s project management and reporting arrangements, and the number and nature of processes in place that contribute to the overall governance of Major Projects within the DMO and the broader Australian Defence organisation. These included:

• the financial framework, particularly as it applies to the project financial assurance and contingency statements and managing project budgets in an out-turned budget environment, (Section 2 of the PDSSs);
• schedule management and test and evaluation processes, (Section 3 of the PDSSs);
• the capability assessment framework, as it relates to the DMO’s evaluation of the likelihood of delivering key capabilities, (Section 5 of the PDSSs);
• ongoing review of the implementation of the enterprise risk management framework and major risk and issue data, (Section 6 of the PDSSs);
• the project maturity framework and reporting and the systems in place to support the provision of this data, (Section 7 of the PDSSs); and
• developments in the areas of acquisition governance including Gate Review Boards, Projects of Concern, Early Indicators and Warnings, Joint Project Directives, business systems rationalisation, and project management and skills development within the Commonwealth and industry, (Chapter 3 in Part 1).

1.13 This review informed the ANAO’s understanding of the systems and processes supporting the PDSSs for the 2013–14 review period, and highlighted issues in those systems and processes that could be beneficially addressed by the DMO in the longer term.

Results of the review

1.14 The following sections outline the results of the ANAO’s review, and which contribute to the overall conclusion in the Auditor-General’s Independent Review Report for 2013–14.

Financial framework

1.15 In 2012–13, the ANAO reviewed the financial framework as it applied to managing project budgets, in an out-turned budget environment, and the project financial assurance statements. The review indicated that all project offices expected to deliver all required capabilities within the allocated budget.

1.16 However, a number of project offices added additional disclosures to their PDSSs, and in particular, AWD Ships, LHD Ships and ANZAC ASMD Phase 2B recognised that available funding for price indexation was a key concern. Prior to 1 July 2010 projects were periodically supplemented for price indexation, whereas the allocation for price indexation is now provided for on an out-turned basis at Second Pass Approval.³⁵ This change in supplementation policy has meant that price indexation has emerged as a risk for some projects, which would generally emerge later in a project’s life cycle.

1.17 As discussed in the 2012–13 MPR, the emergence of indexation risk has, to some extent, changed the nature and use of the contingency budget from dealing only with project risk management to including broader price management. This requires project office finance staff to have a greater understanding of the factors that influence indices and their likely movement over the life of the project.

1.18 A project’s total approved budget comprises:

• the programmed budget, which covers the project’s approved activities; and
• the contingency budget, which is established to provide adequate budget to cover the inherent cost, schedule and technical risks.³⁶

1.19 The DMO’s management of this financial risk is based on a portfolio management approach within the responsibilities of the CFO (refer to paragraph 1.12 in Part 2 of the 2011–12 MPR, where this is explained further).

1.20 In conjunction with the financial assurance statement, introduced in the 2011–12 MPR, the contingency statements were introduced for the first time in this, the 2013–14 MPR. Together, they provide greater transparency of projects’ financial status, following the move to out-turned budgeting in 2010, and highlight the use of contingency funding to mitigate projects risks.

1.21 In 2013–14, the ANAO reviewed the financial framework as it applies to managing project budgets, including contingency, in an out-turned budget environment, and the project financial assurance and contingency statements.

Project financial assurance statement

1.22 The project financial assurance statement was added to the PDSSs in the 2011–12 MPR, to provide readers with a clear articulation of a project’s financial position and to provide transparency in regard to whether there is ‘sufficient remaining budget for the project to be completed.’³⁷

1.23 The 2012–13 project financial assurance statements indicated that all project offices expected to complete within budget. However, AWD Ships, LHD Ships and ANZAC ASMD Phase 2B recognised that available funding for price indexation was a key concern. In the case of AWD Ships, the 2012–13 Statement by the CEO DMO also noted emerging concerns around cost overruns and associated delays in shipbuilding aspects of the AWD Program.³⁸

1.24 In 2013–14, while all projects again continued to operate within their total approved budget, the AWD Ships³⁹, LHD Ships and ANZAC ASMD 2B project offices continued to recognise that available funding may be insufficient as contracted indices escalation may be greater than the approved project budget. In relation to the AWD Ships project, the 2013–14 Statement by the CEO DMO, continues to note concerns in relation to the adequacy of the total project budget, which will be dependent on the results of the AWD Reform Program.⁴⁰

1.25 In addition, during 2013–14, the DMO continued to support the project financial assurance statements with an independent third-party review, considering factors including: remaining budget, Projects of Concern listing, complexity, diversity across divisions and past history.

1.26 Projects selected for third-party review in support of the financial assurance statement assurance process included:

• detailed review—Overlander Medium/Heavy, MH-60R Seahawk, Growler and Additional Chinook; and
• standard review—Joint Strike Fighter, AWD Ships, MRH90 Helicopters, LHD Ships, Battlefield Airlifter and Next Gen Satellite.

1.27 Observations from the review included that both the AWD Ships and LHD Ships projects have significant contractual exposure to indexation factors and that both project offices have recognised and costed a risk in relation to this matter.

1.28 In conclusion, while for the 2013–14 MPR, the CFO’s representation letter to the CEO DMO on the project financial assurance statements was unqualified, the project financial assurance statement is restricted to the current financial contractual obligations of the DMO for these projects including the result of settlement actions and the receipt of any liquidated damages; and current known risks and estimated future expenditure as at 30 June 2014.

1.29 In contrast, for each of the projects discussed in paragraph 1.24, above, the project offices are acknowledging their continuing concerns in relation to price indexation and allocations provided at Second Pass Approval. The ANAO will continue to assess the outcomes of the financial assurance statements in future MPRs.

Contingency statements

1.30 As noted above, the 2013–14 Guidelines introduced the requirement for a ‘contingency statement’ within each PDSS. PDSSs are now required to include a statement as to whether contingency funds have been applied during the year, as well as disclosing the risks mitigated by the application of those contingency funds. The six projects which had contingency funds applied in 2013–14 were Joint Strike Fighter (increased costs of Stage 1 aircraft), AWD Ships (budget and skill/knowledge risks), MRH90 Helicopters (technical and integration risks), LHD Ships (IT standard operating environment risks), FFG Upgrade (workforce resource risks) and Additional Chinook (workforce and facilities risks).

1.31 The examination of the contingency statements also highlighted that:

• where projects had contingency funds applied, the purpose was within the approved scope of the project;
• the clarity of the relationship between contingency application and identified risks varied. Of the 29 projects that have a formal contingency allocation⁴¹, 19 did not explicitly align their contingency log, with their risk log; and
• the method for applying contingency varied, with only seven project offices using the ‘expected costs’ of the risk treatment (as required by the DMO Project Risk Management Manual (PRMM) version 2.4) and the remaining 22 using either a proportionate allocation of the likelihood of the risk eventuating (the method outlined in PRMM version 2.3), or having no application of contingency against risk.⁴²

1.32 Finally, although the ANAO found that all projects offices tracked their contingency budget in some form, the methods of recording the balance of contingency budgets and application of contingency funds differed between projects. For example, project offices varied in whether they maintained a record of reviews of their contingency log, and its adequacy, or included risk identifiers and descriptors for allocations of their contingency budget. All of which are requirements outlined in the PRMM version 2.4.

Project maturity framework

1.33 Project maturity assessments have been a feature within the MPR since its inception in 2007–08. At that time the DMO reflected that they were introduced within the DMO as Project Risks Scores in 2004, and later renamed Project Maturity Scores in 2005.

1.34 The DMO Project Management Manual 2012, defines a maturity score as:

The quantification, in a simple and communicable manner, of the relative maturity of capital investment projects as they progress through the capability development and acquisition life cycle.⁴³

1.35 While the DMO has raised some doubts about the effectiveness of the current framework, the DMO has agreed to retain maturity scores following the recent JCPAA recommendation.⁴⁴ The Committee viewed the retention of maturity scores as important in relation to providing a measure of capability delivered for each project, until a measure equal to or better than current arrangements is available.

1.36 While the ANAO has previously raised inconsistency in the application of project maturity scores as an issue, during 2013–14, the ANAO noted that project offices were more consistently assigning maturity scores than in previous years. However, while some subjectivity necessarily remains, in the context of a framework that relies upon the application of professional judgement across a diverse range of projects (from Military-Off-The-Shelf (MOTS) to developmental), project life cycles and project managers, with the detailed guidance available to project offices, it is a repeatable process for external review or audit.

1.37 The ANAO has also previously noted that while the 2012 Defence Capability Plan (DCP) recognises different benchmarks for Off-The-Shelf and developmental projects at First Pass and Second Pass Approval, the DMO’s project maturity framework does not.⁴⁵ The benchmark for all seven attributes is the same for all projects regardless of acquisition type.

1.38 The DMO’s current guidance⁴⁶ provides a breakdown of the project life cycle gates, what they represent, and the applicable benchmark maturity score.

1.39 Maturity scores are a composite indicator, constructed through the assessment and summation of seven different attributes which cumulatively form a project ‘maturity score’. The attributes are: Schedule, Cost, Requirement, Technical Understanding, Technical Difficulty, Commercial, and Operations and Support, which are assessed on a scale of one to ten.

1.40 Comparing the maturity score against its expected life cycle gate benchmark provides internal and external stakeholders with an indication of a project’s progress. This may trigger further management attention or provide confidence that progress, against a predetermined benchmark, is satisfactory. Refer to Table 6, below.

Table 6: Capability Definition and Acquisition Life Cycle Gates

Source: DMSP (PROJ) 11-0-007, Project Maturity Scores at Life Cycle Gates, September 2010.

Note: Shaded rows are CDG Responsibility; Unshaded rows are DMO Responsibility.

Note: Where multiple elements of a mission system are involved (e.g. 3 surface combatants) this date represents Initial Operational Capability (IOC) of the initial Subset, including its associated operational support, that is, when the Initial Operational Capability is achieved. (DI(G) OPS 45-2 refers).

1.41 The guidance underpinning the attribution of maturity scores would benefit from a review for internal consistency and relationship to the Australian Defence organisation’s contemporary business. For example, allocating approximately 50 per cent of the maturity score at Second Pass Approval, despite acquisition type, is often inconsistent with the proportion of project funds expended, and the remaining work required in order to deliver the project.

1.42 Further, the existing project maturity score model does not always effectively reflect a project’s progress during the often protracted build phase, particularly for developmental projects. During this phase it can be expected that maximum expenditure will occur, and risks realised, some of which will only emerge as test and evaluation activities are pursued through to acceptance into operational service.

1.43 Finally, while the DMO guidance underpinning maturity scores was due for review in September 2012⁴⁷, this review has not yet been finalised. The ANAO will continue to review the framework and attribution of maturity scores in subsequent MPRs.

Enterprise risk management framework

1.44 In the 2012–13 MPR, the ANAO’s review concluded that while the DMO continued to work toward improving the standard of risk management arrangements applying to Major Projects, the inherently uncertain nature of risks and issues meant that PDSS data could not be considered complete because of unknown risks and issues that may emerge in the future. For this reason, under arrangements for the priority assurance review, major risks and issues data in the PDSSs continues to remain out of scope of the Auditor-General’s formal review conclusion, but is included in the analysis to provide an overall perspective of how risks and issues are managed.

1.45 In 2013–14 the ANAO again reviewed the developments with risk management at an enterprise and project level, in order to update its understanding of the DMO’s risk management systems and processes.

1.46 The development of the DMO’s enterprise risk management framework (ERMF), was identified in 2008–09, by the ANAO as a challenging but necessary step for the DMO in achieving its goal of improving project management. This was consistent with advice from the DMO that it would take some time before reliance could be placed on the framework.⁴⁸ The ANAO highlighted particular challenges, such as the gap between project office risk management practices and those preferred practices, as set out in the ERMF.⁴⁹

1.47 Risk management was also a major focus within the sustainment function of the DMO’s business in the Plan to Reform Support Ship Repair and Management Practices (Rizzo Report). The Rizzo Report stated that ‘Navy and DMO need to improve coordination and integrate their interdependent activities more effectively’⁵⁰ and as a result, recommended that Navy and the DMO establish an Integrated Risk Management System.⁵¹ In response, the Interdependent Mission Management System (IMMS) was developed to provide joint visibility of risks at the enterprise level between the DMO and with Defence Capability Managers, and introduce greater accountability in relation to risk management. In May 2014, the Navy Reform Board endorsed the use of IMMS to manage interdependent risk in Navy more comprehensively.⁵²

1.48 The DMO advised that in the maritime sustainment domain, risk registers have been developed at the division, branch and business unit levels, to reflect the different responsibilities for risk management at each level and show both the internal DMO risks and the interdependent risks with the Capability Manager. These registers are then uploaded via the Predict! risk management system to IMMS, to provide visibility of both internal and interdependent risks.

1.49 A system module relevant to the acquisition function of the business, has been integrated into IMMS, and which will eventually provide for a focus on acquisition activities, as outlined in the Materiel Acquisition Agreements (MAAs).⁵³ The DMO has also advised that it is intended that IMMS will also be used to integrate information captured in the DMO Assurance Management Information System (DAMIS), which is expected to collate, manage and report on all DMO assurance activities.

1.50 The ANAO appreciates that these developments at the corporate level are at a formative stage and will monitor progress further in 2014–15.

1.51 In 2012–13, the ANAO reported inconsistencies in risk management processes applied by project offices. At a project level, risk management guidance is provided by the DMO’s PRMM version 2.4, which was updated during the review period, and discussed earlier in this chapter in the financial framework section.

1.52 In 2013–14, the ANAO examined projects’ risk and issue logs, which are created and maintained utilising the mandated Excel or Predict! software.⁵⁴ For the majority of project offices, risks and issues logs were maintained appropriately, however, they were often only updated infrequently and not in line with the requirement of PRMM.⁵⁵ For example, the ANAO noted that some project offices only conduct their risk and issue reviews prior to scheduled Gate Reviews or ANAO site visits in relation to the MPR.

1.53 Other issues the ANAO observed included:

• risk and issue logs were incomplete or an inaccurate representation of project risks and issues;
• risks descriptors within Predict! were inconsistent with PRMM guidance⁵⁶;
• incorrect application of DMO risk matrix calculations, resulting in inappropriate risk score outcomes; and
• where both Excel and Predict! are used by projects concurrently, inconsistencies existed between risk and issues logs.

1.54 In 2012–13 the ANAO also reported that project offices were no longer able to report their major risks and issues in the Monthly Reporting System, following a corporate review to reduce the reporting burden on project offices.⁵⁷ Further, a risk health assessment conducted in September 2013 revealed that the DMO Executive does not receive systematic reporting on risks, nor is there a consistent approach for rolling up or aggregating disparate risks across business units.⁵⁸

1.55 To achieve greater consistency in the approach to risk management and in response to the release of a Commonwealth Risk Management Policy on 1 July 2014, the DMO is developing a single Risk Management Manual, which is expected to be finalised in 2015.⁵⁹

1.56 The ANAO will continue to review the DMO’s progress on risk management and reporting across the Major Projects in 2014–15.

Capability assessment framework

1.57 The DMO’s evaluation of the probability of delivering key capabilities, as denoted by Materiel Release Milestones (MRMs) and Completion Criteria in MAAs, specify the key elements required for the achievement of materiel release to the Australian Defence Force, and are set out as a ‘traffic light’ pie chart in Section 5.1 of each project’s PDSS. These measures of materiel capability delivery performance primarily focus on the anticipated future attainment of particular technical, regulatory and operational requirements.

1.58 During prior reviews, the ANAO found that the measures of capability delivery performance recorded in the Monthly Reporting System (MRS) did not always align with MAAs, and were inconsistent in terms of the degree of detail represented by each measure with reference to platform value and complexity. For example, the AWD Ships project ($7.9 billion) has four MRMs and Completion Criteria (IMR—Ship 1, MR 2—Ship 2, MR 3—Ship 3 and FMR), whereas the 155mm Howitzer project ($0.3 billion) has 23 MRMs and Completion Criteria.

1.59 In addition, MRMs and Completion Criteria often include schedule and cost factors which need to be excised from the PDSS in order to present data solely focussed on materiel capability delivery performance. The ANAO continued to observe these issues in 2013–14, and noted that there can be lengthy delays in receiving advice in relation to achievement of key milestones such as IOC and FOC, given the complexity of some projects.

1.60 Further, in relation to projects’ progression through each of the key milestones, IMR, IOC, FMR and FOC, the Capability Managers are required to review projects’ achievements against the MRMs and Completion Criteria.⁶⁰ This underpins the relationship between the DMO and the Capability Managers in relation to the acquisition and delivery of capability, and is encapsulated within the specific details of the MAAs.

1.61 During 2013–14, the ANAO also observed that in relation to the ARH Tiger Helicopters project, the Capability Manager had co-signed the acceptance of achievement of FMR, albeit with significant caveats.⁶¹ The significance of the Capability Manager’s caveats has been reflected as a high risk to achievement of project completion and FOC, within the acceptance documentation.

1.62 However, the capability assessment provided within the PDSS indicates an expectation by the DMO of 100 per cent delivery for this project despite the risks as assessed by the Capability Manager.

1.63 Inevitably, the assessment of future capability delivery typically involves making certain assumptions in predicting achievements and consequently involves some subjectivity in approach. Taking into consideration this subjectivity and the inherent uncertainty of future events, this information continues to be excluded from the scope of the ANAO’s review. However the ANAO has included the DMO’s capability forecasts in addition to our analysis of projects’ performance in relation to budgeted cost and schedule in Chapter 2 of Part 1.

1.64 Finally, the ANAO’s analysis of capability performance is further detailed in Chapter 2, for all projects in the MPR.

Review conclusion

1.65 The Auditor-General’s Independent Review Report takes into account the overall governance of major projects, the results of our examination of the DMO’s project management and reporting arrangements, and the results of our substantive procedures to gain assurance in relation to key information reported in PDSSs. In 2013–14, the results of the ANAO’s priority assurance review of the 30 PDSSs was that nothing has come to the attention of the ANAO that causes us to believe that the information and data in the PDSSs, within the scope of our review, has not been prepared, in all material respects, in accordance with the Guidelines.

Introduction

2.1 Project performance information is important in the management and delivery of major Defence equipment acquisition projects (Major Projects). Such information provides an important focus for management attention to inform decisions to be made about the allocation of resources, to support advice to government on project progress and performance, and to allow for the Parliament and the public to assess the progress of the Defence Materiel Organisation (DMO) in discharging its responsibilities.

2.2 The Australian National Audit Office (ANAO) has derived three key indicators to analyse the three major dimensions of a project’s progress and performance, utilising data extracted from the Project Data Summary Sheets (PDSSs), and to provide a series of performance information snapshots. These indicators are the:

• percentage of budget expended (Budget Expended)—which measures the total expenditure as a percentage of the total current budget;
• percentage of time elapsed (Time Elapsed)—which measures the percentage of time elapsed from original approval to the forecast Final Operational Capability (FOC)^{62 63}; and
• percentage of key materiel capabilities expected to be delivered (Expected Capability)—which is the DMO’s assessment of the likelihood of delivering the required level of capability.

2.3 These indicators are measured in percentage terms, to enable comparisons between projects of different characteristics, and to provide a portfolio view across project progress and performance.

2.4 As in previous Major Projects Reports (MPRs), the ANAO has included an analysis of the key indicators against the DMO’s assessment of project maturity, at defined milestones⁶⁴, as a percentage of the predefined maximum maturity score of 70 (Project Maturity).

2.5 As explained in the previous chapter, the DMO’s project maturity framework is an assessment methodology used for quantifying, in a practical and communicable manner, project maturity as projects progress through the capability definition and acquisition life cycle. Project maturity comprises a matrix of seven attributes: Schedule, Cost, Requirement, Technical Understanding, Technical Difficulty, Commercial, and Operations and Support, which are each assessed on a scale of one to 10.⁶⁵ Project maturity is a composite performance indicator available to the DMO, Defence Executive and government for decision making, and to assess the overall status of Major Projects.⁶⁶

2.6 However, the DMO has advised the Joint Committee of Public Accounts and Audit (JCPAA) that although project maturity scores are a helpful tool, they ‘…are ultimately indicative and advisory’⁶⁷, and that a Materiel Implementation Risk Assessment (MIRA) which ‘…covers similar matters to the maturity scores but provides a narrative description of the risks and their impacts’ is summarised in the Government Approval Submissions instead.⁶⁸

2.7 Nevertheless, the JCPAA has requested that the project maturity scores be maintained for the MPR until they are no longer required by the Guidelines endorsed by the JCPAA.⁶⁹ The DMO has agreed to this request.

2.8 The first snapshot, Figure 2, at page 45, provides an overview of the three major dimensions of project performance, and sets out Budget Expended, Time Elapsed⁷⁰ and Expected Capability.⁷¹ The figure shows that, for most projects (24 of 30), the Budget Expended is broadly in line with, or lagging behind, the Time Elapsed.⁷² This relationship is generally expected in an acquisition environment predominantly based on milestone payments. However, Budget Expended being greater than Time Elapsed by a significant amount provides management with an indication that the project budget may be at risk or that procurement activities have been undertaken in advance of installation activities.

2.9 Projects where Budget Expended leads Time Elapsed (from government approval to FOC) by more than 10 per cent, include:

• AWD Ships—contracted indexation rates have been higher than the out-turning applied in the project budget and productivity costs have been greater than expected (12 per cent variance);
• LHD Ships—contracted indexation rates have been higher than the out-turning applied in the project budget, and materiel and equipment has been acquired in advance of installation (11 per cent variance);
• Overlander Light—the majority of equipment has been delivered to the DMO, but is awaiting rollout to the Australian Defence Force (ADF), which requires time to complete (12 per cent variance);
• Collins Replacement Combat System and Reliability and Sustainability projects—most of the materiel has been acquired and expenditure undertaken. In addition, originally planned installation dates have been extended, based on submarine availability, reducing the proportion of total Time Elapsed (25 per cent for the Replacement Combat System project and 21 per cent for the Reliability and Sustainability project); and
• UHF SATCOM—significant milestones have been achieved (IS22 Satellite launch and all 20 channels delivered), however FOC (Pacific Ocean) is not forecast to be achieved until June 2018, when the Chief Information Officer Group will be in a position to acquire agreed UHF capacity from the United States (US) as the US capacity builds up in the region (24 per cent variance).

2.10 Where Time Elapsed leads Budget Expended by a significant amount, this may indicate the project has encountered unexpected delays. Projects where the Time Elapsed is approximately 20 per cent greater than the Budget Expended, include:

• Joint Strike Fighter—large scope increase ($10.5 billion) for the purchase of additional aircraft was approved in April 2014 with the project yet to enter contract (30 per cent variance);
• Next Gen Satellite—the project has achieved the delivery of the required capability up to Final Materiel Release (FMR) with significant funds remaining, which is expected to be returned to the Defence budget on closure of the project (28 per cent variance);
• Additional Chinook—delivery and acceptance of helicopters has been delayed, although FMR and FOC are still scheduled to be achieved as planned (23 per cent variance);
• 155mm Howitzer—delivery of the Course Correcting Fuze capability has been delayed. This is currently in the process of being transferred to project LAND 17 Phase 1C.1 (51 per cent variance);
• Battle Comm. Sys.—meeting design requirements remains a challenge as the project is exposed to multiple platform design authorities, and delays in final delivery of some equipment has contributed to ongoing slippage (21 per cent variance); and
• LHD Landing Craft—expenditure for delivery and acceptance of the majority of equipment is scheduled for the next 12 months with FMR and FOC currently scheduled for September 2015 (30 per cent variance).

2.11 The DMO expects that all 30 projects in this year’s MPR will deliver all of their key capability requirements, recognising that some elements of the capability required for some projects may be under threat, but are considered manageable (assessed as either green or amber). This is consistent with the DMO’s 2012–13 assessment, and an improvement on the 2011–12 assessment, where both Wedgetail and MRH90 Helicopters were assessed as unlikely to meet all of their capability requirements.

Figure 2: Project snapshot—Budget Expended, Time Elapsed and Expected Capability

 

 

Source: The ANAO’s analysis of the 2013–14 PDSSs. Refer to paragraphs 2.8 to 2.11 in Part 1, at pages 42 to 44, of this report.

Note: The Expected Capability for Wedgetail has been assessed against the Supplies section of the Materiel Acquisition Agreement, which lists the equipment to be delivered.

Cost performance analysis

Project snapshot—Budget Expended and Project Maturity

2.12 The second snapshot, Figure 3, below, sets out each project’s Budget Expended against Project Maturity.⁷³ As noted previously, Figure 2 shows that, for most projects (24 of 30), the Budget Expended is broadly in line with, or lagging behind, the Time Elapsed, which is expected in an acquisition environment predominantly based on milestone payments. Figure 3 shows that Project Maturity leads Budget Expended for the majority of the projects (23 of 30), and that the variance between expenditure and maturity score progress can be significant.

2.13 Unlike the relationship between Budget Expended and Time Elapsed, and consistent with prior years, there are no major exceptions to Project Maturity leading Budget Expended (greater than 20 per cent). Projects where Project Maturity leads Budget Expended with an approximate differential of 20 per cent or more at 30 June 2014 include: Joint Strike Fighter (58 per cent); Overlander Medium/Heavy (65 per cent); MH60R Seahawk (47 per cent); Growler (52 per cent); Battlefield Airlifter (33 per cent)⁷⁴; Overlander Light (23 per cent); Next Gen Satellite (34 per cent); Additional Chinook (34 per cent); Battle Comm. Sys. (Land) (26 per cent); 155mm Howitzer (46 per cent); and LHD Landing Craft (45 per cent). All of which are either Military-Off-The-Shelf (MOTS) or Australianised MOTS, except Joint Strike Fighter, which is expected to be MOTS by acquisition.

2.14 In part, these variances are the result of the DMO’s project maturity framework attributing approximately 50 per cent of the total project maturity at Second Pass Approval. This reduces the value of project maturity assessments within the acquisition phase, which is predominantly the longest and most expensive component of Major Project acquisition.

Figure 3: Project snapshot—Budget Expended and Project Maturity

 

 

Source: The ANAO’s analysis of the 2013–14 PDSSs. Analysis for the 2013–14 MPR continues to highlight inconsistencies within the application of project maturity, reducing the level of reliability of maturity assessments for key decision makers and other stakeholders; however, improved focus and review by project offices was noted by the ANAO during 2013–14 fieldwork. Refer to paragraphs 2.12 to 2.14 in Part 1, at page 46, of this report.

Note: ANZAC ASMD 2B’s Project Maturity is based on the progress of the lead ship, not on the current eight ship program.

Second Pass Approval and 30 June 2014 approved budget

2.15 For the 30 Major Projects, Figure 4, below, compares each project’s approved budget at Second Pass Approval (the main investment decision by government) and their approved budget at 30 June 2014.

2.16 The total approved budget for the 30 projects at 30 June 2014 was $59.4 billion, a net increase of $16.8 billion, when compared to their Second Pass Approval approved budget of $42.6 billion (the analysis of this variance is included in Figure 5, at page 52).⁷⁵

2.17 Figure 4 indicates relative budget variations from Second Pass Approval of 150 per cent or greater for the following projects:

• Joint Strike Fighter—scope increase of $10.7 billion, mainly for 58 additional aircraft in 2013–14 (total increase of 389.0 per cent);
• MRH90 Helicopters—scope increase of $2.8 billion, mainly for 34 additional aircraft in 2005–06 (total increase of 295.4 per cent);
• Bushmaster Vehicles—scope increase of $955.4 million, mainly for 715 additional vehicles in 2007–08 (437 vehicles), 2011–12 (70 vehicles) and 2012–13 (208 vehicles) (total increase of 323.9 per cent);
• ANZAC ASMD 2B—scope increase of $429.6 million, including $214.7 million for ships 2–8 in 2011–12 (total increase of 172.7 per cent); and
• Collins R&S—increase of $339.7 million, including $302.8 million for additional scope in 2001–02 (total increase of 471.8 per cent).

2.18 Overlander Vehicles, initially provided with Second Pass Approval for a broad suite of platforms in August 2007, was separated into two phases, Phase 3A Lightweight and Light Capability and Phase 3B Medium and Heavy Capability, in December 2011. Phase 3A was reapproved for Second Pass at that time however Phase 3B was not reapproved for Second Pass until July 2013.⁷⁶ As a result of this, the 2012–13 MPR reflected a scope decrease of $2.2 billion. This timing difference has now been readjusted with the inclusion of Phase 3B in the 2013–14 MPR, with an approved budget of $3.5 billion.

Figure 4: Projects’ Second Pass Approval and 30 June 2014 approved budget ($m)

 

 

Source: The ANAO’s analysis of the 2013–14 PDSSs. Refer to paragraphs 2.15 to 2.18 in Part 1, at page 48, of this report.

Note: [vertical white-filled rectangle] indicates that the budget for the project at 30 June 2014 is less than the original budgeted cost. However, for Overlander Light this reflects a transfer of $2.2 billion to Overlander Medium/Heavy on separation of the original project into two phases in December 2011.

Budget performance

2.19 The subsequent figures and tables illustrate the following for the 30 projects in the 2013–14 MPR:

• the three main factors contributing to budget variations in each of the last six years;
• the in-year impact of variations;
• the 2013–14 in-year budget variations; and
• the expenditure forecasting performance against the actual expenditure for the 2013–14 financial year.

Longitudinal and in-year budget variance analysis

2.20 As noted in paragraph 1.16, at pages 28 to 29, from 1 July 2010 price indexation is now provided as part of a project’s budget at Second Pass Approval. This is defined as the budget being ‘out-turned’.⁷⁷ However, for the following analysis, approved budget variations are disaggregated into three main factors: price indexation (material and labour), exchange rate and real variation. The 2013–14 budget variation of $16.8 billion for the 30 Major Projects, when compared to their Second Pass Approval budget, comprises net price variation increases of $6.2 billion, foreign exchange rate movement decreases of $1.4 billion and real variation increases of $12.0 billion.

2.21 Figure 5 examines the three main factors contributing to budget variations in each of the last six years, and highlights the in-year impact of variations for the 30 projects in the 2013–14 MPR.

2.22 Exchange rate variations in project budgets are a result of projects’ exposure to foreign currencies and movement in foreign exchange rates against the Australian dollar.⁷⁸ The exposure of specific projects to currency exchange variations is established through the initial government procurement decision and contractual agreement. The US dollar and the Euro are the main influences, although other currencies also have an impact. This year the Australian dollar varied significantly across the year but by the end of the financial year had stabilised to a similar level to where it began, against the US dollar and Euro.

2.23 The larger than normal in-year exchange variation of $1.7 billion is primarily the result of adjustments in August 2013 and February 2014 totalling $1.2 billion and $0.4 billion respectively. These adjustments accounted for foreign exchange movements from May to August 2013 and from November 2013 to February 2014 which were both periods of pronounced decline in the value of the Australian dollar.

2.24 Real variations in project budgets primarily reflect changes in the scope of projects, transfers between projects for approved equipment/capability, and budgetary adjustments such as administrative savings decisions. The comparatively larger in-year variation this year reflects a $10.5 billion increase in the total approved budget for the Joint Strike Fighter project for additional aircraft as explained in paragraph 2.17 (at page 48) and $0.7 billion increase for the Overlander Medium/Heavy project for general program supplementation and scope transfers from other projects.

Figure 5: Longitudinal budget variations for 2008–09 to 2013–14 ($m)

 

 

Source: The ANAO’s analysis of the 2013–14 PDSSs and Project Budget Approval Histories. Refer to paragraphs 2.20 to 2.24 in Part 1, at pages 50 to 51, of this report.

2.25 Table 7, at page 54, sets out the 2013–14 in-year budget variations for each of the 30 Major Projects. Overall, the 30 June 2014 approved budget for the 30 projects in the 2013–14 MPR increased by $12 775.4 million in 2013–14 or 32.0 per cent, compared to their 30 June 2013 approved budget. This was driven by net real increases of $11 042.9 million (mostly for scope increases), and exchange variation increases of $1 732.5 million.

2.26 In 2013–14, of the real variations affecting projects’ budgets, the most significant change was due to the Joint Strike Fighter scope variation. Projects with larger real cost variations include:

• Joint Strike Fighter—large scope increase of $10 515.4 million for additional aircraft;
• AWD Ships—real cost decrease of $109.9 million for transfer of funding for facilities construction to the Defence Support and Reform Group; and
• Overlander Medium/Heavy—supplementation of $712.2 million and scope transfers from other projects at Second Pass Approval.

2.27 Additionally, as noted in paragraph 2.18 (at page 48), Overlander Vehicles was separated into two phases in December 2011. Phase 3A was reapproved for Second Pass at that time however Phase 3B was not reapproved for Second Pass until July 2013. As a result of this the 2012–13 MPR reflected a scope decrease of $2.2 billion. This timing difference has now been readjusted with the inclusion of Phase 3B in the 2013–14 MPR, with an approved budget of $3.5 billion.

2.28 Projects with larger movements in foreign exchange, as a result of the depreciating Australian dollar, include:

• Joint Strike Fighter—$347.7 million, or 13.4 per cent increase in budget;
• MRH90 Helicopters—$135.2 million, or 3.7 per cent increase in budget;
• Overlander Medium/Heavy—$170.9 million, or 4.9 per cent increase in budget;
• MH-60R Seahawk—$277.8 million, or 9.4 per cent increase in budget;
• Growler—$368.5 million, or 12.1 per cent increase in budget; and
• Battlefield Airlifter—$107.8 million, or 8.4 per cent increase in budget.

Table 7: In-year (2013–14) budget variations by project

Sources: The ANAO’s analysis of the 2012–13 and 2013–14 PDSSs. Refer to paragraphs 2.25 to 2.28 in Part 1, at pages 52 to 53, of this report.

In-year forecast and actual expenditure

2.29 Accurately forecasting expenditure is an important element in the effective management of a portfolio of projects and understandably receives close attention within the Australian Defence organisation and by key stakeholders, including the JCPAA. Figure 6, at page 57, sets out the expenditure forecasting performance of each project against the actual expenditure for the 2013–14 financial year. In total, the actual expenditure for the 30 projects at 30 June 2014 was $3.3 billion, against an initial forecast expenditure of $2.9 billion and revised forecast of $3.3 billion. The main factors contributing to the initial variance were changes to delivery schedules, scope changes and foreign exchange fluctuations.

2.30 As part of its review of the 2012–13 MPR, the JCPAA recommended that expanded information regarding project’s budget estimates and actual expenditure during the financial year be published, including comparisons of variations, the percentage of variance and overall totals and averages where calculable. The DMO agreed to provide this data in their section of this, the 2013–14 MPR and can be found at Table 2.2 in Part 2. In addition, the JCPAA recommended that the ANAO perform analysis on this data.⁷⁹ This table presents the Portfolio Budget Statements (PBS) estimate; the Portfolio Additional Estimates Statements (PAES) budget; the final planned budget for the financial year; actual expenditure; the variance between the final planned budget and actual expenditure, in both amount and per cent terms; and totals for this data across the 30 Major Projects.

2.31 The DMO commentary on the variances identified at Table 2.2 and paragraph 2.10 in Part 2, highlights that the main contributors to these variances are changes in schedule, cost savings, and delays in suppliers billing the DMO for goods. This is consistent with the explanations provided in the individual project PDSSs in Part 3 of this report.

2.32 The JCPAA also recommended that Section 2.2 of the PDSSs be amended to report each Major Project’s budget estimates, as published in the PBS; mid-year estimates, as published in the PAES; any subsequent estimates since the mid-year estimates; actual expenditure for the financial year and explanations of variances between each of the above.⁸⁰ These changes have been incorporated into the 2014–15 MPR Guidelines and will assist in providing greater clarity in budget to actual variance analysis in future years.

2.33 The ANAO’s analysis at Figure 6 highlights that when compared to the initial forecast expenditure, notable in-year underspends occurred in the Wedgetail ($25.7 million), Air to Air Refuel ($29.7 million), and Battle Comm. Sys. (Land) ($29.9 million) projects.

2.34 Compared to the initial budget allocation, notable increases from original in-year budget occurred in the AWD Ships ($34.3 million), MRH90 Helicopters ($46.5 million), MH-60R Seahawk ($42.9 million), LHD Ships ($44.5 million), Growler ($280.3 million) and Additional Chinook ($132.8 million) projects.

2.35 The overall project expenditure in the portfolio of projects was as expected in the revised forecast.

Figure 6: In-year (2013–14) projects’ forecast expenditure performance compared to actual expenditure ($m)

 

 

Sources: The ANAO’s analysis of the 2013–14 PDSSs and Defence Portfolio Budget Statements. Refer to paragraphs 2.29 to 2.35 in Part 1, at pages 55 to 56, of this report.

Schedule performance analysis

2.36 The DMO continues to acknowledge that schedule performance is the key issue in delivering and sustaining equipment for the ADF.⁸¹ Further, in 2013, the Minister for Defence, Senator the Hon. David Johnston confirmed this, stating:

We need to be able to do things much more time effectively, because time is money and this is one of the things we see as the principal causes of problems inside Defence. We want to know when things are beginning to lag, schedule is starting to slip… being forewarned we can give the project appropriate attention.⁸²

2.37 In addition, project slippage could effectively introduce or exacerbate an existing capability gap or require extension to the planned withdrawal date for those platforms being replaced for example, the withdrawal dates for the Sea King and Black Hawk helicopter fleets included consideration of the introduction of replacement capability.

Project snapshot—Time Elapsed and Project Maturity

2.38 The third snapshot, Figure 7, at page 60, sets out each project’s Time Elapsed against the Project Maturity.⁸³ Similar to Figure 3, at page 47, which sets out the relationship between Budget Expended and Project Maturity, Figure 7 shows that Project Maturity leads Time Elapsed for 21 of 30 projects (Figure 3, 23 of 30) with no major exceptions. Projects where Project Maturity leads Time Elapsed with an approximate differential of 20 per cent include: Joint Strike Fighter (28 per cent); Overlander Medium/Heavy (56 per cent); MH-60R Seahawk (52 per cent); Growler (53 per cent); Battlefield Airlifter (27 per cent); Overlander Light (35 per cent); and Battle Comm. Sys. (Land) (35 per cent). Similar to Figure 3 and as noted in paragraph 2.13 (at page 46), all of these projects are either MOTS or Australianised MOTS, except Joint Strike Fighter, which is expected to be MOTS by acquisition.

2.39 The difference between Time Elapsed and Project Maturity for the Joint Strike Fighter project (28 per cent) reflects that a fully defined schedule including an FOC date has been set this year with the approval of the purchase of additional aircraft to acquire the complete capability. A comparison of these two attributes has not previously been possible as the project did not have an FOC date.

2.40 However, the design of the project maturity framework and inconsistencies in its application, reduce the value of Project Maturity as an indicator.⁸⁴ For example, the MRH90 Helicopters project recorded 10 out of 10 for schedule in the 2012–13 MPR, because it had achieved ‘In-Service Date’⁸⁵ (in accordance with one part of the framework). This year it records a score of 8 out of 10 for schedule due to a reassessment against the achievement of Initial Materiel Release (IMR) and FMR (in accordance with other parts of the framework). This example indicates how an overly optimistic assessment of project maturity may result. This also demonstrates the limited ability of the framework to show project maturity progress during the later stages of a project’s life, as a result of adopting 8 out of 10 prior to the achievement of FMR, with the PDSS currently indicating that FMR will not be reached until December 2017.

2.41 As noted in paragraph 2.14 (at page 46), by attributing approximately 50 per cent of the total Project Maturity at Second Pass Approval, the project maturity framework is weighted towards pre Second Pass Approval processes. This reduces the available score for project managers to adequately indicate progress during the build phases. Descriptors used to define project maturity post Second Pass Approval are broader and cover the more complex steps, which generally take longer and where issues are more likely to arise.⁸⁶ For example, in terms of Requirement (one of the seven attributes within the framework), one point is accumulated for endorsing requirements and one point for testing.

Figure 7: Project snapshot—Time Elapsed and Project Maturity

 

 

Source: The ANAO’s analysis of the 2013–14 PDSSs. Analysis for the 2013–14 MPR continues to highlight inconsistencies within the application of project maturity, reducing the level of reliability of maturity assessments for key decision makers and other stakeholders; however, improved focus and review by project offices was noted by the ANAO during 2013–14 fieldwork. Refer to paragraphs 2.38 to 2.41 in Part 1, at pages 58 to 59, of this report.

Note: ANZAC ASMD 2B’s Project Maturity is based on the progress of the lead ship, not on the current eight ship program.

Schedule slippage and acquisition type by approval date

2.42 Examination of the portfolio of projects in the MPR highlights that when dealing with large and complex Major Projects, slippage can occur for a variety of reasons. Projects are delivered through a range of procurement options, and are subject to multiple pressures including: actions by contractors; economic conditions impacting on workforce supply and demand; and procurement decisions by other nations (particularly in Foreign Military Sales cases) which may impact on downstream purchases in terms of time to delivery.

2.43 Providing forecasts on the future schedule performance of Major Projects is difficult, requiring a deep understanding of project technical elements and a realistic assessment of the capacity of the private sector to deliver in the expected timeframe.⁸⁷ Reference to the historical performance of projects may be useful in this context.

2.44 Figure 8, below, illustrates the total schedule slippage since Second Pass Approval for projects in the 2013–14 MPR, demonstrating how this key performance indicator has altered over time. It also depicts the acquisition type and places projects in order of government approval, allowing for more detailed analysis.

2.45 Figure 9, at page 63, illustrates the total schedule slippage for projects which have exited the MPR to allow further analysis. This figure will be updated each year in conjunction with Figure 8 to allow for greater longitudinal analysis of project performance over time.

2.46 It should be noted that a greater focus on MOTS and Australianised MOTS acquisitions is, prima facie, reducing the slippage in the Major Projects profile. However it is not always possible to acquire the necessary capability in this manner, but within the portfolio of Major Projects allows for management focus on a limited number of developmental projects at any one time.

Figure 8: Total slippage post Second Pass Approval and acquisition type by approval date (years)

 

 

Source: The ANAO’s analysis of the PDSSs in published MPRs. Refer to paragraphs 2.44 to 2.50 in Part 1, at pages 61 to 64, of this report.

Note 1: The order of the projects is from latest to earliest approved. All project slippage relates to FOC dates.

Note 2: The MRH90 Helicopters project was originally categorised as a MOTS solution but has since been reassessed as more developmental than envisaged and has been reclassified as Australianised MOTS. This advice was set out in a minute to the CEO Australian Aerospace Ltd from the General Manager Systems DMO, MRH90 Gate Review Outcomes, 15 April 2011.⁸⁸

Note 3: Bushmaster Vehicles has an FOC date for each Production Period (discrete order). The FOC used for this year’s MPR analysis is Production Period Five.

Figure 9: Total slippage post Second Pass Approval and acquisition type by approval date (years) for projects exited the MPR

 

 

Source: The ANAO’s analysis of the PDSSs in published MPRs. Refer to paragraphs 2.45 to 2.46 in Part 1, at page 61, of this report.

Note 1: The order of the projects is from latest to earliest approved. All project slippage relates to FOC dates.

Note 2: Does not include the AIR 5376 Phase 3.2 Hornet Refurb project which exited in 2012 as this project did not introduce a new capability and so did not have an FOC date.

2.47 The data at Figure 8, above, illustrates that older projects, which achieved Second Pass Approval prior to 2005, generally experienced the most slippage. These projects tend to be more developmental (complex) in nature and typically experienced schedule slippage in the past, and have continued to do so. This shows an ongoing trend of slippage in historically late projects including SM-2 Missile, Air to Air Refuel, FFG Upgrade, and Hornet Upgrade.⁸⁹

2.48 This year, three of the four new projects included in the 2013–14 MPR were classified as Australianised MOTS (Overlander Medium/Heavy, Growler and LHD Landing Craft) with the fourth project (Battlefield Airlifter) classified as MOTS. Further information about the classification of the projects can be found in the PDSSs in Part 3, from page 175, of this report.

2.49 The challenge of gaining a full understanding of the complexities of developmental aspects of projects at Second Pass Approval is evident by the extent of slippage over time. The 2008 Audit of the Defence Budget (Pappas Review) identified technical risk as the largest source of post Second Pass Approval schedule slippage for ‘post Kinnaird’ projects⁹⁰, and also observed that schedule slippage causes cost escalation.⁹¹

2.50 In JCPAA Report 442, the Committee flagged its intention to continue to monitor activity in this area, and the ANAO will continue to compile data for further longitudinal analysis in the future.⁹²

Schedule performance

2.51 The subsequent figures and tables illustrate the following for the 30 projects in the 2013–14 MPR:

• the original and 30 June 2014 forecasts for achieving FOC;
• in-year schedule changes to achieving FOC;
• total schedule slippage across the Major Projects; and
• total slippage according to a project’s Second Pass Approval date.

Original and 30 June 2014 Final Operational Capability forecasts

2.52 Figure 10, at page 66, presents information on the projects’ original and 30 June 2014 forecasts for achieving FOC. The total schedule slippage for the 30 Major Projects to date is 1 115 months when compared to the initial prediction when first approved by government. This slippage represents a 36 per cent increase on the expected schedule since the main investment decision.⁹³ Of the 30 projects in the 2013–14 MPR, 19 have experienced schedule slippage.

2.53 The reasons for schedule slippage often include underestimation of the difficulties associated with technical factors such as design problems, industry capacity and capability, difficulties in system integration to achieve the required capability, or emergent work associated with upgrades.⁹⁴ In other cases, a project office’s ability to gain access to the platform (asset) for upgrading, can delay the schedule (for example, the two Collins submarine projects and Hw Torpedo).⁹⁵ Defence’s 2011 Defence Capability Plan Review reported that an estimated 80 per cent of project schedule changes were due to internal factors.⁹⁶

2.54 A closer examination of the reasons for schedule slippage demonstrates the importance of initial assessments of the purchase type, i.e. MOTS, Australianised MOTS or developmental.⁹⁷ Instances of misclassification in this respect have resulted in extended schedule slippage for MRH90 Helicopters⁹⁸ and ARH Tiger Helicopters.⁹⁹ The review of FMR and FOC for MRH90 Helicopters as a result of deed negotiations with the contractor, which was underway last year, has now been finalised and has resulted in 38 and 60 months slippage for these milestones, respectively.

Figure 10: Projects’ original and 30 June 2014 FOC forecasts

 

 

Source: The ANAO’s analysis of the 2013–14 PDSSs. Refer to paragraphs 2.52 to 2.54 in Part 1, at pages 64 to 65, of this report.

Note 1: [vertical white-filled rectangle] indicates that the forecast FOC date for the project at 30 June 2014 is earlier than the original FOC date.

Note 2: Hornet Upgrade FOC date relates to Phase 2.3.

Note 3: Bushmaster Vehicles has an FOC date for each Production Period (discrete order). The FOC used for this year’s MPR analysis is Production Period Five.

In-year schedule performance

2.55 In 2013–14, there was a total schedule slippage of 205 months in the forecast achievement of FOC for the 30 Major Projects. Figure 11, at page 69, shows negative slippage for three projects:

• Joint Strike Fighter—currently expects to achieve FOC in October 2023, two months ahead of the original schedule;
• Overlander Medium/Heavy—currently expects to achieve FOC in January 2023, 11 months ahead of the original schedule; and
• LHD Landing Craft—currently expects to achieve FOC in September 2015, five months ahead of the original schedule.

2.56 In-year schedule slippage involved the following 11 projects¹⁰⁰ (the explanation provided, drawn from the 2013–14 PDSSs, may also include the reasons for prior slippage):

• MRH90 Helicopters—delays resulting from deed negotiations with the contractor and ongoing technical deficiencies;
• Hornet Upgrade—achieved Supplemental Type Certification and Service Release in November 2012, however additional testing was necessary in 2014 with time allowed for finalisation of data analysis prior to Capability Manager sign off;
• Air to Air Refuel—delays resulting from issues around the Aerial Refuelling Boom System and flight testing and maintenance requirements on test aircraft;
• FFG Upgrade—schedule extended to allow for passive sonar (PANORAMA) capability to be included in the project;
• Overlander Light—delays resulting from complexity in finalising the design and manufacture of the Command Post Mobile;
• Collins RCS—slippage resulting from changes to the Full Cycle Docking schedule affecting the installation schedule based on submarine availability;
• Hw Torpedo—slippage resulting from changes to the Full Cycle Docking schedule affecting the installation schedule based on submarine availability;
• SM-2 Missile—delays in receipt of missile spares and arrangement of in-service support requirements;
• 155mm Howitzer—delays resulting from the time taken to approve the transfer of the Course Correcting Fuze capability to project LAND 17 Phase 1C.1;
• Stand Off Weapon—delivery delays due to issues with the reliability of the fuze; and
• Battle Comm Sys.—delays in receipt of final equipment.

Figure 11: In-year (2013–14) schedule changes to achieving FOC

 

 

Source: The ANAO’s analysis of the 2013–14 PDSSs. Refer to paragraphs 2.55 to 2.56 in Part 1, at pages 67 to 68, of this report.

Note: The DMO’s PDSSs indicate that 16 of the 30 MPR projects did not record changes to their FOC dates this year.

Longitudinal schedule performance

2.57 Figure 12, below, shows the schedule slippage of the Major Projects included in the 2007–08 to 2013–14 MPRs (Table 8, at page 71, provides the accompanying details of the projects included in the analysis). The figure shows that the total cumulative schedule slippage for the:

• seven projects in 2007–08 was 26.6 years (319 months) and that the total schedule slippage for these projects in 2013–14 was 48.7 years or 584 months (an increase of 22.1 years or 265 months over six years);
• 11 projects in 2008–09 was 28.8 years (345 months) and that the total schedule slippage for these projects in 2013–14 was 60.2 years or 723 months (an increase of 31.4 years or 378 months over five years);
• 18 projects in the 2009–10 MPR was 55.2 years (663 months) and that the total schedule slippage for these projects in 2013–14 was 87.9 years or 1 055 months (an increase of 32.7 years or 392 months over four years);
• 24 projects included in the 2010–11 MPR was 60.3 years (724 months) and that the total schedule slippage for these projects in 2013–14 was 92.6 years or 1 111 months (an increase of 32.3 years or 387 months over three years);
• 25 projects in 2011–12 was 64.6 years (775 months) and that the total schedule slippage for these projects in 2013–14 was 92.6 years or 1 111 months (an increase of 28.0 years or 336 months over two years); and
• 26 projects in 2012–13 was 75.8 years (910 months) and that the total schedule slippage for these projects in 2013–14 was 92.9 years or 1 115 months (an increase of 17.1 years or 205 months over one year).

2.58 The figure shows that 52.4 per cent (48.6 years or 584 months) of the total schedule slippage across the Major Projects covered in the 2013–14 MPR (92.9 years or 1 115 months) is made up of the schedule slippage from the initial seven projects reported in the 2007–08 MPR.

Figure 12: Longitudinal schedule slippage across years (in years)

 

 

Source: The ANAO’s analysis of the PDSSs in published MPRs. Refer to paragraphs 2.57 to 2.58 in Part 1, at pages 69 to 70, of this report.

Note 1: The total schedule slippage in 2013–14 across the 30 projects is 1 115 months. Overlander Medium/Heavy, Growler, Battlefield Airlifter and LHD Landing Craft, which are new to this year’s MPR, have not experienced slippage against FOC according to the DMO’s PDSS.

Note 2: Bushmaster Vehicles has an FOC date for each Production Period (discrete order). The FOC used for this year’s MPR analysis is Production Period Five.

Table 8: Projects included in Figure 12 analysis by MPR

Source: The ANAO’s analysis of the PDSSs in published MPRs.

2.59 Further disaggregation according to a project’s Second Pass Approval date in Table 9, below, shows that 80 per cent of the total schedule slippage across the Major Projects covered in the 2013–14 MPR is made up of projects approved prior to the DMO’s demerger from the Department of Defence, in July 2005.

Table 9: Project slippage by Project approval

Source: The ANAO’s analysis of the 2013–14 PDSSs. Refer to paragraph 2.59 in Part 1, at page 71, of this report.

Note 1: This figure does not add as new FOC dates were introduced in 2010–11 which were later than the original FOC date. This was not caused by project issues.

Note 2: These figures do not add precisely due to the exclusion of schedule reductions over the life of the project.

Capability performance analysis

2.60 The ADF defines capability as the power to achieve a desired operational effect in a nominated environment, within a specified time, and to sustain that effect for a designated period.¹⁰¹ An operational effect is achieved by combining the eight Fundamental Inputs to Capability: personnel; organisation; collective training; major systems; supplies; facilities and training areas; support; and command and management.¹⁰²

2.61 In acquiring Defence platforms and systems, a range of documentation (including capability definition documents, operational concept documents, function and performance specification, and test concept documents) is developed and sets out the detailed requirements/performance attributes to be achieved. In the case of an aircraft, for example, this would include elements such as its range and speed, handling characteristics, self protection abilities, requirements for runway length and other ground support, civil and military communications and guidance systems, maximum takeoff weights, cargo capacity, maintenance requirements and compliance with military and civil certification regulations and requirements. Depending on the nature of this information, it can be classified for national security reasons and therefore cannot be made public.

2.62 The DMO’s assessment of capability performance (Expected Capability) is against the Materiel Release Milestones (MRMs) and Completion Criteria specified in each project’s Materiel Acquisition Agreement (MAA). The exception is Wedgetail, where the assessment is against the Supplies section of the MAA which lists the equipment to be delivered. This is consistent with prior year MPRs.

2.63 In general, MRMs and Completion Criteria (previously Measures of Operations Effectiveness (MOEs)) are designed to define, at a strategic level, the key elements required to achieve IMR and FMR.¹⁰³ They focus on the achievement of technical, regulatory or operational requirements. Where key requirements are not achieved, this could be expected to have a significant effect on a system’s likely suitability for acceptance into operational service.

2.64 It is important to remember that the DMO’s assessment against MRMs is based on a forecast of future events. This involves a high level of uncertainty as it depends on circumstances that have not yet occurred or may not occur, or may be impacted by events that have occurred but have not yet been identified. In the JCPAA’s Report 442, the Committee has recommended that the ANAO and the DMO consult to apply a more objective method of assessing capability performance and distinguish capability achieved, capability yet to be achieved, capability unlikely to be achieved and capability exceeded.¹⁰⁴

2.65 However, while the DMO has advised that there is no system that universally tracks the progress of inputs to capability, the DMO proposed inclusion in the PDSSs for the 2014–15 MPR, information regarding the key elements which constitute IMR and FMR as stipulated in the MAA for each project. These changes have been incorporated into the 2014–15 MPR Guidelines and will assist in providing greater clarity regarding the amount of capability delivered by the DMO. Further consultation between the ANAO and the DMO will occur in 2014–15.

2.66 The ANAO continues to observe that there is considerable diversity across the projects in the number, level of specification, articulation and focus of MRMs and Completion Criteria in project MAAs. For example, AWD Ships (total budget $7.9 billion) has four MRMs and Completion Criteria (IMR—Ship 1, MR 2—Ship 2, MR 3—Ship 3 and FMR), while 155mm Howitzer (total budget $0.3 billion) reports on 23 MRMs and Completion Criteria.

2.67 The MRMs and Completion Criteria are assessed using traffic light indicators by the DMO’s project management, and reported monthly within the DMO and the Australian Defence organisation. The June 2014 traffic light assessment is represented in the PDSSs as Measures of Materiel Capability Delivery Performance. Due to national security considerations, only the overall status from each project office’s assessment of the likelihood of delivering the required materiel capability is disclosed in the 2013–14 MPR and earlier reports.

2.68 Figures 14 and 15 present the DMO’s assessment of the percentage of materiel capability delivery that:

• has a high level of confidence will be met (green);
• are under threat but still considered manageable (amber); and
• at this stage are unlikely to be met (red).

2.69 For example, Figure 14, at page 80, shows the assessment for Air to Air Refuel as at 30 June 2014, as having a high level of confidence in delivering 75 per cent of the materiel capability, while 25 per cent of the materiel capability is ‘under threat but still considered manageable’. This differs from 2012–13, where 50 per cent of capability was ‘under threat but still considered manageable’.

2.70 The DMO’s Expected Capability should be interpreted with some caution due to diversity across projects in the basis for assessment (MRMs and Completion Criteria) and the high level of risk and uncertainty in forecasting outcomes. As such, the DMO’s current assessment of Expected Capability may differ materially from that stated in the PDSSs and is outside the scope of the ANAO’s assurance review.¹⁰⁵ However, the ANAO has been provided with data from the DMO’s reporting systems in order to examine the accuracy of the disclosure in the PDSSs.

Project snapshot—Expected Capability and Project Maturity

2.71 The fourth snapshot, Figure 13, at page 77, sets out the DMO’s assessment of the likelihood of delivering all of the key capability (Expected Capability) and Project Maturity.¹⁰⁶

2.72 The DMO expects that all 30 projects in this year’s MPR will deliver all of their key capability requirements, recognising that some elements of the capability required for some projects may be under threat but considered manageable (assessed as either green or amber).

2.73 This is consistent with the 2012–13 assessment, and highlights the DMO’s expected improvement on the 2011–12 assessment, where both Wedgetail and MRH90 Helicopters were assessed as unlikely to meet all of their capability requirements. For detail of the current Expected Capability for these two projects refer to paragraph 2.77, at page 78.

2.74 The complexity and type of acquisition (MOTS, Australianised MOTS or developmental) are significant factors affecting the Expected Capability; the risk is appreciably higher for more developmental projects. The DMO’s increased expectations of delivery of key capabilities is in line with the increase in MOTS or Australianised MOTS projects selected for acquisition in recent years.

2.75 In addition, the stage of project life cycle has an impact on the maturity score provided by the DMO, with all significant (greater than 20 per cent) variances for post 2005 approved projects, i.e. projects whose contractual delivery are at earlier stages, reflecting reduced maturity scores. The DMO’s assessment of Expected Capability should become better informed as a project matures along its capability development and acquisition life cycle.

Figure 13: Project snapshot—Expected Capability and Project Maturity

 

 

Source: The ANAO’s analysis of the 2013–14 PDSSs. Analysis for the 2013–14 MPR continues to highlight inconsistencies within the application of project maturity, reducing the level of reliability of maturity assessments for key decision makers and other stakeholders; however, improved focus and review by project offices was noted by the ANAO during 2013–14 fieldwork. Refer to paragraphs 2.71 to 2.75 in Part 1, at pages 75 to 76, of this report.

Note: ANZAC ASMD 2B’s Project Maturity is based on the progress of the lead ship, not on the current eight ship program.

Capability performance

2.76 The subsequent figures illustrate the following for the 30 projects in the 2013–14 MPR:

• the DMO’s assessment of the number of Major Projects having challenges in delivering elements of their system’s planned capability; and
• the DMO’s assessment of Expected Capability to be delivered.

In-year capability performance

2.77 Figure 14, at page 80, highlights the DMO’s assessment that there are five projects with challenges in the delivery elements of system’s planned capability.¹⁰⁷ This shows the DMO’s assessment of an improvement from 2012–13 where six projects had challenges in this area. The five projects currently highlighted as having challenges in delivering elements of their system’s planned capability are:

• Joint Strike Fighter—where there are a number of risks to achieving some of the materiel capabilities required for Initial Operational Capability (IOC) and FOC including final aircraft software builds meeting IOC and FOC requirements and establishing the sustainment of the aircraft;
• Wedgetail—performance deficiencies in some elements of the mission system currently pose a risk to the achievement of FMR. All deficiencies are expected to be addressed under remediation programs agreed by Boeing in accordance with the November 2012 commercial settlement;
• MRH90 Helicopters—notwithstanding the signing of a Deed of Variation in May 2013, there remain some materiel capabilities that are undergoing Operational Test and Evaluation activities. There are associated risks and issues in achieving Service Release of these capabilities, however the project office is actively managing them. An example is the Electronic Warfare Protection;
• Air to Air Refuel—notwithstanding the signing of a Deed of Settlement, Release and Amendment in June 2014, there is a risk to meeting all materiel capability delivery requirements for FOC due to acceptance of aircraft in an initial configuration and the risk of successful remediation of all nonconformances and improvements to the Aerial Refuelling Boom System; and
• FFG Upgrade—acquisition/installation is yet to progress on the Torpedo Defence System requirements.

2.78 However, these results contrast with other available information and highlight the difficulties in estimating future capability. For example, delays in the delivery of ARH Tiger Helicopters and HF Modernisation have increased the risk of technological obsolescence, and Rate of Effort impacts have contributed to the emergence of the risk that ARH Tiger Helicopters will not reach FOC.

Figure 14: Expected Capability at 30 June 2014

 

 

Source: The ANAO’s analysis of the 2013–14 PDSSs. Refer to paragraphs 2.77 to 2.78 in Part 1, at pages 78 to 79, of this report.

Note 1: The Expected Capability assessment for Wedgetail has been against the Supplies section of the MAA, which lists the equipment to be delivered.

Note 2: Expected Capability is as per paragraph 2.2 in Part 1, at page 41, of this report.

Longitudinal capability performance

2.79 A multi-year comparison of Expected Capability should be considered with caution as project’s Measures of Materiel Capability Delivery (MRMs and Completion Criteria) can change from year to year as MAAs are updated. Therefore, any comparison of an individual or a group of projects’ Measures of Materiel Capability Delivery may not involve comparing ‘like with like’. Changes in anticipated capability outcomes should be read in conjunction with the information in the PDSSs.

2.80 Figure 15, below, examines the DMO’s assessment of Expected Capability for the same projects from 2008 to 2014 as reported in the DMO’s PDSSs using traffic light indicators as defined above.¹⁰⁸

2.81 As reflected in the portfolio of projects in the 2013–14 MPR, the DMO’s assessment is consistent with the 2012–13 MPR, that it has a ‘high level of confidence’ in delivering 96 per cent (2012–13: 95 per cent) of the key capabilities associated with the Major Projects in this report. While the delivery of the remaining four per cent (2012–13: five per cent) of the key capabilities is considered to be ‘under threat’, the risk is still considered by the DMO to be ‘manageable’.

Figure 15: Expected Capability across the 2007–08 to 2013–14 MPRs

 

 

Sources: The ANAO’s analysis of the PDSSs in published MPRs. Refer to paragraphs 2.79 to 2.81 in Part 1, at page 81, of this report.

Note 1: * The ANAO did not examine the accuracy of the recording of this data in previous MPRs.

Note 2: ^ Super Hornet did not have MOEs, but does have Measures of Materiel Capability Delivery Performance in the latest MAA.

Note 3: # Hornet Refurb was removed from the PDSSs in 2011–12.

Note 4: ∆ Joint Strike Fighter is excluded from this analysis due to a lack of data.

Note 5: Ω C-17 Heavy Airlift was removed from the PDSSs in 2012–13.

Note 6: ∞ Super Hornet was removed from the PDSSs in 2013–14.

Note 7: √ Armidales was removed from the PDSSs in 2013–14.

Note 8: ∑ C-RAM was removed from the PDSSs in 2013–14.

Note 9: The Expected Capability assessment for Wedgetail has been against the Supplies section of the MAA, which lists the equipment to be delivered.

Introduction

3.1 Major Defence equipment acquisition projects (Major Projects) are large, technically complex, high-cost procurement activities, characterised by risk and long timeframes between concept, delivery and acceptance into service. These characteristics pose significant challenges to the effective governance of these projects and highlight the importance of applying a robust governance framework to them. Such a framework has the capacity to support the consistent assessment of their progress and enhance transparency and accountability of Major Projects across the whole portfolio.

3.2 In addition to those key acquisition governance aspects discussed earlier in Chapter 1 of Part 1 (financial assurance, project maturity, enterprise risk management and capability assessments), the Australian National Audit Office’s (ANAO’s) review included assessing the developments in the governance of projects including: Gate Review Boards; Projects of Concern; Early Indicators and Warnings (EI&W); Joint Project Directives (JPDs); business systems rationalisation; and project management and skills development, to gain a greater understanding of the Defence Materiel Organisation’s (DMO’s) business and to assist in planning the approach to the review.

Developments in the governance framework for projects

Areas of future review focus

3.3 Part of the ANAO’s planning for the Major Projects Report (MPR) each year focuses on reviewing the developments to processes that contribute to the overall governance of Major Projects within the DMO and the broader Australian Defence organisation. These include:

• the Gate Review process¹⁰⁹, which is designed to provide the Chief Executive Officer (CEO) DMO with assurance that all identified risks for a project are manageable, and that costs and schedule are likely to be under control prior to a project passing various stages of its life cycle;
• the Projects of Concern process, which is designed to address project issues of concern to the DMO and government relating to cost, schedule and capability;
• the EI&W system, which is designed to identify problems with projects in the formative stages of the life cycle, for appropriate action to remediate them;
• the introduction of a requirement for Joint Project Directives in 2009–10, for all projects approved by government from 1 March 2010¹¹⁰;
• the DMO’s business systems¹¹¹ rationalisation, which is aimed at consolidating processes and systems in order to provide a more manageable system environment; and
• the project skills professionalisation and development program in the DMO and industry, which is directed at enhancing the skill sets available to manage the DMO’s Major Projects.

3.4 This review informed the ANAO’s assessment of the governance initiatives supporting the PDSSs for the 2013–14 review period, and highlighted areas in those frameworks that could be the subject of the ANAO’s future focus.

Key acquisition governance developments

Gate Review Boards

3.5 Gate Reviews involve a periodic assessment of a project in advance of key ‘gates’¹¹² by an independent Gate Review Board appointed by the DMO.¹¹³ Their purpose is to provide the CEO DMO with assurance that project offices will deliver on government approved objectives, while focusing on their preparedness to progress to the next stage in its project life cycle and in delivering on the required capability, and the continued validity, of projects’ business cases.¹¹⁴

3.6 Gate Reviews are mandatory for Major Projects at four specified gates (First Pass consideration; Second Pass consideration; Contract Solicitation; and Contract Negotiation)¹¹⁵ and optional at other gates, depending on the outcomes of the project risk assessment.¹¹⁶ A project is not permitted to proceed to the next stage of its life cycle unless recommended by the Gate Review Board and subsequently approved by the Senior Executive. The DMO also requires a Gate Review to be undertaken where the known risks merit such a review, for example, where new information about a project triggers an ‘early warning’ of a difficulty arising.¹¹⁷

3.7 The DMO held its first two Gate Reviews in August and September of 2008 (during the period of the Mortimer Review, Going to the Next Level: the report of the Defence Procurement and Sustainment Review)¹¹⁸, and then not until September 2009.¹¹⁹ This initial limited implementation of the Mortimer Review recommendation led to the decision to expand the process so that all Major Projects managed by the DMO undergo at least one Gate Review per year.¹²⁰ The DMO has since formalised the directive that projects will normally be reviewed annually from First Pass to Final Materiel Release (FMR).

3.8 This expansion was followed by a directive from government to establish an Independent Project Performance Office (IPPO).¹²¹ IPPO’s responsibilities also include overseeing the remediation of all Projects of Concern (discussed further at paragraphs 3.15 to 3.23, at pages 87 to 90), and implementing the Early Indicator and Warnings system (discussed further at paragraphs 3.24 to 3.26, at page 90).¹²²

3.9 While the Gate Review program has been strengthened and is better regulated as a result of these reforms, the ANAO Audit Report No.52 2011–12, Gate Reviews for Defence Capital Acquisition Projects, tabled in June 2012, concluded that opportunities for further improvement and rigour remain.

3.10 Subsequent to the DMO’s responses to the ANAO’s recommendations in the above report, the DMO advised in July 2014 that periodic updates were provided to the Executive Committee on the status of Gate Reviews, with a new set of metrics focusing on root cause analysis being tested to measure project performance during 2013–14. However, this is still in the formative stages.

3.11 During 2013–14 the ANAO assessed the Gate Reviews, and their outcomes (where available), for 23 Gate Reviews that had been conducted on 22 of the projects in the 2013–14 Major Projects Report¹²³ (2012–13: 25 MPR projects were subjected to Gate Reviews).

3.12 During the 2013–14 review, the DMO advised that projects which had undergone a Gate Review in the lead up to FMR did not warrant further Gate Reviews, despite continued residual issues. This includes the FFG Upgrade and SM-2 Missile projects which have experienced a further 12 month and 11 month in-year slip to FMR, respectively. Further, this also includes the ARH Tiger Helicopters project, despite the Capability Manager’s key concerns with the capability delivered at FMR.¹²⁴ There is a risk that significant issues in these projects will not be remediated without the oversight that the Gate Review process provides.

3.13 Significant recent developments to the Gate Review program include the introduction of Initiation Gate Reviews (pre First Pass), Acquisition Concept Gate Reviews (i.e. the continuation of those between First and Second Pass Approval), and Sustainment Gate Reviews.¹²⁵ Further, the DMO proposes that projects deemed ‘low risk’ as a result of the preliminary independent analysis should not be subjected to a Gate Review. The ANAO was also informed that the introduction of Sustainment Gate Reviews is in line with the future expectation that fewer acquisition projects will be eligible for Gate Reviews (as a result of a reduction in anticipated project approvals) with more projects being transferred into sustainment as they approach FMR.

3.14 The ANAO appreciates that the Gate Review process is evolving and will monitor its progress in 2014–15.

Projects of Concern

3.15 The Projects of Concern list was established in 2008 to focus the attention of the highest levels of government, Defence and industry on remediating problem projects. Projects identified by the DMO with significant issues that are beyond the normal capacity of a project team to manage may be recommended by the CEO DMO for inclusion on the list. The CEO DMO then makes a recommendation to the Minister for Defence for final determination.¹²⁶

3.16 Projects are removed from the Projects of Concern list once the Minister for Defence is satisfied that remediation activity has been completed successfully or government has decided to cancel the project.¹²⁷

3.17 Project of Concern summits¹²⁸ were also implemented in February 2011 by the then Minister for Defence Materiel to further increase focus from government, Defence and industry to remediate Projects of Concern. The DMO advised that although summits are yet to be reinstated by the new government, Project of Concern reports continue to be provided to the Minister for Defence on a monthly basis.

3.18 The IPPO is responsible for overseeing and administering the process for remediation of all Projects of Concern, and has advised the ANAO that policy and procedures for Projects of Concern are currently in a draft stage.¹²⁹

3.19 During 2013–14 two MPR projects were continuing Projects of Concern¹³⁰, however in June 2014, the Ministers for Defence and Finance announced¹³¹ that AWD Ships was being placed on the list to better address the increasing commercial, schedule and cost risks identified in ANAO Audit Report No.22 2013–14, Air Warfare Destroyer Program, tabled in March 2014, and following the earlier announcement of an independent review.¹³² The project was added as a Project of Concern on the basis of the recommendation of the CEO DMO to ‘ensure that this important project is delivered successfully’.¹³³

3.20 In May 2013, the then Minister for Defence and the then Minister for Defence Materiel announced that the MRH90 Helicopters project would be considered for removal from the Projects of Concern list by the end of 2013, following the signing of a Deed of Variation, termed Deed 2, to the original contract, to address commercial, technical and schedule issues.¹³⁴ As at June 2014, MRH90 Helicopters is still listed as a Project of Concern¹³⁵ as remediation actions are yet to be completed.¹³⁶ In addition, Air to Air Refuel is still considered a Project of Concern by government.

3.21 As noted by the Australian Strategic Policy Institute (ASPI) in August 2013:

• The introduction in 2008 of the ‘Projects of Concern’ management process has seen some projects that faced serious challenges remediated (and in a couple of instances terminated).¹³⁷

3.22 Since the Projects of Concern process was established, 10 MPR projects have been on the list and seven have been removed due to government decisions that the projects have been remediated.¹³⁸ The seven projects considered remediated are:

• Wedgetail;
• Overlander Medium/Heavy;
• ARH Tiger Helicopters;
• FFG Upgrade;
• ANZAC ASMD 2B;
• HF Modernisation; and
• Stand Off Weapon.

3.23 While specific concerns may have been addressed for ‘remediated’ projects, it should be noted that residual or emergent risks and issues, for example for ARH Tiger Helicopters (see paragraph 3.12, at page 87), exist post FMR and may even impact the achievement of Final Operational Capability.

Early Indicators and Warnings

3.24 In May 2011, the then Government announced that it would implement an Early Indicators and Warnings (EI&W) system designed to help identify and correct potential problems in the formative stages of the project life cycle.¹³⁹

3.25 In June 2013, the Defence Capability and Investment Committee (DCIC) reviewed the performance of the EI&W system and considered potential improvements. The DMO has since advised that while the EI&W reporting system has provided benefits across Defence, it is no longer providing the required outcomes in its current form.¹⁴⁰ In response, the CEO DMO proposed a new reporting regime which could improve the consistency of reporting and moderate the shortcomings of disparate data collection and reporting systems.

3.26 As a result, the IPPO advised that the Quarterly Project Performance Report (QPPR) process is expected to replace the EI&W system with the objective of identifying problems across all DMO projects providing a more allencompassing report, however this is currently pending approval.

Joint Project Directives

3.27 Joint Project Directives (JPDs) were introduced by Defence for all projects approved by government from 1 March 2010 to identify the scope and limits of acquisition projects and the party responsible for the major components of the work.¹⁴¹ They are also intended to be an authoritative document which enables any other necessary project documentation, including Materiel Acquisition Agreements (MAAs)¹⁴², to be based on a reliable source. Therefore, JPDs and MAAs are intended to aid sound governance and accountability in Defence acquisition projects.¹⁴³

3.28 During the 2013–14 review, the ANAO continued to examine a range of governance documents which define project deliverables. Traditionally, MAAs have defined a range of project attributes with the DMO currently able to provide JPDs for eight MPR projects.¹⁴⁴ Of these, nothing has come to the attention of the ANAO that the JPDs are not consistent with the MAAs and other necessary project documentation.

3.29 However, as a consequence of the slower than anticipated implementation, the ANAO will take JPDs into account in its review program in future years, where these have been prepared. With another two post 2010 projects to be included in the 2014–15 MPR, better alignment with government approved requirements is expected and an initial internal audit would assist in determining that the predefined impacts of JPDs are being met.

Business systems rationalisation

3.30 In previous MPRs, the ANAO has reported that the control environment of each project differed, due to the large range of corporate and project management Information Technology (IT) applications being employed by the different project offices. During the 2013–14 review, the same observations apply across the 30 Major Projects. This has again resulted in an inconsistency between the information produced by each of the project’s IT systems (i.e. risk management, financial management, and document management systems) and highlights an issue for the DMO in ensuring reliable and consistent information to properly inform project management and decision making. However, the DMO advises it is making progress in this area through the development of an Integrated Project Management System (IPMS) which will interface with current IT systems to provide reports to management. IPMS is expected to improve reporting practices across the DMO and will be tested on selected project offices in 2014–15.

3.31 In its Report 436, Review of the 2011–12 Defence Materiel Organisation Major Projects Report, the Joint Committee of Public Accounts and Audit (JCPAA) expressed its ongoing concern with the consistency of information in DMO’s business systems, and recommended that ‘the DMO develop a business systems improvement plan which prioritises projects, assigns completion dates and allocates senior level ownership for implementation. A progress update on achievements against the plan should be included in the 2012–13 Major Projects Report’.¹⁴⁵ The DMO has since advised of the development of an Information Management (IM) Investment Plan to prioritise investment in business systems and other IT applications and the development of an overarching IM Framework and Strategic Plan, including an IM Governance Model.

3.32 During 2012–13, the DMO advised that there were 540 separate applications in use, and that the DMO was identifying and confirming where duplicate licenses and other inefficiencies existed, as well as researching software and systems, in order to achieve greater visibility over Information and Communications Technology expenditure in the DMO. As a result of this process the total number of applications in use as at June 2014, has been reduced to approximately 300. The process of identifying inefficiencies in software is now being undertaken across the whole of Defence by the Chief Information Officer Group (CIOG), with a total of approximately 3 400 applications identified and 276 recommended for removal.

3.33 There have also been developments in the Standalone Network Remediation Process (SNRP), being undertaken by the CIOG, which aims to rationalise DMO networks and programs onto the Defence Restricted Network (DRN). There are currently 113 standalone networks identified with approximately 50 expected to remain as local area networks due to compatibility issues (e.g. simulator software is not suitable to be maintained on the DRN) and the remainder to be rationalised as the SNRP continues. The SNRP has also led to the identification of networks that are not compliant with best practice on network security, as well as the remediation of these non-compliances for legitimate standalone networks.

3.34 Separately, in 2010 the DMO informed the ANAO that it was focusing on rationalisation of the finance domain, due to resource constraints on progress in prior reviews of the business systems and implementing recommendations arising from these reviews. This included the transition from the previous program level budget management system (CEPPlan) to the enterprise level system (BORIS), which was implemented across the DMO in February 2014 (see paragraphs 1.24 to 1.25 in Part 2).

3.35 The Chief Finance Officer (CFO) DMO is expected to make strategic decisions on the future of DMO finance. The DMO Financial Information Management Advisory Committee exists to make recommendations to the CFO DMO on financial information management, with a view to having a coordinated approach to implementing business rules, process improvements and system rationalisation in the finance domain. The DMO Future Finance State Initiative has been endorsed by the Committee, and is incorporating the work done by the Finance Information Management Improvement Program to inform future systems developments including those to be delivered via the Finance Systems Modernisation Project JP 2080 Phase 3 and 4.

3.36 Finally, as part of a program to rationalise its information and communications technology systems, the DMO also rolled out the Defence endorsed single records management system called Objective in 2013, which has centralised the DMO’s electronic document management.¹⁴⁶ While some problems were experienced, the implementation went ahead without significant issues and benefits were expected to flow to project management in the DMO over 2013–14. However, due to issues around access rights, incompatibility with other applications and lack of buy-in from staff, not all the expected benefits of Objective have eventuated. Guidance and training is being offered to all employees, along with regular Communities of Practice on records management, in order to ensure these expected benefits are realised.

Project management and skills development

3.37 A key challenge for both the DMO and the Australian Defence industry is to improve the project management, scheduling, logistics, procurement and engineering services provided to the Australian Government, within current and future workforce constraints. The DMO’s professionalisation initiative is a central strategy in meeting the DMO’s challenges now and into the future¹⁴⁷, as well as to attract and retain a skilled workforce.

3.38 To assist with professionalising DMO staff, the DMO’s Directorate of Professionalisation Policy, Accommodation and Coordination is responsible for the development of certification programs that focus on developing specific competencies and gaining professional qualifications. This supports the DMO core business of acquisition and sustainment across the Procurement, Financial Management, Project and Sustainment Management, Materiel Engineering and Materiel Logistics streams.

3.39 Prior to the start of the initiative in 2005, the DMO had only 153 staff certified in areas of project management, engineering, and accounting.¹⁴⁸ The Directorate advised that during 2013–14 approximately 1 050 DMO staff have either been certified or are enrolled in a certification program with a professional body compared with 900 staff in 2012–13 and 1 200 staff in 2011–12.¹⁴⁹

3.40 To assist with the development of participants within the Defence industry, the Skilling Australia’s Defence Industry (SADI) program was established by the Australian Government in 2005. The aim of the SADI program is to up-skill existing employees, improve the quality and quantity of skills training, and provide funding support to the Defence industry for training activities where there is an identified skills shortage in technical, trade and professional skill sets.¹⁵⁰

3.41 The SADI program provides funding support to companies and industry associations for training and skilling activities where that training is linked to a Defence capability. The DMO advised that since 2005, the program has funded more than 30 000 training places. Also, approximately 200 industry participants have been provided with funding support in trade, technical and other professional skill sets.¹⁵¹

3.42 To meet the needs of the Australian Defence Force, SADI aims to invest $215 million¹⁵² within the period 2005–06 to 2015–16, with more than $5 million allocated in 2014–15 under round one of the SADI program to boost the skills of workers in the Defence industry.¹⁵³

3.43 For 2014–15, the SADI Grant Program has been updated in administrative and functional aspects to optimise program outcomes, reduce red tape, and improve the effectiveness of the program. This has been done by allowing greater training flexibility when it comes to scheduling of training by industry participants, and by improving the cash-flow of companies and industry associations.¹⁵⁴ Industry participants will be paid at the commencement of each quarter based on the number of scheduled (or enrolled) training activities to be undertaken in that quarter. At the end of each quarter, training activities will then be acquitted, based on the submission of a Quarterly Statement Report and supporting documents by the industry participants.¹⁵⁵

3.44 Other government approved programs aimed at up-skilling participants within the Defence industry include the Industry Skilling Program Enhancement (ISPE) package and the Priority Industry Capability Innovation Program (PIC IP). ISPE, which was announced in 2008, provides up to $60.8 million.¹⁵⁶ In 2012–13, the DMO advised that there was no funding programmed past 30 June 2014. Since then, the DMO has advised that some initiatives under this program have been extended until the new Defence Capability Plan, Defence White paper and Defence Industry Policy Statement have been prepared and released, expected in 2015. This program incorporates a number of initiatives aimed at enhancing work and future career pathways in the sector by implementing programs at schools and offering Defence technical and engineering scholarships.

3.45 PIC IP, which was implemented in 2011–12, provides up to $45 million over seven years¹⁵⁷, and is aimed at providing Australian Defence companies direct support in the form of repayable and matched grants (the recipient matches the funds provided by the Australian Government on a dollar for dollar basis).¹⁵⁸ The program particularly focuses on assisting small to medium sized enterprises to pursue innovative Defence industry projects.

3.46 The DMO has since advised that the PIC IP will cease with no further funding rounds to be offered under this program, and the PIC Development Fund (PIC DF) will be instituted in its place. It is currently anticipated that PIC DF will be funded from the Defence Capability Plan and will be administered by the DMO, though oversight will be provided by the Capability Development Group. However, this transition is still in development.

3.47 In addition, on 5 September 2011, the then Minister for Defence Materiel underlined the need for the replacement or upgrade of up to 85 per cent of its military equipment over the next 15 years, and asked Skills Australia to work with Defence and the Defence industry to develop a plan to ensure Australian industry has the skills to meet this challenge.¹⁵⁹ Skills Australia noted the following in their September 2012 report Building Australia’s Defence Supply Capabilities – Main Report for the Defence Industry Workforce Strategy:

…the current skilling programs offered by the DMO suffer from a lack of longitudinal evaluation processes or strategies in place to assess the effectiveness of these programs. The lack of these mechanisms for either SADI or ISPE means that the DMO and Government are not in a position to determine when success is achieved in these programs and whether they have provided value for money.¹⁶⁰

3.48 Skills Australia made three recommendations regarding this issue aimed at undertaking performance evaluations of SADI and ISPE and developing Key Performance Indicators.¹⁶¹ In response, the DMO advised that they have considered outcome focused reporting for new and upcoming programs however, they advised that it would be onerous to re-establish reporting requirements for existing programs in the current climate, particularly for programs such as ISPE, which are currently due to cease in 2014–15.

3.49 Finally, in June 2014, the Productivity Commission released the Trade & Assistance Review 2012–13 which also commented on Defence industry assistance programs, noting that:

… defence industry assistance programs do not appear to have been reviewed (externally and using an economic framework) to the same extent as many other industry assistance programs… All defence programs with material industry assistance objectives, explicit or not, should be separately reported to Parliament, and from time to time reviewed to ensure relevance to strategic and government priorities.¹⁶²

3.50 The DMO advised that reviews of industry assistance programs will likely be encapsulated in the wider Defence First Principles Review, due to be released in early 2015. The ANAO will continue to monitor the SADI and other industry assistance programs in 2014–15.

CEO DMO Foreword

I am pleased to present the 2013-14 DMO Major Projects Report (MPR). The first DMO MPR, which was tabled in Parliament in November 2008, reported on a sample of nine of Defence’s major projects. This year, 30 projects are reported. The differences from last year include the removal of three projects: the Bridging Air Combat Capability; Armidale Class Patrol Boats; and Counter-Rocket Artillery and Mortar. New projects included this year are: Medium Heavy Capability, Field Vehicles, Modules and Trailers; EA-18G Growler Airborne Electronic Attack Capability; Battlefield Airlift – Caribou Replacement; and Amphibious Watercraft Replacement. Together, the 30 projects are worth more than $59 billion.

The DMO MPR plays an important role in providing transparency to Parliament on the management of these projects, and provides quantitative data on DMO’s project management performance. Importantly, the MPR consistently demonstrates that the DMO continues to deliver Government approved materiel capability within budget. This is a point worth repeating for emphasis - DMO projects, on average, continue to deliver the materiel capability sought at Government approval within the approved budget.

The delivery of complex new military capabilities on schedule remains challenging. Factors affecting schedule performance (in this year’s sample) include: platform availability; industry delays in resolving technical design issues; unforseen technical problems; and other contractual issues. The DMO continues to work with Capability Development Group, Capability Managers and industry to reduce risks within major project schedules in order to ensure more executable schedules are presented to Government when projects are approved at Second Pass.

This said, I am pleased to note that the MPR projects which were initiated post the 2003 Kinnaird review continue to demonstrate improvement in schedule performance, confirming the value of the reforms implemented in the DMO since that time, and in particular following the DMO becoming a prescribed agency on 1 July 2005. It is interesting to observe that the pre-Kinnaird projects within the MPR sample continue to have a disproportionate influence on the reported schedule performance.

The format of this year’s report remains the same as that of last year. Chapter One provides an overview of the DMO’s performance and discusses key issues, such as financial, schedule and risk management. Chapter Two provides a detailed analysis of schedule, cost and delivery of the materiel elements of capability.

• The capacity to engage a flexible, professional workforce to maintain project performance remains a challenge for the DMO. In 2014-15 the DMO will continue to implement reform across all elements of its business operations, striving always to improve its support to Government and Defence outcomes while fostering a positive strategic relationship with Defence industry. I welcome the First Principles Review of Defence, including the DMO, and anticipate further opportunities for improvement.

I am pleased to report that in April 2014 the Electronic Support Measures Upgrade Project for the AP-3C Orion aircraft was removed from the Projects of Concern list. This is a credit to the DMO and industry teams who have worked hard to remediate the project. The Air Warfare Destroyer project was added to the list in June 2014 and implementation of reform is underway, including a program to improve productivity involving the integration of a specialist management team.

The Chiefs of Army, Air Force and Navy and the major contractors for each project have reviewed the relevant project data published in this MPR. Their views have been considered in finalising this report.

The DMO and the Australian National Audit Office MPR teams continue to have a strong professional working relationship, and the teams have further improved the MPR consistent with guidance provided by Parliament’s Joint Committee of Public Accounts and Audit.

I thank the Auditor-General, Mr Ian McPhee, and his staff for their contribution to the overall report. I also commend the various DMO project office staff, the DMO MPR Directorate, and other DMO business areas involved in bringing together this DMO MPR.

Warren King
Chief Executive Officer
8 December 2014

Executive Summary

In 2013-14, the Defence Materiel Organisation (DMO) managed over $9.4 billion¹⁶³ in expenditure for the Department of Defence across the acquisition and sustainment program and other management services (including about $30 million in industry support programs). As at 30 June 2014, the DMO managed 180 major acquisition projects with an average value of $527.1 million. The acquisition budget for FY 2013-14 was $3.968 billion.

The 2013-14 Major Projects Report (MPR) examines 30 of the largest and most technically challenging of these. The key aspects of this MPR are:

• the removal of three projects reported in the 2012-13 MPR (Bridging Air Combat Capability; Armidale Class Patrol Boats; and Counter-Rocket Artillery and Mortar);
• the inclusion of four projects (Medium Heavy Capability, Field Vehicles Modules and Trailers; EA-18G Growler Airborne Electronic Attack Capability; Battlefield Airlift – Caribou Replacement; and Amphibious Watercraft Replacement) bringing the total number of reported projects to 30;
• all projects are currently delivering capability within the approved budget, although the final budget for Air Warfare Destroyer under the Reform Program is yet to be established; and
• analysis has identified that average schedule slippage to Final Materiel Release ((FMR) - the point at which the DMO has delivered all of the approved materiel requirements) is 11 per cent for the 30 projects in this year’s sample compared with 8 per cent for the 29 projects in last year’s sample (noting that 26 projects are common to both years).
• The DMO’s budget performance betters that of large commercial projects, and compares favourably with DMO’s counterpart agencies in other countries. To put the DMO’s cost performance in perspective:
• the DMO major projects tend to be software intensive, the average cost overrun for large commercial software projects is 66 per cent¹⁶⁴; and
• the United Kingdom Ministry of Defence’s 2013 Major Projects Report found an overall cost increase of 12.3 per cent since approval across its 11 projects.
• Similar to project delivery organisations around the world, schedule performance remains an ongoing challenge. As noted above, the current average slip to FMR is 11 per cent. The average slip to Final Operational Capability ((FOC) - where the Capability Manager declares that the whole capability has been realised) is 35.8 per cent. Most of the in-year FMR schedule slip experienced by the MPR projects was caused by factors such as technical problems, platform availability and contractual issues (further detail is in Chapter 2). The DMO is continuing to implement reforms that will deliver improvements in schedule estimation and management (further detail is in Chapter 1). However, the benchmarks below are useful in setting current major project schedule performance in context:
• McKinsey’s 2012 study found that commercial software projects average schedule overruns of 33 per cent¹⁶⁵; and
• the United Kingdom Ministry of Defence’s 2013 Major Projects Report found overall slippage of 21 per cent across its 11 current projects and 35 per cent for projects approved since 2000.

For 2013-14, the 30 projects included in the MPR reported that 253 of the 264 (or 96 per cent) capability elements had a high level of confidence in being deliverable. While risks exist, the remaining four per cent of capability elements are still considered achievable. For the second year in a row, no capability elements have been identified as being unlikely to be met.

Reforms stemming from the 2003 Kinnaird review have driven measurable improvements in project delivery performance: such as the two pass Government approval process; and the financial separation of the DMO from Defence to create a formal customer-supplier relationship. This is reflected in the superior schedule performance of post-Kinnaird projects in the MPR. Average schedule slip for these projects is 15 per cent¹⁶⁶, compared with 71 per cent for projects approved prior to the Kinnaird reforms.

The Projects of Concern (POC) regime continues to be of benefit to the DMO in addressing project risks. Since its introduction in 2008, 22 projects have been placed on the POC list. Of these, 13 have been remediated; two cancelled; one scope reduced; and six remain the subject of intensive remediation.

DMO Overview

1.1 During 2013-14 the DMO managed 180 major acquisition projects in support of the Australian Defence Force with an average value of $527.1 million. In capability acquisition, the DMO is primarily responsible for managing the acquisition of the materiel systems (the mission and support systems) component of capability. Although these are often the most complex and expensive elements of a capability solution, they are typically dependent on support from other areas across Defence. The materiel elements are just one element alongside all of the fundamental inputs to a full capability solution including the personnel, training, supplies, facilities, training areas, logistics, support, command and management.

1.2 The DMO role extends across the totality of the acquisition lifecycle, from contributing to the early stages of project development before formal Government approval, to delivering the materiel elements of major projects as approved by Government. The DMO also sets in place the in-service sustainment arrangements. Separate benchmarking studies conducted by the Helmsman Institute have concluded that the DMO delivery environment and projects are more complex than the average industrial project. Complexity factors include: the level of innovation required; the level of technical development; the number of stakeholders; ambiguity in requirements; and organisational complexity. Helmsman also noted that complexity will continue to increase across future projects.

1.3 During 2013-14, 19 projects were closed having met the required capability with an average of seven per cent under the approved budget. This continues the trend of under budget performance with project closures across the previous three financial years having recorded an average of five per cent under budget.

DMO Achievements

1.4 An overview of cost and schedule performance is shown in Chapter 2. Examples of significant achievements in 2013-14 are:

• Joint Strike Fighter: The Government has approved the acquisition of an additional 58 F-35 Joint Strike Fighter aircraft¹⁶⁷.
• Air Warfare Destroyer: The first major combat system element (Mark 41 Vertical Launch System) has been loaded into the first Air Warfare Destroyer being built at Port Adelaide¹⁶⁸.
• MRH90 Helicopters: The 29^th of 46¹⁶⁹ MRH90 Multi Role Helicopters was accepted in June 2014.
• MH-60R Seahawk: Navy commenced Seahawk Romeo dipping sonar operations¹⁷⁰ in May 2014, a capability that Navy has not had since the 1980s.
• LHD Ships: The hull of Australia’s second and final Landing Helicopter Dock Ship arrived in Port Phillip Bay in February 2014, completing its voyage from Ferrol, Spain¹⁷¹.
• ARH Tiger Helicopters: The project achieved Final Materiel Release (FMR) in March 2014. All 22 Tiger helicopters, simulators, ground support systems and contracted training devices have been accepted and introduced into service.
• Next Gen Satellite: The project achieved FMR in June 2014.
• 155mm Howitzer: The project achieved FMR in September 2013.
• Stand Off Weapon: The project achieved FMR in September 2013.

Projects of Concern

1.5 Projects of Concern (PoC) are those projects identified as having very significant technical, cost and/or schedule difficulties. The primary objective of the PoC regime is to assist with the implementation of an agreed remediation plan. Projects listed as PoC receive a higher level of oversight and management and undertake increased reporting to Government. Since 2008, 22 projects, with a total value of $30.4 billion, have been managed as PoC. There are six active PoC (listed in Table 1.1) with a total value of $14.6 billion as at 30 June 2014. In 2013-14, the Electronic Support Measures Upgrade Project for the AP3C Orion Aircraft was removed, and the Air Warfare Destroyer was added as a PoC.

Table 1.1 – List of Projects of Concern at 30 June 2014

DMO Challenges

1.6 The DMO delivers the materiel elements of capability as described in the Materiel Acquisition Agreements (MAAs) set between the DMO, the Capability Development Group and the relevant Capability Manager at project approval. The average schedule slip to Final Materiel Release (FMR) is 11 per cent for the 30 projects included in this MPR versus eight per cent for the 29 projects included in the 2012-13 MPR. Note that three projects have been removed from and four new projects added to this MPR. The main factors driving the schedule slip against the FMR milestone are: platform availability; industry delays in resolving technical design issues; unforseen technical problems and other contractual issues.

1.7 For the 30 projects in this MPR, the average schedule variance at Final Operational Capability (FOC) is 35.8 per cent. In a number of cases, such as the FFG Upgrade, core capabilities have been successfully delivered and have been in use for some time, but final capability elements remain outstanding. The first of the upgraded frigates, HMA Ship Melbourne, commenced security operations in the Middle East in 2010 and all upgraded frigates received Operational Release in June 2012.

DMO Major Acquisition Program – Financial Management

1.8 The DMO Chief Finance Officer provides financial assurance statements in relation to individual projects included in the MPR delivering the remaining intended scope within approved project budgets. This arises from concerns raised by the Joint Committee of Public Accounts and Audit regarding adequacy of project budgets following the move to out-turned budgets in 2010.

1.9 When considering and approving budgets, the Government takes account of the estimated impact of inflation over the life of a project. This forecasting of future inflation is known as ‘out-turning’. From 1 July 2010, all DMO major capital projects have been managed using out-turned budgets. At the time of project approval, project managers estimate the impact of indices tendered (or estimated) for the life of the project. These estimates are built into the project budget as part of the out-turning process.

1.10 Financial assurance statements have been validated by an external independent company for ten of the projects reported in this MPR.

Table 1.2 – List of 2013-14 MPR Projects by Total Approved Budget¹⁷³

DMO Project Performance

1.11 Based on internal and external analysis, the DMO has steadily improved its effectiveness in delivering materiel capability since being established in 2000. In the last four financial years, 80 major projects have completed their MAA, after successfully introducing the required capabilities. The materiel capability for these 80 projects was delivered on average five per cent under budget, allowing over half a billion dollars in savings to be returned to Defence.

1.12 The cost performance mentioned above compares favourably to the private sector, as well as the DMO’s counterpart agencies in other countries. To put the DMO’s cost performance in perspective:

1. noting that the DMO’s major projects tend to be software intensive, large commercial software projects average cost overruns of 66 per cent¹⁷⁸; and
2. the United Kingdom Ministry of Defence’s 2013 Major Projects Report found an overall cost increase of 12.3 per cent since approval across its 11 projects.

1.13 The current average schedule slip of post-Kinnaird (that is approved via the two pass Government Approval process implemented following the 2003 Kinnaird Review) projects of approximately 15 per cent compared with 71 per cent for pre Kinnaird projects (approved prior to the Two Pass process) indicates the project performance improvements realised since the 2003 Kinnaird review and, in particular, following the DMO becoming a prescribed agency on 1 July 2005. Even so, as is the case for project delivery organisations around the world, schedule performance remains a challenge. The benchmarks below put current major project schedule performance in perspective:

1. a McKinsey’s 2012 study found large commercial software projects experience average schedule overruns of 33 per cent¹⁷⁹; and
2. the United Kingdom Ministry of Defence’s 2013 Major Projects Report found overall slip of 21 per cent across its 11 current projects and 35 per cent for projects approved since 2000.

Continuous Improvement

1.14 During 2013-14, the DMO continued to deliver reform across all elements of its business operations. Current reform priorities focus on:

1. defining the core set of DMO roles required to deliver assigned outcomes while minimising overheads and costs through:

(1) prioritising the DMO’s activities and resources to meet the ADF’s urgent operational demands, deliver government-approved projects, and sustain the force-in-being;

(2) optimising the efficiency of core specialised DMO functions; and

(3) revising contract and performance arrangements to better use industry innovation, capability and capacity.

1. embedding stronger controls to heighten personal accountability and provide lead indicators for management intervention.
2. improving schedule performance through enhanced schedule estimation and risk assessment methodologies.
3. contributing to ongoing tailoring of the two pass process for major project approvals, as recommended in the Mortimer Review, to better balance the preparatory work for project approvals, commensurate with their respective scope and complexity. Opportunities exist to lessen costs to industry by reducing unnecessary ‘red tape’ by clarifying accountabilities, constraining the level of detail sought in tender responses, and speeding up project approvals.
4. providing specialist commercial, technical/engineering and through-life support advice earlier in the capability development process to ensure that technical and safety risks, costs and appropriate acquisition and sustainment strategies are considered in the development of capability options (thereby limiting rework in the later phases of capability delivery).
5. enhancing the DMO’s advice to Government on industry’s capacity to deliver the required capabilities and the accuracy of industry cost and schedule estimates.
6. improving the delivery of assistance programs to further the skills, productivity, innovation and competitiveness of Australian defence industry, with a focus on those areas deemed critical to retaining indigenous industry capacity to deliver approved Defence projects and through-life support.

Negotiation Cell

1.15 In 2013-14, the DMO Negotiation Cell established a standing offer panel for negotiation services with around 20 nominated highly-performing individuals. These individuals are supporting various project and sustainment teams to develop their commercial expertise and to achieve the best outcomes from their dealings with industry.

Acquisition and Sustainment Planning Framework

1.16 The 2008 Mortimer Review recommended improvements to the DMO’s acquisition strategy development process during the requirements phase of the capability systems life cycle. To support this improvement, the DMO has developed new Acquisition and Support Implementation Strategy (ASIS) policy and guidance that places greater emphasis on implementation of the support system to ensure whole of life issues are better addressed early in the project life cycle. The ASIS policy and guidance was released as an Exposure Draft in September 2013. Following stakeholder consultation and feedback from projects that are developing ASIS documents, the ASIS policy is being updated prior to formal release. Pilot ASIS training was undertaken for around 250 DMO staff in March 2014 and the DMO is working to transition the ASIS pilot training to provide an ongoing program of training.

Schedule Management

1.17 Accurate schedule estimation is essential for predictable schedule performance. To this end, the DMO is undertaking various initiatives to improve schedule estimation across Defence major capital acquisition projects, including:

1. developing a Tier 1 Defence Cost and Schedule Estimation Manual in conjunction with Capability Development Group;
2. integrating the Schedule Compliance and Risk Assessment Method (SCRAM) with the Integrated Baseline Review Process to improve early identification of schedule risks. This new integrated process will be piloted on selected high risk projects over the next 12 months to refine and prove the process;
3. analysing current reporting systems to determine if a more integrated approach to quantitative and qualitative performance information can be instituted to improve project performance management;
4. developing a Project Controls Manual to consolidate, update and improve Work Breakdown Structure, scheduling and earned value management processes.

DMO Schedule Compliance Risk Assessment Method

1.18 In 2013-14 there were a number of independent SCRAM reviews performed on DMO projects to identify risk, issues and recommendations for schedule achievement. Notably, further SCRAM assessments were conducted on the Joint Strike Fighter Program using scientific Software Parametric Analysis techniques to model and forecast Mission System software development duration. Furthermore, the DMO SCRAM team members participated in the United States Congressional mandated review of software development for the F-35 Joint Strike Fighter aircraft.

1.19 The DMO is extending SCRAM methodology to better integrate with the DMO-mandated Integrated Baseline Review Process. This includes the feasibility of a practical method to identify potential schedule pressures prior to contract signature. Work is also underway to develop a standard Technical Implementation Risk Assessment (TIRA) methodology. The intention is for TIRAs to provide a framework and guidance for the conduct of early DMO technical risk assessments prior to the formation of the Contract Project Management Baseline. Hence, TIRAs would also be an input for Materiel Implementation Risk Assessments which are developed by the DMO’s Project Managers as part of the Project Approvals process

DMO Project Risk Management Framework

1.20 A consolidation of risk policy is being undertaken to provide a more consistent approach to effective risk management within the DMO. A single DMO Risk Management Manual will be created and will ensure compliance with the new Commonwealth Risk Management Policy under the Public Governance, Performance and Accountability Act 2013.

Information and Communications Technology (ICT) Services

1.21 Under various formal Agreements, the Chief Information Officer Group (CIOG) delivers the ICT infrastructure and services to the DMO that are necessary to enable materiel capability to the war fighter as well as the corporate and other functions of the DMO. The DMO and CIOG acquire and sustain a range of ICT applications to meet capability and other business requirements. During 2014-15, the DMO will review its Information Management Strategy to ensure alignment with Defence.

1.22 The DMO Information Management Investment Plan 2013-18 (The Plan) encompasses all significant Information Management investment across the DMO, excluding Defence Capability Plan projects and initiatives as well as programs and projects being undertaken by CIOG or other groups for which the DMO is a stakeholder rather than the lead. The Plan has a specific focus on business information and complements the Defence Information Management Strategy and Plan.

1.23 The DMO applies a range of ICT applications to support its management of projects, including applications for: project scheduling and management; risk management; project performance reporting; and electronic records management. There is ongoing work to ensure that these applications meet current DMO business requirements, including planning and implementation of new versions as well as improvements in the integration of systems to reduce duplication and improve the accuracy of information. Among other changes, during 2013-14, the DMO implemented a new version of its project scheduling and management tool along with a significant upgrade of the architecture which supports that system.

1.24 The limitations in the enterprise systems for managing project financial information have been highlighted in previous editions of the MPR. Improvements are being pursued through a range of strategies as in previous years. The DMO is continuing to pursue its Acquisition Budget Management Improvement Program. The DMO has successfully transitioned the functions from its current program-level budget management system to the enterprise budget management system (Budget and Output Reporting Information System).

1.25 In addition to improvements to program-level budget management and reporting, the DMO proposes to standardise project financial management tools across its projects to support data integration between the project and program levels. It is anticipated that the first phase of work for an Integrated Project Management System will commence in 2014-15.

Document Management systems

1.26 Objective is the mandated records management tool in Defence and all DMO System Program Offices have been using Objective since 30 June 2013. It enables the DMO to meet its recordkeeping obligations while at the same time facilitating information management and workplace collaboration.

Professionalisation

1.27 In 2013-14, the DMO continued to refine education and skilling pathways for its staff. Professional certification for project managers remained on the agenda. During the year, the DMO facilitated 34 courses in the project management discipline; 15 in the logistics domain; six in the commercial and business domain; 32 in the procurement and contracting domain; and two engineering courses.

Projects Exited the MPR

1.28 Table 1.3 lists the projects that have exited the MPR. The Expenditure to date is as at 30 June 2014.

Table 1.3 – List of Projects Exited from MPR

Introduction

2.1 This chapter presents a performance overview of the 30 projects included in the 2013-14 MPR. It includes detailed analysis of the three key variables of cost, schedule and materiel systems delivery against the Government’s approved budget, schedule and scope. The analysis commences at an aggregate level, discussing performance of all 30 MPR projects collectively, before addressing some project specifics.

2.2 Given that the MPR only covers 30 of the current 180 major projects, or 17 per cent of the total major acquisition portfolio, caution must be applied when extrapolating any analysis to the entirety of the DMO’s acquisition effort. This is because the projects in the MPR are not necessarily representative of all projects: the 30 projects are the largest by budget, at the time of inclusion, and, in general, involve a much higher level of complexity than average or lower budget projects.

Project Performance

2.3 Table 2.1 provides a summary of the total cost performance and Table 2.2 gives an overview of the in-year budget movements and cost performance. Table 2.3 shows schedule performance for the 30 projects in this year’s MPR and Table 2.4 provides a summary of the key characteristics of each project in terms of maturity and level of development required. The DMO’s analysis shows that, while projects have been managed within approved budgets, schedule performance, as identified in previous MPRs, continues to be the key issue for both the DMO and Defence.

Table 2.1 – Project Total Budget Status

Table 2.2 – Project In–Year Budget Status

Table 2.3 – Project Schedule Status

Table 2.4 – Project Characteristics

Budget performance

2.4 Project budgets, against which cost performance is measured, are subject to variations arising from exchange rate variations, Government approval of changes in scope and transfers to Defence Groups. Following the move to out-turned project budgets in July 2010, there are no further variations to project budgets as a result of inflationary effects.

2.5 Table 2.1 includes a summary of the project budget variations from date of Government approval to 30 June 2014. Only one significant real variation occurred during 2013-14:

• Joint Strike Fighter: budget increase of $10.5b due to Government approval to acquire an additional 58 F-35 Joint Strike Fighter aircraft.

2.6 Figure 2.1 provides a comparison of expenditure as at 30 June 2014 compared to the total approved budget at that date. While this provides an indicator of project progress, the percentage of budget spent is dependent on the nature of the project and the level of early investment that may be required for project start-up and non-recurring engineering effort.

2.7 Importantly, no project has exceeded its total approved budget and the net variation to the original Government approvals is negative 1.3 per cent, reflecting accumulated real savings to date of $533 million.

Figure 2.1: Comparison of Overall Project Budget and Expenditure to 30 June 2014 (in $m)

 

 

2.8 The relationship between project expenditure and project progress is not necessarily linear. The profile of expenditure against total approved budget is determined by several factors including the level of development required for the acquisition. For example, a project with a low level of development acquiring a Commercial or Military Off-The-Shelf (COTS or MOTS) product will generally have an expenditure pattern closely matched to the production of the materiel elements. In comparison, a highly developmental project usually requires a degree of initial ‘seed capital’ on commencement with expenditure declining during the development phase and increasing as the project shifts into the build/integration phase.

2.9 Another factor is the evolution of the project and its performance to date. A project may be well advanced in years but show a low level of expenditure against the total budget. This may result from poor contractual performance culminating in withholding of payments against specific milestones. This is, in effect, a deferral of payments that will be re-instated upon contractor achievement of milestones.

2.10 Table 2.2 provides a summary of the in-year spend against budget for 2013-14. Figure 2.2 compares in-year expenditure at 30 June 2014 with budget for 2013-14. Most projects have expended close to their given budget with minor variations attributed primarily to exchange rates or rounding issues.

• One project had a significant overachievement of 10.3 percent:
  • HF Modernisation – The overachievement was as a result of being able to bring forward an IT Refresh milestone.
• Five projects had significant underspends:
  • FFG Upgrade – The underspend is due to systems costing less than originally budgeted for, delay in invoicing due to technical difficulties and spares being paid for from sustainment budget.
  • Next Gen Satellite – The underspend is due to a delay in billing by the USA and realisation of cost savings in production under the Wideband Global Satellite Program. The DMO process underspend relates to activities that have been moved to 2014-15.
  • Hw Torpedo – The underspend is primarily due to the postponement of Pre-Full Cycle Docking work on HMA Ship Collins.
  • SM-2 Missile – The underspend is primarily attributed to delayed contracted scope with Australian and Foreign Industry and subsequent savings as well as deferred Foreign Military Sales (FMS) payments. Establishment of In-Service arrangements has not proceeded as quickly as planned.
  • Battle Comm. Sys. – The underspend is primarily due to a number of supplies and quotes that are: no longer required; have been transferred to sustainment; or have taken longer than anticipated to approve.

Figure 2.2: Comparison of In-year Project Budget and Expenditure at 30 June 2014 (in $m)

 

 

Contingency Management

2.11 In keeping with standard commercial practice, budgets for major Defence capital investment programs are approved by Government with a contingency provision that varies between projects depending on the complexity and risk of the acquisition. This allows Project Managers to proactively manage risk, and, when necessary, treat risk events that have materialised into issues. Contingency funding provides a limited financial margin for Project Managers against inherent uncertainties, risks and unexpected events that may arise during the course of a project. It is especially important in Defence projects that typically have greater inherent risk, longer timeframes and are generally more complex than commercial projects.

2.12 Previous reports have included the aggregate amount of contingency that has been spent over the life of the relevant MPR projects. This year each PDSS includes advice on whether contingency has been applied to the project during the financial year. The application of contingency is assured by the ANAO. Consistent with this, the aggregate amount of contingency reported over the life of the relevant MPR projects in paragraph 2.13 is the aggregate amount of applied contingency rather than contingency actually spent. The term ‘applied contingency’ is the amount of contingency that the project has been authorised to expend to either manage, treat or retire project risks. It is a combination of both the amount of contingency that has already been spent and the amount that is expected to be spent.

2.13 Across the life of the 30 projects in this year’s MPR (that is, from August 1996 to June 2014), the aggregate amount of applied contingency is approximately $1.9b (or 3 per cent of the total of approved project budgets of $59.4b). The areas where risk has been retired using contingency include:

• Systems development;
• Systems integration;
• Logistics and Support;
• Schedule constraints; and
• Project resourcing.

2.14 The DMO has established a framework which provides full accountability and traceability of all management decisions related to the use of the project contingency budget.

Schedule Performance

2.15 In 2010, Defence introduced Initial Materiel Release (IMR) and Final Materiel Release (FMR) as more appropriate milestones for measuring the DMO’s acquisition performance in contributing to the Capability Managers’ coordination of all capability elements to achieve Initial and Final Operational Capability (IOC and FOC) milestones. Schedule analysis presented in previous MPRs was based on achievement, or expected achievement, of FOC.

2.16 Figure 2.3 represents the schedule performance for each of the projects covered in this year’s MPR. The chart shows the original project approval date, the originally approved FMR estimate and the forecast FMR as at 30 June 2014.

2.17 The chart also highlights the time it takes to deliver complex Defence acquisition projects. Five projects commenced prior to 2000: ARH Tiger Helicopters; Hornet Upgrade; FFG Upgrade; Bushmaster Vehicles; and HF Modernisation. A further seven projects have commenced just over a decade ago.

2.18 There are four projects: ARH Tiger Helicopters, Next Gen Satellite, Stand Off Weapon, and 155mm Howitzer, that achieved FMR during the 201314 financial year. Another four are scheduled to achieve FMR during 2014-15: Wedgetail, Battle Comm. Sys., UHF SATCOM, and SM-2 Missile.

Figure 2.3: Schedule from Approval to Current FMR Estimate

 

 

2.19 The period between FMR and FOC dates indicates the time required by Capability Managers to bring together all the other Fundamental Inputs to Capability. This usually occurs after the DMO has delivered the final materiel elements of capability¹⁸⁵. Figure 2.4 provides an indication of the estimated time required to complete this work.

2.20 One project in the MPR - Stand Off Weapon - achieved FOC during 2013-14; another six are scheduled to achieve FOC during 2014-15. Those six projects are: Wedgetail, Hornet Upgrade, Next Gen Satellite, SM-2 Missile, 155mm Howitzer, and Battle Comm. Sys. The declaration of FOC marks the formal transition of capability from the acquisition to the sustainment phase of the capability life cycle at that time.

Figure 2.4: Current FMR and FOC Estimates

 

 

In-year Final Materiel Release Schedule Variance

2.21 Figure 2.5 illustrates the in-year schedule variance for FMR as a percentage of the total time expected to reach the FMR milestone. In the 2013-14 MPR, 14 projects re-assessed their estimated FMR date.

2.22 The largest individual in-year variance to forecast FMR dates is 34 per cent for Battle Comm. Sys., reflecting an anticipated delay of 22 months to align installation of equipment with Capability Manager priorities. The MRH90 Helicopters has the second largest in-year FMR variance of 31 per cent. This is attributed to a number of technical and reliability issues.

2.23 Three projects are forecasting to be ahead of their original FMR estimates: Joint Strike Fighter - two percent, the Overlander Medium/Heavy - four percent and the LHD Landing Craft - nine percent. Collins RCS also improved its FMR forecast by three percent (more details are available in the Project Data Summary Sheets for each project).

Figure 2.5: In-year FMR Schedule Variance, by Project Type

 

 

Total Final Materiel Release Schedule Variance

2.24 Figure 2.6 shows the variance to forecast FMR dates, as a percentage of originally estimated duration, for each of the 30 projects in the MPR. At 30 June 2014, the average FMR variance for all of the MPR projects was 11 per cent.

2.25 Three projects are forecasting early achievement of FMR – Joint Strike Fighter, Overlander Medium/Heavy, and LHD Landing Craft.

2.26 Eight projects reported an FMR schedule variation of 20 per cent or greater across the life of the project. The largest individual variance to forecast FMR dates for the life of the project is 54 per cent for Battle Comm. Sys. reflecting a delay of 22 months due to the need to align installation of equipment with Capability Manager priorities. Hw Torpedo has a variance of 38 per cent due to FMR date being set before the Full Cycle Docking program had reached maturity in terms of the length of dockings and impact of emergent work. As a result, the installation schedule has been delayed.

2.27 FFG Upgrade has a variance of 32 per cent, across the life of the project, due to the extended implementation period required for the Torpedo Decoy System. The majority of the upgraded FFG capability has been delivered and employed by FFGs on operations for several years.

Figure 2.6: Total Schedule Variance for FMR, by Project Type

 

 

In-year Final Operational Capability Schedule Variance

2.28 Figure 2.7 shows in-year schedule variance to forecast FOC dates, as a percentage of the 2012-13 MPR project duration estimate. In the 2013-14 MPR, 14 projects re-assessed their forecast FOC date. Key points to note are as follows:

• The largest in-year FOC schedule variances were recorded by Hw Torpedo, MRH90 Helicopters and Battle Comm. Sys. projects which had in-year schedule variances exceeding 20 per cent. The Hw Torpedo variance is attributed to restricted platform availability resulting from the Full Cycle Docking Program and the installation completion dates vary according to boat availability. MRH90 Helicopters and Battle Comm. Sys. variances are discussed in paragraph 2.31.
• Joint Strike Fighter, Overlander Medium/Heavy and LHD Landing Craft projects gained some schedule and are about two, eleven and five months ahead of their original FOC estimates, respectively.

Figure 2.7: In-year FOC Schedule Variance, by Project Type

 

 

Note: Joint Strike Fighter is currently developmental in nature but should ultimately become MOTS when it enters production line delivery.

Total Final Operational Capability Schedule Variance

2.29 The average variance to forecast FOC dates, as a percentage of originally estimated duration, for the 30 projects in MPR 2013-14 was 35.8 per cent at 30 June 2014.

2.30 Figure 2.8 charts the schedule variance as a percentage of the originally estimated project duration from Government approval to FOC. The projects are grouped in the categories of MOTS, Australianised MOTS (AMOTS) and Developmental. The chart shows that, generally, MOTS projects are more likely to be delivered on time while AMOTS and Developmental acquisitions are more prone to underestimating technical complexity and systems integration effort.

2.31 The number of projects reporting an FOC schedule variation of 50 per cent or greater has increased from 10 to 12. Delays to FOC over the life of these projects reported are as follows:

• MRH90 Helicopters delays are due to a number of technical and reliability issues.
• ARH Tiger Helicopter project delays are a result of the reduction in the flying Rate of Effort experienced by ARH fleet.
• The two Collins projects – RCS and R&S – have been negatively impacted by limited platform availability due to operational requirements arising from the implementation of the Coles Review and unscheduled maintenance requirements having a detrimental impact on the Full Cycle Docking program.
• Deferral of FOC for the Anzac Anti-Ship Missile Defence program (Phases 2A and 2B) resulted from a Government approved change of project delivery strategy and scope, including the decision to substitute the Very Short Range Air Defence System option with a phased array radar capability. This is a significant capability advantage over the originally approved scope and allowed leading edge Australian technology to be proven in one ship before committing to the upgrade of the remaining seven ships.
• The Stand Off Weapon project incurred delivery delays caused by a fuze reliability issue.
• Battle Comm. Sys. delayed the FOC date to align installation of equipment with Capability Manager priorities.
• Wedgetail, FFG Upgrade and HF Modernisation project delays are fundamentally due to an initial underestimation of project complexity. More specifically, the FFG Upgrade issues concerned the complexity of large scale platform integration efforts; HF Modernisation underestimated the level of effort required to deliver the system; and Wedgetail has faced difficulties integrating the phased array radar and other mission critical elements into an operational system. Most of the capability required of HF Modernisation has been delivered and has been in operation for ten years.
• Air to Air Refuel delays are mainly due to development and improvements to the avionics and the air refuelling boom systems required to meet full capability.

Figure 2.8: Schedule Variance for FOC since Government Approval, by Project Type

 

 

Note: Joint Strike Fighter is currently developmental in nature but should ultimately become MOTS when it enters production line delivery

Effect of Kinnaird Reforms on Schedule

2.32 Implementation of the 2003 Kinnaird reforms has delivered a marked improvement in schedule performance for Defence capability acquisition projects. As shown in Table 2.5, average schedule variance to FOC for pre-Kinnaird MPR projects is 71 per cent, while the average variance for post-Kinnaird projects is 15 per cent. This is indicative of a general shift to more MOTS rather than developmental projects, as well as the age and stage of the post-Kinnaird projects.

Table 2.5 – Pre and Post Kinnaird Schedule Variance

Schedule Variance Attribution

2.33 Schedule slippage for developmental and Australianised MOTS projects is attributable to the higher levels of technical complexity and system integration risk. The higher the technical challenge, the higher the inherent risk to the schedule. Developmental projects are new and often at the leading edge of available technology. As such, it is not uncommon for projects, as they progress, to encounter unforeseen technical difficulties requiring significant modification that results in delay. Similarly, Australianisation can also encounter unforeseen technical difficulties and have an unanticipated impact on existing features of the baseline MOTS product. Rectifications of such issues often require extensive, time consuming remediation work.

2.34 Further analysis of 2013-14 MPR data has revealed additional drivers of schedule delays as shown in Table 2.6. Of the 10 projects that underestimated technical complexity, five could also identify some overestimation of industry capability and performance, and another two were also affected by approved changes to scope (Anzac ASMD Phases 2A/2B attained Government approval to acquire the more capable phased array radar based solution over the original very short range air defence system solution, though without an increase to schedule). In September 2012, Government announced a re-baselining of the AWD construction schedule¹⁸⁶, and limited platform availability was the primary driver for delays of Collins class submarine projects.

2.35 Some projects also have a greater schedule delay to FOC as a result of Capability Manager decisions tying the project equipment to expected capability outcomes or operational imperatives.

Table 2.6 – Attribution of Schedule Variance Factors

Materiel Scope Performance

2.36 Materiel Scope Performance measures represent the key materiel capability performance attributes of a project delivered by the DMO and industry, which if not satisfied could have a significant detrimental effect on the eventual suitability of equipment for operational service. The Materiel Scope Performance measures for each project, as defined in the Materiel Acquisition Agreement, are identified from the project approval documentation, including the Operational Concept Document and the Function and Performance Specification, that detail the capital equipment assets to be delivered.

2.37 For security classification reasons the MPR does not identify the individual measures for each of the projects. However, each PDSS has a percentage breakdown on how the project is tracking against its particular suite of capability elements.

2.38 The subjective ‘traffic light’ assessment of each element is indicative of:

• Green: a high level of confidence that the capability outcome sought will be met;
• Amber: the capability outcome being under threat but still considered manageable and able to be met; and
• Red: at this stage, the capability outcome is unlikely to be fully met.

2.39 Materiel Capability Performance measures indicate the status of the materiel element of capability against the FMR milestones definition as at 30 June 2014 and are not necessarily indicative of each project’s ultimate ability to deliver the final intended scope. For the 2013-14 DMO MPR, there are 264 individual Materiel Capability elements across the 30 projects with 253 green, 11 amber and no red.

Figure 2.9: Materiel Capability Performance measures for the 2013-14 Report

 

 

Figure 2.10: Trends for Materiel Capability Performance measures

 

 

2.40 Figure 2.9 indicates that a high percentage (96 per cent) of Materiel Capability Performance measures are currently reported as green.

2.41 For the four per cent reported as amber, the reasons are:

• Joint Strike Fighter – The project assesses that Phase 2A & 2B (Combined Stage 1 and 2) will deliver its materiel requirements, noting there are a number of risks to achieving some of the materiel capabilities required to deliver IOC and FOC. These risks include: integration of JSF into the ADF system; final software builds meeting required functionality by IOC and FOC; and establishing the sustainment capability;
• Wedgetail – Performance deficiencies in some elements of the mission system currently pose risk to full achievement of capability. All deficiencies are expected to be addressed under remediation programs agreed by Boeing in accordance with the November 2012 commercial settlement;
• MRH90 Helicopters – There are some materiel capabilities that are undergoing Operational Test and Evaluation (OT&E) activities and working towards Service Release. There are associated risks and issues in achieving Service Release of these capabilities, however the project is actively managing these risks;
• Air to Air Refuel – There is a risk to meeting all materiel capability requirements for FOC due to acceptance of aircraft in an initial configuration and the risk of successful remediation of all non-conformances and improvements to the Air Refuelling Boom System;
• FFG Upgrade – Maritime Interception and Anti Ship Missile Defence operational releases have been achieved, however Underwater Warfare Systems remain an area of risk. Navy has assessed a Defence Science and Technology Organisation developed passive sonar system in operation with the Albatross Towed Array as meeting Towed Decoy System requirements; however, production/acquisition is yet to progress. A contract with Thales Underwater Systems will be signed in July 2014 for acquisition and installation. The Capability Manager reached a decision on 10 June 2014 to support Mine and Obstacle Avoidance System on FFGs as an obstacle avoidance sonar.

2.42 Figure 2.10 shows previous MPRs’ Capability Performance Measures, as at 30 June for the respective years. Whilst the number of performance measures has increased from 138 in 2009-10 to 264 in 2013-14, analysis of these measures indicates that the DMO’s performance has improved over the last five years, with the majority of current performance measures likely to be achieved.

Appendix 1: Categorising Acquisitions

The DMO categorises its acquisition projects to enable it to differentiate between the complexities of business undertakings, focus management attention, provide a basis for professionalising its workforce and facilitate strategic workforce planning. Projects are graded into one of four Acquisition Categories (ACATs) which are as follows:

• ACAT I – These are major capital equipment acquisitions that are normally the ADF’s most strategically significant. They are characterised by extensive project and schedule management complexity and very high levels of technical difficulty, operating, support and commercial arrangements;
• ACAT II – These are major capital equipment acquisitions that are strategically significant. They are characterised by significant project and schedule management and high levels of technical difficulty, operating, support arrangements and commercial arrangements;
• ACAT III – These are major or minor capital equipment acquisitions that have a moderate strategic significance to the ADF. They are characterised by the application of traditional project and schedule management techniques and moderate levels of technical difficulty operating, support arrangements and commercial arrangements; and
• ACAT IV – These are major or minor capital equipment acquisitions that have a lower level of strategic significance to the ADF. They are characterised by traditional project and schedule management requirements and lower levels of technical difficulty, operating, support and commercial arrangements.

As the complexity of a project will vary over its life cycle, Defence reviews project acquisition categories at defined milestones between entry into the Defence Capability Plan and project completion.

The ACAT framework provides a recognised, consistent and repeatable methodology for categorising projects and aligning Project Managers’ certified experience and competencies to the complexity and scale of projects under management. Project Managers are assigned to acquisition projects on the basis that their Certified Professional Project Manager status is consistent with the project’s ACAT level.

The ACAT level of a project is assessed against six project attributes:

1. acquisition cost: the approved budget for the project;
2. project management complexity: the complexity of project management necessary for its execution;
3.  schedule complexity: the inherent complexity brought about by delivery pressures on the project;
4. technical difficulty: the complexities associated with technical undertakings such as design and development, assembly, integration, test and acceptance;
5. operation and support: the complexity associated with readying the organisation and environment in which the system will be operated, supported and sustained; and
6. commercial: the readiness and capability of industry to develop, produce and support the required capability, and the complexity of the commercial arrangements being managed.

Appendix 2: Project Maturity Scores – Monitoring Progress

The DMO’s Project Maturity Score quantifies the maturity of a project by way of a score based on the Project Managers’ judgement at defined milestones in its capability development and acquisition phases. This score is then compared against an ideal or benchmark score for that milestone. A project’s maturity is assessed on 16 milestones across its lifecycle and for each of these milestones the ideal or benchmark condition is represented by a benchmark score as shown in Figure 1.

Figure 1 - Benchmark Maturity Scores

 

 

The Project Maturity Score comprises a matrix of seven attributes:

• Schedule;
• Cost;
• Requirement;
• Technical Understanding;
• Technical Difficulty;
• Commercial; and
• Operations and Support.

The Project Manager assesses the level of maturity that a project reaches at a particular milestone for each of these attributes on a scale of 1 to 10. Score assessment is made by selecting the most appropriate description that fits the question under the attributes columns. Project Maturity Scores provide a means of communicating in a simple fashion an indicative ‘as is’ versus a ’should be’ condition to inform decision making for each project. The scores are not precise and are not intended to enable exact comparisons across projects. Following is a description of the Project Maturity Score Attributes.

Project Maturity Score Attributes

 

 

 

 

Appendix 3: Lessons Learned

The Joint Committee of Public Accounts and Audit recommended in Report 422: Review of the 2009-10 Defence Materiel Organisation Major Projects Report, that the DMO include a Lessons Learned section in the MPR for projects that have met the exit criteria to report on their lessons learned at both the project level and the whole of organisation level.

Table 1 lists their lessons at the project level, against a whole of organisation level category.

Table 1: Lessons Learned – Project Level

Lessons Learned – Whole of Organisation Level

The 2013-14 MPR builds on the lessons learned (at the organisational level) reported in last year’s report. Set out below is a summary of progress against the categories of lessons learned.

Military Off-The-Shelf Equipment

The lessons learned in relation to Military Off-The-Shelf equipment are incorporated in DMO policies.

Schedule Management

The lessons learned in relation to Schedule Management are incorporated in Chapter 1, Schedule Management, 1.17, pages 114-115.

Requirements Management

To address issues associated with understanding the level of maturity of requirements and progression through the various systems engineering reviews, the standard procedure covering Function and Performance Specification (FPS) development was amended to provide further guidance on the maturity levels for the FPS when it is progressing through the capability development process. Additionally, policy, guidance handbook and a standard procedure was promulgated to assist with assessing the suitability and risks for system reviews conducted in acquisition.

To improve the quality of objective evidence, or data, relating to acquiring and sustaining materiel systems, emphasis has been applied in the areas of configuration management and Verification & Validation (V&V). Improvements in the V&V policy, handbook and a standard procedure were issued to provide a more robust approach in utilising risk-based methodology to ensure that the necessary objective evidence is obtained. To reinforce the data quality of materiel systems an overarching configuration management policy has been promulgated to ensure reliable materiel system status throughout acquisition and sustainment.

Appendix 4: Glossary