Introduction

1.1 Defence established its Projects of Concern regime to manage the remediation of underperforming projects. The objective of the regime is:

to remediate these projects by implementing an agreed plan to resolve any significant commercial, technical, cost and/or schedule difficulties. Projects of concern receive targeted senior management attention and must be reported on more regularly to the Government.⁵

1.2 The Projects of Concern process is essentially a risk identification and management process which is intended to escalate troubled projects to the attention of senior managers and ministers.⁶ Defence states:

The Projects of Concern regime is a proven process for managing underperforming capability projects at a senior level. Once a project is listed as a Project of Concern, the primary objective of the regime is to remediate the project by implementing an agreed plan to resolve any significant commercial, technical, cost and/or schedule difficulties. Projects of Concern receive targeted senior management attention and must be reported regularly to the government.⁷

1.3 The Projects of Concern regime is managed by Defence’s Capability Acquisition and Sustainment Group (CASG which was formally known as the Defence Materiel Organisation). As of 30 September 2018, CASG had responsibility for the management of 117 major acquisition projects and 110 sustainment projects.

The Projects of Concern list

1.4 Since 2008, when the Projects of Concerns list was made public⁸, there have been 25 Projects of Concern in total. The list has included some of the most significant capital equipment projects and sustainment products in the portfolio, such as the Air Warfare Destroyer build, the Multi-Role Helicopter (MRH90) acquisition and the sustainment of the Collins Class Submarine fleet.⁹ As of December 2018, there were two projects on the Projects of Concern list — the Multi-Role Helicopter (MRH90) acquisition and the deployable air traffic management system.

1.5 Of the 25 projects listed as Projects of Concern since 2008, Defence has cancelled two of these projects and returned most of the remainder to normal management arrangements. Thirteen are reported to have reached Final Operational Capability.¹⁰ A complete list of projects that have appeared on the list since 2008 is set out in Table 1.1.

Table 1.1: Projects of Concern, from 2008 forward

Note a: In December 2007, Defence advised the incoming Minister for Defence that these ten projects — among some 215 projects then being managed within Defence by the Defence Materiel Organisation — were considered by Defence to be Projects of Concern. AIR 87 — ARH Tiger Armed Reconnaissance Helicopter and AIR 5077 Phase 3 — Wedgetail, Airborne Early Warning and Control Aircraft are known to have been on the Projects of Concern list from at least 2002.

Note b: These six are among the 27 projects reviewed in the 2016–17 Major Projects Report (Auditor-General Report No. 26 2017–18, 2016–17 Major Projects Report). The Project Data Summary Sheets in that report (p. 119 forward) give extensive detail on each project. Many of the projects that have been on the Projects of Concern list have been the subject of individual Auditor-General performance audits, including: Auditor-General Report No.26 2015–16, Defence’s Management of the Mulwala Propellant Facility; Auditor-General Report No.52 2013–14, Multi-Role Helicopter Program; Auditor-General Report No.22 2013–14 Air Warfare Destroyer Program, and Auditor-General Report No.23 2008–09, Management of the Collins-class Operations Sustainment.

Note c: These projects have been reviewed in earlier years’ editions of the Major Projects Report.

Note: No project which has been removed from the Projects of Concern list has returned to the list later. Defence advises that it has no rule, written or unwritten, prohibiting such an occurrence: it has simply not occurred.

Source: ANAO analysis of various Defence documents and ministerial press releases. Final Operational Capability (FOC) dates advised by Defence.

Length of time on the Projects of Concern list

1.6 Entry to the Projects of Concern list, and exit from it, is decided by ministers. The duration the 25 projects have spent on the Projects of Concern list differs widely, from two months (Super Seasprite helicopter) to eight years and 11 months (Collins Class submarine sustainment). The elapsed time is illustrated in Figure 1.1.

1.7 On the face of it, the mean elapsed time on the list of this set of projects is three years and two months. However, it should be borne in mind that:

• a range of projects had already been recognised by Defence as Projects of Concern before January 2008. For example, the Super Seasprite had been described internally as a Project of Concern in 2002; the Lightweight Torpedo replacement project had been described in this way in 2004; and the M113 Armoured Personnel Carrier went on the list in 2006. It is not known how long other projects had been there that were identified as being on the list at the start of 2008;
• Collins Class Submarine Sustainment activity had been considered a Project of Concern earlier than November 2008, but that is the effective date accepted by Defence;
• two projects remained on the list at the time this report was being prepared and their completed duration on the list will remain unknown until they are removed; and
• the audit has identified instances of projects whose entry onto the list has apparently been delayed for various reasons (see Chapter 3).

1.8 Since 2008, the number of projects on the Projects of Concern list at any one time reached a maximum of 12 during 2008–09 and 2010–11, but has a declining trend, overall, from mid-2011 to the present (Figure 1.2 on the following page). Defence has advised the ANAO that this reflects the nature of the equipment program following the changes introduced by the Kinnaird and subsequent reviews of Defence procurement.¹¹ These reviews led to a greater priority being given to military-off-the-shelf (MOTS) acquisitions and less on developmental acquisition projects.¹²

Figure 1.2: Number of concurrent Projects of Concern, by month, January 2008–December 2018

 

 

Source: ANAO analysis of Defence records.

1.9 The Projects of Concern list attracts a profile in Parliamentary consideration of Defence matters and in media discussion. For most of the history of Projects of Concern, Defence has used specific criteria to provide a basis to recommend that a project be placed on the list. The approach was then enhanced with further mechanisms to invigorate and formalise the process in mid-2011. From 2017, a set of (unpublished) principles has been followed rather than specific criteria (see Appendix 2).

1.10 Defence has also maintained separate Projects of Interest and Sustainment Products of Interest lists in various forms since around 2005 (see Appendix 3).¹³ A Project or Product of Interest is one which Defence management considers to be underperforming and in need of senior management attention and close monitoring to prevent it deteriorating to the point of becoming a Project or Product of Concern. The Projects of Interest and Products of Interest lists are not published by Defence.

1.11 Projects of Concern and Projects of Interest form only a small proportion of the capital acquisition projects that Defence has under management. Defence reported that it had 198 major and minor capital projects under way as at 30 June 2018.

Audit approach

Rationale for undertaking the audit

1.12 The reason for undertaking the audit is that Projects of Concern include projects that contribute substantial capability to the Australian Defence Force and involve a major resource commitment by the Australian Government. As a mechanism for resolving difficulties with Defence projects, there is a clear link between the effectiveness of the Projects of Concern regime and Defence’s strategic priorities as stated in one of its purposes under the Public Governance, Performance and Accountability Act 2013 (PGPA Act): ‘Deliver and sustain Defence capability and conduct operations’.¹⁴ Further, the Projects of Concern regime regularly receives Parliamentary attention and this audit is intended to provide insight into how Defence operates and manages the Projects of Concern regime.

Audit objective, criteria and scope

1.13 The objective of the audit is to assess whether Defence’s Projects of Concern regime is effective in managing the recovery of underperforming projects.

1.14 The following high-level criteria were adopted for the audit:

• Defence has established an appropriate framework for the Projects of Concern regime, including processes for the entry, management and exit of projects.
• Defence applies the Projects of Concern regime with an appropriate degree of consistency.
• Defence has established appropriate internal and external reporting arrangements on the progress of Projects of Concern.
• Defence can demonstrate that the Projects of Concern regime contributes materially to the recovery of underperforming projects and products.

Audit method

1.15 The audit was conducted by:

• examining the frameworks used by Defence for identifying projects that require additional management attention as Projects of Concern;
• examining documentation on the management, implementation, monitoring and reporting of Defence projects, scrutinising, in particular, the records of Defence Gate Reviews and Independent Assurance Reviews;
• examining three prominent case studies, of which two comprise projects that have recently been removed from the Projects of Concern list and another which is expected to be removed within a year or so; and
• interviewing key Defence officials and representatives of the defence industry. Contributions were also received from an Australian industry peak representative body and a representative of a major supplier organisation.

1.16 This audit was conducted in accordance with the ANAO Auditing Standards at a cost to the ANAO of approximately $398,000.

1.17 The team members were David Rowlands, Natalie Whitely, Sonia Pragt and Sally Ramsey.

Does the Projects of Concern regime have a clear purpose and scope?

Purpose

2.1 The Projects of Concern regime has two purposes:

• to ensure that projects in serious difficulty attract senior (including ministerial) attention with a view to resolution through remediation or cancellation. The successful recovery of the project could involve a change in scope, cost or schedule¹⁵; and
• to keep ministers informed both of projects in difficulty and their progress to resolution.

Drawing underperforming projects to the attention of decision-makers

2.2 Under the PGPA Act, the Secretary of Defence has a duty to keep the responsible minister informed of any significant issue that has affected Defence.¹⁶ This is a key accountability relationship between a department and its minister. Making and keeping ministers aware of problematic projects is also a prerequisite to their involvement in resolving difficulties for the project. The first stated purpose for the Projects of Concern regime is to ensure that projects in difficulty receive ministerial and senior management attention within Defence and engagement at the most senior levels of the vendor of the project.

2.3 Further, by making the project’s difficulty known to the public, Defence expects that commercial pressure on the vendor will facilitate action on the vendor’s part that will lead to or help substantially with remediation. Defence has put the view that it has ‘few better mechanisms for influencing commercial behaviour’ than the Projects of Concern list, including the threat to a vendor’s reputation of being put on it and the possibility of being excluded from future tenders.¹⁷ There is an underlying presumption in this mechanism that vendor performance is the aspect of the project most in need of attention. Only projects where an opportunity for improvement could flow from applying pressure to industry are regarded as potential candidates for nomination to the list.

Timely consideration of underperforming projects is important

2.4 Timely consideration of underperforming projects by decision-makers is important, particularly if cancellation may be an appropriate outcome where an underperforming project is unlikely to provide good value for money or there is a substantial risk of a waste of public resources. For example, the Super Seasprite helicopter project, a Project of Concern that was subsequently cancelled, cost over $1.4 billion and yielded no capability whatsoever.¹⁸

2.5 Delay in making the cancellation decision — or failure to make that decision — raises the risk of incurring additional sunk costs, as explained by a Senate Committee inquiry into Defence procurement:

a delayed or unsuccessful project creates a capability gap, fails to meet the government’s strategic requirements, damages Defence’s relationship with industry and undermines public and parliamentary confidence in Defence’s procurement program.¹⁹

2.6 Delaying the entry of projects onto the list, as noted in Chapter 3, may undermine the purpose of the list and result in wasted public resources if the project must subsequently be cancelled.

Keeping ministers informed

2.7 The Minister for Defence set out an additional purpose for Projects of Concern in 2010:

[The Minister for Defence Materiel] and I will announce later today that project AIR 5418, the acquisition of the Joint Air-to-Surface Standoff Missile (JASSM), has been added to the Projects of Concern list. This listing is not primarily because of industry delays or cost increases. It is because of our poor management, our failure to keep Government properly and fully informed about the Project and its difficulties.²⁰

2.8 The Minister raised this matter with Defence on numerous occasions.²¹ Defence records show that Defence did not provide full and frank information in a timely manner on at least one occasion where a classified project with an approved $50 million facilities component had suffered a ‘cost blow-out’ to $150 million in those costs. After being aware of this for several months the relevant Defence groups could not agree to advise the Minister.²² Further, Defence records indicate concerns about providing frank assessments of Projects of Concern to the Minister, in particular, if that information might then also be available to the Auditor-General.²³

2.9 In direct response to this finding, Defence advised the ANAO in February 2019 that ‘Defence is of the view that anything pre-2015 is of limited value due to the significant shift in the nature of the relationship with Industry and the Defence Operating Model as a result of the First Principles Review’.

Scope

2.10 Of the 25 projects listed as Projects of Concern between 2008 and 2018, 24 have been projects to develop and acquire military capability (acquisition projects).²⁴ In addition, one ‘product’ under sustainment (Collins Class Submarine sustainment) has been listed.

2.11 Projects have generally been listed after ‘Second Pass’ approval of the project by government, where the vendor has been selected and contracts are in place or nearly so. In two cases, acquisition projects have entered the list before contracts have been signed with vendors. These have been AIR 5431 Phase 3, the Civil and Military Air Traffic System (CMATS), and LAND 40 Phase 2, Direct Fire Support Weapons.²⁵

Performance Exchange Program

2.12 Since 2001, Defence has monitored and reported on the performance of major Defence contractors for capital acquisition and in-service support. Previously known as the ‘Company Scorecard Program’ and now known as the ‘Performance Exchange Program’, the program encompasses contracts that exceed the thresholds of $10 million (acquisition); $5 million (sustainment) and $5 million (services). Defence has advised that, ‘under the Performance Exchange Program, decision-makers across both Defence and industry now have a set of measures that are directly relevant to decision-making and performance across domains, military capabilities and tiers across the Defence industry sector’.²⁶

2.13 The Performance Exchange Program and the Projects of Concern regime have a common objective, better performance, and they have proceeded in parallel for nearly two decades. Defence has raised concerns that inconsistent views can develop between the two processes. Defence has advised that:

… an updated Performance Exchange process has been developed and implemented to address this issue, as well as to deliver broader, and honest, two-way performance reporting between Defence and its contracted key industry partners.²⁷

Has Defence established policy and procedures for the entry, management and exit of projects from the Projects of Concern regime?

2.14 The development of Defence’s approach to managing its Projects of Concern regime is reflected in the statements of processes, criteria and principles set out at various times since the commencement of the regime in 2001 (see Appendix 2):

• In its earliest form (2001), Defence adopted a simple set of ‘hard’ criteria for making a project a Project of Concern, relating to governance, capability, schedule and budget. The criteria at this point were substantially numerical and the regime was largely internal to Defence.
• Ministers became involved in 2008; the subsequent statement of criteria in 2009 moved away from numerical triggers and introduced a greater element of judgement. It introduces procedures for entry to the list, and exit, both by ministerial approval. A requirement for a plan for resolution (through cancellation or remediation) was required from this point.
• The 2011 statement, announced in a press release by the Minister for Defence, introduced substantial additional elements, including incentives for industry to fix problem projects; a new ‘Early Indicators and Warnings’ system to flag potential problem projects; the use of Gate Reviews to analyse each such project; more detailed requirements for remediation plans and greater ministerial involvement through bi-annual Projects of Concern reviews (summits), being face-to-face meetings with the minister ‘to hold responsible individuals to account’.
• In 2017, the ‘Statement of Principles’ introduced an emphasis on a ‘collegiate culture’ and included no reference to incentives for industry. There was no mention of review meetings (summits) with the minister.
• The 2018 principles set out more procedural activity, including reporting to government and Parliamentary committees. The principles again mention summit meetings with the minister but these are to be held ‘when required’.

Ministerial involvement in strengthening the Projects of Concern regime

2.15 In mid-2008, the General Manager, Programs, Defence Materiel Organisation flagged ‘a major shift in thinking’ at ministerial level to his colleagues. He told them that ‘Projects of Concern are a serious issue for Defence and are currently attracting high public and political visibility and could be subject to precipitous decisions’. He stated:

From my meetings with the Parliamentary Secretary and the staff in the Minister’s office it is clear to me there has been a major shift in thinking about these projects. The new default way forward for a “Project of Concern” is to assume that the project will be cancelled unless sufficient justification is provided each month that the project can deliver its scope dependably within any agreed (revised) schedule and funding. Failure to do so will result in the project being recommended for cancellation by the Minister and Parliamentary Secretary. This puts the onus on the project office (assisted by the sponsor and capability manager) to show that they have an acceptable way forward to delivering the approved materiel capability within the agreed schedule, budget and risk profile.²⁸

2.16 With projects identified as Projects of Concern to be subject to ‘intense management’, with a prospect of cancellation, Defence required a remediation plan that set out for each Project of Concern ‘an acceptable way forward to delivering the approved materiel capability within the agreed schedule, budget and risk profile’. Ministers became more involved, including by the Minister for Defence Materiel deciding entry to the list and exit, on Defence’s advice.

2.17 The General Manager emphasised that ‘While cancellation may be a suitable outcome in a small number of cases, in most cases it would be very unacceptable from a capability and reputation point of view’. He also advised Service Chiefs that if a Project of Concern were cancelled, they should not assume there would be a replacement program. He also stated that it was important to reduce the Project of Concern list to zero in the shortest possible time.

2.18 Defence then set up a Projects of Concern unit to report to the Parliamentary Secretary on progress with remediation:

The unit has been tasked with providing assistance and advice to the projects to help them get back on track, while also drawing detailed information from each project and providing it to Government on a monthly basis. This has allowed the Government to become more active in the management of these projects and has informed its decision making when it comes to how we should handle these projects.²⁹

2.19 The process was further strengthened in 2011 with the Minister for Defence engaging with industry on how to improve the process. Industry advice formed the basis for reforms announced on 29 June 2011 by the Minister for Defence and the Minister for Defence Materiel. The reforms, which applied from 1 July 2011³⁰ included:

• incentive arrangements for industry to focus on fixing problems — through weighing a company’s performance in remediation when evaluating tenders for other projects;
• a formal process for adding projects to the list — using the triggering of Early Indicators and Warnings (discussed below) as a basis for advising ministers and potentially leading to a Gate Review;
• a requirement for formal remediation plans — prepared by Defence and industry;
• a process for removing projects from the list — involving government decision; and
• increased ministerial involvement — through bi-annual reviews with Defence and industry.

Early Indicators and Warnings

2.20 Earlier concern that Defence senior management — and ministers — be kept aware of projects in difficulty led Defence to develop a system of mandatory ‘Early Indicators and Warnings’ on project progress. The triggering of one or more of these criteria was expected to lead to a Gate Review and consideration of whether the project should be nominated for inclusion on the Projects of Concern list.³¹

2.21 The Early Indicators and Warnings system emphasised early identification of risks and problems, as the name indicates.³² The Ministers’ announcement stated:

Defence assesses that 80 per cent of problems with Defence capability projects occur in the first 20 per cent of the project’s life.

That is why it is important to pick up problems early.

One of the biggest challenges in Defence procurement is projects running late. The earlier these issues are picked up, the earlier the problem can be fixed.

The Government will implement an Early Indicators and Warning System. This system will help identify and correct potential problems with projects.³³

2.22 Before the 2011 reforms, nomination depended upon project staff identifying that the project had crossed a threshold and/or senior managers in Defence identifying the underperformance from internal reporting. A significant change to the process of nominating projects flowed from the use of Gate Reviews and their successor, Independent Assurance Reviews, to provide a critical review and considered approach. That process continues.³⁴

Operation of the Projects of Concern regime in 2018

2.23 Defence has stated in evidence to the Senate Foreign Affairs, Defence and Trade Legislation Committee that the Projects of Concern regime is a disciplined process, both for entry and exit.³⁵ In December 2018, that process was set out in Defence’s operating principles on its intranet site.³⁶ In contrast to the 2011 principles, Defence does not make the 2018 principles public.

2.24 There has been a change over the years in the character of the criteria used to nominate projects to the Projects of Concern list (see the sets of criteria at various points of time set out in Appendix 2). In the earlier years, entry to the Projects of Concern list was based on a range of ‘hard’ criteria, such as a percentage change in expected cost or schedule being exceeded. A more recent characterisation is that ‘There are no set quantitative measures or thresholds …’ (2017). The rationale has been that the complexities that characterised the projects under consideration cannot be captured in such criteria.

2.25 A significant change is that, whereas the 2011 process included the Minister for Defence Materiel holding bi-annual reviews of Projects of Concern (summits) with Defence and Industry representatives, that element had been omitted from the 2017 edition (see Appendix 2) and referred to in the current principles as occurring ‘when required’. There is also a greater emphasis on collaboration between Defence and industry and less on mechanisms such as providing incentives for industry to fix problem projects. Defence has stated to the ANAO that ‘The main focus is on collaboration to obtain the best possible capability outcomes within the schedule and cost constraints of the program’.³⁷

2.26 In February 2019, Defence advised that project cancellation remains an option:

Placing a project on the PoC list aims to provide additional support to remediate the project and deliver capability and value-for-money outcomes to the Department of Defence. If these outcomes cannot be achieved, project or contract cancellation may be considered.

Industry view

2.27 The ANAO received a submission from the Australian Industry Group, on the Projects of Concern process stating:

Industry feedback noted that the Projects of Concern process can be valuable; however, the current system lacks transparency. Industry comments stated that the criteria used to establish a Project of Concern are unclear, as well as the processes used to manage and resource the project once on the list.³⁸

2.28 There is no public information on either the Defence Internet site or the Defence Annual Report 2017–18 on how the Projects of Concern process operates.³⁹ The Australian Industry Group stated:

Overall, our members’ view is that the Projects of Concern process should have a high degree of visibility and status, a clear and transparent set of criteria and scheme for elevation, and be part of the regular conversation between Defence and industry.

Management of Projects of Concern following the First Principles Review

2.29 Following the First Principles Review, Defence introduced its Integrated Investment Program to govern investment in Defence’s strategic goals. The First Principles Review also led Defence to introduce its new Capability Life Cycle and the notion of ‘capability streams’:

To allow effective high level prioritisation and communication with Government, the Integrated Investment Program is structured into six Capability Streams which were developed through the 2015 Force Structure Review process. Defence capabilities (Programs) were defined and mapped to the Capability Streams.⁴⁰

2.30 The Air Warfare Destroyer build case study undertaken as part of this audit shows that a risk which came to attention during the project — declining shipyard productivity — may now be more significant to the progress of the wider capability stream Maritime and Anti-Submarine Warfare than to the remainder of the Air Warfare Destroyer build project itself.⁴¹

2.31 However, the operation of the Projects of Concern regime is not well-defined in the context of the new Capability Life Cycle. There is no reference to Projects of Concern in: Defence’s Capability Life Cycle — detailed design; Capability Acquisition and Sustainment Group’s Complex Procurement Guide (March 2018); the 2016 Defence White Paper or the accompanying 2016 Integrated Investment Program and 2016 Defence Industry Statement. The absence of clear guidance on the operation of the Projects of Concern regime as part of the risk management framework gives the appearance that the regime is of marginal importance in the context of the Capability Life Cycle.

2.32 As a result of the First Principles Review, a proposal was developed within CASG in early 2017 for the potential extension of the Projects of Concern regime to align it with the One Defence framework. Such an extension could usefully encompass the issue of broader application of risks identified in any particular project to the capability stream of which it forms a part.⁴² In October 2018, Defence indicated that, following a discussion by its Investment Committee in September 2018, it supports expanding the scope of the Projects of Concern regime to encompass all projects managed across Defence, and this would be consistent with the One Defence approach.⁴³

Does Defence apply a consistent process to the entry of projects to the list?

Procedures for recommending a project as a Project of Concern

3.1 From June 2011 forward, the Projects of Concern processes require a recommendation that a project be considered for inclusion on the Projects of Concern list to originate from a Gate Review or Independent Assurance Review.⁴⁴ One of the primary purposes for Independent Assurance Reviews is the early identification of problem projects and sustainment products.

3.2 The existing 19-page formal instruction governing the operation of Independent Assurance Reviews does not instruct an Independent Assurance Review Board to consider whether a project should be nominated for inclusion on the Projects of Concern.⁴⁵ As Projects of Concern are not directly mentioned in the instruction, there is a risk that troubled projects might not be drawn promptly to the attention of senior management for consideration for listing as a Project of Concern. There would be merit in updating instructions for Independent Assurance Reviews to reflect their role in recommending whether a project should be a Project of Concern.

Consideration of projects as Projects of Concern

3.3 From a review of records of 191 Gate Reviews and Independent Assurance Reviews conducted since 2015, the following challenges in the management of entry of projects to the Projects of Concern list were identified:

• in a minority of cases, no record of whether consideration was given by an Independent Assurance Review as to whether a project warranted nomination to the list;
• potentially delayed entry of projects to the list;
• delays in advice to ministers of a project in difficulty;
• restrictions on information available to an Independent Assurance Review;
• deciding not to nominate a project because it might attract additional workload;
• incidental identification of a potential Project of Concern; and
• an inconsistent approach to publicising additions to the Projects of Concern list.

3.4 These instances are discussed below.

Inconsistent consideration as to whether a project warrants nomination

3.5 Of the 191 Gate Review Boards and Independent Assurance Review Boards reviewed, in 144 cases (75 per cent) the Board both considered whether the project should be recommended for Project of Concern status and recorded its view (as outlined in Table 3.1). Of the remaining 47 cases, there were:

• 27 cases where the project was in the early or late stage; and
• 20 cases (10 per cent) with no record of consideration of whether the project should be nominated as a Project of Concern or as a Project of Interest (see paragraph 1.10).

Table 3.1: Gate Review and Independent Assurance Review consideration

Source: ANAO analysis of Defence records

Delayed entry to the Projects of Concern list

3.6 Defence records indicate that there have been occasions when considerable time elapses between the first record of Defence contemplating the nomination of a project to be entered onto the Projects of Concern list and its actual entry. Any unnecessary delay in action leading to the resolution of projects carries a number of risks. Some of these risks are that:

• as Defence is aware (paragraph 2.21), the more advanced the project the greater the costs of any corrective action;
• delays can reduce the capacity of Defence to influence, on terms favourable to the Australian Government, the vendor’s actions in support of problem resolution; and
• delays increase reputational risk associated with more radical options available to Defence (such as project cancellation) and increase the risk of the public resources being wasted as observed, for example, in the Super Seasprite project.⁴⁶

3.7 Instances of lengthy periods, typically two years, elapsing between Defence’s observing problems with a project and the project entering the Projects of Concern list are set out in Box 1.

3.8 In relation to delayed entry of projects onto the list, Defence advised the ANAO in February 2019 that:

It needs to be acknowledged that there is a qualitative/management judgement aspect to [Projects of Concern] that has always existed and always will. Arguably, some projects could have been added sooner, however, Defence officials are always cognisant of the trade-off between allowing a company more time to ‘come good’ and the right time to add to the PoC list, including the cost/benefit analysis. A more detailed analysis would reveal that behind the scenes there is a considerable amount of engagement, negotiation, consultation, expectation setting, between the company (and/or Parent Company), Defence and often the Minister’s office. Usually, the time is related to companies working toward contractual milestones, or complex technical solutions which can be lengthy but have been negotiated in good faith. Defence often needs to wait for the expiry of timeframes (some of which are legally bound) before further firm action is taken. That does not imply that nothing is done, nor that the implication of becoming a project of concern has not been socialised at the senior level with a company.

Delays in advice to ministers

3.9 AIR 5431 Phase 1, mentioned among the case studies above, also represents a delay in Defence drawing the attention of ministers to a project that is underperforming. The November 2015 Gate Review, having observed a 16-month slippage only 11 months into the contract, recommended that government be informed of current project status, as the schedule was unlikely to be recovered.

3.10 A year later, the Defence Agendum paper prepared for the subsequent Independent Assurance Review (November 2016) noted that:

There has not yet been any specific updated advice to Government on the status of this project, despite the Gate Review recommendation last year. DGCP-AF [Director-General, Capability Planning–Air Force] advises that this project is a lower priority, both for Air Force and for Ministers, and that there is not an urgent need for such an update. Government will be advised ‘in due course’, as part of broader advice on the full range of air traffic control projects.⁵⁰

3.11 As noted above, the project was added to the Projects of Concern list a year later, in August 2017, where it remained in December 2018.

Restricting information supplied to Independent Assurance Reviews

3.12 Given the important role of Defence’s Independent Assurance Reviews in identifying problem projects and products, including potential Projects of Concern, it is essential that they be supplied with all the information they need to carry out their work. The Independent Assurance Review conducted for SEA 1180 (Offshore Patrol Vessel) on 17 July 2017 reported as follows:

The team were restricted in being able to provide any meaningful detail, even when there was no suggestion that they provide the outcomes for each tenderer. Consequently, I am not able to provide an informed independent assessment of the deliverability of this project, nor of the appropriateness of the schedule, cost and risk assessments. I do feel that this is a missed opportunity for the project given the breadth of experience of the Board Members. Furthermore, I am not sure of the value of conducting an Independent Assurance Review for Gate 2 projects that have such restrictive information access …

Given the limitation for access to information for reviewers and the restrictions imposed on the project team and the Capability Sponsor’s ability to speak to the specifics of tender responses, particularly as they relate to cost, schedule and risk at the Board Meeting, I am not in a position to advise on the true status of the project, whether the cost estimates, schedule or risk assessments are a sound basis for progressing to Second Pass Approval, nor what more needs to be done to support the Approval Process or to enhance the actual delivery [Emphasis in original].

3.13 The reason for the restriction in supplying information to the Independent Assurance Review is not evident in the documentation. This approach has several consequences:

• such restrictions reduce the value of holding an Independent Assurance Review;
• the approach could keep from senior management risks that the Independent Assurance Review is best placed to identify. Those risks could then develop and crystallise; and
• even if this instance is unusual, it may form an undesirable precedent that, if followed, could lead to failure to detect risks in other projects.

3.14 In response to this finding, Defence advised the ANAO:

This example is isolated and was particularly disappointing…However, it should be noted that there is no identified systemic issue of information being withheld from IAR reviewers.⁵¹

Deciding not to nominate a project because it might attract additional workload

3.15 An Independent Assurance Review for project JP 2008 Phase 5a, UHF SATCOM, took place on 1 August 2017. The Review Board considered that there were several significant issues that put the project in jeopardy. The Board concluded that ‘the Government must be advised about the schedule slippage to Final Operational Capability’.⁵² It expressed concern at the lack of senior executive oversight and found that the budget was a ‘major concern’.

3.16 Nevertheless, the Board’s ‘summary assessment’, considered and approved by senior officers (SES and star-ranking) in Defence, included the following finding:

The Board did discuss the possibility of elevating the project to a Project of Concern from a Project of Interest but agreed that this could create more work and challenges for a Project Office that is already overtasked.⁵³

3.17 If the Projects of Concern regime is working effectively it should not be viewed by senior officers as creating bureaucratic overhead. Rather, it is intended to resolve problems that are proving intractable at lower levels of management and decisions not to employ the regime may result in delays and lost opportunities to progress a troubled project.

Incidental identification of a potential Project of Concern

3.18 An Independent Assurance Review of LAND 2072 Phase 2B, Integrated Tele-communications System — Land, took place in July 2017. The Board found that the project had made good progress, was ahead of schedule and the project team was handling issues professionally. Nevertheless, there were issues beyond the team’s control that presented contingencies to the project’s success. One of these was the delivery of the next generation of deployable local area network, a project being undertaken as a sustainment item. The Board was advised that the vendor’s performance had been ‘extremely disappointing’.

3.19 The Board concluded:

I am concerned by the eDLAN [enhanced deployable local area network] situation. It is unclear why an undertaking of this obvious complexity is being performed in a sustainment environment with inadequate governance and oversight. HJS [Head, Joint Systems] made an observation that if eDLAN was a Project, it would be ACAT II [Acquisition Category II, the second most significant] and on the current performance it would be a Project of Concern. CASG should review carefully the lessons being learned from the eDLAN experience and should engage with other stakeholders to ensure that follow-on DLAN projects are subject to more appropriate governance and management arrangements.

3.20 Even though there is a precedent for sustainment products being elevated to the Projects of Concern list (for example, CN10, Collins Class Submarine), there is no evidence of any further consideration within Defence of this finding.

Inconsistent approach to publicising additions to the Projects of Concern list

3.21 Defence has indicated that the operation of the Projects of Concern list depends substantially on the commercial pressure on vendors created by the public listing as a Project of Concern of the project they are involved in (paragraph 2.3). However, there have been occasions when a project has been added to the Projects of Concern list but this fact has not been made public, or at least, not until some months later, limiting the effect intended by elevating the project to that status. Instances include:

• JP 2008 Phase 3F, Defence SATCOM Terrestrial Enhancement.
• JP 2088 Phase 1A, Air Drop Rigid Hull Inflatable Boat Trailers.
• AIR 5402, Air-to-Air Refuelling Capability.
• JP 2048 Phase 1A, Landing Platform Amphibious Watercraft.

Reforming risk management practices

3.22 Following a review of the effectiveness of the CASG’s risk management framework, practices, systems and methodologies, Defence approved a Risk management reform program and implementation plan in June 2017 to remodel the Group’s risk management.⁵⁴ This reform program noted that CASG had, at that time:

no standardised approach to risk identification, assessment, risk reporting or risk reduction. [Thirty-seven] different risk management software tools were identified alongside hundreds of different templates, methods and formats for capturing and presenting risk.⁵⁵

3.23 The reform program set out priorities including to ‘develop business rules regarding Projects of Interest and Projects of Concern’. These would:

enhance the means by which Projects of Interest and Projects of Concern are identified, assessed and managed as part of the CASG Risk Management Model. The goal of this effort is to proactively move projects with this status to a level where the risk is in line with the organisational appetite for risk.⁵⁶

3.24 In terms of Projects of Concern, the approved program of work set out the following:

For the Specialist Risk Area of Projects of Interest or Projects of Concern, develop a suitable module that will integrate and connect with the CASG Risk Management Model for this application. This module may include the “Practice Guide” for this Specialist Risk Area (p. 8).⁵⁷

3.25 Subsequently, in August 2017, the Major Programs Control Directorate advised those leading the work on the Reform Program that the governance and processes for Projects of Interest and Projects of Concern already existed and that specific risk management practice was not required. There is no record of any further action having been taken on the planned work.

3.26 A subsequent Defence Internal Audit ‘Management of Selected Projects of Interest’ (April 2018) found that formal documentation of the process for Projects of Interest was not yet complete.⁵⁸ The process relied on ‘professional judgement’. With no standard approach to risk identification, assessment, risk reporting or risk reduction, there can be no assurance that Projects of Concern — or Projects of Interest — are being identified appropriately and in a timely way.

3.27 Regardless of the detailed reasons for the difficulties that arise in any individual case, every project approved by government has a defined estimate of cost and schedule, and is expected to deliver a specified set of capabilities. These will all have been agreed by government and understood at the time of that approval. It should be possible to set tolerances — a risk appetite — which, when exceeded, form a clear and sound basis for concern, and a trigger to alert ministers to emerging risks.

3.28 The Projects of Concern regime could benefit from establishing a simple set of primary entry, management and exit criteria based on cost, schedule and capability, with further technical detail supplied to ministers and senior managers as and when required. These could be embodied in a short, transparent, formally-approved set of procedures which would help provide confidence that the process is being administered rigorously, fairly and consistently.

Does Defence apply a consistent process to the management of projects while on the Projects of Concern list?

3.31 Defence has used three principal processes to manage Projects of Concern while they are on the list: the preparation of remediation plans (commenced in 2009); the holding of regular summit meetings with the minister, attended by Defence officials and vendors (from 2011); and the application of commercial pressure through contract arrangements (from 2011).

Remediation plans

3.32 Since 2009, there has been a requirement that when a project is placed on the Projects of Concern list, a remediation plan would be prepared and agreed between the parties. Remediation plans were to be agreed at, or prepared in light of, Projects of Concern summits. For example, the department advised the Minister for Defence in October 2011 that:

During the recent Projects of Concern summit, 27–28 September 2011, you directed that a Remediation Plan and Action Log be developed by each project in consultation with the companies … Remediation Plans define the agreed strategy to remediate a project and the basis for removing the project from the Projects of Concern list. The Action Log defines the commitments made by the project manager, company CEO and others in each six month period between Projects of Concern summits.⁵⁹

3.33 With an agreed remediation plan, it becomes feasible to measure progress against that plan with the action log. Remediation plans were to include the basis on which a project would be removed from the Projects of Concern list (‘removal criteria’). The prospect of further summit meetings at a regular frequency provides an incentive for participants to discharge their commitments before the next summit.

3.34 Defence’s June 2018 Projects of Concern report provided Defence senior leaders and ministers with:

• a summary of remediation expectations for the short, medium and long term and removal criteria for all three projects on the list, a forecast date for removal and statement of progress towards that; and
• a more detailed description of the expected path towards remediation but not specific remedial actions and accountability. Some of the activities are expressed in general terms such as ‘engage with the vendor’, rather than setting out specific remedial action and who is to do it.

3.35 The audit examined whether remediation plans had been prepared for the three projects selected as case studies (Air Warfare Destroyer, MRH90 helicopters and Collins Class submarine sustainment) and the other projects that remain on the Projects of Concern list or which have been removed over the preceding twelve months (AIR 5431 Phase 3, CMATS; AIR 5431 Phase 1, Deployable Air Traffic Management and Control System; JP 2086 Phase 1, Mulwala Redevelopment Project; and JP 2008 Phase 3F, Defence SATCOM Terrestrial Enhancement).

Remediation plans for case study Projects of Concern

3.36 Remediation plans had been prepared for each of the three projects comprising the major case studies:

• For Air Warfare Destroyer (see Appendix 4, paragraph 19 forward) the remediation plan flowed from the Winter-White Review of the project, commissioned by both Defence and Finance, before the project entered the Projects of Concern list. The project is now expected to be completed with Initial Operating Capability forecast for December 2019. However, a primary basis for the project entering the Projects of Concern list, shipyard productivity, has deteriorated.
• For MRH90 helicopters (see Appendix 5, paragraph 10 forward) the remediation plan was centred on two deeds to the acquisition contract. These, too, were agreed before the project entered the Projects of Concern list. Evolving remediation requirements and action were then discussed at subsequent Projects of Concern summit meetings. Remediation is not complete and the project remains on the list.
• Collins Class submarine sustainment (see Appendix 6, paragraph 9 forward) was declared a Project of Concern before the remediation plan was prepared. The major element of that plan comprised the Coles Review. Remediation has been successful in terms of boat availability (materiel-ready-days) though sustainment costs remain an ongoing concern.

Remediation plans for other projects

3.37 For the other projects that remained on the Projects of Concern list at the time that audit fieldwork was undertaken or had been removed over the preceding twelve months, the approach to preparing remediation plans varied:

• Defence documentation indicates that, whereas no formal remediation plan was agreed with the vendors for the Mulwala Redevelopment Project, Defence was satisfied that the project had successfully completed Defence’s objectives for remediation.
• In the case of the Defence SATCOM Terrestrial Enhancement, a remediation plan was developed by Defence as an agenda item for the summit with the Minister on 20 July 2015. Defence and the vendor continued to engage on the remediation plan thereafter. However, the project was not completed as originally envisaged and a deed of settlement was signed with the vendor in November 2017.
• In the case of CMATS, which entered the Projects of Concern list in July 2017, no remediation plan was prepared upon it becoming a Project of Concern. Following contract execution, CMATS was removed from the list.⁶⁰
• For the Deployable Air Traffic Management and Control System, a remediation plan had been developed in October 2016, before the project entered the Projects of Concern list. However, an Independent Assurance Review in November 2017 found that the remediation plan had not delivered the expected performance and the risk profile was bleak. Defence stated in February 2019 that a further remediation plan had been agreed in-principle on 11 September 2018. However, a contract change plan was yet to be finalised as Defence awaited outstanding documentation from the vendor.

The status of remediation plans

3.38 In October 2018, Defence advised the ANAO that the role of remediation plans set out in paragraph 3.32 no longer describes the process:

Since [the First Principles Review], industry is a Fundamental Input to Capability and remediation plans are agreed by all parties and progress reported against. The main focus is on collaboration to obtain the best possible capability outcomes within the schedule and cost constraints of the program.

3.39 On the current status of remediation plans for Projects of Concern, Defence further advised the ANAO in October 2018 that :

some were drafted during the [2010–13] era and some were drafted for the last full-on summit with [the then Minister] but not endorsed by any of the parties. It is important to keep in mind that PoC is not about introducing additional overhead, it is about getting the appropriate help for the project.⁶¹

3.40 This advice conveys a diminished view or expectation of remediation plans within Defence and leaves their current status uncertain, even though they remain a requirement under the current principles for the Projects of Concern regime.

Summit process

3.41 Under the Projects of Concern processes adopted in 2011, an important element was a six-monthly review or ‘summit’ meeting with the responsible minister. These meetings were held regularly from 2011 through to 2013, usually over a two-day period, and chaired by the responsible minister.⁶² They were attended by Defence officials and senior industry representatives. Their purpose was to address the problems of the Projects of Concern and prepare a remediation plan.

3.42 Typically, these summit meetings took place at Parliament House, Canberra. An hour or more was allocated for each Project of Concern, with successive groups of vendor representatives for respective projects in attendance, each only for the relevant period, along with senior Defence staff responsible for that project. The Chief Executive Officer of the (then) Defence Materiel Organisation stated:

I have observed many cases where significant progress has been achieved in the days leading up to the summit, demonstrating that their frequency is key to sustaining momentum.⁶³

3.43 By late 2014, however, no summit had been held for over a year, which had led, in Defence’s view, to ‘a stagnation of this key element in the Projects of Concern regime’. Defence advised the Minister in November 2014 that:

Should the summits not proceed in the near term, there is a risk that some of the commercial leverage and momentum developed over the past six years would be lost. This is particularly important given the recent addition of the $8 billion Air Warfare Destroyer (AWD) programme to the list. A summit will provide a forum where the relevant senior stakeholders can be brought together to discuss the unique commercial, schedule and cost sensitivities involved, and begin the remediation work at the most senior levels.

3.44 Defence took the view that the Minister’s presence generated gravitas, which was a key element of the success of the program. Defence wished to maintain what it saw as the momentum of the program. A summit meeting was subsequently held, on 20 July 2015.

3.45 Defence has advised that two project-specific summits have been held since that time:

• 12 September 2018 for AIR 5431 Phase 1, Deployable Defence Air Traffic Management and Control System; and
• 16 August 2017 for AIR 5431 Phase 3, Civil Military Air Traffic Management System.⁶⁴

3.46 As noted earlier, the Projects of Concern principles of 2017 (see Appendix 2) and those currently in effect (see paragraph 2.23) do not refer to six-monthly or any other review or summit meetings with the minister. There is no longer an expectation that there be regular summit meetings. Defence has advised that they are ‘held on an “as required” basis, depending on the nature of the issues with the project and the wishes of ministers.’⁶⁵ A submission provided to the ANAO in November 2018 by an industry peak body stated that:

From a practical point of view, one company involved in the process noted that attendance and scheduling of the Projects of Concern meetings tended to be ad hoc. Similarly, the structure/ agenda for the meetings could vary greatly dependent on the Ministers at the time. The most positive meetings were when the Minister personally engaged with all the participants in an open and direct conversation.⁶⁶

Using Project of Concern status to apply commercial pressure

3.47 Under the processes adopted in 2011, for Defence to employ its Projects of Concern regime to best commercial effect, the Department was required to take account of these projects in other dealings with vendors, in particular, ‘a negative weighting of tenders’ when contemplating further acquisitions (see Appendix 2).

3.48 To strengthen the consequences for companies that significantly underperform on Defence contracts, Defence incorporated requirements into its contract (ASDEFCON) templates.⁶⁷ This was intended to place commercial pressure on vendors to rectify projects on the Project of Concern list.

3.49 There are two elements to this mechanism:

• tenders are evaluated based on tender evaluation criteria which include taking account of performance on past and present contracts including those that are, and those that have been, a Project of Concern; and
• the Commonwealth has an express discretionary right to exclude a tender from further consideration during a tender process if that tenderer is materially involved in a Project of Concern.⁶⁸

3.50 These arrangements have been in place since 2011.

3.51 In 2012, Defence wrote to tenderers highlighting these conditions:

Should [Vendor’s name] tender be submitted for [the project], you can expect that a key factor that the Commonwealth will consider in exercising its rights under this clause during the tender process will be the progress [the Vendor] has made in remediating [an earlier project] as a Project of Concern, including the content of and timeliness in agreeing the remediation plan in the first instance, and then the subsequent implementation of the plan.⁶⁹

3.52 Defence records included formal tender evaluation plans that explicitly included consideration of any Project of Concern connections. However, Defence advised the ANAO in July 2018 that it is not aware of any company being excluded from any further tender until a Project of Concern has been remediated.⁷⁰ The audit did identify an occasion when Defence elected not to exclude a vendor from subsequent tenders. This was in relation to the Civil-Military Air Traffic System (CMATS) project where this decision was attributed to the unusual circumstances of the project.⁷¹

Does Defence apply a consistent process to the exit of projects from the Projects of Concern list?

Removing projects requires more than meeting remediation objectives

3.53 In 2010, removal from the Projects of Concern list depended on ‘both a completely clean overall bill of health/risk retirement and achievement of remediation objectives’.⁷² Defence has stated that it still requires a high degree of confidence in the outlook for the project before recommending to ministers that the project be removed from the list. In particular, ministers want to be ‘very much assured that something we have identified as being a project of concern is not going to come back’.⁷³ That is, when making a recommendation to ministers to remove a project from the list, Defence management needs assurance that the project is unlikely to underperform again to the point that Defence will recommend it be returned to the list.

3.54 There is evidence that Defence has sought to keep a project on the Projects of Concern list after all the specifically articulated remediation objectives have been satisfied. In a particular case, CN10 Collins Class submarine sustainment, where all remediation objectives had been met, Defence senior managers took the view that residual risk remained with a remote possibility of the project coming back on the list. As a result, the project remained on the list for reassessment some six months or so later, after which it was removed. No specific milestones were set to form the final basis for removal. Defence has advised:

The removal criteria for CN10 Collins Class Submarine sustainment was consistent achievement of submarine availability in accordance with the international benchmark as per the Coles review. Post all remediation objectives were being met, CN10 remained on the PoC list for another six months to ensure consistent submarine availability was being achieved prior to recommending the removal from the PoC list. This is a good business practice for such a complex capability.⁷⁴

3.55 In practice, under the various versions of the regime, removal of a project from the Projects of Concern list has consistently turned on a management judgement that risks have been sufficiently contained as well as the satisfaction of a specific metric or similar criteria. An instance where this approach has not been observed is the Civil-Military Air Traffic Management System (see Box 2, below).

High element of judgement in selection of projects for removal

3.56 Selection of a project for removal from the Projects of Concern list has not always flowed from any apparent objective criteria, as exemplified by a case in late 2009. In this instance, the decision to recommend removal turned on the desire within Defence to give the vendor (which had other projects on the Projects of Concern list) some relief.

3.57 There is no reason documented by Defence for giving the particular vendor ‘some relief’ nor deliberation on the merits (or otherwise) of doing so in relation to the effectiveness of the Projects of Concern regime. The correspondence identified by the audit indicates a highly discretionary approach. The apparent fluidity with which project status could be varied would likely become apparent to the vendor, and possibly other vendors, thereby risking the undermining of the value of the entire mechanism.

Has Defence reported internally and externally on the operation of the Projects of Concern list?

Internal Reporting

4.1 As well as bimonthly reporting on acquisition projects and sustainment products to the Deputy Secretary, Capability Acquisition and Sustainment Group (CASG), CASG reports to senior Defence leaders and ministers on Projects of Concern through:

• Quarterly reporting to the Secretary and the Chief of the Defence Force through the CASG Quarterly Performance Report.
• Biannual reporting to the CASG Executive Advisory Committee.
• Papers presented to the Defence Audit and Risk Committee.

Quarterly Performance Report

4.2 CASG’s primary method for internal reporting on Projects of Concern is through the CASG Quarterly Performance Report⁷⁷ which includes a status update and a description on Defence’s progress towards remediation for each project on the list. The report includes a ‘traffic light’ reporting system with supporting quantitative and qualitative information.

4.3 The quantitative information reported is drawn from Defence’s information technology reporting systems for acquisition projects and sustainment products — the Monthly Reporting System and the Sustainment Performance Management System respectively.⁷⁸ Qualitative information reported by Project and Product Managers outlines the key challenges relating to the project and progress towards remediation. The reports contain appropriate information, describing the project and what went wrong, providing (among other data) an update on each current Project of Concern, progress towards remediation and an expected date for removal from the list.

4.4 After an internal approval process, which includes noting by the Defence Investment Committee and approval by the Secretary of Defence and Chief of the Defence Force, the report is generally provided to the Minister within two months of the end of each quarterly reporting period. In August 2017, the Deputy Secretary, CASG acknowledged to a Parliamentary Committee that the report was not sufficiently timely and that he had concerns about its accuracy and completeness.⁷⁹ In early 2018, Defence records indicate that the timeliness of the report remained an issue due to limitations with Defence’s information technology systems.⁸⁰

4.5 In June 2018, Defence began developing a new Quarterly Performance Report section within the Sustainment Performance Management System. The new section is to be populated with a summary of key product information on budget, year to date spend, and Australian industry capability which cannot be drawn easily from the system. The data from that section would then be used to populate the Quarterly Performance Report. In October 2018, Defence advised the ANAO that the section was used initially for reporting on sustainment products in the April-June 2018 Quarterly Performance Report and for all top 30 sustainment products in the July-September 2018 Quarterly Performance Report.⁸¹

External Reporting

Reporting to Parliament

4.6 Defence has reported on Projects of Concern (project number and name, and the date it was added to the list) in its Annual Report including changes to the list during the reporting period (for example, the removal or addition of a project).⁸² Changes to the Project of Concern list are often also announced via ministerial media release.⁸³

4.7 Projects of Concern and Projects of Interest are reported on as part of the Portfolio Budget Additional Estimates Statements. The Portfolio Budget Statement Outcome 2 is:

We protect and advance Australia’s interests through the provision of strategic policy, the development, delivery and sustainment of military, intelligence and enabling capabilities, and the promotion of regional and global security and stability as directed by Government.

4.8 The associated performance criterion is: ‘Capability proposals, once approved by Government, meet agreed schedule and are delivered within agreed costs and scope’. For 2017–18, the information included under the section on performance in achieving the criterion is:

Of the 123 post-second pass approved major capital equipment projects, four projects (or 3.3%) had issues with capability, schedule or cost which were significant enough to be included in the Projects of Concern report. A further nine projects (or 7.3%) were identified as Projects of Interest with risks associated with capability, schedule or cost that warrant attention from senior executives.

4.9 Defence has reported to Parliament on the Projects of Concern regime both at Senate Estimates and other Parliamentary committee hearings and in a more structured way through a regular statement to the Senate Standing Committee on Foreign Affairs, Defence and Trade.⁸⁴ These reports include current Projects of Concern, any changes to the list since the previous report, and the previous Projects of Concern including the outcome (for example, cancelled or remediated) and the date removed from the list. The report also includes a one page summary for each Project of Concern that provides an overview of the project’s current status.

Reporting to Ministers

4.10 Defence reports to ministers on Projects of Concern regularly and has done so since, at least, 2007. Briefs initially included detailed information on each of the identified Projects of Concern including: project schedule with the original approved and rebaselined dates; expected risks to milestones; original approved budget and any cost increases over the life of the project; technical issues and their impact; current and planned remediation activities; commercial sensitivities; and any likely Ministerial guidance or decisions required in the future.

4.11 Over time, the reports have become more focused. In March 2018, Defence reduced the frequency of reports for Projects of Concern from monthly to quarterly and incorporated those reports within its Quarterly Performance Report.⁸⁵ This reflected an internal view that the content of the Projects of Concern report did not change significantly from one month to the next and the reporting frequency may not be beneficial for the Ministers or the projects involved.

4.12 Reporting does not provide an indication of how cost or schedule may have shifted over time, and in some cases during the project’s time on the Projects of Concern list, such as the MRH90 helicopter. Including the original approved costs or milestones as well as the current approved schedule and budget for the Projects of Concern would make reporting more useful.

4.13 The Minister may also receive updates on specific projects at key milestones or decision points, and briefing materiel in preparation for Projects of Concern summits, and any outcomes of the summits.⁸⁶ The Minister is provided a brief to approve a project’s addition or removal from the Projects of Concern list.

Can Defence demonstrate that the Projects of Concern regime contributes materially to the recovery of underperforming projects?

4.14 Defence has consistently stated that its Projects of Concern regime is ‘one of the Department’s most successful management tools for recovering problem projects’ and that ‘the Projects of Concern regime is a proven process for managing seriously underperforming projects at a senior level’.⁸⁷ Senior involvement in Projects of Concern is evident in:

• the flow of documentation that must be put to ministers to obtain a decision on entry or exit from the list; and
• ministerial summit meetings, and their associated processes (such as remediation plans).

4.15 Defence has not set any criteria by which it could evaluate or assess the Projects of Concern regime, nor has it conducted any systematic evaluation of the regime nor any analysis to highlight those features of the process that contribute most to achieving a successful outcome. When asked whether any persuasive analysis had been done that shows that the Projects of Concern process yields a better outcome than otherwise would have eventuated, Defence advised the ANAO that it was not aware of any.

4.16 The three major case studies (see Appendices 4 – 6) do not provide a basis for concluding that elevation of the project (or product) to the Projects of Concern list has had a significant effect on the progress or recovery of the project:

• The Air Warfare Destroyer build project was placed on the Projects of Concern list well after the project’s progress had been extensively reviewed, had attracted senior engagement and a remediation plan developed (Appendix 4).
• Similarly, remediation plans for the MRH90 helicopter were already in place when it was placed on the Projects of Concern list in 2011. The Minister regarded the latter step as ‘somewhat empty’. The project remained on the list in early 2019 with a prospect of removal during the year (Appendix 5).
• Project of Concern status and senior engagement did precede development of the remediation plan for sustainment of the Collins Class submarines (Appendix 6).

4.17 There are positive and negative views of the Projects of Concern regime as outlined below.

Positive views of the effectiveness of the Projects of Concern regime

4.18 The audit encountered a range of positive views about the effectiveness of the Projects of Concern regime. For example, evidence in late 2014 from previous ministers in the Defence portfolio shows that they viewed the Projects of Concern list as good practice and useful in helping the Defence Minister and the Department to press industry for better service.⁸⁸

4.19 Also, on occasions in the past, senior staff in Defence have recorded a view indicating that the Projects of Concern regime can be an effective mechanism:

In my experience the only way we get traction on these matters is to let them know that their behaviour will threaten new business. [A particular vendor] refused to come to the table on [a project on the Projects of Concern list] … until they lost [a second project] contract.⁸⁹

4.20 Notably, both the comments from previous ministers and those from senior staff relate to the Projects of Concern regime as it operated before 2014. Their reflections encompass the period when regular summits and greater ministerial involvement were a part of the process.

4.21 There is written evidence of some vendors paying attention to Project of Concern status. However, both of these instances date from the earlier years of the Projects of Concern regime.

• The chief executive of a major vendor issued a public press release which stated, ‘It is important to emphasise that, contrary to some reports, the [name of the asset] has not been placed on the list of projects of concern.⁹⁰
• In a very early example, the managing director of another vendor wrote to colleagues in his own organisation advising that a particular project was a Project of Concern: ‘the consequences of this are straight to reputation … If we have a problem with schedule deal with it now take the punishment and then make up and move forward. If this means buying back schedule then so be it’.⁹¹

4.22 The Australian Industry Group put this view to the audit:

The general sense from industry is that a Projects of Concern process is a valuable part of the project management and assurance process between Defence and industry. The system is useful as it provides a direct and open feedback mechanism to Government on major projects.

4.23 The Australian Industry Group’s submission to the ANAO passed on the view of a member organisation, a large Defence industry company, as stating:

the POC [Projects of Concern] process serves a useful purpose in the management, mediation and resolution of our very challenging defence projects when they are in trouble. It is very rare for a POC defence project to be completely the fault of either industry or defence — invariably there is a co-dependence on the root causes, and in reality the effective treatment and resolutions.

I would encourage that the POC process remains available in the toolkit for effective governance of defence programs.

Possible negative consequences of the Projects of Concern regime

4.24 Defence documents have raised the question of whether Projects of Concern had an ‘adversarial element inconsistent with [Defence’s] aim of improving its relationship with industry’ and whether the public naming of vendors in Defence industry had a negative impact. This was because ‘there is the potential for industry to perceive that they are being primarily targeted as the cause for all underperformance’.⁹²

4.25 Concern about the negative effects of the Projects of Concern process on the relationship with specific vendors has arisen. In one case in 2018, the reputational implications of a project remaining a Project of Concern were highlighted as having negative consequences for the future of the project.

4.26 A senior Defence committee was advised in mid-2017:

While the companies listed as Projects of Concern have provided advice that the last summit was a useful and collaborative activity, there is a risk that the public may not view the process as positively and may still view it as a ‘name and shame’ list … Consideration should also be given to whether listing the project as a Project of Concern will likely have a positive impact and improve outcomes on the project overall.⁹³

4.27 These concerns indicate a degree of ambiguity in perceptions within Defence as to whether the whole Projects of Concern mechanism has a benefit, overall, for Defence in its relationship with industry. In February 2019, Defence advised the ANAO that:

listing a project on the PoC report is not a ‘threat’ or a ‘name and shame’ exercise. It is a management tool for projects or products with significant risks or issues with regards to capability, schedule and cost.

4.28 Evidence indicates that in at least one case, concerns have been raised about inappropriate responses to Project of Concern status. In March 2018, an Independent Assurance Review Board recommended that a particular project be considered a candidate for the Projects of Concern list. Defence officials subsequently expressed concern that the vendor appeared to have focused on removal of Project of Concern nomination rather than delivery of an executable and value-for-money plan for the project’s Initial Operational Capability milestone.

Lessons learned

4.29 Following a briefing Defence provided to a Parliamentary Committee on Projects of Concern on 30 November 2017, Defence was asked, among other questions: ‘What is the systemic approach to Projects of Concern, e.g. how are the lessons learnt from remediating individual projects being applied systematically?’ Defence responded that ‘… CASG have established and are building a Lessons Learned Program …’ Review of this material indicated it was still in early development and contained no specific analysis of Projects of Concern or lessons flowing identifiably from these projects or their history.

Future evaluation

4.30 After a decade or more of operation of this high profile process, Defence has made no systematic assessment of its operation nor established criteria by which it could assess whether making an underperforming project a Project of Concern adds value to the particular project’s recovery or limits the project’s further deterioration. Defence has pointed out that there are challenges in doing so, given the complexity of large materiel acquisition projects and the difficulty of establishing a counterfactual:

There is no counterfactual for ANAOs argument – we have no way of definitely determining how much worse these projects could have been if they hadn’t been subject to additional oversight/reporting/implementation of remediation plans etc.⁹⁴

4.31 An opportunity for establishing criteria to judge success or failure of the process may lie in formalising the practice of agreeing a remediation plan and remediation criteria as part of a future Projects of Concern policy.

Appendix 1 Department of Defence response

 

 

 

 

Appendix 2 Projects of Concern — processes, criteria and principles, at various dates from 2001 to 2018

ANAO note: The set of criteria listed below for 2001 applied at a time when the Projects of Concern regime was managed as a largely ‘internal’ process by the Defence Materiel Organisation. For the sets of criteria/principles listed for 2009 through 2018, the regime has involved decision-making by ministers.

(1) Projects of Concern criteria, July–August 2001

---

CORPORATE OVERVIEW

Exception Reporting

The Corporate Overview is a monthly report, which the USDM [Under Secretary for Defence Materiel] presents to the Defence Committee. The document is also used for reporting in the Defence Matters Scorecard, the DMO Scorecard and is the foundation for a report, which goes to the Minister each month.

Recently the requirements for the Corporate Overview have changed. Not only does the DMO report on the Top 20 projects, two new sections have been added: “Projects of Concern” and “Significant Issues”.

The details below outlines draft criteria to determine if a project needs to be reported as a Project of Concern or Significant Issue.⁹⁵

1. Governance:

• If a project is anticipating a potential Major Breach of Contract or

• Contractor Default.

2. Capability

• Any essential/critical requirements now forecasted that won’t be met.

3. Schedule

• If the schedule has slipped by more than 10% from the contracted baseline schedule.

• On Projects NYTC [not yet through committee] but approved if the schedule slippage is greater than 10% from the contract baseline schedule.

4. Budget

• The Project may go over budget.

• A change to the forecasted expenditure of 10% or greater.

• Project contingency money being used.

• If there is a risk that the remaining contingency is less that what is required for the project.

---

(2) Criteria for Listing and Removal from the List of Projects of Concern, September 2009

1. Entry to and exit from the list of Projects of Concern is decided by the Minister for Defence Personnel, Materiel and Science. There are two ways that a project will be removed from the list:

1. Project or contract cancellation — only exercised in the most extreme cases, e.g. Seasprites and JP 129; or
2. Project remediation.

2. Specific criteria have not been established for listing as a Project of Concern or for removal from the list. The complexities which characterise many of the listed projects tend to cross a number of aspects which are not always likely to captured entirely within specifically defined criteria. As general guidance an indicative set of principles for listing might include a synthesis of some or all of the following issues:

1. Uncontained instability in schedule, cost, requirements or risk;
2. Scope outside or non-compliant with Government approval;
3. Unlikely to meet Capability Manager’s directed level of capability;
4. Potential adverse impact on current or known future deployments;
5. Unresolvable technical issues;
6. Unresolvable commercial, or relationship with contractor, issues — eg: an unresolved dispute over 3 months duration; or
7. Insufficient platform availability for installation and test.

3. Underlying any set of principles or criteria, a common quality is that issues or risk are unbounded or uncontained and lack any demonstrable or sufficiently assured path to closure or completion. Establishment of a demonstrably assured plan and path to resolution should be an essential precursor to consideration of a project’s removal from the list of Projects of Concern.

(3) Projects of Concern processes and criteria, June 2011

---

Project of Concern Reforms

Incentive for Industry to focus on fixing problem projects:

Where a company has a project on the list, Government and Defence will weigh their performance in remediating the project when evaluating their tenders for other projects.

When a company is not satisfactorily implementing an agreed remediation plan, this will result in a negative weighting of tenders received from the company, and in extreme circumstances could result in exclusion from further tenders until the project is remediated.

Formal process for adding projects to the list:

The process for determining whether a project should be added to the Projects of Concern list will be as follows:

• When an Early Indicator and Warning is triggered [see below], Defence will advise Ministers, including whether a full diagnostic review (Gate review) of the project is required.
• If a Gate review is to be conducted, Ministers will write to the Chair/CEO of the prime contractor advising them that the project has triggered an early warning, requesting their involvement in the Gate review, and emphasising the potential for the project to be added to the Projects of Concern list.
• Following the Gate review, Defence will provide Government with recommendations on how to fix the problems with the project. If the problem is very serious, it may be listed as a Project of Concern immediately.
• Alternatively, Defence will propose a series of actions that it and the company involved will undertake to fix the problems. This will include timelines, targets and thresholds which if not met will trigger a further Gate review to consider listing the project as a Project of Concern.

The decision to add a project to the list will be made by the Minister for Defence and Minister for Defence Materiel.

Remediation plans:

The Defence Materiel Organisation (DMO) and Industry will develop formal remediation plans for all projects that undergo Gate reviews.

In the case of projects confirmed as Projects of Concern, these plans will:

• identify remediation objectives;
• identify key milestones and the timeline for their achievement; and
• detail an end-state for remediation and the agreed basis on which a project will be removed from the Project of Concern list.

Where DMO and Industry cannot agree a satisfactory remediation strategy, DMO will provide formal advice to Government on whether the project should be cancelled.

For all existing Projects of Concern, formal remediation plans will be developed and agreed with Industry. These will include the basis on which these projects will be removed from the current list.

Removal of projects from the list:

There are two events that will enable a project to be removed from the Project of Concern list:

• Government satisfaction that remediation is completed in accordance with the agreed plan; or
• a decision is taken by Government to cancel the project.

Increased Ministerial Involvement:

Ministerial involvement has been a cornerstone in driving improved outcomes for Project of Concern projects.

The Minister for Defence Materiel will hold bi-annual reviews of Projects of Concern with Defence and Industry representatives.

Biannual face-to-face meetings with the Minister will ensure responsible individuals are being held to account for the progress of projects, and will give the Minister a better understanding of the progress of remediation strategies.

---

Early Indicators and Warnings: Criteria and thresholds for post-second pass projects

Performance criteria and thresholds for post-second pass projects against Government-endorsed milestones were as follows:

1. Schedule. Projects where the schedule for meeting Initial Operating Capability (IOC) or Final Operating Capability (FOC) will be delayed by:
   1. 10% or more for acquisition of an off-the-shelf capability,
   2. 20% or more for acquisition of a modified off-the-shelf capability, and
   3. 30% or more for acquisition of a developmental capability.
2. Cost. Projects with costs that are forecast to exceed the approved budget.
3. Contractor Performance. Projects where the delivery contractor is identified as not meeting:
   1. capability targets,
   2. schedule milestones, or
   3. approved costs.
4. Policy or Legislation Impacts. Projects where policy or legislative changes are likely to increase the schedule or cost.
5. Capability. Projects where an essential capability requirement will not be met.
6. Emerging Requirements. Emerging requirements or regulatory or safety standards that are different to those in place when the Project was approved by Government, and that will materially affect the outcome.
7. Industry Capacity. Industry engaged in the Project does not have:
   1. sufficiently skilled and/or qualified workforce,
   2. financial capacity, or
   3. management commitment to meet contracted project milestones.
8. Risk. Project risks have increased beyond the parameters agreed by Government

---

(4) Projects of Concern principles, February 2017

---

Projects of Concern

Statement of Principles

1. Projects of Concern are a the most challenging projects in the Defence portfolio – they are:

• at the extreme end of underperformance;
• at risk of failure;
• unable to manage their way out of difficulty through standard reporting chains; and
• require buy-in from Senior stakeholders.

2. PoC provides senior intervention to underperforming projects. This intervention will supplement the existing standard senior management oversight in a more targeted and focused manner. This senior intervention will focus Defence and Industry stakeholders to align and concentrate their efforts on solving underlying issues and causes.

3. There are no set quantitative measures or thresholds to determine whether a project should be added to the Projects of Concern list:

• Qualitative measures that take into account stakeholder relationships and emerging risks, and the extent to which these factors impact the prognosis for future performance are considered.
• The Quarterly Performance Report also identifies underperforming projects and sustainment products, and these may be referred for further investigation.
• Once identified as a candidate, CASG Independent Assurance Reviews provide a mechanism for assisting in determining whether a project would benefit from being added to the Projects of Concern list.

4. A non-advocate assessment provides the independence and perspective required to see the project or product’s risks and issues. Remediation plans must be centred on a clear strategy with identified milestones against which progress can be checked.

5. Behaviours: PoC supports a ‘hands up’ culture.

• A collegiate approach allows projects to be remediated and deliver the best value for money to Defence.
• The regime is built on trust, not blame.
• Innovation is at the core — new ways of solving problems is to be encouraged.

6. A project’s removal from the Projects of Concern list will be determined by successfully achieving remediation outcomes, where there are no untreated risks or issues, and approval by the responsible Minister. If remediation of a Project of Concern is not likely to be achieved, there may be no alternative but for Government to cancel the Project or its contract.

7. The Projects of Concern regime provides innovative solutions to complex and serious performance issues. As part of this regime, CASG will capture, understand, and apply lessons learned from previous Projects of Concern.

---

(5) Projects of Concern principles, 2018⁹⁶

---

Projects of Concern

The Projects of Concern (PoC) team is part of the Business Analysis and Reform Branch (BARB). Projects or sustainment activities identified as Projects of Concern have very significant technical, cost, and/or schedule difficulties that are beyond the normal project team management.

The Projects of Concern regime is one of our most successful management tools for recovering challenged projects. This methodology includes:

• independent diagnostics;
• developing an agreed remediation plan; and
• monthly project reporting.

Since its introduction in early 2008, the Capability Acquisition and Sustainment Group (formerly DMO), has worked closely with Industry, Defence, and the Government to successfully remediate these projects with the goal of returning them to the standard project management framework. It is important to note that the Projects of Concern list allows for all parties to form a common plan of action and avoids ‘blame’.

Projects of Concern will continue to have strong Ministerial engagement and hold summit meetings when required. The summits focus Industry’s attention on Projects of Concern, and provide the Commonwealth with real commercial leverage for improved project outcomes.

Key activities

The Projects of Concern team will assist listed projects with their reporting requirements and the implementation of their agreed remediation plans through these key activities:

• Prepare and analyse data for the Projects of Concern monthly reporting to Government and Defence stakeholders
• Prepare and submit a Projects of Concern statement to the Joint Standing Committee on Foreign Affairs, Defence, and Trade, immediately prior to each Senate Estimates Hearing
• Coordinate oversight of Projects of Concern including face to face briefings and reporting
• As and when required, coordinate the Projects of Concern summits with senior stakeholders from Government, Defence, and Industry to discuss and agree the way forward for challenged projects
• Engage with individual projects of concern, particularly in relation to:
  • Ministerial submissions.
  • Stakeholder forums including Project Management Stakeholders Groups (PMSGs), CASG Independent Assurance Reviews (CASG IARs); and
  • Progress against agreed remediation objectives.

Policy

Entry to and exit from the list of Projects of Concern is decided by the Minister for Defence (MINDEF), and the Minister for Defence Industry (MDIND) either at the recommendation of the DEPSEC CASG or at the Minister’s own instigation.

The formal process for determining whether a project should be added as a Project of Concern begins when the Quarterly Performance Report (QPR) highlights a Project of Interest and an Independent Assurance Review is conducted for a diagnostic review. Following this, DEPSEC CASG will provide a recommendation to the MINDEF and MDIND to list the project or sustainment activity as a Project of Concern.

Projects are removed from the list through either remediation or cancellation with the approval of the MINDEF and MDIND.

Appendix 3 Projects of Interest

1. Projects of Interest are identified by Defence as those underperforming at any point in the capability lifecycle that require management action to avoid becoming a Project of Concern. Defence has not published a list of Projects of Interest; however, Projects of Interest have been identified in the annual Major Projects Report.⁹⁷

2. The current Projects of Interest list is updated quarterly, and reported in the Capability Acquisition and Sustainment Group Quarterly Performance Report.⁹⁸ The graph below illustrates the number of Projects of Interest over the past four years. Before July 2016, Projects of Interest were referred to as ‘underperforming projects’ or ‘underperforming products’. Sustainment Products of Interest have been reported from March 2016.

Figure A3.1: Numbers of concurrent Projects/Products of Interest July 2014–March 2018

 

 

Source: ANAO analysis of Defence documentation

3. As of December 2018, there were eleven acquisition projects with an approved budget of $22 billion, and nine sustainment products with an annual approved budget of $975 million on the Projects of Interest list. In terms of the role of the list as a means to manage project and product performance, the Secretary of the Department of Defence has stated that⁹⁹:

These frameworks [the Projects of Interest and Projects of Concern frameworks] provide an escalation mechanism for increased management of capabilities under development for the Australian Defence Force. A project may be identified as a Project of Interest when scope, schedule or cost variances warrant heightened senior management attention.

4. The projects and products on the list as of December 2018 are included in the table below. The performance issues that have warranted further attention from senior executives for these projects and products are: schedule delays, risks to capability, and budget overspend.

Table A3.1: Projects and Products of Interest as at December 2018

Note: The approved budget and spend to date for sustainment products of interest is for 2017–18 only. The Acquisition Category (ACAT) framework is Defence’s method for categorising acquisition projects. The framework is based on strategic significance, project and schedule management complexity, and levels of technical difficulty, operating and support and commercial arrangements. Higher complexity projects with a greater degree of strategic significance are assigned an ACAT I level, with low complexity minor acquisitions assigned an ACAT IV level. Defence assigns a Materiel Sustainment Category (MSCAT) to sustainment products (similar to the ACAT). The criteria is based on sustainment budget, management complexity, technical difficulty, life cycle stage, demand and availability and commercial arrangements. The more complex and higher budget products are assigned a MSCAT I level.

Source: Defence documentation.

Development of Projects of Interest

5. Defence’s use of Projects of Interest as an identifiable category has evolved. There are references to Projects of Interest dating back to 2005, with ‘projects and products of emerging interest’ reported in Defence’s Annual Report.

6. In 2011, the then Minister for Defence and the then Minister for Defence Materiel jointly announced a set of accountability and procurement reforms aimed at addressing outstanding recommendations from the Mortimer Review.¹⁰⁰ These included establishing an Independent Project Performance Office to review projects and assist project teams to solve problems. Part of these reforms also included the implementation of the Early Indicators and Warnings System.

7. The Independent Project Performance Office was established in the Defence Materiel Organisation in July 2011, and part of its role was to: ‘Implement the reforms … to the Projects of Concern process and oversee the remediation of all Projects of Concern’. The office also monitored Projects of Interest and provided information on these projects to Division Heads responsible for their management and senior leadership within the organisation.

8. The Early Indicators and Warnings System was aimed at identifying problems early in a project’s lifecycle and undertake appropriate remedial action. The system included a set of performance criteria around cost, schedule and risk. Any project that exceeded performance thresholds identified in the Early Indicators and Warnings System would be referred to senior management within Defence for review.

9. In 2013, Defence sought to formalise the process for identifying Projects of Interest and developed a set of business rules. These rules outlined the purpose of the Project of Interest list, the data sources used to determine whether a project should be included on the list and how the list should be used within Defence.

10. In 2014, the Early Indicators and Warnings System was replaced with the Quarterly Project Performance Report (now the Capability Acquisition and Sustainment Group Quarterly Performance Report). This report focused on the performance of major capital acquisition projects, and in late 2015 also included information on the performance of sustainment products.

11. Defence has advised the ANAO that the current approach for determining Projects of Interest has been in place since July 2016. The current approach relies on the Strategic Performance Reporting team¹⁰¹ using information from various data sources in Defence — including from performance reporting systems and Independent Assurance Reviews — to identify potential Projects of Interest. These potential Projects of Interest are confirmed with relevant stakeholders within Defence such as Project or Product Managers and cleared at the Division Head level.

12. Defence informed the ANAO that Independent Assurance Reviews form a key role in identifying potential Project of Interests. However an ANAO review of 191 Independent Assurance Reviews since 2015 found:

• Inconsistent approaches across Independent Assurance Reviews — the reviews are inconsistent in recording whether they considered if a project should become a Project of Interest or Concern. There are also projects with significant delays in schedule and budget overspends but which have not been recommended for inclusion on the Projects on Interest list (see paragraph 3.3 forward).
• A disconnect between information in the Quarterly Performance Report and Independent Assurance Reviews — there are instances where a project has been identified a Project of Interest through the Quarterly Performance Report, but the Independent Assurance Review has not made this recommendation. In some instances, the review has made a recommendation that the project be included on the Project of Interest list after the project has already been included on the list in the Quarterly Performance Report, or recommended that the project stay on the Project of Interest list after it has already been removed from the list in the Quarterly Performance Report.

Appendix 4 Major case study 1 — SEA 4000 Phase 3, Air Warfare Destroyer

What was the project about?

1. The project is to build and deliver three Air Warfare Destroyers and a support system.

2. Government commitment to Defence’s requirement for a long-range air-defence capacity in the fleet was set out in the 2000 Defence White Paper.¹⁰² The project, known as ‘SEA 4000’, was identified in the Defence Capability Plan 2001–10 as an ‘affordable Maritime Air Warfare capability’ with an estimated acquisition cost of $3.5 – $4.5 billion and an expected in-service date of 2013.¹⁰³ In the 2004–14 edition of the Defence Capability Plan this was revised to identify separate phases to the project: Phase 3 of SEA 4000 would be the ‘build’ phase with an estimated cost for that phase in the range $4.5 – $6 billion.¹⁰⁴

3. On 1 May 2005, Navantia was announced as the preferred designer of the ship. On 31 May 2005, ASC was selected as the preferred shipbuilder from among three responses to a request-for-proposal. The Minister for Defence said the Government made the decision on the basis that ASC offered a superior bid in terms of value for money. Following this, the government’s first pass approval, Phase 2 (design) commenced.

4. In June 2007, at second pass approval, it was announced that Navantia would work with the AWD Alliance (Defence Materiel Organisation, ASC and Raytheon Australia) to deliver three air warfare destroyers to the Royal Australian Navy at a cost of nearly $7.2 billion. SEA 4000 Phase 3 (detailed design and construction), to build three Australianised air warfare destroyers, based on the F-105 design in service with the Spanish navy and equipped with an Aegis combat system, then commenced.

How did the project become a Project of Concern?

5. Evidence of low productivity in the build project was evident in 2011, three years before it became a Project of Concern. The DMO Program Management Office reported to the Government–ASC Steering Committee meeting on 19 December 2011 that First Marine International (FMI), an international specialist consulting company, had been engaged to assess, independently, the actual and planned productivity at each of the Australian AWD shipyards in September–October 2011.¹⁰⁵ DMO stated that FMI had found:

Productivity against each ship has only been assessed on an overall basis which means the data includes the productivity of the block subcontractors BAE and Forgacs. Although ASC-specific performance cannot be extracted from the report data, the overall assessment does reflect the ASC plan and budgets for overall ship production and, as shown by Figure 1, this shows that far from the Project achieving world class productivity, the plan and budget do not even approach [the] realistic target identified by FMI.¹⁰⁶

6. Figure 1 from the FMI report, labelled ‘Current and target productivity’, and the associated notes are included below (see Figure A4.1):

Figure A4.1: Current and target productivity

 

 

Note: This figure shows productivity measured in ‘man-hours’ per compensated gross tonne for ships ➀, ➁ and ➂ in the air warfare destroyer fleet. ‘Compensated’ means that account is taken of different ship types of varying complexity requiring differing amounts of effort per tonne of vessel built.

Source: Defence documentation.

7. The FMI report describes how shipyard productivity tends to be much lower for the first ship in a series and increase with each successive vessel (the heavy black curved arrow is described as the ‘learning curve’). FMI had concluded, based on its observations, that the potential core productivity of the enterprise was 60 man-hours/compensated gross tonne. However, FMI found this could not be expected to be achieved in a three-ship program: the realistic target was 80 – 85 man-hours/compensated gross tonne (the green band). The notation ‘Lead ship factor = 100%’ indicates that then current estimated at-completion productivity on the first ship of the series was 120 man-hours/compensated gross tonne, which represents 100 per cent greater input than the potential productivity of 60 hours/compensated gross tonne estimated by FMI.

8. The DMO report went on to state:

Evidence of concerted efforts by ASC to improve productivity were not observed by FMI; they did not identify a plan to achieve improved productivity targets, nor did they witness a high degree of focus in improving productivity or evidence of productivity-based targets defined by appropriate metrics particularly at the production shop-floor level, where productivity can be implemented.

9. The report concluded that ‘the Alliance with ASC, will need to address … productivity, cost control, schedule control and risk for the shipbuilding and consolidation/ integration.’

10. In September 2012, ministers announced a ‘re-baselined production schedule’ — that is, project slippage — with a ‘keel-to-keel’ interval extended to 18 months.¹⁰⁷

11. An update from FMI (8 February 2013) found that estimated productivity at completion for all three ships had declined, with the lead ship productivity now 130 man-hours/compensated gross tonne.

12. Further concerns about productivity and cost overruns came to light over the next twelve months:

• On 17 December 2013, the 2012–13 Major Projects Report was tabled.¹⁰⁸ It identified the approved budget for the Air Warfare Destroyer build program as having risen through price indexation to $7.9 billion and noted ‘emerging concerns … around cost overruns’. The report identified shipbuilding productivity, among other items, as a major project risk.
• In March 2014, Auditor-General Report No.22 2013–14, Air Warfare Destroyer Program was tabled in Parliament. It reported a continuing decline in construction productivity.¹⁰⁹

13. The secretaries of the Departments of Defence and Finance commissioned a three-month independent review of SEA 4000 by two experts (Professor Donald C. Winter and Dr John White), in March 2014. The review would:

recommend remediations [sic] and mitigations to improve the cost and schedule performance of the AWD Program and to realise the national security benefits of the program and the long term benefits of the program for the Australian shipbuilding industry. The Review will also identify the risks to the future completion of the AWD Program, and provide recommendations to mitigate those risks.¹¹⁰

14. The Winter–White report, completed in May 2014, identified ‘two principal direct causes for the AWD Program difficulties, along with three significant contributing causes and two systemic issues’.¹¹¹

The two direct causes that were both categorised as findings are:

1) The initial program plan for AWD development and production was unrealistic in its cost and schedule estimates.

2) The Alliance, as structured, composed and staffed, has been unable to effectively manage the AWD Program.

The contributing causes include:

1) Systems engineering on the AWD Program has been of limited effect.

2) The AWD Alliance and ASC were unable to effectively manage the AWD block subcontractors.

3) The oversight provided by the Commonwealth of Australia has been of limited effect.

The two systemic issues, which could affect any other naval shipbuilding program in Australia, were identified as:

1) The limited base of shipbuilding activity in Australia materially impacted the AWD Program.

2) The Commonwealth of Australia has not developed a long term shipbuilding plan that can cost effectively support the needs of the RAN while sustaining the Australian industrial shipbuilding base’.¹¹²

15. The Review recommended inserting a new management team from a suitable company. It evaluated ‘a number of potential sources’ and found that only BAE Systems met all of the identified requisite criteria for such a role.¹¹³

16. The Winter–White Review was focused as much on future naval shipbuilding in Australia as on addressing the Air Warfare Destroyer build: it saw the two as strongly linked. In their letter to ministers setting out their early findings (4 April 2014) the authors recommended that ‘no final commitments are made beyond the planning phases of other inter-related major naval industry programs until the AWD Program issues are showing clear signs of resolution’.¹¹⁴ Their final report also found that ‘If appropriate internationally bench-marked performance levels are not achieved through the AWD Program, consideration could be given to future naval surface combatants being procured overseas’.¹¹⁵

17. In light of the Auditor-General’s performance audit and the Winter–White Review, Defence proposed to the Minister that the Air Warfare Destroyer project be placed on the Projects of Concern list. It advised him that these reports had confirmed that the project’s cost estimates would be ‘significantly exceeded due to poor shipbuilding productivity’, and:

Unless addressed quickly, the ongoing poor performance on the Project could also have a wider detrimental effect on Australia’s maritime sector, including projects such as SEA 5000 — Future Frigate — that are likely to rely on a high quality and productive ship building capability for success.

18. Defence considered that listing was warranted ‘because of the high profile of the Project and particularly its significance to the future of Australian naval shipbuilding in Australia’. The Minister agreed, and announced his decision in June 2014.¹¹⁶

What was the remediation plan?

19. At the same time, the Minister for Defence and Minister for Finance announced a ‘way forward for the $8.5 billion Air Warfare Destroyer program’.¹¹⁷ They stated that the Government was ‘totally committed to put this important Defence project back on track and to stop the growing cost and schedule overruns by implementing the recommendations for a Reform Strategy made by the Winter Review.’¹¹⁸

20. The ministers also announced that the Reform Strategy would:

• Improve shipbuilding productivity at the Air Warfare Destroyer shipbuilder ASC and its subcontractors BAE Systems, Forgacs and Navantia;
• Include the urgent insertion of an experienced shipbuilding management team into ASC; and
• After augmented shipbuilding capacity has been put in place, pursue the reallocation of blocks between shipyards to make the Air Warfare Destroyer program more sustainable.

21. The reform strategy became the remediation plan for the Air Warfare Destroyer project. As mentioned above, a major element was to insert a new team to take control of production management.

22. At the Ministers’ press conference on the matter the Minister for Finance stated: ‘In summary, the reform strategy that Professor Winter has recommended to the Government will seek to improve ship building productivity at the Air Warfare Destroyer Ship Builder ASC and its sub-contractors’. The Minister for Defence referred directly to the analyses of productivity by FMI (discussed above): ‘We balanced baseline production at about 80 man hours per gross tonne for these three ships. Currently it is running at 150 man hours per tonne’.¹¹⁹

23. By October 2014, four months later, a Gate Review found ‘The schedule for delivery of the ships is to a large extent “unknown”’ and a cost increase of ‘around $310 million above the increase of $140 million in the 2012 baselining’.¹²⁰

The Board noted that, commensurate with ‘pain-share’ under the Alliance agreement of 50/50 pain share, the Commonwealth (Defence) and the industry participants each bear half the cost of over-runs above the Alliance’s Total Cost Estimate. The Industry participants’ fee is reduced by the amount of industry’s pain share until all fee is exhausted. After that point is reached (an overrun of some $600m), the Commonwealth bears the full costs while industry receives no fee. The projected outcome, even with future productivity reforms, would likely see the overrun exceeding $600m and thus industry participants receiving no fee which is a challenging environment within which to incentivise the participants.

24. Nearly a year later, on 22 May 2015, the Minister for Defence and the Minister for Finance announced that an additional estimated $1.2 billion would be required to complete the project and that delivery of the destroyers had been ‘significantly delayed’.¹²¹ On 4 August 2015, the Prime Minister and the Minister for Defence announced that the additional $1.2 billion would be provided to deliver the program.¹²² The insertion of a Navantia management team to ASC to manage the project was announced in December 2015 and reforms to the structure of ASC were announced in October 2016.¹²³

25. On 5 May 2016, the Joint Committee of Public Accounts and Audit tabled Report 458: Defence Major Projects Report (2014–15): Review of Auditor-General Reports No 16 (2015–16). This included a recommendation that Defence review Air Warfare Destroyer contractual arrangements, examining the distribution of liabilities for project problems. Defence’s response indicated that, under the ’pain share/gain share’ arrangements one partner had repaid the full amount it was liable to repay but that ASC had committed to repay its share, ($82.7 million) by 2019. Defence has informed the ANAO that it expects this to occur by 31 January 2019.

How did the project leave the Projects of Concern list?

26. Defence’s Independent Assurance Reviews are a primary opportunity to consider whether a project on the Projects of Concern list should be removed. Defence conducted an Independent Assurance Review of the Air Warfare Destroyer project in July 2016. The review noted that the project had been added to the list in June 2014 but did not identify the reasons, nor did it refer to any remediation plan. It concluded that the project ‘had come a long way in the past 12 months’ and recommended that ‘consideration be given to setting achievement of IOR [Initial Operational Release] for HMAS Hobart as a suitable milestone for removal of SEA 4000 Phase 3 from the Projects of Concern [list]’.

27. Defence conducted a further Independent Assurance Review of the project in July 2017.¹²⁴ It drew positive conclusions about progress: ‘My assessment is that Phase 3 (the build) is now well on track to meet the re-baselined schedule, notwithstanding some defects on Hobart remain to be rectified’. It noted that Navy was not supportive ‘in light of the number of remaining risks and defects’ but, nevertheless, the Review recommended ‘SEA 4000 Phase 3 be considered for removal from POC list at IOR/commissioning as that milestone will demonstrate significant burn down of risks and the adherence to the re-baselined schedule …’ Moreover, ‘the Board considered that by IOR there would be sufficient evidence that the original concerns that triggered POC status (namely cost overruns, low productivity, technical and quality issues and subcontractor delays to block delivery) had been addressed effectively’.

28. Defence subsequently sought the agreement of ministers to remove the project from the Projects of Concern list. In its advice, Defence stated that ‘Through the long-term reform arrangements set in place by the Government in December 2015, the Project is now meeting its revised budget and schedule targets, with shipyard productivity improving.’ Ministers agreed, and their decision was announced on 1 February 2018.¹²⁵

29. Shortly after the announcement, FMI provided a further update of its analysis of ASC productivity (2017 update, dated 15 February 2018).¹²⁶ This shows that productivity in man-hours/compensated gross tonne estimated at completion — the measures quoted by the minister at the time the project went on the list — has improved from the first ship to the second ship, and the second to the third, as expected, but overall, productivity for each ship has declined over the life of the project.

30. The ‘adjusted’ productivity based on the 2017 EAC [estimated at completion] data is illustrated in Figure A4.2, below.

Figure A4.2: Air Warfare Destroyer build productivity (2018)

 

 

Source: Defence documentation.

31. The FMI report explained this diagram thus:

The historical adjusted budgets for 2012 to 2016 are also shown. These illustrate how the expected out-turn productivity has continued to deteriorate with each successive year. The horizontal bands across the chart represent the historical and current budget-based productivity targets. FMI’s original (2011) suggested target for Ship 3 and the estimated core productivity of the AWD enterprise are also shown.¹²⁷

32. This FMI report also provides evidence that some elements of the remediation strategy may not have worked as effectively as expected:

Navantia took control of production management in February 2016 … the goals of ASC Shipbuilding and Navantia are not wholly aligned. This misalignment has created friction which at the time of this assessment had led to deterioration in the relationships between senior managers in both teams. This now appears to be having a negative impact on performance. It is essential that the senior managers work effectively together and focus on the common goal of delivering quality vessels to schedule and under budget. Working together to achieve this in the remaining two years of construction is the best way to maintain the credibility of all parties involved.¹²⁸

33. FMI also found that Navantia was ‘not incentivised to provide a shipbuilding continuum at Osborne’:

Thus it is only motivated to provide technology transfer and process improvements that will make a difference on AWD. On the other hand, ASC is focused on process improvements for the future and on further developing the local workforce.

34. Defence advised the ANAO that FMI benchmarking reports are not provided to its ministers.

Analysis of the case

35. Work on the analysis of the project (Winter–White report) and subsequent remediation plan began before the Minister placed the project on the Projects of Concern. Further, the concerns that developed before the Winter–White review began had a twofold focus:

• the slow progress of the Air Warfare Destroyer build project and its increasing cost; and
• the productivity of the ASC shipyard, where that build was taking place.

36. The question of productivity was relevant to the Air Warfare Destroyer build but was also a concern for future shipbuilding capability within Australia, one of the central concerns of the Winter–White review (see paragraph 12, above) and included in the review’s terms of reference.¹²⁹

37. Defence’s Air Warfare Destroyer Project Management Office states that it was difficult to address the productivity problem when it first came to light and attributes credit to the Department of Finance’s involvement in remediation:

low productivity in itself did not initially cause concern as it was expected to improve through the build program. The rate of improvement, however, was well below expectation. The EV [earned value] data in the monthly PMRs [project management reports] made the deterioration of cost and schedule clear to the AWD PMO [Air Warfare Destroyer Project Management Office], but, as stated above, it was hampered in not being able to initiate remedial actions in the Shipyard and of ASC, quickly enough.

The Government’s Reform program brought the Department of Finance into the process, which allowed remediation to move forward.

38. When the Air Warfare Destroyer project was removed from the Projects of Concern list, Defence’s Independent Assurance Reviews concluded that, after the implementation of the reform program (including insertion of the Navantia management team) the prospects for completion within the available time and budget were good. In this light, there was a basis of substance from the perspective of the project that it be removed from the Projects of Concern list. This view does not seem to have changed subsequently, with provisional acceptance of the third and final ship expected in December 2019.¹³⁰ Initial Operating Capability was forecast to be achieved in December 2018 (36 months behind the original planned schedule).

39. Notably, there was no reference in the Defence deliberations on removal from the list of the meeting of any milestones on a remediation plan or, indeed, any reference whatsoever to the remediation plan. The primary consideration was the prospect of completion of the required capability within the time and funding available.

40. On the other hand, the regular FMI analyses were not showing any increase in productivity of the shipyard. Defence has pointed out that ‘Even in mature shipyards, productivity levels for a first of class ship are always much lower than the shipyard’s core productivity, even up to 50%’.¹³¹ Thus, productivity in the build of Ship 2 and Ship 3 were progressively better than Ship 1 — as expected. Nevertheless, by FMI’s calculation, that productivity remained, even with Ship 3, over 120 man-hours/compensated gross tonne, twice the benchmark originally set by FMI of 60–65 man-hours/compensated gross tonne and well outside the modified target of 80 man-hours/compensated gross tonne.

41. A further consideration with Ship 3 in this case is that its construction may suffer from ‘last ship syndrome’ — where progress is adversely affected by labour anticipating a need for new work. This has been recognised by Defence as a risk in need of mitigation.

42. There is no indication that placing the Air Warfare Destroyer on the Projects of Concern list, of itself, made any difference to the remediation work, given that remediation was initiated independently and already had senior engagement. It is also highly probable that the project would have continued to command senior management and ministerial attention, given the project’s importance and cost and the fact that other work was under way on a future shipbuilding strategy.

43. On the matter of what difference being placed on the Projects of Concern list made to the Air Warfare Destroyer project, Defence’s Air Warfare Destroyer Project Management Office provided the following account:

While placing the AWD project on the Projects of Concern list in 2014 may have garnered the attention of Government and senior members in Defence and DMO, it made little difference to the operation of the AWD Program, as the Commonwealth and the Alliance were already working to remediate the project.

In 2012, internal reporting made it clear that there were issues within the project that needed to be addressed: the quality of work in the block subcontractors, and that budget and schedule were slipping. The Alliance made sure overseas shipbuilding expertise was inserted into BAE’s Williamstown shipyard and ASC personnel embedded at Forgacs, and the Alliance reallocated module work away from BAE to Forgacs and Navantia.

Even with these actions, cost and schedule were still slipping due to the inexperience of the shipbuilders, but the DMO Program Office was unable to take positive control of the deteriorating situation, in part due to the nature of ASC being a Government Business Enterprise, and in part due to the nature of the ABTIA [Alliance-Based Target Incentive Agreement] contract, which required unanimous approval at the AWD Project Board level before any remedial action could be taken.

It was the publication of the ANAO report and the publicity surrounding the Winter-White report, rather than being placed on the PoC list, that initiated the Reform program which culminated in Navantia being installed as shipbuilding manager in December 2015. Since then, the AWD budget and schedule have stabilised and shipyard productivity improved.¹³²

From the AWD Program’s perspective, being placed on the PoC list appeared to be a necessary formality, required for administrative monitoring of DMO/CASG programs under stress and warranting increased senior stakeholder oversight.¹³³

Appendix 5 Major case study 2 — AIR 9000 Phases 2, 4 and 6, Multi-Role Helicopter

What was the project about?

1. In 2002, Defence released the Australian Defence Force’s Helicopter Strategic Master Plan which sought to reduce the number of helicopter types in the ADF from nine to five with the aim of realising training, support and sustainment efficiencies. The acquisition of the Multi-Role Helicopter (MRH) formed a key component of the Strategic Master Plan as one of three fundamental classes of platform.¹³⁴

2. The MRH Program consists of three phases of AIR 9000:

• Phase 2 (12 MRH90 helicopters) is the acquisition of an additional Squadron of troop lift aircraft for the Australian Army;
• Phase 4 (28 MRH90 helicopters) will replace Army’s Black Hawk helicopters in the Air Mobile and Special Operations roles; and
• Phase 6 (6 MRH90 helicopters) will replace Royal Australian Navy (RAN) Sea King helicopters in the Maritime Support Helicopter role.

3. The Government approved AIR 9000 Phase 2 in 2004, and Phases 4 and 6 in 2006. The Phase 2 acquisition contract was signed in June 2005 for $912.5 million, and was amended in June 2006 to include the additional 34 MRH90 aircraft acquired under Phase 4 and 6, and their support systems, at a cost of $2,656.5 million. The sustainment contract, for $353 million, was signed in July 2005.

How did the project become a Project of Concern?

4. In September 2010, Defence discussed internally the MRH90 project becoming a Project of Concern. In November 2010, Defence had stopped accepting aircraft as they did not meet all the contractual requirements which, in turn, affected the MRH90 program’s ability to achieve capability milestones. Defence briefed the then Minister for Defence Materiel in late 2010 recommending that the project be included on the Projects of Concern list due to continuing delays, and outlining the planned action to stop aircraft acceptance and key system issues including instances of engine failure. The Minister did not approve the listing, requesting instead that Defence undertake a diagnostic gate review to enable further consideration of the matter.

5. A Defence Gate Review in February 2011 identified a number of issues with the project including: technical issues being complicated by contractual shortcomings, poor contractor performance, low aircraft availability, and value for money concerns. The Gate Review Board raised 18 action items to address these issues, of which the key focus was the renegotiation of the acquisition and sustainment contracts commencing with a deed to be negotiated with the Contractor by June 2011.

6. A Gate Review was held in September 2011, and was followed by a further meeting of the Gate Review Board in October 2011 with the prime contractor, Australian Aerospace.

7. Following the September and October 2011 Gate Reviews, the Defence Materiel Organisation provided advice to the Ministers for Defence and Defence Materiel that: ‘the program has triggered schedule and contractor performance Early Indicator and Warning thresholds, and recommend that the project be listed and managed as a Project of Concern by Defence.’

8. The Gate Review Assurance Board further recommended, if the Ministers agreed to the Project of Concern listing, that:

• Industry should be invited to participate in an initial Project of Concern summit, if Defence is to achieve the desired impact.
• Project of Concern remediation objectives should encompass whole of program aspects, including full rectification of technical issues and the delivery of a reliable and available capability at an affordable price through the capability life cycle.
• Specific triggers and milestones would need to be achieved for the program to be removed from the Project of Concern list and these will need to be defined by Defence senior management in conjunction with the Minister and industry.

9. The MRH90 program subsequently became a Project of Concern in November 2011 due to: ‘schedule and contractor performance.’

What was the remediation plan?

10. The 2011 Gate Reviews that recommended that the project become a Project of Concern also identified a number of actions to remediate the project. Remediation was centred on the negotiation of two Deeds with the contractor: Deed 1, to rectify technical issues with the aircraft; and Deed 2, to negotiate the Commonwealth’s desired sustainment-related performance and value-for-money outcomes.

11. The first Deed was signed on 5 October 2011 just before the project became a Project of Concern. The Deed sought to address a number of technical, legal and commercial issues.¹³⁵ Deed 1.1 was signed in March 2012 as a variation to Deed 1. Through this variation Defence agreed to accept an initial tranche of MRH90 aircraft at a lesser configuration than the contract required and, in return, Australian Aerospace agreed to an interim sustainment performance regime for 1 February 2012 to 31 January 2013.

12. A further deed, Deed 2, was signed on 9 May 2013. Defence and Australian Aerospace entered into Deed 2 to agree how certain issues between them would be resolved, vary the AIR 9000 program contracts in accordance with the terms of the deed, and release each other from certain claims each of them had at 1 July 2013.¹³⁶

13. The two-Deed approach to remediating the MRH90 Program was already underway before the project became a Project of Concern in November 2011. A brief to the then Minister for Defence provided advice on the outcomes of the September 2011 Gate Review and noted that while the Gate Review Assurance Board had considered termination of the project due to excessive delays, the Board supported continuing the two-Deed approach and that the project would likely become a Project of Concern. In response the Minister commented that: ‘[Project of Concern] status seems somewhat empty if one has already gone down the Deed 1 [and] 2 route.’

Development of remediation criteria and action logs

14. In December 2011, Defence began developing the remediation criteria for the project. The Project Director of the MRH90 Program noted that the first remediation criterion should be ‘full access to costing and technical data’.

15. Remediation criteria and action logs for the project were developed as part of the Projects of Concern summits. The first summit after the project was listed as a Project of Concern took place in April 2012. Early remediation plans focused on the negotiation of Deed 2 and the achievement of Operational Capability Maritime 2¹³⁷ by 30 August 2013. At the last Project of Concern summit in July 2015, remediation focused on testing of a sub-system to ensure deficiencies had been rectified; conducting an Airworthiness Board to achieve an Australian Military Type Certification and Service Release; and holding a Configuration Board to enable further aircraft acceptance.

16. Defence advised the ANAO in August 2018 that the CASG Quarterly Performance Reports contain a summary of short term, medium term and long term remediation plans. The June 2018 report for the MRH90 program outlines the short-term (1–3 months) remediation goal for the project. These include addressing issues with the Electronic Warfare system to allow operations in medium and high-threat environments, and to seek Army’s and Navy’s position in removing the program from the Projects of Concern list.

17. The Project of Concern monthly reports illustrate that the reason for the Project of Concern listing, expected removal dates and removal criteria have changed over time for AIR 9000 Phases 2, 4 and 6.

Removal from the Projects of Concern list

18. The project is still on the Projects of Concern list as of October 2018; nevertheless, there have been attempts to remove the project from the list over the years.

19. A November 2013 brief to the then Minister for Defence noted that MRH90 was one of three projects approaching a point where it would be appropriate to consider removing them from the list:

The signature of Deed 2 along with improved reliability in the helicopter, means that this project could be considered for removal from the list upon achieving Initial Operating Capability for the Navy. This milestone is expected in late 2013 or early 2014. While some challenges will remain, this project would be able to return to normal management arrangements.

20. There were no further attempts to remove the project from the Projects of Concern list until May 2018 when an internal Defence brief recommended that the project be removed due to a ‘demonstration of an acceptable level of capability for Navy and Army’ and that the ‘resolution of remaining issues are being managed through existing leverage under the contract and via high level engagement with Industry.’

21. The brief noted that the project may need to remain on the Projects of Concern list due to the delays to the Operational Capability Maritime 2 milestone; unresolved commercial matters relating to capability delivery; and cost of ownership concerns.

22. The MRH90 project team sought the Capability Manager’s position on the assessment of the Early Warning Self Protection System and the ability to progress declaration of further capability milestones. As of October 2018, Defence forecast removal of the project from the list during mid-2019.

Analysis of the case

23. The following sections provide an overview of the implications of the Project of Concern listing for the MRH90 Program.

Effect of the Project of Concern listing on budget, schedule and capability

24. The table below outlines the change in budget and schedule at key points for the MRH90 project. While the project’s budget has remained largely the same over the last 12 years, the scheduled Final Operating Capability for the project has slipped by 89 months (or around 7.5 years).

Table A5.1: Budget and schedule at key points for MRH90

Note a: This figure includes $953.9 million for Government Second Pass Approval for AIR 9000 Phase 2, and $2,565 million for Government Second Pass Approval for AIR 9000 Phases 4 and 6.

Note b: At the time of Project of Concern listing, the Final Materiel Release and Final Operational Capability dates were subject to review based on the rebaselining of aircraft delivery as part of Deed 2 negotiations.

Note c: December 2021 is the current forecast date for FOC; however, July 2019 remains the date in the current Materiel Acquisition Agreement and the approved date. An update is being progressed to Government seeking agreement for the 2021 FOC date (Defence advice of October 2018).

Source: ANAO analysis of Defence documentation.

25. Over the life of the program, operational capability has been affected by several technical issues including the cargo hook release mechanism, the fast roping and rappelling device, cabin seating and the self-defence gun mount. While capability over the life of the project may not have been reduced, the schedule has been delayed. For example, the initial remediation plan in April 2012 for the project sought to achieve Operational Capability Maritime 2 by 30 August 2013; however, as of July 2018 this milestone had not yet been achieved.

Leveraging industry

26. Defence have sought to use the Project of Concern listing for AIR 9000 Phases 2, 4, and 6, and also the potential listing of the project, to leverage industry.

27. The February 2011 Gate Review of the MRH90 project noted that:

The indicators for making the MRH Project a [Project of Concern] appear to have been triggered. However, [Australian Aerospace] are keen to avoid this action and have indicated a strong commitment to resolving engineering problems and raising aircraft capability … the Commonwealth should use the [Project of Concern] leverage to require [Australian Aerospace] to improve performance.

28. In April 2011 prior to the project being listing as a Project of Concern, the General Manager Systems of the Defence Materiel Organisation wrote to the Chief Executive Officer of Australian Aerospace regarding the project and the strategies needed to address various issues.

29. The General Manager Systems of the Defence Materiel Organisation also proposed that a joint steering group from Defence and Australian Aerospace be established to monitor and guide the parties. Defence sought written confirmation from the contractor to agree to this strategy and indicated that Defence would recommend to the Minister of Defence that he not list the Program as a Project of Concern.

30. Following this letter, a briefing paper to the Defence Capability and Investment Committee noted that the February 2011 Gate Review of the program had set a trigger date for consideration as a Project of Concern for September 2011 and that:

[Australian Aerospace] have been actively campaigning to not become a [Project of Concern] (reputational impacts), and the set time period will allow them time to demonstrate their commitment to remediation of the program.

31. In June 2018, Defence’s First Assistant Secretary, Helicopter Systems Division, wrote to the Chief of Navy and the Chief of Army to seek their agreement on the Project of Concern removal criteria for AIR 9000 Phase 2, 4 and 6. The letter noted that Airbus and Defence agreed that the project would be considered for removal from Projects of Concern when an acceptable level of capability for Navy and Army was achieved, commercial resolution of technical deficiencies was resolved and there was a demonstrated improvement in aircraft serviceability that supported achievement of Navy and Army capability milestones.

32. The letter further noted that a key technical issue for resolution is the Enhanced Cargo Hook System, and that Defence was managing the resolution of this issue through high level engagement with industry and a delay to the Final Acceptance Milestone payment. Defence assessed that tying the resolution of this issue to the project’s Project of Concern listing offered Defence: ‘the best chance to extract the last remaining leverage and further incentivise Industry beyond measures already in place’.

Use in other Defence acquisitions

33. In February 2012, Defence wrote to Australian Aerospace regarding the implication of the Project of Concern listing for AIR 9000 Phase 2, 4 and 6. The contractor was advised that if it were to tender for AIR 9000 Phase 7, Defence would be exercising its rights under the Conditions of Tender to ensure Australian Aerospace had made progress in remediating the project, including the content of and timeliness in agreeing the remediation plan in the first instance, and then its subsequent implementation.

34. The tender evaluation plan for AIR 9000 Phase 7, Helicopter Aircrew Training System, dated 6 March 2012, referred to Projects of Concern in its initial screening process:

Non-compliant Tenders, Tenders associated with Projects of Concern, and any Alternative Tenders will be identified and a recommendation will be provided to the Delegate regarding appropriate treatment, including whether any non-compliant Tenders or Tenders associated with Projects of Concern should be excluded and whether any Alternative Tenders should be considered.

35. Additionally, the RFT criteria included: ‘Past performance of contractual obligations of the Tenderer, including involvement in any contract that is or has been listed as a Project of Concern.’

Disagreement in Defence on the removal criteria and timeframe for the Project of Concern listing

36. There has been some internal disagreement within Defence on the timeframe and criteria for removing the MRH90 Program from the Projects of Concern list.

37. In November 2013, a brief to the Minister noted that it might have been possible to remove the project from the list in early 2014. A Gate Review in December 2014 concluded that, despite costs and schedule risks, the Deed 2 agreement had led to significant improvement and the project was proceeding well. The CEO of DMO noted that the findings of the Gate Review Board were in direct contrast to the views of Navy as Capability Manager: ‘I have recently received a minute from Chief of Navy which is much more critical of the deficiencies [with the project] than this review indicates.’ A July 2018 minute from the Chief of Navy to Helicopter Systems Division, Capability Acquisition and Sustainment Group, noted that he had very little confidence in the contractor’s ability to deliver and that Defence should consider in-house solutions for the cargo hook.

38. Defence has forecast that the MRH90 Acquisition Project is expected to be removed in mid-2019.¹³⁸

MRH90 Sustainment

39. The sustainment ‘product’ corresponding to the acquisition project, CA48 Multi-Role Helicopters¹³⁹, has been on the Projects of Interest list since March 2017.

40. The October 2017 Independent Assurance Review for CA48 identified a number of issues with sustainment comparable to the issues for the MRH90 acquisition project including contractor delays. The Board noted that the most significant issue was the ‘high operational maintenance burden and associated inability to achieve an acceptable ratio of maintenance-hours to flying-hours, and a lack of responsiveness by [the sub-contractor] in a number of areas.’

Summary

41. It is not apparent that making the MRH90 program a Project of Concern made a difference in remediating the acquisition project. The key remediation activities — negotiating Deed 1 and Deed 2 — were already underway at the time of the listing. As noted in paragraph 13, the Minister commented in 2011 that ‘[Project of Concern] status seems somewhat empty if one has already gone down the Deed 1 [and] 2 route’.

42. The Project of Concern listing for the MRH90 program was used by Defence Senior Management to try to leverage industry before the listing (paragraphs 30–32, above), and in attempting to influence industry to resolve the ongoing cargo hook issue. It is not apparent that any benefit flowed from this attempt to influence industry behaviour, given the continued lengthy delays to the project schedule, and the ongoing technical issues. For example, the cargo hook issue was first identified in early 2012, and remains unresolved as of October 2018. Defence advised the ANAO in October 2018 that it expects the issue will be resolved by June 2019.

Appendix 6 Major case study 3 — CN10, Collins Class Submarine Sustainment

What was the project about?

1. Navy has six Collins class submarines that fulfil the roles of maritime strike and interdiction, maritime surveillance, reconnaissance and intelligence collection, undersea warfare and Special Forces operations. Construction of the first of the six submarines, HMAS Collins, began in 1990 with delivery in 1996. The final submarine, HMAS Rankin, was delivered in 2003.¹⁴⁰

2. In December 2003, Defence entered into a Through Life Support Agreement with the then Australian Submarine Corporation (now known as ASC). The agreement was worth $3.5 billion over 25 years (comprising a 15 year agreement with a further two five-year extension options).

How did the project become a Project of Concern?

3. The program was made a project of concern in November 2008 as the submarine fleet had failed to meet acceptable benchmarks for operational capability.

4. The Through-Life Support Agreement signed with ASC in 2003 had a number of shortcomings, and Defence noted in internal briefings that:

The terms of the [Through-Life Support Agreement] reflected, in part, the [Commonwealth of Australia’s] recognition of ASC’s weak financial position at the time and limited assured future work, and the strategic importance of having an Australian industrial capacity to provide submarine maintenance and capability enhancement.

5. Defence also noted that the Agreement provided little leverage to drive ASC to achieve greater efficiencies in sustainment of the submarines, but that achieving greater efficiency would be essential to the Commonwealth affording the submarine capability.

6. The Defence Budget Audit (Pappas Review),¹⁴¹ released in April 2009, indicated that the ASC sustainment activity was inefficient and that a 15 to 30 per cent increase in submarine availability could be achieved for the same cost.

7. By mid-2009, submarine availability had worsened — for the period from 19 to 21 June 2009¹⁴² there were no available submarines for training or operations, a situation Defence deemed unacceptable given the need to train submariners for the current and future capability. This prompted a review of submarine availability, completed in August 2009. The review identified a number of issues including that submarine availability was impacted by design defects, logistics support and poor industry performance. The review concluded there was a lack of understanding of the resource implications and management effort involved in Collins sustainment. This issue was further compounded by a lack of an integrated master schedule¹⁴³, and inefficiencies in industry support.

8. The review identified that the key focus should be providing the greatest number of materiel-ready days¹⁴⁴ to progress training in support of maintaining sufficient Navy submarine crew.

What was the remediation plan?

9. The Collins Class Sustainment program was regarded by Defence and its ministers as the ‘number one’ Project of Concern. In February 2010, a speech by the Minister for Defence Personnel, Materiel and Science noted that ‘Collins Class Submarine sustainment remains a challenge, and it is at the top of the list of Projects of Concern.’ The speech outlined the key remediation activities for Collins Class sustainment including: developing a new contractual approach with ASC; appointing a senior DMO executive to oversee Collins Class sustainment and establishing the Australian Submarine Program Office with the goal of improving submarine output.

10. The new contractual approach included the negotiation of an In-Service Support Contract, to replace the Through-Life Support Agreement with ASC. A Gate Review in June 2010 assessed the readiness of the program to enter the In-Service Support Contract. The Gate Review Board concluded that the submarine program was not ready and raised a number of significant issues with the program including continuing concerns with submarine availability, and significant financial shortfalls in the order of a half to one billion dollars across the coming decade.

11. The Gate Review listed 21 action items to assist in the program’s remediation, and recommended that the Submarine Program Manager conduct a fundamental review of Defence’s strategy for submarine sustainment before the In-Service Support Contract was negotiated.

12. The Collins Class Sustainment Review Phase 1 Report (Coles Review Phase 1) was completed in November 2011. Coles Review Phase 1 sought to make a number of early recommendations to improve the availability and overall program management of Collins Class submarines, before a more detailed Phase 2 report due for completion in late 2012. The Phase 1 report identified: poor availability of submarines caused by a crew shortfall; lack of spares and unreliable equipment; lack of cohesion in strategic leadership including a lack of clarity of accountability, authority and responsibility; thinly spread submarine domain knowledge; a lack of robustness of Navy’s contribution to manning and sustainment; no long term strategic plan for efficient asset utilisation; unclear requirements and unrealistic goals. The report recommended that the In-Service Support Contract should proceed as planned and should be used to drive the requirement for an early move to performance based contracting.

13. In June 2012, the In-Service Support Contract became operational. This contract was performance based and placed responsibility on ASC for delivering outputs using key performance indicators.

14. In November 2012, the Study into the Business of Sustaining Australia’s Strategic Collins Class Submarine Capability (Coles Review) was released. The study sought to identify the problems with Collins fleet sustainment performance and the initiatives that would address problems and resolve issues. The study assessed Collins Class sustainment against four categories: availability, planned maintenance duration, maintenance overrun and days lost to defects when not in maintenance. The report found that, compared to international benchmarks, the Collins Class: achieved 56 per cent of the benchmark for availability, spent 39 per cent more time on planned maintenance days, and had more than twice the expected maintenance overrun days and days lost to defects when not in maintenance.

15. The Coles Review made 25 recommendations aimed at remediating the Collins Class sustainment program including: implementing the In-Service Support Contract to encourage performance based behaviour, developing a through-life capability management plan, developing and implementing a workforce strategy to address skills shortages at the management level, and developing and implementing an information management strategy to inform decision making.

How did the project leave the Projects of Concern list?

16. In March 2014, the Study into the Business of Sustaining Australia’s Strategic Collins Class Submarine Capability (Coles Progress Review 1) was released, 15 months on from the November 2012 report, to assess the progress made towards improving submarine availability and performance.¹⁴⁵ Overall, the report noted an improvement in submarine availability, and progress towards achieving benchmark availability.

17. Following the Coles Progress Review, Defence considered removing the program from the Projects of Concern list due to the improvement in sustainment performance. However, after further consideration, the program remained on the list until the international benchmark availability mentioned in the Coles Progress Review had been achieved and maintained for a two year period.

18. The Study into the Business of Sustaining Australia’s Strategic Collins Class Submarine Capability – Beyond Benchmark (Coles Progress Review 2) was released in May 2016. The review noted a steady improvement in submarine availability and performance against international benchmarks. The table below illustrates the progress in remediating the Collins Class sustainment program across the three Coles Reviews.

Table A6.1: Progress in remediating the Collins Class sustainment program across the three Coles Reviews

Note a: This figure is the average from 2006–07 to 2010–11.

Note b: Defence defines a Materiel Ready Day as a day in which a submarine is not in planned maintenance or does not have defects that prevent it from being at sea. The Coles Reviews determined that cost per Material Ready Day was an indicator of cost effectiveness but noted in the 2014 review that increased cost effectiveness was a result of the increase in Materiel Ready Days and not a reduction in budget.

Source: ANAO analysis of: J.Coles, Study into the Business of Sustaining Australia’s Strategic Collins Class Submarine Capability, November 2012; J.Coles, Study into the Business of Sustaining Australia’s Strategic Collins Class Submarine Capability – Beyond Benchmark, May 2016; and J. Coles, Study into the Business of Sustaining Australia’s Strategic Collins Class Submarine Capability – Progress Review, March 2014.

19. The review also noted that 17 of the 25 recommendations made in the 2012 Coles Review had been implemented, with the remaining eight partially implemented.

20. The Coles Progress Review 2 noted that: ‘with benchmark availability on the verge of being achieved, the focus should now be on efficiency improvements and cost reductions across the sustainment program’.

21. Following the Coles Review, there were internal discussions within Defence to remove the Collins Class sustainment program from the Projects of Concern list in November 2016. In February 2017, the Projects of Concern report to the Minister for Defence noted that:

[The Collins Class Sustainment Program] has achieved significant improvement in areas identified by the 2012 Coles Report and has achieved international benchmark levels of availability. Removal may be considered if benchmark availability is maintained.

22. An Independent Assurance Review in May 2017 noted that while benchmark availability had been delivered for 2016–17, continued rigorous governance and reporting processes, including high level reporting to Government would be needed to keep them apprised of changing circumstances and action. The Independent Assurance Review Board recommended to the Minister that the Collins Class sustainment program be removed from the Projects of Concern list. The Minister did so in September 2017.

Analysis of the case

Effect of the Project of Concern listing on budget and capability

23. Despite being placed on the Projects of Concern list in late 2008, the costs of Collins Class sustainment continued to rise and the availability of submarines continued to decline. At the February 2011 Senate Estimates hearings Defence advised that, from 2004 to 2010, submarine sustainment costs went from $203 million per annum to $325 million per annum — a 60 per cent increase. At the same time, the number of submarines available was in decline. At the May 2011 Senate Estimates hearing, Defence advised that the sustainment cost had climbed by a further $90 million to $443 million per annum — a 25 per cent increase across five months and a 120 per cent increase over seven years.

24. In March 2011, a Defence Workforce and Funding Request was made to address the funding shortfall for Collins Class sustainment on behalf of the Chief of Navy. The proposal requested:

• Additional funding for 2011–12 and 2012–13 for the delivery of Navy’s Collins Class Submarine availability to meet Defence strategic guidance.
• That the In-Service Support Contract negotiation with ASC would require an additional $141.06 million be in place by 30 April 2011.
• Additional funds of $96.65 million were required to be in place by 31 May 2011 for inventory and obsolescence remediation.
• Further, the proposal noted a funding shortfall for 2016–21 of $1341.12 million.

25. The request noted that if funds were not made available, there was a risk to the schedule and operational availability of submarines.

26. The Coles Progress Review 2, released in 2016, noted that:

Attaining benchmark performance was a higher priority than efficiency. With benchmark availability on the verge of being achieved, the focus should now be on efficiency improvements and cost reductions across the sustainment program. Such cost reductions may be required to re-invest into inventory, obsolescence remedies, new infrastructure to manage an ageing fleet and the transition to future submarines.¹⁴⁶

27. The August 2017 Projects of Concern monthly report to the Minister for Collins Class sustainment outlined the improvement to availability and the achievements of benchmarks as set out in the Coles Reviews. However, the report also noted that to maintain this level of performance the program budget needed to remain as set out in the current Materiel Sustainment Agreement (the 2017–18 budget was $570.8 million), and that there were several issues to resolve requiring funds from sources other than the sustainment budget, to ensure they did not affect Collins Class performance.¹⁴⁷

Leveraging with Industry

28. During the program’s time on the Projects of Concern list, Defence used senior executive and Ministerial engagement to leverage with industry in an attempt to resolve issues. This included the Program Manager for Collins writing to the Managing Director of ASC in April 2011 to resolve key issues around the In-Service Support Contract, and meetings between the then Minister for Defence Materiel and the Managing Director and Chief Executive Officer of ASC to discuss progress on the Collins program, the status of the In-Service Support Contract and sustainment remediation. Explicit reference to the Projects of Concern listing was not used in correspondence with industry.

29. In a February 2015 brief to the Minister, Collins sustainment was used by Defence as an example of how Projects of Concern could be used to influence companies and see them apply significant efforts and resources to recover key programs. The brief noted that Collins sustainment is ‘a technically complex and challenging program requiring additional oversight to manage the sustainment of this strategically important capability. During its time on the Projects of Concern list, availability has improved and continues to become more reliable.’

Other Collins Class projects in Defence

30. In addition to the Collins Class Sustainment program, there are 13 major capital projects under management within the Collins submarine program to improve reliability and capability of the submarine fleet. Twelve of these projects have been approved by the government, and one has not yet been approved. Of these twelve, two are complete, and three are nearing completion and will be consolidated into one project.

31. One of the projects, SEA 1439 Phase 5B.1 Communications Antenna Capability Enhancement, was the subject of an Independent Assurance Review in May 2013. The review concluded that the management of the project was unsatisfactory resulting in the need for a significant cost increase and slippage to the original scheduled final operating capability of around 10 years. The project was not made a Project of Interest, or a Project of Concern and the Independent Assurance Review Board noted instead that: ‘While not an individual Project of Concern, SEA1439 Phase 5B.1 is part of the Collins Submarine Program, which is a Program of Concern as a whole.’

32. The relationship between Projects of Concern listing and remediation of the program is, on the face of it, more positive than in the other case studies. Defence has claimed that the listing for Collins Class sustainment played a significant role in the program’s remediation. Additionally, many of the Collins program reform initiatives began in 2009, after the November 2008 Project of Concern listing, including the establishment of the Australian Submarine Program Office, the agreement of the Integrated Master Schedule and the negotiation of the In-Service Support Contract with ASC.

33. However, the Coles Review Phase 1 report suggests that the remediation of the program and, in particular, the renegotiation of the Through Life Support Agreement, was prompted by the then Government’s announcement in 2008 that it would consider privatising the ASC. The Coles Review and its subsequent progress reports do not refer to the Project of Concern listing for the program, nor is it included as a consideration as part of the terms of reference for the reviews.