Background

1. Defence utilises satellite communications as the primary system for high and medium speed communication with mobile Australian Defence Force (ADF) platforms such as ships, aircraft and vehicles. Vulnerabilities in satellite communications make it necessary for Defence to have alternative means of long distance communication. The purpose of High Frequency communication capabilities is to provide an alternative means of long range communications; and the primary communication capability for ADF mobile platforms not fitted for satellite communication. Joint Project (JP) 2043 (the Project) was established to modernise the ADF's High Frequency communication system.

2. The scope of Phase 3A of the Project involves the development and implementation of a modernised High Frequency communication fixed network and the upgrade of communications equipment on selected mobile platforms (see Figure 1). This Phase was approved in 1996, with a budget of $505 million. By June 2006, the budget had increased by $111 million to $616 million, primarily through price and currency adjustments.

3. The Prime Contract was awarded to Boeing Australia Limited (the Prime Contractor) in December 1997 for completion in May 2004. The Contract was a fixed price contract initially comprising 40 per cent milestone payments and 60 per cent earned value payments. The negotiated price for the Prime Contract was $312.20 million (1996 prices). A Network Operation and Support Contract was also executed in December 1997 for $72.48 million (1996 prices). The Network Operation and Support Contract commences after the final stage of the fixed network is accepted which is currently scheduled for late 2007.

4. The Prime Contract required the development of a High Frequency communication fixed network in two stages which are:

• the Core Communication System, to provide a High Frequency communication capability at a level no less than that provided by the networks being replaced¹; and
• the Final Communication System, to provide an enhanced High Frequency communication capability².

Figure 1 High Frequency Modernisation – communications with mobile platforms

Source: Defence Documentation

5. The High Frequency communication equipment in selected ADF sea, land and air mobile platforms is required to be upgraded to utilise the full capability to be provided by the Final Communication System³. The types of mobile platforms to be upgraded by the Project were altered significantly in 1999 after a review by the Defence Capability Committee to reflect changing strategic circumstances. Following that review a significant number of mobile platforms that were to be upgraded under the original Prime Contract were removed from the Contract in 2004⁴. By early 2007 arrangements for the majority of mobile platforms now required to be upgraded using Project funds were yet to be incorporated into a contract.

6. The objective of the audit was to assess the effectiveness of the management by Defence and the DMO of the procurement of the modernised High Frequency communication capability for the ADF. The audit focussed on Phase 3A of the Project which commenced in the mid 1990's and involved the selection of the Prime Contractor; negotiation of the Prime Contract and the Network Operation and Support Contract; and the development and implementation of the Communication System.

Overall audit conclusions

7. The modernisation of the High Frequency communication system is a complex Project involving the upgrade and construction of facilities and infrastructure; systems engineering; software development; and platform integration. The ANAO found that at the execution of the Prime Contract in late 1997, requirements were not clearly identified and risks associated with systems engineering and software development were acknowledged as being significant to the Project. These risks have materialised because risk mitigation measures applied in the formation of the Prime Contract were less successful than both the Contractor and Defence expected. As a consequence of these issues, and the overall project complexity, the contract schedule was extended three times, each being documented in a Deed. The outcome of these negotiations has been a delay to the Project schedule combined with resolution of a number of issues that had been in dispute.

8. In the period from contract signature to late 2003, difficulties were encountered in achieving project deliverables associated with systems engineering and software development. A number of significant scope changes were made to the Prime Contract over this period that were not resolved with the Prime Contractor until the Contract was rebaselined in 2004⁵. Rebaselining included scope increases regarded as necessary to support the operation of the Core Communication System; and to ensure the Final Communication System was capable of meeting current and future ADF requirements. The contractual costs of these scope changes were largely offset by cost reductions associated with the removal of the High Frequency Direction Finding capability and the removal of nine of the 10 types of mobile platforms to be upgraded later in the Project.

9. The removal of these platforms from the Prime Contract effectively represents a deferral of capability and expenditure, in that the ability to achieve the contracted levels of system functionality is contingent on mobile platforms being fitted with equipment fully compatible with the Final Communication System. Some of the capabilities such as Automatic Link Establishment will, however, be able to be used by some platforms. The ANAO considers that significant risk remains to the schedule for acceptance of Final Communication by late 2007, and the upgrade of mobile platforms by the end of 2010.

10. This Project identified the need to carefully control risks associated with projects that have a large developmental component. Key areas where the Project outcomes may have been improved include: the identification and resolution of risks in the pre contract phases; the transition of risk into the contract; and the management of risks as they transpired in the post contract phase. The ANAO acknowledges that over the life of this Project, the DMO and Defence have undertaken significant procurement reforms, including the 2003 Defence Procurement Review. Lessons learnt from this Project provide a range of insights into areas where the DMO and Defence might further refine these processes which are particularly relevant given the prominence of information integration projects in delivering future capability to the ADF.

Key Findings

Tender Process

11. The Project has involved three phases with the first two phases involving a series of studies conducted by potential suppliers to reduce the Project risk. During these phases the two short listed suppliers established teams which needed to be retained in anticipation of being awarded the Prime Contract under Phase 3A. The cost associated with retaining these teams influenced Defence's approach to the tendering process. The normal project approval process was altered to expedite the tendering process; with the Request for Tender being released to the short-listed suppliers in July 1996, prior to the Government's approval of Phase 3A in August 1996⁶. Only one of the two tenders received provided a quoted price, that when combined with other estimated Project costs, was within the approved Project budget.

12. Throughout the tender process a series of internal and third party reviews commented on weaknesses in the requirements set out in the Request for Tender documentation. Both of the tenders submitted had limitations in terms of compliance with the tender requirements and the level of detail provided. These factors and tender evaluation procedural issues resulted in a decision being taken to reassess various aspects of both tenders. Following this process the initial assessment of the preferred tenderer's approach to systems engineering and software development was upgraded. The need to expedite contract negotiations resulted in the resolution of risks being carried forward into the Prime Contract.

13. Risks associated with requirements instability, software development and systems engineering were identified at the time of contract signature. These risks were inadequately addressed through pre-contract negotiations. These risks subsequently had a significant impact on the Project in terms of schedule and the capability delivered under the Prime Contract. This experience underlines the importance of Defence having well defined and stable requirements and contractors having sound systems engineering and software development processes.

Contract Management

14. By November 2006, the DMO had expended $328.82 million, representing 53 per cent of the Project budget. The bulk of this expenditure was against the Prime Contract. The remaining Project budget, as at November 2006 was $285.4 million with the largest areas of planned expenditure being $81.7 million for the remaining mobile platform upgrades; $69.6 million expenditure against the Prime Contract and associated Contract Change Proposals; and $36.6 million for the first three years of the Network Operation and Support Contract.

15. Over the period from contract signature in December 1997 to mid 2003, contract management issues proved to be problematic with important Contract Change Proposals taking several years to resolve and a series of contract milestones not achieved. Arrangements applied by the DMO were unsuccessful in resolving issues and requirements instability, payment arrangements and ongoing delays compounded project management difficulties. Inappropriate processes for the escalation of issues within the DMO at the time placed the Project at significant risk.

16. The ongoing inability to achieve milestones, particularly those associated with systems engineering and software development, resulted in the DMO suspending earned value payments in April 2003. In early 2004, the Prime Contract was rebaselined with the execution of a Deed of Settlement and Release. Rebaselining involved three key elements including: agreement on delays to the delivery of the Core and Final Communication Systems and mobile platform upgrades; scope changes; and compensation for costs incurred by Defence due to the delayed delivery of the Core Communication System. Following rebaselining, payment arrangements under the Prime Contract transitioned from a combination of earned value and milestone payments to solely milestone based payments.

17. Scope increases associated with the delivery of the fixed network had a combined value of $32.74 million. These scope increases were considered necessary to support the operation of the Core Communication System and to ensure that the Final Communication System satisfied future requirements⁷. Cost reductions due to the removal of the requirement to deliver certain capabilities under the Prime Contract meant that cost increases associated with rebaselining only marginally increased the value of the Prime Contract.

18. The amendments to the mobiles to be upgraded under the Prime Contract were based on decisions taken by Defence in 1999 and 2002 to reflect changed circumstances within the individual Services. Following these changes all Air Force airborne mobile platforms and Navy Major Fleet units were removed from list of platforms to be upgraded by the Project. These changes resulted in the number of types of military platforms to be upgraded under the Prime Contract being reduced from 10 platforms to a single platform. The total price reduction to the Prime Contract achieved through the removal of nine mobile platforms types was $21.90 million (1996 prices).

19. An ongoing requirement under the Prime Contract is the upgrade of a First of Type Chinook helicopter and the provision of five Chinook follow-on upgrade kits which are scheduled for delivery in mid to late 2007. The requirement to develop generic mobile upgrade systems was inserted into the Prime Contract through rebaselining in order to prove concepts prior to upgrading platforms and thereby reduce risk⁸. Defence advised the ANAO that the generic mobile systems will be used to demonstrate enhanced mobiles communications capabilities to the user communities in the individual Services. A cost increase to the Prime Contract of $5.81 million (1996 prices) was agreed to fund the development of the generic upgrade systems and other costs associated with the upgrade of the Chinook helicopter.

20. The communications equipment on a further eight mobile platforms types is required to be upgraded using Project funds. In early 2007, planning was ongoing for these mobile platform upgrades and for the incorporation of follow-on upgrade kits into the five remaining Chinook helicopters. The cost of upgrading these eight platforms was yet to be clearly identified by the DMO in early 2007 as contractual arrangements for these upgrades were yet to be established.

Capability Delivery

21. Due to requirements instability and scope change within the Project, a Deed was signed in 1999 granting a seven and a half month schedule delay. The Project continued to encounter schedule slippage through to late 2003. Through contract rebaselining in early 2004, the acceptance of the Core Communication System was rescheduled to August 2004, and the acceptance of the Final Communication System was rescheduled to November 2007. Table 1 outlines the extent of these schedule delays.

Table 1 Schedule for DMO acceptance of fixed network

 Source: Developed for DMO acceptance of fixed network

22. The delay in the acceptance of the Core Communication System resulted in the DMO claiming that additional costs would be incurred due to the requirement to keep legacy communication stations operational for an extended duration. Through the 2004 Deed of Settlement and Release the DMO reached agreement with the Prime Contractor for the recovery of significant liquidated damages and the provision of work-in-kind as compensation for costs incurred due to the delay.

23. The Core Communication System was delivered in July 2004 and accepted by the DMO in October 2004. Following Initial Operational Release⁹ in November 2004, the system experienced a range of operational issues that required resolution prior to the closure of all legacy communication stations. Following the implementation of measures to resolve these issues the last legacy communication station was closed in late 2005, with the ADF becoming fully reliant on the Core Communication System for High Frequency communications.

24. In May 2005 a Deed of Agreement was negotiated to address schedule risk associated with the delivery of the Final Communication System. Through this process the DMO gained a series of benefits related to warranty provisions for the Final Communication System and reduced costs associated with the ongoing operation of legacy communication stations. These were offset against a reduction in review processes; changes to the software development approach; and the removal of an Intermediate Communication System that was to be delivered as an interim stage between the delivery of the Core and Final Communication System. Challenges remain in achieving the schedule for the Final Communication System as achievement of this schedule is largely contingent on a critical software development milestone being met in mid 2007.

25. The DMO has applied a significant level of effort towards resolving issues associated with the development and integration of the Modernised High Frequency Communication System into the mobile platforms. Much of the work in this area was ongoing in early 2007. A range of complex issues are yet to be fully resolved to finalise the mobile upgrade program by late 2010 and conclude this Project. These encompass platform specific software development and integration and are subject to the acceptance of the generic mobile upgrade systems and the availability of platforms for upgrade. Defence advised the ANAO in January 2007 that a reasonable risk remains in this aspect of the Project, mainly in terms of schedule, and that this risk will remain, if for no other reason than platform availability.

Defence and DMO response

26. The Department of Defence provided a response on behalf of the DMO and Defence (see Appendix 1). The Defence Response stated that:

Defence notes that the report provides a summary of key events that have occurred over the life of the Project. The High Frequency Modernisation Project is a complex software intensive and high risk project involving geographically diverse sites at five major locations around Australia. The Core System, which provides the majority of the contracted functionality, has now been operational for over two years and is providing excellent service to the ADF.

Defence received significant compensation from the Prime Contractor for delays arising from problems with the contractor's system engineering and software development effort. The value of the compensation was consistent with the level of estimated losses caused by the delay and there were no scope changes as a result. Risk mitigation measures incorporated into the contract based on recognised international practice at that time were not as successful as the Contractor and Defence had expected.

Defence notes that the level of delay experienced in this project is comparable to international experience with similar projects reviewed by the Standish Group International over the period 1994 to 2004. Internationally, the processes, tools and techniques for managing complex software development projects have matured significantly over the past ten years resulting in measurable improvements in project performance. For example, according to the latest Standish Report the average project delay for similar projects has improved from 160% in 1994 to 84% in 2004. The delay to the Core System of the High Frequency Modernisation Project was 72%. DMO continues to monitor progress in these areas and adopt relevant practices.

Since 1997 when the Prime Contract was awarded the DMO has implemented a suite of acquisition initiatives including standard contracting templates for software intensive projects, benchmarking of process improvement based on international best practice using the Capability Maturity Model Integrated developed by the Software Engineering Institute in the US, and improved measurement regimes to further improve project outcomes.

To reduce delays resulting from unstable requirements Defence has instituted more rigorous requirements development processes. Projects now require an Operational Concept Document, Function and Performance Specification and Test Concept Document before approval. These requirements have been further strengthened by changes made following the Defence Procurement Review.

Footnotes

1 Core Communication System capabilities include the transmission of organisational messages, voice (both non-secure and encrypted), facsimile and data. The Core Communication System involved the construction and redevelopment of facilities and infrastructure; the procurement of equipment; systems engineering; and software development and accounted for a significant proportion of the value of the Prime Contract.

2 Enhancements to be provided by the Final Communication System include greater levels of automation, improved communication protocols, a higher traffic capacity, new traffic types and greater reliability. Following implementation of the Final Communication System there will be separate primary and back-up Network Management Facilities.

3 The system is intended to enable communications with platforms that have not been upgraded and with allied forces.

4 The Prime Contractor advised the ANAO that the development of a generic mobile for air platforms and a generic mobile for land/sea platforms will support the reintroduction of specific mobile platforms in the future.

5 The Prime Contractor advised the ANAO that during 2003-04 both the DMO and the Prime Contractor put in place a new project management team that has supported much of the Project recovery and many of the successful results being achieved since that time.

6 Defence advised the ANAO that the practice of obtaining tender quality prices prior to approval is consistent with the processes introduced following the 2003 Defence Procurement Review.

7 The Prime Contractor advised the ANAO that the scope changes negotiated for Core and Final Systems required new or amended requirements to be engineered into the systems design with a corresponding impact on the delivery schedule. Through a series of Deeds and subsequent Contract Change Proposals many of these delays were approved into the Contract baseline.

8 There are two generic High Frequency upgrade systems: a generic air upgrade system and a generic land and sea upgrade system. Development of these systems includes non-recurring and platform-independent engineering activities and is scheduled for completion in late 2007.

9 Acceptance by the DMO under the Prime Contract is followed by a two stage process involving Initial Operational Release; and Operational Release. Initial Operational Release is generally undertaken shortly after contractual acceptance. It is the milestone where the relevant authority is satisfied that the operational state of the equipment including deficiencies, training and supportability elements, are such that it is safe to proceed into the Operational Test and Evaluation Period. Operational Release represents the in-service date at which the relevant authority is satisfied that the equipment is, in all respects, ready for operational service. Operational Release is not planned to occur until after the acceptance of the Final Communication System.