Background

1. Defence materiel sustainment is about the maintenance and support of Defence’s fleets of specialist military equipment: the provision of in-service support for naval, military and air platforms, fleets and systems operated by Defence. Effective sustainment of naval, military and air assets is essential to the preparedness of the Australian Defence Force (ADF) and to enable Defence to conduct operations. Defence spends similar amounts each year on sustainment and the acquisition of new equipment. In 2015–16, Defence spent $6.3 billion—21 per cent of its total departmental expenditure—on the sustainment of specialist military equipment.

2. Parliamentary committees have frequently stated an interest in Defence’s reporting of its sustainment performance and, in particular, obtaining greater insight into that performance. During the course of this audit the Joint Committee of Public Accounts and Audit (JCPAA) announced that it had commenced an inquiry into Defence sustainment expenditure.

3. Part of the context for this audit is to inform the Parliament regarding Defence’s approach to managing sustainment expenditure, including in the context of the current JCPAA inquiry, and to inform the ANAO’s future program in this area.

Audit objective and criteria

4. The objective of the audit was to assess whether Defence has a fit-for-purpose framework for the management of materiel sustainment. To form a conclusion against the audit objective, the ANAO adopted the following high-level criteria:

• Defence has established an appropriate governance and operational framework for the management of materiel sustainment;
• Defence has established and implemented a high quality performance framework to support the management and external scrutiny of materiel sustainment; and
• Defence has achieved key outcomes expected from the Smart Sustainment reforms and has progressed its implementation of the reforms to sustainment flowing from the First Principles Review.

Conclusion

5. The fundamentals of Defence’s governance and organisational framework for the management of materiel sustainment are fit-for-purpose. However, Defence continues to address specific operational shortcomings and there remains scope for Defence to improve its performance monitoring, reporting and evaluation activities to better support the management and external scrutiny of materiel sustainment.

6. Defence has clear and long-standing governance and organisational arrangements for managing the sustainment of specialist military equipment.

7. Research and reviews conducted for Defence have revealed a range of specific operational problems that are detracting from the efficient and effective sustainment of Defence capability, including the functioning of Systems Program Offices. Defence has initiated a reform project as part of its First Principles Review implementation.

8. The development of an effective sustainment monitoring system remains a work in progress, and the effectiveness of Defence’s internal reporting system for sustainment could be improved in several areas. Opportunities also remain to increase the completeness and transparency of publicly reported information regarding materiel sustainment.

9. The 2015 First Principles Review was preceded by earlier major reform initiatives, notably the Strategic Reform Program (SRP) begun in 2009. Defence records show that the department made substantial efforts to keep track of the large number of diverse initiatives identified across the department under the ‘smart sustainment’ reforms associated with the SRP, including internal reporting to management. However, Defence did not adequately assess the outcomes from the ‘smart sustainment’ reforms.

10. Reforms to the management of sustainment flowing from the First Principles Review remain at an early stage, and this stream of activity is likely to take much longer than the expected two years. For example, the Systems Program Office reviews are not yet complete and Defence has provided no evidence that decisions have been taken on changes to their structure and functioning.

11. Defence has engaged industry expertise to guide and help it with the First Principles reforms relating to acquisition and sustainment, including $107 million with a single company where the contract for services is not performance-based. Reform is expected to lead to greater outsourcing of functions currently performed in-house by Defence’s Systems Program Offices.

Supporting findings

Defence’s governance and operational framework for the management of materiel sustainment

12. Defence has clear and long-standing arrangements for managing the sustainment of specialist military equipment. Key elements of the governance and organisational framework include: a specialist organisational unit—currently the Capability Acquisition and Sustainment Group—staffed by a mix of civilian and military personnel; contract-like arrangements for sustainment between that unit and the Chiefs of Navy, Army and Air Force (Defence’s Capability Managers); and day-to-day responsibility for most sustainment activities falling to Systems Program Offices, which carry out that work using a mix of in-sourced and out-sourced service provision. Military units also undertake some operational-level sustainment activities.

13. Nevertheless, an internal review conducted by Defence and research conducted for Defence following the 2015 First Principles Review have identified a range of operational problems that detract from the efficient and effective sustainment of Defence capability, including: adherence to procurement principles; staff capabilities; duplication of effort and transparency of internal costs. A project to reform and consolidate Systems Program Offices is currently underway, with 24 out of 64 Systems Program Offices (37.5 per cent) reviewed as at February 2017.

Defence’s performance framework for materiel sustainment

14. With the introduction of its Sustainment Performance Management System (SPMS), Defence continues to develop a basis for an effective monitoring system for sustainment. Once fully implemented this system should be capable of systematically reporting against a suite of performance indicators settled in agreement with Capability Managers.

15. There remains potential to improve some core key performance indicators used within the system—for example, to more usefully determine the total cost of the capability to Defence. The SPMS system was not fully implemented during audit fieldwork but is expected to be fully operational by the end of June 2017. In the longer term the system is to be expanded to cover acquisition.

16. The effectiveness of Defence’s internal reporting system for sustainment could be improved in several areas. The Quarterly Performance Report is the primary way by which Defence provides information to government and senior Defence personnel about the status of major acquisition and sustainment activities. However, based on the ANAO’s review of a Quarterly Performance Report produced during the audit, its contents are neither complete nor reliable, it takes two months to produce and its contents are sometimes difficult to understand. The ANAO’s analysis found that the report may not include additional information available to Defence that is critical to the reader’s ability to understand the status of significant military platforms. It provides only a partial account of materiel sustainment within Defence and is potentially at odds with the ‘One Defence’ model promoted by the First Principles Review. The ANAO has recommended that Defence institute a risk-based quality assurance process for information included in the Quarterly Performance Report.

17. Defence conducts reviews of sustainment performance through sustainment gate reviews that help Defence to obtain insight into a project or product’s progress and status. The effectiveness of these reviews could be increased if the lessons obtained from gate reviews were routinely incorporated into management reporting on sustainment and if gate reviews were extended to contribute to the proposed quality assurance mechanism for Quarterly Performance Reports.

18. Defence has not implemented measures of efficiency and productivity for all sustainment products. Reviews have consistently emphasised the need for Defence to improve the efficiency of its operations, including in sustainment. Most recently, the First Principles Review recommended immediate implementation of measures of productivity, a related concept. The ANAO found that, 18 months after implementation commenced, there had been limited progress.

19. Defence has recently improved its whole-of-life costing of proposals to acquire major capital equipment but remains unable to measure or report reliably the total cost of ownership. It is now planning to implement a new model, which seeks to capture the full cost of ownership throughout the life of an asset, with implementation planned for completion in July 2018. The First Principles Review has pointed out that Defence has treated Systems Program Office staff costs at the project level as a free good, reducing the transparency of the cost of sustainment work and providing inaccurate price signals and a distorted incentive structure to Capability Managers.

20. Opportunities remain to improve the quality and transparency of Defence’s publicly reported information regarding materiel sustainment, while being sensitive to national security concerns. Areas for improvement include:

• achieving a clearer line of sight between planning and reporting documents—for both expenditure and descriptive information at the corporate and project levels; and
• the use of consistent time series data and analysis of sustainment expenditure.

21. Defence’s second corporate plan prepared under the Public Governance, Performance and Accountability Act 2013 (PGPA Act) identifies sustainment more clearly than in the first corporate plan. Under the PGPA Act, Defence will need to report meaningfully against this plan in its annual performance statement.

22. Defence’s 2015–16 public reporting of sustainment activity included expenditure information and other descriptive material. However, it was not: complete; consolidated in one easy to locate area; prepared in a manner which permitted the comparison of actual expenditure against estimates; or consistent in its presentation of clear reasons for full-year variances. Further, performance summaries were highly variable and inconsistent between public planning documents—Defence’s Portfolio Budget Statements and Portfolio Additional Estimates Statements—and the Annual Report.

23. Defence has not published program level expenditure data on a consistent basis over time, or time series analysis, to assist with external scrutiny of its sustainment expenditure.

Smart Sustainment reforms

24. There is no record of whether the intended savings to sustainment costs were achieved from the Defence Materiel Organisation’s (DMO) 2008 initiative to identify efficiencies. A target expenditure cut of five per cent of sustainment costs over a financial year was set by the Chief Executive Officer of the DMO in February 2008. Defence expected savings from: reducing spares; making more use of performance-based contracting; discouraging unnecessary end-of-year spending; travel efficiencies and improved maintenance philosophies.

25. Defence has not kept a systematic record of the outcomes of the Smart Sustainment initiative associated with the 2009 Strategic Reform Program. In 2011, a ‘health check’ by external consultants found some early successes with cost reductions and changes in practice. Nevertheless, the consultants considered the program was failing because of shortcomings in governance, program management, and Defence’s approach to reform described by a major vendor as ‘minor reform, driven by piecemeal, top down budget pressure’.

26. Smart Sustainment had a ten-year savings target of $5.5 billion. In its 2014–15 Annual Report, Defence claimed to have achieved $2 billion of savings from the initiative in its first five years. Defence has not been able to provide the ANAO with adequate evidence to support this claim, nor an account of how $360 million allocated as ‘seed funding’ for Smart Sustainment initiatives was used.

Materiel sustainment reform—First Principles Review

27. Defence has drawn heavily on contracted industry expertise to support its implementation of the program of organisational change relating to acquisition and sustainment that has followed the First Principles Review. This represents a major investment of at least $120 million, including contracts for services from the principal provider (Bechtel) valued at some $107 million. Contracts with the principal provider are not performance-based.

28. Systems Program Office consolidation and reform is one of the largest reforms to Capability Acquisition and Sustainment Group deriving from the 2015 First Principles Review. This stream of activity is likely to take much longer than the two years expected for implementation. In June 2017, Defence advised the ANAO that implementation plans for significant reforms to Systems Program Offices would be developed in the second half of 2017.

29. Other reforms flowing from the First Principles Review remain underway:

• Introducing performance-based contracting into Defence sustainment has been underway for over a decade. Defence does not yet have a completed register of its acquisition and sustainment contracts though, since January 2016, it has had a facility in place and had commenced populating it.
• Initial establishment of ‘centres of expertise’ in Capability Acquisition and Sustainment Group is underway. Defence expects full implementation to take a further two years. Similarly, the new Capability Acquisition and Sustainment Group Business Framework is expected to take ‘many years’ to fully implement.
• Defence has developed its own meaning for the term ‘smart buyer’, which does not clearly articulate the intent of the First Principles Review recommendation. This introduces risks related to ensuring that Systems Program Office staff working on outsourcing have the necessary skills and competencies.

30. Defence has also been developing its approach to asset management over many years, having obtained substantial advice from internal and external sources to adopt a Defence-wide asset management strategy to underpin a sustainment business mode for specialist military equipment. It is now not clear whether Defence will continue with this work.

31. Defence has not put in place plans to evaluate either the reforms themselves or its implementation of them. There is a risk that insights into a very substantial reform process could be lost. This was the case with the earlier Smart Sustainment reforms. The ANAO has recommended that Defence develop and implement an evaluation plan.

Parliamentary interest

32. In light of the Parliamentary interest in Defence sustainment (see paragraph 1.4 forward) the ANAO has considered, on the basis of the current audit’s findings, the issue of enhanced scrutiny of sustainment through a process similar to the existing Major Projects Report (MPR).¹ The MPR has been prepared annually over the last decade to provide independent assurance of the status of selected Defence major acquisition projects. The MPR has added value through the review process and the transparency of information providing increased assurance to the Parliament and the Government. This has been achieved while managing risks to national security.²

33. A key step in the MPR process is the preparation of a Project Data Summary Sheet for each project being reviewed. It is apparent from this audit that Defence has developed information systems that could support the preparation of similar information for its sustainment work. Defence has undertaken work on performance measures for sustainment and has developed the infrastructure to collect and report on its sustainment work (see Chapter 3).

34. Three principal issues remain for the conduct of a process for sustainment parallel to the MPR for acquisition:

• First, the effective management of risks to national security which can arise with the exposure of details of the readiness and availability of Defence capability. This issue could be managed provided the scope of any sustainment review is appropriately selected. For example, a small number of products could be selected, with rotation or other variation from year-to-year, limiting the risks that may flow from time-series analysis and the release of other material whose aggregation could add risk.³
• Second, the material to be produced for scrutiny of sustainment performance would need to reflect a ‘One Defence’ (that is, whole of portfolio) view. As noted in this audit report, the focus of some current arrangements requires clarification as it may reflect only the performance of Capability Acquisition and Sustainment Group.
• A final consideration is that of resourcing. In setting the scope and methodology of any review of sustainment performance, the costs and benefits of the proposed program of work and its relationship to existing scrutiny arrangements would need to be considered, both for Defence and for the ANAO.

35. The ANAO has also observed in Chapter 3 of this audit report that there is scope to improve the quality and transparency of public reporting on sustainment in existing Defence reports—the Portfolio Budget Statements, Portfolio Additional Estimates Statements and Defence Annual Report.

Recommendations

Entity response

The Department of Defence welcomes the Australian National Audit Office (ANAO) performance audit on Defence’s management of materiel sustainment. Defence’s comments, and suggested editorial amendments, have been provided to the ANAO.

Defence notes the findings of the ANAO, and agrees to both recommendations.

Defence concurs with the ANAO’s conclusion that the fundamentals of the governance and organisational framework for the management of sustainment are clear, and fit for purpose.

Defence notes that the audit was conducted at a strategic level, and acknowledges the opportunities for improvement identified by the ANAO. Defence continues to strive towards efficiencies in the delivery of sustainment outcomes for Defence capability and to report on outcomes, noting the need to balance transparency and accountability to the Australian public and Parliament, with the national security interests of the Commonwealth.

Defence’s implementation of the recommendations of the First Principles Review has also introduced a single end-to-end capability development function, the Capability Life Cycle, which will reduce previous delineations between the management of acquisition and sustainment activity.

1.1 Defence materiel sustainment is about the maintenance and support of Defence’s fleets of specialist military equipment. Defence defines sustainment as involving the provision of in-service support for specialist military equipment, including platforms, fleets and systems operated by Defence. Typically, sustainment entails repair and maintenance, engineering, supply support and disposal of equipment and supporting inventory.⁴ Effective sustainment of naval, military and air assets is essential to the preparedness of the ADF and to enable Defence to conduct operations in accordance with the Government’s requirements.⁵

1.2 Defence has commented that sustainment management is not a technical discipline: ‘it is an over-arching business-oriented management function focused on meeting the outcomes required of Capability Acquisition and Sustainment Group customers’.⁶

1.3 In recent years, Defence has spent, each year, similar amounts on sustainment and the acquisition of new equipment (Figure 1.1, below). In 2015–16, Defence spent about $6.3 billion—21 per cent of its total departmental expenditure—on sustainment of specialist military equipment.

Figure 1.1: Defence acquisition and sustainment expenditure

Note: For 2005–06 to 2014–15, expenditure was made through the Defence Materiel Organisation. Figures for 2015–16 are based on Defence advice with one-third of Capability Acquisition and Sustainment Group staff costs allocated to acquisition and two-thirds to sustainment.

Source: Defence Annual Reports 2005–06 to 2014–15, and Defence advice.

Parliamentary interest

1.4 Parliamentary committees over several years have stated an interest in Defence’s reporting of its sustainment performance and, in particular, obtaining greater insight into that performance.⁷ In May 2014, the JCPAA recommended that the DMO prepare a suitable methodology for reporting sustainment activity and expenditure, within six months.⁸ The Government disagreed on the basis that current arrangements balanced Parliamentary scrutiny of sustainment expenditure with the protection of classified information on the military capability, readiness and availability associated with that sustainment.⁹ In September 2014, the Committee sought a sustainment options paper from the ANAO, which was made public by the Committee. Defence subsequently provided the Committee with an in camera briefing (November 2015), consistent with an option in the paper put forward by the ANAO. The Committee expressed the view, in May 2015, that:

Sustainment expenditure is currently at approximately $5 billion per annum and predicted to increase significantly over time. The Committee considers sustainment expenditure to be an area requiring further parliamentary scrutiny on the adequacy and performance of Defence involving billions of dollars in the future.¹⁰

1.5 The Committee noted, however, that the final structure for sustainment reporting was as yet undecided. During the course of this audit (9 January 2017) the Committee announced that it had commenced an inquiry into Defence Sustainment Expenditure. Part of the context for this audit is to inform the Parliament as to whether Defence has developed its management reporting sufficiently to facilitate a program of ANAO assurance reviews of selected sustainment activities.

1.6 The ANAO’s observations on this aspect of the audit are reflected in the Summary and Recommendations section of this report (paragraph 32 forward).

Audit approach

1.7 The objective of the audit was to assess whether Defence has a fit-for-purpose framework for the management of materiel sustainment. To form a conclusion against the audit objective, the ANAO adopted the following high-level criteria:

• Defence has established an appropriate governance and operational framework for the management of materiel sustainment;
• Defence has established and implemented a high quality performance framework to support the management and external scrutiny of materiel sustainment; and
• Defence has achieved key outcomes expected from the Smart Sustainment reforms and has progressed its implementation of the reforms to sustainment flowing from the First Principles Review.

1.8 The scope of this performance audit is the management of sustainment of specialist military equipment. It addresses sustainment management at a high level and is not focused on the sustainment of individual platforms or the technical aspects of their maintenance. Defence’s broader sustainment program also includes the maintenance and support of corporate information technology and estate and infrastructure assets. These activities are outside the scope of the audit.

1.9 The audit was conducted in accordance with ANAO Auditing Standards at a cost to the ANAO of approximately $422 000.

1.10 The team members for this audit were Dr David Rowlands, Kim Murray and David Brunoro.

Does Defence have a clear governance and operational framework for the management of materiel sustainment?

2.1 The ANAO examined Defence’s governance and operational arrangements for managing the sustainment of specialist military equipment, with a particular focus on the clarity of:

• roles and responsibilities; and
• processes for documenting capability managers’ requirements around sustainment.

Governance and operational arrangements

2.2 Defence has clear and long-standing arrangements for the management of materiel sustainment for specialist military equipment. Key elements are:

• since 2000, a specialist acquisition and sustainment unit—previously the Defence Materiel Organisation (DMO) and most recently the Capability Acquisition and Sustainment Group—staffed by a mix of civilian and military personnel;
• maintenance by the specialist unit of physically dispersed sub-units—known as ‘Systems Program Offices’—with day-to-day responsibility for Capability Acquisition and Sustainment Group sustainment activities¹¹;
• since 2005, contract-like arrangements known as ‘Materiel Sustainment Agreements’ between the specialist unit and the Australian Defence Force’s capability managers—primarily the Chiefs of Navy, Army and Air Force;
• a mix of in-sourced and out-sourced service provision for sustainment activity; and
• reliance on a mix of public and private equipment and facilities for sustainment activity.

2.3 While a number of reviews have focused on the optimal distance of the specialist entity from the Department of Defence¹²—with changes over time in its degree of separation—and have made recommendations on the unit’s operations and practices, the fundamentals of the sustainment management framework have been stable for over a decade.

Capability Acquisition and Sustainment Group

2.4 The sustainment of specialist military equipment was formerly a core responsibility of the DMO. Following government acceptance of a recommendation of a review of Defence—the First Principles Review, which reported in April 2015—DMO was delisted from 1 July 2015.¹³ DMO’s sustainment role then passed immediately to Defence’s Capability Acquisition and Sustainment Group.¹⁴ As at November 2016, Defence employed about 5500 people in that Group.¹⁵

2.5 Defence describes the accountabilities of the Group as follows:

The Capability Acquisition and Sustainment Group (CASG) purchases and maintains military equipment and supplies in the quantities and to the service levels that are required by Defence and approved by Government.¹⁶

We are accountable to:

• The Australian Government through the Defence Ministers (our owner).
• The Secretary of Defence and Chief of the Defence Force.¹⁷
• The women and men of the ADF through the capability managers (our customers).
• Defence industry (our partners).

Systems Program Offices

2.6 As at February 2017, Defence had some 64 Systems Program Offices, which manage the acquisition, sustainment and disposal of specialist military equipment. They do this through a combination of internal work and commercial contracts. In January 2017 Defence’s Systems Program Offices managed the sustainment of 112 fleets of equipment and services. Systems Program Offices are located in different parts of the country but are organisationally all part of the Capability Acquisition and Sustainment Group within Defence. They may also be involved in acquisition projects but the greater bulk of their work involves sustainment.

2.7 In most cases, a major platform—such as an aircraft type or class of ship—is managed by a single Systems Program Office.¹⁸ Systems Program Offices maintain a relationship with industry bodies, particularly those providing maintenance services, spares, engineering and other support, and with the representative of the relevant capability manager.¹⁹ Each Systems Program Office has contracts with industry suppliers of services, parts and consumables.

2.8 As a consequence of the 2015 First Principles Review and the decision to consolidate and reform Systems Program Offices, Defence has adopted a new definition of ‘Systems Program Office’. A number of acquisition projects had been managed in project offices which are now counted as Systems Program Offices.²⁰ Systems Program Offices do not endure indefinitely: for example, after the withdrawal from service of an asset and its disposal the relevant Systems Program Office may be dissolved or amalgamated with another. Similarly, a new Systems Program Office may be created for a major acquisition project. A list of Systems Program Offices (including staff numbers, assets under sustainment and contracts) is at Appendix 1.

Source: Abstracted from Defence, ‘Reform Issues’, April 2011.

Internal Agreements

2.9 Agreements, internal to Defence, between the Capability Acquisition and Sustainment Group and each capability manager, set out the level of performance and support required by the capability manager.²¹ These are ‘Materiel Sustainment Agreements’ (MSAs). MSAs include an agreed ‘price’ for the sustainment work and performance indicators by which Capability Acquisition and Sustainment Group internal service delivery is measured and reported.²²

2.10 Defence intends to replace MSAs (and the associated Materiel Acquisition Agreements) progressively with Product Delivery Agreements. The Product Delivery Agreement will cover both acquisition and sustainment of a capability system over its life, through to disposal.

2.11 A simplified representation of the essential relationship among elements responsible for the sustainment of specialist military equipment is in Figure 2.1 below.

Figure 2.1: Simplified representation of organisational arrangements supporting Defence sustainment of specialist military equipment

Note: Military units also undertake some operational-level sustainment activities.

Source: ANAO

2.12 Navy, Army and Air Force policy requires six-monthly reviews of MSAs. These are known within Defence as ‘Fleet Screenings’.²³ While ‘Fleet Screening’ procedures vary among the Services, the intent is broadly similar: that is, to review the funding allocated to sustainment through the MSAs, and make decisions about changes to funding levels, equipment operation, or performance indicators.²⁴ For example, additional funding may be required for equipment that has recently returned from operations. Fleet Screenings comprise a meeting or series of meetings between the Services and Capability Acquisition and Sustainment Group.

Relationship between sustainment and acquisition

2.13 Generally, the lifecycle of Defence’s military assets falls into two periods: acquisition and sustainment. There has long been a strong internal perception in Defence that sustainment attracts less management attention than acquisition. An internal survey a decade ago found that Systems Program Offices attributed the problem to Defence management in Canberra:

sustainment still does not get sufficient level of attention and needs to be recognised as the major component of the life cycle. [Systems Program Offices] claim that policies and new tools are developed which do not sit well with sustainment and, because Canberra is acquisition-focused, sustainment advice is not taken.²⁵

2.14 Five years later, the Rizzo Review into naval sustainment drew a similar conclusion:

The need for the sustainment of assets is understood in Defence and DMO, but it is not given the same rigorous attention as asset acquisition. Sustainment costs can exceed those of the original procurement and the challenges can be more complex.²⁶

2.15 Coles took a similar view a few months later, in the Collins Class Sustainment Review, arguing that Defence should give sustainment much higher attention and priority during the initial phases of the asset’s lifecycle (‘Needs’, ‘Requirements’, and ‘Acquisition’ phases). Failure to pay sufficient attention to sustainment early on would increase the cost of ownership to Defence. That review found ‘sustainment is still being treated as a “poor relation” compared to the generally higher-profile acquisition work’.²⁷ The Sustainment Complexity Review (2010), which found ’50 plus’ sustainment business models in DMO’s Systems Program Offices, explained this diversity by reference to the greater attention Defence had given acquisition. Defence had developed, by the time of that review, a consistent approach to acquisition. But ‘no such global improvement approach’ had been implemented across sustainment. The Review found, in terms similar to those of Coles: ‘In fact, the general comment is that “sustainment is the poor cousin to acquisition.”’²⁸ In June 2017 Defence advised the ANAO that it was attempting to address this perception through the current round of reforms, such as the new Capability Life Cycle and the December 2016 release of Support Procurement Strategies.

Review of Systems Program Offices

2.16 Two recent reviews of the operations of Systems Program Offices have identified major challenges for the future management of sustainment. The first is a Defence internal review conducted in late 2015; the second comprises work done following the First Principles Review as part of the Systems Program Office reform and consolidation project.

Review of contracted services arrangements in Systems Program Offices

2.17 In late 2015, Defence’s Audit and Fraud Control Division reviewed the contracted service arrangements within three Systems Program Offices in response to a request by Defence’s Chief Finance Officer Group. That review found shortcomings in the application of procurement principles, which exposed Defence to risks. Specifically, the review identified instances where:

• Defence had not established a strong case for value for money for the continual renewals of supplier contracts. This practice: has created the risk of dependency on particular suppliers; raises concerns about probity as contractors and APS staff involved in the procurement may not be dealing with each other at ‘arms-length’; and means that Defence may not be getting value for money.
• Defence suppliers have access to potentially commercially sensitive Defence financial information which could provide an unfair advantage to some service providers.
• Defence has entered into sole source contracts with suppliers outside of the Defence system established to manage such arrangements. As a result: service providers may have been contracted to provide services for which Defence has previously determined they were not suitable to deliver, and/or charge higher rates than previously negotiated with Defence; and there is a risk that the documentation required to support the procurement decision may not be adequate.
• Defence procurement guidance did not reflect some of the requirements of the Commonwealth Procurement Rules in place at the time of the review.

2.18 Defence makes extensive use of external service providers and contractors to deliver acquisition and sustainment services. Further, Defence’s Capability Acquisition and Sustainment Group, is implementing reforms that will potentially increase reliance on industry to deliver these services. Defence therefore requires a robust framework to manage the risks associated with the widespread use of contracted services. Defence informed the ANAO that it had updated its procurement guidance in April 2017 and intends to address the other concerns raised in the 2015 review over the next 18 months.

Reform and consolidation of Systems Program Offices

2.19 Following the First Principles Review, Capability Acquisition and Sustainment Group is undertaking a project to reform and consolidate Systems Program Offices. In its submission to the JCPAA inquiry into sustainment expenditure about this project (February 2017), Defence advised that:

[Capability Acquisition and Sustainment Group] Systems Project Offices Reform project is undertaking a review of all SPOs across [Capability Acquisition and Sustainment Group]. The outcome from this review is alignment of the SPO activity to Planning, Governance and Assurance roles, and enabling industry to undertake the management and delivery of sustainment activity, in particular the transactional functions required to maintain capability.

2.20 Defence records indicate that, as at February 2017, the project had reviewed 24 Systems Program Offices across three ‘domains’ (Maritime, Aerospace, and Joint), comprising 37.5 per cent of the Group’s business delivery units. Systemic issues identified by Defence from the reviews of Systems Program Offices to date are set out in Box 3. They include:

• the experience, skills and competencies of Systems Program Office staff;
• duplication of effort; and
• overhead costs.

2.21 The review is examined in more detail in Chapter 5 of this audit report, while the transparency of internal costs is examined in Chapter 3.

Note a: Defence, CASG Executive Advisory Committee paper position paper reflecting the views of the SPO Reform Project, 13 February 2017.

Note b: ‘Man-marking’ is where Defence staff in the Systems Program Office supervise the performance of one or a small number of contractor personnel.

3.1 To assess whether Defence has in place an effective performance framework to support the management and external scrutiny of materiel sustainment, the ANAO examined:

• monitoring systems for sustainment;
• internal reporting practices for sustainment;
• reviews of sustainment performance;
• whether Defence has established measures of efficiency and productivity for sustainment activity;
• whether Defence assesses whole-of-life costs and the total cost of ownership of its assets; and
• public reporting on sustainment.

Does Defence have effective monitoring systems for sustainment?

The Monthly Reporting System (MRS) and the Sustainment Performance Management System (SPMS)

3.2 The reporting system relied upon by Capability Acquisition and Sustainment Group and, before it, the DMO, to track major acquisition projects since mid-2004 has been its web-based Monthly Reporting System (MRS). At the time DMO developed MRS it was noted internally that ‘DMO currently spends around 3 billion dollars per year on sustainment activities and at this stage has no system that can adequately report on the effectiveness or efficiency of these activities’.²⁹ Work began in September 2005 to introduce sustainment reporting capacity into MRS.

3.3 Several major reports have criticised sustainment reporting over the last decade, including Mortimer (2008) and Rizzo (2011).³⁰ Rizzo was highly critical of existing performance indicators:

Each current [MSA] Product Schedule has inadequate Key Performance Indicators … They only include a small subset of the ‘contractual’ measures that should be placed on each party and there are no consequences associated with non‐compliance. At worst, non‐delivery will result in a red traffic light status in DMO Sustainment Overview Reports which it seems that few stakeholders read.³¹

3.4 This led to the commencement of work on a new IT system, the Sustainment Performance Management System (SPMS), in mid-2011 to replace the sustainment module in MRS.³² Defence agreed in August 2012 to the ‘continuing refinement and implementation of a shared Sustainment Performance Management Framework and reporting system for use in the sustainment of all products jointly managed by Defence and DMO’.³³ An objective of developing SPMS is that the system should provide ‘the central source of truth for sustainment management data and reporting for sustainment management personnel at all levels of both Capability Acquisition and Sustainment Group and Capability Manager organisations’.³⁴ This was contrasted with the MRS, which reportedly required substantial work-arounds requiring double and triple handling. MRS and, to an increasing extent, SPMS, are now used to populate a Quarterly Performance Report on both acquisition and sustainment (see paragraph 3.11 forward).

How SPMS works

3.5 In common with MRS, SPMS is a web-based system designed to provide performance reports for Capability Acquisition and Sustainment Group and Capability Managers. Data is entered monthly by subject matter experts, usually based in the relevant Systems Program Office. The Systems Program Office Director reviews the data and comments for each measure and provides comments for a set of ‘key performance indicators’ and an additional set of ‘key health indicators’.³⁵ Further comments can be added up the hierarchy to the relevant Capability Acquisition and Sustainment Group division head. By way of example, Appendix 2 of this audit report lists Navy’s MSA Performance Framework core Key Performance Indicators and Key Health Indicators.³⁶ Defence advised the ANAO that Air Force has also developed a corresponding set of indicators and Army is progressing similar work.

SPMS implementation

3.6 SPMS was introduced, first, for Navy (May–July 2015) following work to develop its sustainment performance framework with a standard suite of performance indicators.³⁷ The system was being implemented in Air Force and Army from late 2015 through 2016. Implementation is incremental and subject to feedback and review. Defence expects that all sustainment products will be reporting in SPMS by the end of June 2017.

3.7 A joint review of Navy’s performance framework, completed in March 2016 by Navy and Capability Acquisition and Sustainment Group, concluded that there was ‘universal acceptance that the framework provides a sound basis for the assessment of sustainment outcomes’.³⁸ However, the review decided to remove one of Navy’s core key performance indicators (cost per materiel-ready day achieved) on the basis that the measure required the Systems Program Office to enter four separate data elements and ‘stakeholder compliance’ was reportedly low. The indicator was replaced with a measure ‘CASG-related Product Cost per materiel-ready day’, whose value could be generated automatically and would not require comment by the Systems Program Office.³⁹ This would be reported in SPMS but not be represented as a key performance indicator⁴⁰ because it did not include the cost of all the fundamental-inputs-to-capability and did not measure the cost per materiel-ready day. Defence informed the ANAO that both measures are based on direct product costs incurred by the Capability Acquisition and Sustainment Group and do not include sustainment costs incurred by other parts of Defence.

3.8 Neither indicator is useful in determining the total cost of the capability to Defence. Knowledge of the total cost could contribute to a better understanding of the whole-of-life costs of Navy assets and, over time, could provide useful insight into whether cost-effectiveness is improving or deteriorating.⁴¹

Further development of SPMS

3.9 In September 2016, Defence decided to expand SPMS to cover acquisition. When implemented, the system will be renamed the ‘Program Performance Management System’.⁴² In the long term, Defence intends to build an ‘enterprise solution’ to encompass both sustainment and acquisition management and reporting using SAP software.

Does Defence have effective internal reporting for sustainment?

3.10 Reviews of Defence over the years have highlighted the importance of management reporting on its activities. The Mortimer Review (2008, p. 48) noted that the efficiency and effectiveness of DMO sustainment performance will not improve unless it is measured.

The Quarterly Performance Report

3.11 The Quarterly Performance Report is the primary way by which Defence provides information to government and senior Defence personnel about the status of major acquisition and sustainment activities.⁴³ The report was developed in 2015 in consultation with the office of the Minister for Defence. Staff in the Capability Acquisition and Sustainment Group compile the report manually, incorporating information contained in MRS and, increasingly, SPMS. The report includes performance summaries for the Top 30 major acquisition projects, all major acquisition projects reported in the Major Projects Report and the Top 30 sustainment products. The report also includes an overview of projects of concern and underperforming acquisition projects and sustainment products.

3.12 In a recent submission to the JCPAA, Defence stated that ‘The Defence Ministers are provided with a Quarterly Performance Report, which includes Projects of Concern, Projects and Products of Interest and Performance Summaries for the Top 30 Sustainment Products’.⁴⁴ The stated intent of the Quarterly Performance Report is to provide ministers and senior Defence personnel with:

a clear and timely understanding of emerging risks and issues in the delivery of capability to our Australian Defence Force end-users. These risks and issues are highlighted so that stakeholders can respond in a coordinated manner to guide the conduct of remediation actions.

3.13 This broad focus is consistent with Defence advice to the ANAO that some sustainment services for some equipment items are also undertaken by Capability Managers and other enabling Groups, including Joint Logistics Command.⁴⁵ The primary data source for the report is SPMS—the ‘central source of truth for sustainment management data and reporting for sustainment management personnel at all levels of both Capability Acquisition and Sustainment Group and Capability Manager organisations’.⁴⁶ The approach, as advised to the JCPAA, also reflects the First Principles Review recommendation that an end-to-end approach should be taken for accountability for capability—the ‘One Defence’ model.⁴⁷

Figure 3.1: Preparation of the Quarterly Performance Report

Source: ANAO

3.14 However, Defence advised the ANAO in June 2017 that the Quarterly Performance Report’s ‘primary focus is on products and projects delivered by the Capability Acquisition and Sustainment Group, and does not encompass the complete sustainment enterprise. This narrower focus risks telling only a partial account of sustainment within Defence and is potentially at odds with the ‘One Defence’ model promoted by the First Principles Review

Assessment of the report

3.15 Under current arrangements for its production, the Quarterly Performance Report is neither timely—it is more than 50 days old by the time it gets to ministers—nor clear—it is dense with acronyms and jargon. Some terms are used in an unusual way, for example, expenditure of more funds than had been budgeted is referred to as an ‘overachievement’.

3.16 One potentially useful feature is a list of underperforming sustainment products.⁴⁸ This is based on information provided through MRS, SPMS and gate reviews. Eight underperforming products were listed in the second quarter of 2016. Ten such products are listed in the report for the subsequent quarter (July – September 2016). In that later report the list was labelled ‘sustainment products of interest’ rather than ‘underperforming’, though the items are still listed ‘in order of concern’.

Case study—performance reporting on Army’s Armed Reconnaissance Helicopter

3.17 The ANAO observed marked differences in the information provided about Army’s Armed Reconnaissance (Tiger) helicopter between:

• a Quarterly Performance Report to the Defence Minister produced during the audit (April – June 2016) and, in contrast,
• a recent ANAO performance audit⁴⁹, the 2015–16 Major Projects Report⁵⁰, a sustainment gate review⁵¹ and the Houston review of Army Aviation.⁵²

3.18 A comparison shows that the Quarterly Performance Report does not include information available to Defence that is critical to the reader’s ability to understand the gap between the expectation of capability and reality for the Tiger helicopter program (See Appendix 3).

3.19 In this case, the Quarterly Performance Report’s usefulness is reduced by the following:

• The use of the ‘traffic light’ indicators ‘Green’ (acceptable performance) or ‘Amber’ (early signs of underperformance). Neither of these is a reasonable assessment of the aircraft’s capacity to meet the required level of capability expected by government.
• The report’s traffic lights, in particular, focus attention on the measures contained in contractual and intra-Defence agreements (for example, aircraft ‘availability’) and not the measures that matter to Army, the end-user (aircraft actually able to be flown).⁵³
• The year-end spend performance indicator is blank.

3.20 The contrasting information identified in this case indicates that the Quarterly Performance Report would benefit from the application of a quality assurance process to ensure that government and senior Defence personnel are presented with a frank and balanced assessment of performance (see paragraph 3.27 below).

Does Defence conduct effective reviews of sustainment performance?

3.21 Defence has implemented a strategy in recent years that should help to draw problems in individual sustainment products to senior management attention. This has occurred with the extension of DMO’s earlier program of gate reviews to cover sustainment as well as acquisition.⁵⁴

3.22 In 2009 Defence commenced gate reviews for capital acquisition projects as an internal assurance process.⁵⁵ They have provided insight into a project’s progress and an opportunity for project staff to discuss difficult issues with senior management and seek guidance. Defence later recognised that gate reviews also have potential value for sustainment, especially where it allows management to become aware of maintenance problems that might otherwise remain hidden. The Rizzo Review into naval sustainment raised this problem in the following terms:

To avoid being seen to fail personally, there is a danger (especially in the can do, make do environment) that staff will choose to not raise bad news. This can result in bad news remaining at lower levels in the organisation, increasing enterprise risk and only becoming apparent when recovery is expensive, difficult or even impossible.⁵⁶

3.23 Defence commenced sustainment gate reviews in December 2015. Gate reviews are conducted to ‘provide high quality and reliable advice to Defence and Government regarding the health and outlook of both acquisition projects and sustainment products’. Defence requires that the first sustainment gate review be held within 12 months of the review held before final operational capability is reached.⁵⁷ Thereafter, further such reviews are to be undertaken periodically (every one, two or three years) with the exact timing determined on a risk basis. Defence has stated that gate reviews ‘review overall performance of the sustainment system and its fitness for purpose rather than just its outcomes for the past month.’

3.24 Gate reviews can draw management attention to sustainment risks in individual fleets. For example, sustainment cost has been identified as a ‘key risk’ for the MRH90 helicopter:

• An ANAO performance audit in 2013–14 found that, at the time of approval (June 2004), Defence had estimated the sustainment cost for 40 aircraft at $85.2 million a year.⁵⁸
• By 2009–10 when only 15 aircraft had been delivered, the cost of sustainment for those aircraft had already exceeded $85 million per annum.
• During 2012 DMO endeavoured to capture and model the expected cost of ownership for the 47 MRH90 aircraft over the planned life of the aircraft. That modelling indicated cost of ownership of between $240 million and $360 million per year (October 2012 prices), or $5.1 million to $7.7 million per aircraft per year.
• In May 2016, a sustainment gate review identified the cost of sustainment as a key risk for this program. Defence confirmed in March 2017 that this risk remains.

3.25 Information from sustainment gate reviews is not routinely incorporated into management reporting on sustainment. The ANAO suggests that Defence could devise a means of doing this, thereby capturing additional useful information on sustainment performance and current risks. For example, Quarterly Performance Reports could include a section providing an update on how the issues identified in the sustainment gate reviews are being addressed over time.

3.26 Moreover, there is an opportunity for Defence to extend the use of its Sustainment Gate Review program to have those reviews consider the quality of the information provided in the most recent Quarterly Performance Report for the asset under review, to help quality assure the accuracy, clarity and completeness of that information. As discussed, a quality assurance process for the Quarterly Performance Report would provide a firmer basis for internal and external scrutiny of sustainment reporting.

Has Defence established measures of efficiency and productivity for sustainment activity?

3.29 The efficiency of sustainment has been raised repeatedly over the years. In 2003, a few years after the formation of the DMO, a ‘zero-based’ review found that there was a ‘significant opportunity to reduce resourcing levels by re-engineering processes and improving systems’—in other words, improve efficiency.⁵⁹ Specifically, there was potential to ‘optimise resource utilisation through judicious outsourcing to original equipment manufacturers in appropriate circumstances’. Later (April 2004), senior DMO managers stated that there was scope to achieve a better risk balance in sustainment program management, where well-defined tasks had, in the past, been transferred to industry successfully with net savings, thereby improving efficiency.

3.30 More recently, after the release of the 2015 First Principles Review, one of the Review panel members characterised inefficiency as the most prominent problem across Defence:

The big problem that the Review team saw was one of efficiency and not of effectiveness. By any reasonable standard the military output—the effectiveness of the ADF—is world class, but does the Department arrive at that effectiveness in the most efficient way possible? Can the Department assure the Government of the day and the Australian taxpayer that the resources—the people, processes and tools—used to create that output are being utilised in the most efficient way? The Review team thought the answer to that was ‘No’.⁶⁰

3.31 To develop sound measures of efficiency, Defence also needs to understand the cost of ownership of its assets (see paragraph 3.34 forward) and to take account of staff costs in assessing the total cost of its activities (see paragraph 3.44 forward).

3.32 The First Principles Review found no direct measures of productivity in the existing management information systems, though some cost and schedule information is included in MRS and SPMS. The Review proposed the immediate implementation of productivity measures at a project level:

It is also incumbent upon the organisation to have a process in place that effectively measures its cost, schedule and productivity at a project level. This should be implemented immediately and individual leaders from the first line to the Deputy Secretary Capability Acquisition and Sustain­ment must be accountable for cost and schedule targets.⁶¹

3.33 On the question of the above recommendation about measuring productivity, Defence referred the ANAO to the Capability Acquisition and Sustainment Group Business Framework and, within it, the proposed process for the Capability Acquisition and Sustainment Group Business Review Cycle. The Business Review Cycle provides for monthly Project/Product Performance Reviews (PPRs) and, at increasing levels of aggregation, Branch monthly reviews and Domain monthly reviews. Deployment of the PPR was, at that point (September 2016), ‘on pause’ pending further review of the prototype arrangements. The envisaged end state is the deployment of a Program Performance Management System (PPMS), see paragraph 3.9 of this audit report), covering both acquisition and sustainment, commencing 30 June 2017.⁶²

Does Defence effectively assess whole-of-life costs and the total cost of ownership of Defence assets?

3.34 Achieving value for money is the core rule of the Commonwealth Procurement Rules. When conducting procurement, an official must consider the relevant financial and non-financial costs and benefits including, among other things, whole-of-life-costs.⁶³ Sustainment expenditure is a major contributor to whole-of-life costs—in some cases, the largest contributor.

3.35 Defence major capability decisions have substantial long‐term financial consequences for Defence and the Commonwealth budget. Nevertheless, developing whole‐of‐life cost estimates for major capital projects has been a challenge for Defence for decades. A Defence instruction on lifecycle costing has been in place since at least November 1992.⁶⁴ In addition, a ‘Through Life Support Manual Volume 6—Life Cycle Costing Analysis’ has been available since 2001 (and remains current).⁶⁵ The issue of instructions and availability of guidance has not provided assurance of adequate costing action.

3.36 Numerous reviews and audits have urged greater focus on whole-of-life costs and better knowledge of the total costs of ownership of military assets, including sustainment, but have found that Defence has struggled to establish the skills, systems and data for this activity⁶⁶:

• Defence was urged to use lifecycle costing throughout the acquisition lifecycle by an ANAO performance audit in early 1998. The audit had found, among other things, that it was not generally used in the ‘in-service’ (sustainment) stage.⁶⁷ The audit included, as an appendix, a brief better practice guide to lifecycle costing.
  • Defence agreed to the recommendation that total cost information be provided to relevant Defence committees, with qualification.⁶⁸ When Defence reported in 1998 on the implementation of the then recent Defence Efficiency Review, it stated that ‘A project has been initiated by the Joint Logistic Systems Agency to establish a common process to address through-life support arrangements, including … lifecycle costing … ’.⁶⁹
• In 2008, the Mortimer Review recommended that decisions to purchase new equipment or maintain existing systems should be based on the through-life cost of each option.⁷⁰
• By July 2011, the Rizzo Review stated that ‘The Team recognises that Defence has a policy instruction on life‐cycle costing … However, it is considerably out‐of‐date and compliance is inadequate’.⁷¹

Presenting whole-of-life cost estimates to government for approval

3.37 In recent years, Defence has generally estimated whole-of-life costs when seeking government agreement to a proposed acquisition. An ANAO performance audit in 2013 found that, in most cases, submissions to government included whole-of-life cost estimates. However, this was not presented as prominently as acquisition costs nor was its significance made clear to decision-makers.⁷²

3.38 In the course of this audit, the ANAO examined nine Defence major capital equipment submissions approved by government between July 2015 and April 2016 to review the presentation of whole-of-life costs.⁷³ In all but one submission examined, whole-of-life cost estimates are more prominent than in the submissions examined for the 2013 ANAO performance audit. In one case, whole-of-life cost estimates were not prominent with only the acquisition cost and ‘net personnel and operating costs’ referred to in recommendations to government.⁷⁴

Defence has plans to improve its whole-of-life costing

3.39 The 2015 First Principles Review found that: ‘Costing methodology does not account for all of the inputs to capability, at acquisition and over project life, and the true total cost of ownership is opaque’.⁷⁵ It also found that, to manage its major equipment efficiently and effectively, Defence needs to manage on a whole-of-life basis. This requires visibility of the total cost of ownership of major equipment. Such visibility would enhance Defence’s ability to benchmark its sustainment performance against allied countries with similar equipment.

3.40 In October 2016, Defence concluded that inadequate attention to managing its equipment on a whole-of-life basis had resulted in funding shortfalls for ongoing operating, maintenance and support costs. Defence is seeking to address the issue.⁷⁶ In the same month, it endorsed a total cost of ownership approach to estimating whole of life costs for all major capital equipment, infrastructure, and information and communications technology projects. The model seeks to capture the full cost of ownership throughout the life of an asset.⁷⁷

3.41 Defence also decided that, in the absence of tender quality cost information it would mandate an analogous⁷⁸ or a parametric⁷⁹ approach to developing cost estimates for all Defence major capital equipment projects. It also agreed that deviation from the endorsed approach will be permitted only in exceptional circumstances, and with the approval of Defence’s costing authority—its Chief Finance Officer.

3.42 To provide the historical cost data to support analogous or parametric cost estimates, Defence’s Chief Finance Officer will build a database of historical costs and cost attributes. The data will be obtained from: Defence’s internal cost data; industry suppliers; industry-maintained databases; and cost data made available to Defence from the United States, United Kingdom, New Zealand and Canada.⁸⁰

3.43 In developing this new approach to cost estimating, Defence sought information from Defence partner nations about the structures, data and approaches they use to develop cost estimates for major capital projects. Defence’s view is that, once implemented, this initiative will improve Defence’s cost estimates for major capital equipment projects and will bring Defence practice into line with the approach adopted by capital intensive industries and Defence partner nations. Defence has an implementation plan with target dates, concluding in July 2018.

Staffing costs are not included in Materiel Sustainment Agreements

3.44 The Mortimer Review (2008) observed that ‘through-life maintenance and support account for more than half of the DMO annual budget and involve about two-thirds of its workforce’. The cost of that workforce is not included in the internal price paid by Defence capability managers for the sustainment services provided by DMO/Capability Acquisition and Sustainment Group through MSAs. Those prices comprise only the costs incurred by the relevant Systems Program Office for goods and services supplied under contract.

3.45 The First Principles Review found that, at the project level, Defence treats staff as a ‘free good’ across the department:

Employee costs are not considered part of a project’s costs and a manager cannot adjust numbers of staff based on project need. An alternative approach is to manage a total project budget that includes employee expenses. This encourages managers to exercise judgement and discretion to ‘trade’ within that operating budget (assessing their need for how many staff and what levels, skills and contracted expertise are required) to most efficiently and effectively deliver the outcomes required.⁸¹

3.46 Treating staff as costless lessens transparency of the true cost of activity and introduces risks because of the inaccurate price signals and distorted incentive structure then facing Systems Program Office managers and the capability manager. For example, this approach could make ‘in-sourced’ labour appear costless and put outsourcing at a comparative disadvantage. This risks encouraging insourcing in preference to outsourcing even where the total real cost to the Commonwealth of the former is greater.⁸² Defence has noted this internally (February 2017):

Any reduction of SPO [full-time equivalent staff], without a commensurate reduction of activity, will result in work being outsourced to industry and therefore increase the cost to the [Capability Manager].

Acquisition and sustainment salaries are held by [Capability Acquisition and Sustainment Group] and not funded by [Capability Managers], so reductions in this Commonwealth APS workforce do not generate a saving for [Capability Managers]. … The anticipated transfer transactional workloads to industry will therefore increase the unit cost of capability delivery as far as the [Capability Managers] are concerned, regardless of the expected overall improvement in Defence’s budget.⁸³

3.47 Systems Program Offices have had between three and four thousand staff over the last decade (Figure 3.2, below). Defence advised the ANAO that Capability Acquisition and Sustainment Group’s expenses on all employees in 2015–16 were $490.4 million. In this light, the cost of those staff is a substantial omission from the apparent price of services facing the capability manager. This is especially so, given the recent observation by Defence that, in some cases, ‘[Systems Program Offices] have been spending … more money on salaries than they were on actual acquisitions or sustainment’.⁸⁴

Figure 3.2: Numbers of staff in DMO/Capability Acquisition and Sustainment Group Systems Program Offices, 2004–16

Note a: In compiling the data underlying this table PMKeys Reporting is not able to identify all those organisational units now recognised by Capability Acquisition and Sustainment Group as Systems Program Offices. This data is likely therefore to understate the total number of staff in Systems Program Offices.

Note b: These figures include Australian Public Service and Australian Defence Force staff.

Source: Defence, PMkeys Reporting, Full-time equivalent staff in all Systems Program Offices DMO/Capability Acquisition and Sustainment Group, by month, July 2004 to June 2016.

3.48 On the inclusion of staff costs in the new Product Delivery Agreements, Defence informed the ANAO in March 2017 that ‘The PDA remains under development, and it is inappropriate to speculate in any detail on how anticipated elements of the document will be used’.

3.49 In October 2016, Defence’s Chief Finance Officer stated that, under current proposals, actual costs will be reported in Defence’s central financial management system and that system will be consistent with the total cost of ownership model. Capability Acquisition and Sustainment Group also undertook in July 2016 that ‘over the next 12 months, Capability Acquisition and Sustainment Group will commence effort capture (where staff time is tracked against the projects and products we support), … led by the Group Business Manager’.⁸⁵

Does Defence produce high quality public reporting for sustainment?

3.50 To assess Defence’s public reporting for the sustainment of specialist military equipment, the ANAO considered:

• public reporting principles and requirements, including under the Public Governance, Performance and Accountability Act 2013;
• Defence’s public annual reporting of sustainment activity; and
• Defence’s pubic reporting of program level estimates over time.

Public reporting principles and requirements, including under the Public Governance, Performance and Accountability Act

3.51 The Public Governance, Performance and Accountability Act 2013 (PGPA Act) ‘aims to improve the line of sight between what was intended and what is delivered’. A key focus of the enhanced Commonwealth performance framework established under the PGPA Act is ‘ensuring that programme managers, accountable authorities, ministers, the parliament and the public are able to use performance information to draw clear links between the use of public resources and the results achieved’.⁸⁶

3.52 Under the PGPA Act, Commonwealth entities must prepare and publish a corporate plan by the last day of August each year. The plan must include the purposes and activities that the entity will pursue, the results it expects to achieve and planned performance measures. It must cover a minimum of four years.

3.53 Commonwealth entities must report—in an annual performance statement—their actual results against the planned performance criteria outlined in their corporate plan and Portfolio Budget Statements. The report is expected to be a direct acquittal of the performance measurement and reporting intentions identified in the corporate plan.⁸⁷

3.54 Under section 17 of the PGPA Act, one of the functions of a Commonwealth entity’s audit committee is to provide independent advice and assurance to the entity’s accountable authority—the Secretary of Defence—on performance reporting for the entity. The Defence Audit and Risk Committee will therefore require appropriate information on sustainment reporting to enable it to provide this advice and assurance to the Secretary.

3.55 Sustainment has been identified as a distinct category of expenditure since 1 July 2005.⁸⁸ In effect, this was the point at which the term ‘sustainment’ came into regular Defence use.

3.56 The first planning and reporting cycle under the PGPA Act requirements was for 2015–16. The 2015–16 Defence Corporate Plan did not include an explicit reference to sustainment in the Purposes statement. The 2016–17 Defence Corporate Plan incorporated a revised set of Defence Purposes and performance measures.⁸⁹ These identify sustainment more clearly than in the previous plan, under the revised Purpose 2, ‘Deliver and sustain Defence capability and conduct operations’. A performance measure under this Purpose is ‘Military capability is sustained consistent with Government requirements’. This, it states, is to be measured tri-annually and reported annually for each year of the corporate plan. The assessment method is to be ‘Assessment of sustainment against capability manager requirements’. Defence has informed the ANAO that it will report against this measure in its Annual Report for 2016–17.

Defence’s public annual reporting of sustainment activity

3.57 Defence provides information about sustainment in its Portfolio Budget Statements (PBS), Portfolio Additional Estimates Statements (PAES) and Annual Reports.

3.58 Since 2007–08 Defence has reported financial and descriptive information for the Top 20 or Top 30 sustainment products (Top 20 to 2012–13 and Top 30 from 2013–14). In general, reporting has included: a description of sustainment activity; estimated expenditure; actual expenditure; the variation between estimated and the actual expenditure; and some explanation of the variation between estimates in the PBS and PAES.

3.59 At the time of this audit, the latest set of these documents where both prospective and retrospective elements are available was for the financial year 2015–16. The following paragraphs provide an overview of the contents of the 2015–16 performance documents. Further detail is provided in Appendix 4.

3.60 The Defence PBS 2015–16 reports estimates for Defence’s Capability Sustainment Program and Top 30 sustainment products, as well as some descriptive information on Defence’s Top 30 sustainment products. This information is presented in multiple tables and sections of the PBS and an explanation of the variation between tables is not provided.⁹⁰ Further, some of the descriptive information about sustainment activity is not always expressed in clear language, which detracts from its usefulness. The Defence PAES 2015–16 provides revised estimates for the 2015–16 financial year, the variation between the 2015–16 estimate from the PBS, and, for the Top 30 sustainment products, some explanation of the variation, plus a ‘project performance update’.

3.61 The Defence Annual Report 2015–16 includes financial information and some descriptive material for the same Top 30 sustainment products reported on in the PBS and PAES.⁹¹ However, the Annual Report does not report the actual expenditure for Defence’s Capability Sustainment Program, for which estimates were reported in the Defence PBS and PAES.⁹² Additionally, there is no reconciliation of the ‘total sustainment product funds available’ amounts reported in the Annual Report⁹³ and the total funding for the Capability Sustainment Program reported in the PBS and PAES tables.

3.62 In summary, the ANAO’s review of public reporting on sustainment activity indicates that:

• reported sustainment expenditure is not complete in that it does not include staffing costs, which are significant⁹⁴;
• information on sustainment expenditure and descriptive material is not consolidated in one easy-to-locate area;
• actual expenditure against estimates is not reported for the Defence Capability Sustainment Program overall;
• reported expenditure information is not fully reconciled between different information sets;
• there was no explanation in the 2015–16 annual report for expenditure variations between the PAES estimate and the actual expenditure for the Top 30 sustainment products—therefore the reasons for the full year variances are unclear; and
• the performance summaries are highly variable and inconsistent between planning documents (PBS and PAES) and the Annual Report. For some Defence systems the planning documentation includes a description of the system followed by a statement of the primary focus for that asset for the forthcoming financial year, and in the annual report there is an account of what has been achieved. In other cases, this information is so general as to be of limited value.

Defence’s public reporting of program level estimates over time

3.63 Defence has not published program level estimate data on a consistent basis over time, and no time series analysis is published by Defence to assist with external scrutiny of sustainment expenditure.

3.64 Published estimates comprise of two separate data series. From 1 July 2005 until the DMO was delisted on 30 June 2015, Defence reported its sustainment expenditure estimates in its Portfolio Budget Statements (and Portfolio Additional Estimates Statements). This was funding to be spent by DMO on sustainment on behalf of Defence.⁹⁵ From 2013–14 forward, Defence also reported the estimated cost of capability sustainment for the whole of Defence. These estimates, based on the latest available figures each year (generally from the Portfolio Additional Estimates Statements) are set out below (in Figure 3.3) for the period 2005–06 to 2020–21 (the end of the current forward estimates period as at May 2017).

Figure 3.3: Defence estimates of the cost of its capability sustainment program, 2005–06 to 2020–21

Source: Defence, Portfolio Budget Statements and Additional Estimates Statements, 2005–06 to 2017–18.

3.65 During DMO’s existence, the data represents expenditure by DMO on capability sustainment excluding DMO service fees/DMO departmental appropriation (representing DMO staff and other operating costs). The second series, commencing in 2013–14, represents the reported totals for Defence’s capability sustainment program.

3.66 There is no direct explanation in the estimates documents reconciling or explaining the difference or helping the reader to identify any trends. It is apparent, however, that different and additional items are included in the latter series.

3.67 The estimates documents generally provide components of these estimates, divided among the Services and other Defence Groups (Navy sustainment, Army sustainment and so on). Again, this presentation is not consistent as in certain years, for example, individual product sustainment costs are identified as separate line items.⁹⁶ Nevertheless, Navy, Army and Air Force sustainment costs can be traced to identify trends. This is reflected in Figure 3.4. (These figures do not include DMO/Capability Acquisition and Sustainment Group staff costs.)

Figure 3.4: Estimates of Service sustainment costs, 2006–07 to 2020–21

Note: Excludes Defence staff costs.

Source: Defence, Portfolio Budget Statements and Additional Estimates Statements, 2006–07 to 2017–18.

3.68 For each Service, comparing the amounts presented in the first and last years of this analysis yields an average annual (compound) rate of growth in the estimates.⁹⁷

Table 3.1: Increases in sustainment estimates by service, 2006–07 to 2020–21 ($m)

Source: Defence, Portfolio Budget Statements and Additional Estimates Statements; ANAO analysis.

3.69 When sustainment costs are presented cumulatively, the contribution of items other than Navy, Army and Air Force sustainment to sustainment cost growth over the forward estimates can be seen. As others have observed, the apparent growth flows mainly from the inclusion of certain Estate and Infrastructure Group costs (such as garrison support) and the balance of Defence’s estimated future sustainment funding that Defence has yet to allocate to a Defence Group within the broader Capability Sustainment Program in the last years of the period.⁹⁸ This is apparent for the period following the delisting of DMO and the inclusion of items other than specialist military equipment in Defence’s overall sustainment estimates (Figure 3.5).

Figure 3.5: Cumulative estimates of Defence sustainment costs, by year, 2006–07 to 2020–21

Source: Defence, Portfolio Budget Statements and Additional Estimates Statements, 2006–07 to 2017–18.

4.1 Sustainment practices have been subject to numerous reform initiatives over the last decade, especially since the Mortimer Review (2008).⁹⁹ In 2015, the First Principles Review concluded that the change programs associated with previous reviews ‘have resulted in only incremental change’.¹⁰⁰ This remark explicitly included all phases of the capability lifecycle, including sustainment. The most substantial reform effort was Smart Sustainment under the Strategic Reform Program, associated with the 2009 Defence White Paper. The ANAO examined first, a 2008 DMO initiative to cut sustainment costs and, second, Smart Sustainment to identify what became of those reforms.

Did DMO achieve the intended savings to sustainment costs from its 2008 initiative?

4.2 In February 2008, the Chief Executive Officer (CEO) of DMO announced a target of reducing by five per cent in real terms, in 2008–09, the sustainment cost of each of the 100 or so fleets of equipment then sustained by DMO.¹⁰¹ The CEO announced this measure as ‘an ongoing reform program that aims to identify efficiencies within the DMO and reduce the cost of ownership to the Commonwealth in the operation of sustainment fleets’.¹⁰² This, he wrote, was to satisfy a key priority of the government to reduce inefficiencies across the public sector.

4.3 The CEO wrote to all Systems Program Office directors seeking their help. He described the areas where he expected they could make cost reductions, including by: reducing spares; more performance-based contracting; discouraging end-of-year spending to meet budget; travel efficiencies; and improved maintenance philosophies. The initiative was not prescriptive except that capability and safety should not be compromised by any savings proposal. The cut was subsequently referred to within DMO as ‘a harvesting of “fat” within the products … readily absorbed by relevant Systems Program Offices and Divisions’.¹⁰³ The CEO DMO also wrote to the CEOs of companies undertaking sustainment work for DMO seeking their support. That letter also identified a savings target of $200 million a year for re-use elsewhere in the portfolio.

4.4 Defence informed the ANAO that it believed it had achieved a five per cent sustainment budget cut in 2008–09. However, the department’s approach to monitoring savings does not provide sufficient information to determine whether that occurred, and how the reported savings were achieved.

What did Smart Sustainment achieve?

The 2009 Smart Sustainment initiative

4.5 The government announced Defence’s Strategic Reform Program in May 2009. Defence began its Smart Sustainment reforms in early 2010 as part of the Strategic Reform Program Implementation package. The Program was largely based on the findings of the 2008 Audit of the Defence Budget (the Pappas Review)¹⁰⁴, led by consultant Mr George Pappas and undertaken in parallel with the development of the 2009 Defence White Paper, companion reviews, and the Mortimer Review. The Pappas review had observed that sustainment was ‘an area with significant opportunities for efficiencies’.¹⁰⁵

4.6 The Strategic Reform Program comprised 15 separate streams of reform, several of which were intended to reduce costs.¹⁰⁶ Overall, Defence expected the program to yield $20.6 billion, to be reallocated within Defence over the following decade, with Smart Sustainment contributing $5.5 billion of that figure.¹⁰⁷ This would make sustainment savings the largest single source of funds under the Strategic Reform Program. Defence planned to reallocate the $20.6 billion to other areas of Defence capability and contribute to the government’s 2009 Defence White Paper, Defending Australia in the Asia Pacific century: Force 2030. To facilitate the reforms, Defence made $360 million of ‘investment funds’ available for projects that would yield net savings.

4.7 Defence’s 2010 publication The Strategic Reform Program—Making It Happen stated:

Defence’s budget to 2019 has already been adjusted to take account of the $20.6 billion in reinvestment required for Force 2030—this requires us to think less in terms of adding up savings and more in terms of living within our means.¹⁰⁸

4.8 The harvesting and reallocation of the $20.6 billion from Defence’s Strategic Reform Program are not apparent in Defence’s Portfolio Budget Statements and Annual Reports. Further, the ANAO understands that the adjustments related to the expected savings were not identified in the Commonwealth’s central budget management system. This indicates that any actual adjustments, if made, were to internal Defence allocations.

4.9 Defence’s Sustainment Complexity Review in mid-2010 warned that most Systems Program Offices ‘did not have the capacity to undertake Smart Sustainment initiatives’.¹⁰⁹ Nevertheless, in late 2010, an internal ‘health check’ on Defence’s governance arrangements for Smart Sustainment found that, despite some difficulties, Defence had the fundamentals in place to support their success.¹¹⁰

4.10 Defence gave examples of activities it considered successful in its 2010–11 Annual Report (pp. 22 and 81):

• More efficient support services for the Jindalee Operational Radar Network, reportedly saving $100 million over ten years;
• Navy using Telstra’s NextG network for several minor war vessels, relinquishing satellite communications subscriptions and saving $2.3 million a year; and
• A service optimisation program for Army Land Rovers, saving $590 000 a year.¹¹¹

4.11 In 2011, a Chief of Service advised DMO’s audit committee¹¹² that an emerging issue from the Strategic Reform Program, common across all the ADF, was a shortage of future funding for sustainment. In July 2011, a further health check—this time by external consultants—found that Smart Sustainment had achieved some early successes with cost reductions and changes in practice. Nevertheless, it was at risk of failure. This was because of shortcomings in governance, program management, and Defence’s approach to reform.¹¹³ Its major findings include:

• Defence’s devolved and fragmented leadership approach to the program meant that Defence had no holistic vision of what the reform program was setting out to achieve, and therefore no logical program of work designed to support it.
• Fragmented management meant Defence did not have an enterprise-wide view of the many Smart Sustainment initiatives, and was missing opportunities for wider reform.
• Defence did not track expenditure or benefits from the $360 million allocated to support the program.
• The lack of understanding of whole-of-life costings and poor management of sustainment funding within Defence meant that the increasing levels of future cost reductions expected were at risk (the review estimated a shortfall of $1.8 billion could increase to a $3 billion shortfall).
• The multiple approaches and varying quality for determining baselines and metrics for the various improvements expected for the program, and inadequate tracking of benefits would limit Defence’s assurance about improvements and cost reductions.
• Defence had not developed robust capability and cost baselines for all capabilities, did not fully understand sustainment cost drivers nor whole-of-life costs for platforms.
• Sustainment costs were increasing because of the inappropriate application of DMO policies and procedures, and a lack of business acumen in contracting and engagement with industry.
• Defence had not defined the cultural changes it expected from the program and therefore may not deliver the behavioural changes required for long-term reform.
• Many Systems Program Offices had concerns that capability and safety were at risk.

4.12 A year later, in August 2012, a Defence internal audit of Strategic Reform Program savings and performance reporting questioned the integrity of reporting and found it had focused mainly on reporting savings rather than achieving reform.¹¹⁴ A Defence update to Defence Ministers on the Smart Sustainment reform program (August 2012) reported that:

Currently more than 800 Smart Sustainment initiatives are either underway or planned; about half of these have quantified the estimated savings, equating to about $3.7b across the remaining years of SRP (against a residual target of about $4.6b). Work continues with the Capability Managers to develop more systemic demand-related reform activities for implementation in the medium to longer term.

4.13 In contrast to the advice to ministers, in September 2012, an internal progress report on the action items from the 2011 health check showed that Smart Sustainment was still struggling to gain traction. There had been limited progress on the 35 action items identified in the 2011 health check with seven action items implemented, five abandoned, and 23 outstanding (among which Defence intended to re-write 14).

4.14 Also in September 2012, Defence advised the minister that the performance of the broader Strategic Reform Program was continuing to decline, losing credibility as a transformation program, and that the program’s financial performance ‘was particularly difficult to assess’. Defence informed the minister there was a significant risk that Defence would fall short of the cost reduction target for the Program ($20.6 billion) by some $4 billion, almost one quarter of the remaining target ($17.6 billion). Smart Sustainment was showing ‘significant stress’. One major contractor noted that ‘Smart sustainment … has achieved limited success to date through cost cutting and minor reform driven by piecemeal, top down budget pressure’.¹¹⁵

4.15 Defence records show that the department made substantial efforts to keep track of the large number of diverse initiatives identified across the department under Smart Sustainment, including internal reporting to management. However, Defence did not adequately assess that their intended outcomes were realised or enduring.

Did Defence achieve the expected savings from Smart Sustainment?

4.16 Defence documents indicate that by June 2013, it had determined that the Strategic Reform program targets were no longer valid and Defence was developing a revised approach to reform. This was to ‘focus more holistically on benefits realised to the organisation through reform, rather than only acquitting financial targets’. Defence expected that the comprehensive reform benefits and reporting for the revised reform model would be further refined in the near future.¹¹⁶ The government agreed that Defence could cease reporting savings from Smart Sustainment and Defence undertook to report to government annually, detailing achievements and benefits of the new reform approach across productivity, cultural and ADF Service-specific reforms.

4.17 There is no evidence that Defence further developed a planned approach to a reform program supported by clear objectives or benchmarks, or that it provided reports annually to government on the achievements and benefits of this new approach. Following the change in government in 2013, it terminated work on the Strategic Reform Program.

4.18 By June 2015, Defence was claiming approximately $2 billion in savings from the Smart Sustainment Program since its inception in 2009.

4.19 Notwithstanding Defence’s effort to keep track of the claimed and planned savings from Smart Sustainment, Defence has not been able to provide the ANAO with adequate evidence that the savings were realised. Defence was able to provide the ANAO with copies of a suite of standard letters of assurance from managers throughout DMO, each stating that, ‘to the best of their knowledge’, cost reductions of various values had been implemented for the respective financial year. These letters of assurance provide no details of the specific initiatives or how the claimed savings were calculated.

4.20 In October 2015, Defence’s First Principles Review Implementation Team decided that work on Smart Sustainment, along with other reform activities underway within Capability Acquisition and Sustainment Group at that time, would cease and the ‘relevant elements’ were to be incorporated into work on the new Capability Life Cycle.

4.21 There are reasons to doubt some of the savings estimates for Smart Sustainment:

• It is likely that some unquantifiable portion of claimed savings derive from the five per cent across-the-board cut among Systems Program Offices imposed by the Chief Executive Officer before the Strategic Reform Program commenced. This was, strictly speaking, a previous initiative.
• Internal Defence briefs and letters of assurance reveal that Defence believed it could claim some $500 million in savings from no longer having to sustain the Super Seasprite helicopter following the project’s cancellation. This equipment never made it into service before the project was cancelled, yielding no capability whatsoever at a cost of $1.5 billion.¹¹⁷ It is not clear how this windfall reduction in future sustainment costs can properly be attributed to the program. Moreover, the project was cancelled in March 2008, over a year before the government announced Defence’s Strategic Reform Program.
• An unspecified proportion of Smart Sustainment savings of $197 million in 2009–10 flowed from a change in fuel pricing. This is another windfall benefit which cannot be properly attributed to the program.¹¹⁸
• Some $360 million was allocated as ‘seed funding’ to be invested in projects intended to yield savings under Smart Sustainment. In 2011, a review of the Smart Sustainment program by consultants engaged by Defence identified that there were no records to support how these funds were spent or what benefits had been realised. Defence advised the ANAO in March 2017 that it was ‘looking into what became of the [unquantified] residual seed funds’.

Lessons from Smart Sustainment

4.22 The Smart Sustainment initiative lost momentum and its costs and benefits are uncertain. There was no systematic evaluation of the constituent projects at their conclusion nor overall evaluation of this major program. The main insights into what became of the initiative can be derived only from external consultancy reports. It is now too late to revisit these events; nonetheless, Defence could benefit from evaluating future major change programs. This matter is also taken up in the next chapter.

Discussion of sustainment in the First Principles Review

5.1 The most recent major review of Defence, the 2015 First Principles Review, was a comprehensive examination with a suite of recommendations for reform. The Review’s terms of reference included one specific reference to sustainment—a requirement that Defence ‘ensure a commercially astute, focused and accountable materiel acquisition and sustainment capability’. The terms of reference were also accompanied by a range of issues for consideration, among which was ‘How Defence sustains and supports capabilities and extending lessons from the Coles review [of Collins Class submarine sustainment] across all areas of sustainment’. The 2012 Coles Review, by subsequent reports, has been both successful and insightful.¹¹⁹

5.2 The report of the First Principles Review refers to sustainment in the discussion of its recommendation that Systems Program Offices each be examined and analysed:

In some cases it would be appropriate to outsource completely the acquisition and sustainment functions conducted by a Systems Program Office. In other cases it may be appropriate to outsource sustainment and partner with industry for the acquisition phase.¹²⁰

5.3 Otherwise, the report does not discuss sustainment as a topic separate from the acquisition stage of the equipment lifecycle. It does not explicitly analyse the Coles Review and its lessons (such as the value of international benchmarking, discussed below).¹²¹ Rather, the report passes the task of learning from Coles back to Defence, stating that the implementation of the First Principles Review’s proposed approach ‘should encompass the progress made through the Rizzo and Coles reforms’.

Note a: The Collins Class submarine sustainment program has been a ‘project of concern’ since November 2008.

Note b: The availability of the Collins Class was about half that achieved by comparable submarine fleets; time spent in planned maintenance was about one third greater; and maintenance overruns and percentage days lost due to defects were approximately double.

Note c: Defence defines a ‘materiel ready day’ as a day in which a submarine is not in planned maintenance or does not have defects that prevent it from being at sea.

Note d: In late 2016, Defence estimated that the Collins Class submarine sustainment program would achieve international benchmarks for cost per materiel ready day by 2022–23.

Note e: The 2016 report from the Coles review team states that ‘Attaining benchmark performance was a higher priority than efficiency. With benchmark availability on the verge of being achieved, the focus should now be on efficiency improvements and cost reductions across the sustainment program. Such cost reductions may be required to re-invest into inventory, obsolescence remedies, new infrastructure to manage an ageing fleet and the transition to future submarines. The achievement of this should be greatly enhanced by the application of the cost model’. Defence has advised that Collins Class sustainment cost per materiel ready day has decreased by approximately 40 per cent. This has largely been achieved by increasing the number of materiel ready days being obtained by adopting a 10+2 year Usage and Upkeep Cycle, but also by reducing costs of maintenance periods, particularly Full Cycle Dockings.

Reforms which will affect sustainment and its management

5.4 Although sustainment does not attract a separate focus in the First Principles Review, the review’s findings and recommendations have major implications for how sustainment is performed. The management of sustainment arises mainly in the discussion leading to First Principles Review Key Recommendation 2:

Establish a single end-to-end capability development function within the department to maximise the efficient, effective and professional delivery of military capability.¹²²

5.5 A subsidiary recommendation (‘2.2 disbanding the DMO and transferring its core responsibilities in relation to capability delivery to a new Capability Acquisition and Sustainment Group’) has led to the creation of Capability Acquisition and Sustainment Group in Defence. Although this largely represents the absorption and re-badging of the former DMO, other subsidiary recommendations are directed at substantial restructuring and reformulating the business practices in this part of Defence. In particular, the other recommendations with substantial implications for the management of sustainment are as follows:

We recommend:

2.3 developing a new organisational design and structure as part of the implementation process for the Capability Acquisition and Sustainment Group with reduced management layers;

2.4 examining [and analysing] each Systems Program Office to determine where each fits within the smart buyer function, [and] the most appropriate procurement model [for delivering the capability] and achieving value for money; and

2.11 significant investment to develop an operational framework which comprehensively explains how the organisation operates and the roles and responsibilities within it; a detailed set of life cycle management processes which provide project and engineering discipline to manage complex materiel procurement from initiation to disposal; and reviewing architecture to reinforce accountability at all levels and bringing together information upon which good management decisions can be made.¹²³

5.6 Recommendation 2.11 aggregates three First Principles Review recommendations which appear after the following conclusions in the text:

In an organisation which routinely manages complex projects and programs we found it remarkable that there is no common project management architecture or artefacts to support it. There are no standardised reporting mechanisms (reporting is informal, anecdotal, local or crisis based) or management processes, with all divisions having different methodologies and manage­ment systems. There has been no consistent application of fundamental tools such as Earned Value Management. Such tools will be required to support the new leadership team.¹²⁴

5.7 Defence’s Capability Acquisition and Sustainment Group has its own ‘CASG First Principles Review Implementation Plan’, which contains a range of elements, some of which correspond directly to the summary set of recommendations in the First Principles Review (such as ‘[Systems Program Office] Consolidation and Reform’). There are others which relate to the more detailed findings of the First Principles Review, including those identified in the paragraph above (such as a common approach to project management).¹²⁵ All elements have implications for both acquisition and sustainment.

5.8 The ANAO identified the following streams of work—either directly flowing from the First Principles Review or from Capability Acquisition and Sustainment Group’s own list of deliverables—as having the most apparent potential to affect the management of sustainment:

• examining each Systems Program Office to determine where each fits within the smart buyer function, and the most appropriate procurement model for delivering the capability and achieving value for money. The primary element of this reform is the consolidation and review of the Systems Program Offices;
• performance-based contracting;
• establishing centres of expertise as part of a matrix organisation within Capability Acquisition and Sustainment Group to make best use of specialist skills that are common across its Systems Program Offices¹²⁶;
• developing the new operational framework (‘Business Framework’) to explain comprehensively how the organisation (specifically, Capability Acquisition and Sustainment Group) operates;
• development of the ‘smart buyer’ model; and
• though not part of the First Principles Review, any further work Capability Acquisition and Sustainment Group carries out on its asset management strategy, a stream of activity that has been under way for some years.

5.9 The First Principles Review specifies few targets or measures of performance. However, it does set two years as a timeframe for delivery of the changes it recommends.¹²⁷ In setting that objective it allows for a ‘tail of activity’ that might run beyond two years for which it gives, as an example, completing the roll-out of its new ‘smart buyer’ model.

5.10 The ANAO considered how Capability Acquisition and Sustainment Group is managing the reform process, including by the extensive use of industry expertise. The ANAO considered progress with two of the above elements briefly (business framework, centres of expertise) but in more detail for three others (smart buyer, Systems Program Office reform and performance-based contracting).

Has Defence developed relevant capability to support its implementation of sustainment reforms?

5.11 A major feature of the implementation of the First Principles Review has been the use of industry expertise. Capability Acquisition and Sustainment Group officers have stated that, because of the lack of internal capacity and expertise, the group has depended heavily on consultants/contractors with its reforms. Reflecting the First Principles Review theme of better engagement with industry, Defence has engaged a USA-based engineering company, Bechtel, to help with implementing the Review’s recommendations. Bechtel was formerly a contender to manage and operate the UK counterpart organisation to Capability Acquisition and Sustainment Group, Defence Equipment and Supply. Bechtel is now, with others, providing the UK with tailored private sector expertise and human resources, indicating relevant experience in the Defence procurement field.

5.12 Bechtel had made contact with Defence in July 2013 seeking an opportunity to help with reform of the DMO, presenting a paper ‘Australian Defence Organisation—DMO Transformation Opportunities’. Following the First Principles Review, Defence first engaged Bechtel from October 2015 to April 2016 (extended to June 2016) to provide initial implementation help.¹²⁸

5.13 Senior Bechtel representatives met with Defence officials again in December 2015 and subsequently presented a new paper ‘Delivering CASG Transformation’. This expanded on various models of contract, including ‘Management & Operations (M&O) contracts, as vehicles for enterprise transformation’ and offered suggestions as to how Capability Acquisition and Sustainment Group might approach the following 18 months of reform.¹²⁹ The paper set out a range of options for organisational transformation.

5.14 Defence offered Bechtel significant additional work in May 2016, engaging them to provide advisory services and a senior manager, seconded into a Capability Acquisition and Sustainment Group division head role.¹³⁰ The estimated cost of all current contracts with Bechtel is about $107 million.¹³¹

5.15 The contracts with Bechtel are not performance-based. The Commercial Division in Capability Acquisition and Sustainment Group expressed concern that Defence needed to articulate clearly its ‘objective/end state’ before agreeing to contracts of this cost and magnitude. It pointed out that a similar contract with a counterpart organisation in another jurisdiction included a performance element where Bechtel’s fee was based on outcomes achieved. It expressed concern that without such a feature Defence was exposed to risk. Defence advised the ANAO that it would have been difficult to implement the contract as a performance-based contract:

Unlike standard Defence capability projects, Bechtel has been contracted to develop Defence’s capacity and capability to deliver rather than for Bechtel to deliver itself. Bechtel’s role is to support, train and transfer knowledge to Defence staff. The success of the contract will be measured by Defence’s ability to operate without Bechtel support at the end of the contract, rather than by any specific product delivered by Bechtel.

5.16 As at November 2016, the total estimated value of contracts awarded to support Defence activity in Capability Acquisition and Sustainment Group directly or indirectly associated with implementing the First Principles Review, excluding the Bechtel contract, was $40 million.

What progress has been made with the reforms?

5.17 As set out above (paragraph 5.8), the activities most likely to affect the management of sustainment are:

• consolidation and review of Systems Program Offices;
• performance-based contracting;
• establishing centres of expertise;
• developing a new operational framework;
• development of the ‘smart buyer’ model; and
• any further development of Defence’s approach to asset management.

Consolidation and review of Systems Program Offices

5.18 When Defence established Systems Program Offices in 2000 as the focus of the newly created DMO, Systems Program Offices were to have the following characteristics:

• a collection of multi-functional, multi-disciplined teams;
• staffed to achieve a balance between environmental, capability system and technical needs in whole-of-life management;
• organised along class/fleet/weapon system lines; and
• ideally located with the principal user but, dependent on the maturity and nature of the current, new and enhanced materiel systems, may be located at more than one site.¹³²

5.19 Defence has more recently (April 2016) defined Systems Program Offices as:

a business unit that works directly with Capability Managers and Suppliers as necessary to deliver agreed capability across the [Capability Life Cycle], coordinating the [Fundamental Inputs to Capability] elements required to achieve this objective.¹³³

5.20 The purpose and structure of Systems Program Offices have remained essentially the same since their inception in 2000. A consultancy found in March 2013 that:

DMO evolved through the amalgamation of many dispersed business units, without undergoing typical post-merger integration activities needed to drive efficiency. Consequently, DMO retains over 50 SPOs [Systems Program Offices] each operating relatively independently of others and in different positions in the value chain. SPOs vary widely in their shape, size, supplier relationships and services to Capability Managers.¹³⁴

5.21 The report found that the lack of Systems Program Office commonality added complexity, blurred accountability and confounded the ability to build enterprise solutions for common services. Moreover, the many changes to DMO, in response, for example, to the Mortimer and Kinnaird reviews, had not significantly altered its size, shape or footprint.

5.22 In May 2016, Defence completed a stocktake of its Systems Program Offices to determine how many existed, and which were candidates for further reform. Defence subsequently increased the numbers of organisational units it considered to be Systems Program Offices from 54 to 78, with staff numbers ranging from three to 178.¹³⁵

5.23 As discussed in paragraph 3.47 of this report, Systems Program Offices have had between three thousand and four thousand staff over the last decade. Defence advised the ANAO that Capability Acquisition and Sustainment Group’s expenses on all employees in 2015–16 were $490.4 million. The First Principles Review observed that, at the project level, Defence treats staff as a ‘free good’ (see paragraph 3.45), and the cost of those staff is a substantial omission from the apparent price of services facing the capability manager. Systems Program Office staff are involved in both acquisition and sustainment. Defence is not easily able to identify the effort, in terms of staff numbers, expended on each. In March 2017, Defence informed the ANAO that work was underway to address this.

5.24 According to Defence documentation, the degree of outsourcing among Systems Program Offices has long been diverse, ranging from a totally outsourced model to a high degree of transaction-based activities carried out in-house.¹³⁶ A 2015 review of Defence’s Systems Program Offices concluded that Defence manages the majority of its acquisition and sustainment activity in-house.¹³⁷

Purpose of the reform of Systems Program Offices

5.25 The 2009 report from the Pappas Review identified opportunities for efficiencies in maintenance activities carried out by Systems Program Offices by consolidating maintenance facilities; restructuring outsourcing contracts to ensure the contractual conditions create the right incentives for performance improvements; and implementing lean techniques in workshop management.¹³⁸ From the formation of DMO, Defence received similar external advice from different sources that substantial improvements in efficiency were possible in the Systems Program Offices.

5.26 In 2016, the First Principles Review concluded that Defence had too many Systems Program Offices¹³⁹, and that Defence could make savings by restructuring them and reducing their staff.¹⁴⁰ The Review drew on a 2013 consultant’s report that identified ways to reduce the DMO workforce.¹⁴¹

5.27 The First Principles Review recommended that each Systems Program Office be examined and analysed to determine:

• Where it fits within the smart buyer function; and
• The most appropriate procurement model for delivering the capability and achieving value for money.¹⁴²

5.28 The First Principles Review also saw outsourcing as an important element of the reform of Systems Program Offices.¹⁴³ Under the First Principles Review reforms, Defence expects to reduce the number of Systems Program Offices and, from 2015 levels, the number of staff directly undertaking sustainment activity, though no estimates have been made known and no targets have been set.¹⁴⁴

Progress with the reform of Systems Program Offices

5.29 In September 2015, Defence engaged a consulting firm to review and develop an optimal design for one of its Systems Program Offices. Defence received the final report in December 2015.¹⁴⁵ In August 2016, Defence re-engaged the same consulting firm to complete reviews of a further 53 Systems Program Offices. Defence advised that, during the course of this contract, a number of Systems Program Offices have merged into larger organisations, resulting in a total of 48 being reviewed (as at end February 2017). The value of the contracts for this work is $8.8 million. Defence is now undertaking a rolling program of reviews of all Systems Program Offices, adding a further 17 to those being reviewed. Defence advised that a contract variation has been applied to accommodate this additional work, bringing the total cost to $16.3 million.

5.30 In August 2016, Defence finalised a Systems Program Office Design Guide to provide Capability Acquisition and Sustainment Group senior managers ‘with sufficient guidance to design or redesign’ Systems Program Offices.¹⁴⁶

5.31 In March 2017, Defence advised that the first 25 Systems Program Office reports had been provided to the Capability Acquisition and Sustainment Group Executive Advisory Committee for exposure and awareness. There is no evidence of any decisions having been made.¹⁴⁷

5.32 Defence is in the early stages of the Systems Program Office reform envisaged by the First Principles Review. Completing this work is likely to take longer than the two years intended for the First Principles Review Implementation.

Performance-based contracting

5.33 Defence defines performance-based contracting as ‘an outcomes-oriented contracting method that ties a range of monetary and non-monetary consequences to the contractor based on their accomplishment of performance requirements’.¹⁴⁸

5.34 Performance-based contracting rose to prominence in the USA for its military in the late 1990s. The 2003 Australian Defence Aerospace Industry Sector Strategic Plan, produced jointly by Defence and industry representatives, recommended this approach, as well as a move to long-term through-life support contracts.¹⁴⁹ Aerospace Systems Division then led in this reform area, producing a ‘Performance-Based Contracting Handbook’ in 2005.

5.35 The problem of existing contract arrangements was described as follows by an RAAF officer in 2005, with a special focus on ageing aerospace platforms:

The typical [support entity] entails a Systems Program Office managing a variety of small to large volume-based, process-oriented contracts. The nature of the contracts provides an inherent disincentive for contractors to improve platform supportability of their own accord as the quantum of their financial reward is often directly linked to the volume of repairs conducted.¹⁵⁰

5.36 Performance-based contracting was described in a chapter on managing sustainment in the DMO’s 2007 Acquisition and Sustainment Manual, but not made a requirement or a priority.¹⁵¹ In 2008, a Defence internal audit evaluating the effectiveness of Defence’s use of performance-based contracting found that: the maturity of management systems for such contracts varied, there was inadequate central oversight and support, inconsistent contracting arrangements, and less effective outcomes in many cases.

5.37 Defence undertook substantial work in 2009 and 2010 to explore greater use of the technique so as to enhance Defence’s ability to meet capability preparedness requirements, and reduce the total cost of ownership.¹⁵² DMO advised industry CEOs at a meeting in July 2009 that ‘For greenfield sustainment and other new contracts there will be a greater focus on contracting for availability’.¹⁵³ Defence established a Performance-Based Contracting Centre of Excellence at that time and produced a new Defence contracting template with performance-based contracting integrated into it.¹⁵⁴ Defence has claimed that ‘almost all projects since 2011 have used a performance-based contracting template for materiel support services and have enjoyed the support of the Performance-Based Contracting Centre of Excellence’.¹⁵⁵ It has not been able to substantiate this claim (Box 6).

5.38 Major industry organisations have continued to urge Defence to progress performance-based contracting as a means of improving capability while reducing sustainment costs.

Note a: Many entries on the AusTender site represent individual payments rather than separate contracts. The system supporting the current AusTender site is described by Defence as ‘in a state of decay as it is over a decade old and no longer viable to upgrade and sustain’. Defence advises that ‘Over the past 12 months the CASG e-Procurement Team and the DoF AusTender Team, and SAP Australia Canberra Office have been progressing a whole of Government project to address the technical and business process risks posed by the current SAP AusTender Contract Reporting data capture and reporting solution’. In Defence’s case, this should eliminate the need for some 20 000 manual update transactions to AusTender each year.

Note b: This is part of a substantial project within Defence to improve both the recording of contracts and purchasing within Defence. Major steps include the development and delivery of a single (electronic) procurement form.

Note c: Defence, ‘Current e-Procurement Projects’, June 2016.

Current position

5.39 To work out whether performance-based contracting is yielding the outcomes it expects, Defence is undertaking an ‘academic analysis of the Critical Success Factors for PBC and their role in Program success’. Defence expects that this research will be complete in 2018.

5.40 The analysis could usefully have regard to the outcomes of past reviews and audits, which provide insight into the effectiveness of Defence’s contracting arrangements for specific fleets of equipment. For example, in ANAO Report No.52 2013–14, Multi-Role Helicopter Program, the ANAO concluded that the pre-2012 sustainment contract provisions for Defence’s Multi-role Helicopter Program were largely ineffective.¹⁵⁶ The audit report noted that this was largely due to Defence not applying its own processes to the standards required. The report also acknowledged that contract amendments were made to introduce a revised performance management regime. In May 2016, the Sustainment Gate Review for the program found that the commercial incentive in the contract was insufficient to obtain the desired rate improvement.

Establishing centres of expertise

5.41 The proposal to establish Centres of Expertise is reminiscent of the ‘business improvement teams’ envisaged under the Sustainment Business Model project (Appendix 6), and a further similar proposal to establish a ‘centre of procurement excellence’ recommended in 2000.¹⁵⁷ The novel element is the adoption of a matrix management arrangement for the control and deployment of a range of areas of expertise. The idea is to establish a managed resource pool that provides skilled personnel for CASG activities. This proposal is regarded as a key principle, integral to the acquisition and reform agenda. It is also seen as needed to streamline and consolidate the Systems Program Offices.

5.42 A design guide for the establishment of the Centres of Expertise has been produced and initial establishment of these centres is scheduled for April 2017, with the establishment of a ‘balanced matrix’ expected to be over a two-year period.¹⁵⁸

Developing a new operational framework

5.43 Developing the Capability Acquisition and Sustainment Group Business Framework is the first component of First Principles Review Recommendation 2.11, which provides: ‘that there be significant investment in the development of an operational framework which briefly but comprehensively explains how the organisation operates and the roles and responsibilities within it’ (the full recommendation is set out in paragraph 5.5, above). Such a framework would need to articulate how an organisation responsible for both acquisition and sustainment would carry out those major tasks. It is an overarching element in progressing First Principles Review implementation in Capability Acquisition and Sustainment Group.

5.44 Defence began development of the Capability Acquisition and Sustainment Group Business Framework as of 31 July 2015 with a view to completion on 1 February 2016. In February 2016, the Deputy Secretary, Capability Acquisition and Sustainment Group, informed the First Principles Review Implementation Committee that ‘the business framework project has commenced and been resourced to deliver a fundamental rebuild of all Group processes’.¹⁵⁹ The completion date was extended to 1 July 2016 and the objective was to have a ‘High-level business framework approved’.

5.45 A review of progress by a consultant in May–June 2016 found that no significant progress had been made: ‘As a result, my review changed from “progress made” to why none had been made’.¹⁶⁰ The consultant concluded that ‘Lack of decision making has frankly constipated this project’.¹⁶¹

5.46 Work was then reinvigorated and Defence’s First Principles Review Implementation Committee approved the Business Framework on 1 September 2016. Defence published a final, complete edition, comprising 18 pages plus appendices with a date of 12 December 2016. Although the document provides a description of the intended new organisational structure and the functions and responsibilities of its elements, it is not apparent that a ‘fundamental rebuild of all Group processes’ has yet been delivered. Defence advised that the new Business Framework ‘will take many years to fully implement’.¹⁶² Between February 2016 and February 2017, Defence entered into contracts with the International Centre for Complex Project Management (ICCPM) worth a total of $2.94 million for work described as relating to the development and implementation of the business framework. Based on Austender records, Defence also engaged two other companies, Cordelta and Eveille Consulting, to assist with the Business Framework. The value of these contracts totals about $835 000.

Development of the ‘smart buyer’ model

5.47 Operating as a smart buyer is an essential function for an organisation seeking to outsource work. To be a smart buyer requires the capacity and technical knowledge to articulate its organisation’s requirements and recognise, independently of the vendor, whether the vendor is meeting those requirements.¹⁶³ Past Australian use of the ‘smart buyer’ concept (discussed in Appendix 7 of this audit report) is consistent with this approach.

5.48 The First Principles Review recommended that Defence establish a single end-to-end capability development function and that this would require, among other changes, ‘Moving to a leaner “smart buyer” model that better leverages industry, is more commercially oriented and delivers value for money’.¹⁶⁴ The Review cites a US Government Accountability Office report which is also consistent with past Australian usage:

A smart buyer is one who retains an in-house staff who understands the organization’s mission, its requirements, and its customer needs, and who can translate those needs and requirements into corporate direction. A smart buyer also retains the requisite capabilities and technical knowledge to lead and conduct teaming activities, accurately define the technical services needed, recognize value during the acquisition of such technical services, and evaluate the quality of services ultimately provided. As long as the owner retains the in-house capabilities to operate as a smart buyer of facilities, there does not appear to be any greater risk from contracting out a broad range of design review-related functions, so long as such functions are widely available from a competitive commercial marketplace. If the owner does not have the capacity to operate as a smart buyer, the owner risks project schedule and cost overruns and facilities that do not meet performance objectives’.¹⁶⁵

Capability Acquisition and Sustainment Group’s implementation of ‘smart buyer’

5.49 Defence has developed its own meaning for the term ‘smart buyer’, which does not clearly capture the intent of the First Principles Review recommendation. At its core, this is a two-step decision-making framework comprising risk analysis and a tailored strategy done at the earliest practicable stage of a project:

• Step 1—develop a risk profile for the project, which identifies the severity of risk in nine categories for the acquisition phase and eight for the sustainment phase; and
• Step 2—develop a risk-based tailored project execution strategy, covering approvals, project management, and through life asset management, including a coherent approach to the acquisition and sustainment phases.¹⁶⁶

5.50 Defence considers that ‘The game changer for Capability Acquisition and Sustainment Group will be the “Smart Buyer” model that will unleash our organisational capability’.¹⁶⁷ Defence sees the main benefit of this process as early identification of risks and, as a result, better allocation of Defence resources.¹⁶⁸ Its ‘smart buyer’ paper sets out detailed issues for consideration in areas of risk separately for both acquisition and sustainment.

Note a: Defence, ‘Smart Buyer—Detail Design’, approved 20 October 2016, p. 1.

5.51 In its implementation of the smart buyer concept, Capability Acquisition and Sustainment Group does not clearly articulate the competencies required to specify outsourcing requirements. These competencies include: specifying requirements independent of suppliers, working out where best to obtain those requirements; and knowing how to assess what they have received on behalf of the Commonwealth. These are all fundamental skills in an organisation seeking to engage more with industry and contract out sustainment work wherever that represents value for money. This is a risk in light of the systemic issue identified by Defence in its work towards reforming Systems Program Offices: ‘SPO staff have limited experience, skills, and competencies needed to effectively establish, govern, and assure industry delivery of capability’ (see Box 3, p. 23 of this audit report).

5.52 The same risk has been highlighted in a number of internal Defence reviews including:

• a 2016 sustainment gate review of the Multi-Role Helicopter (MRH) project which commented on the need to clarify the roles and responsibilities of Defence and industry, and ensure the Systems Program Office has the appropriate skills and sufficient capacity to discharge its responsibilities¹⁶⁹;
• the 2016 Houston Review into Army Aviation, which observed that the Systems Program Office for the Tiger and the MRH helicopters relied heavily on several key contractors¹⁷⁰; and
• a sustainment gate review of Air Force’s Airborne Early Warning and Control fleet, which observed that all Systems Program Offices staff needed to be fully capable to deal with a complex platform and small fleet of aircraft.¹⁷¹

5.53 A relevant finding also emerged in the recent Parker Review in the UK of the shipbuilding strategy for the Royal Navy. The Review found a need for the Ministry of Defence ‘to rebuild its capability as an intelligent client for warship design and build’.¹⁷² It concluded that, to ensure pace could be maintained ‘additional technical expertise in the form of a Client Friend (or Friends) is procured early, in particular to assist in the development of a detailed specification’. This suggests that, where in-house expertise is not (or is no longer) available, expertise from industry could be bought in. Care would need to be taken to avoid any conflict of interest among vendors in these circumstances.¹⁷³

Asset management

5.54 Defence has long aspired to introduce a systematic approach to asset management for its specialist military equipment (see Appendix 6 of this audit report). Significantly, the Coles review recommended that Defence develop an asset management strategy for sustainment (see Appendix 5 of this audit report) and Chief of Navy subsequently expressed a strong desire for a total asset management framework.¹⁷⁴

5.55 In November 2016, Defence informed the ANAO that it intends to use the principles of the ISO 55000 Asset Management Standard as a guide of good practice and to further develop Defence’s framework for the management of sustainment.¹⁷⁵ In March 2017, it advised that the policy issued under the earlier DMO governance framework was gradually being reviewed and updated. Defence has continued a program offering its staff professional development options in asset management.

Proposals that asset management underpin a Sustainment Business Model

5.56 Appendix 6 of this audit report summarises proposals between 2008 and 2011 to introduce a Defence Sustainment Efficiency Office and a Sustainment Business Model. Three significant reviews into Defence sustainment around this time observed that Defence did not have an asset management approach to sustainment.¹⁷⁶ An asset management approach seeks to derive the best possible value and return on investment from assets across their lifecycle.¹⁷⁷ According to the standards, the approach is ‘vital for organisations that are dependent on the function and performance of their physical assets in the delivery of services or products, and where the success of an organisation is significantly influenced by the stewardship of its assets’.¹⁷⁸

5.57 A briefing to DMO Division Heads in September 2011 summarised the shortcomings of Defence’s approach to sustainment in the absence of an asset management approach:

• The absence of an asset management approach within Defence makes it ‘difficult to piece together a holistic asset management view across the myriad individual policies’ which focused on discrete processes.
• Reporting and compliance was ‘devoted to “soft” governance rather than to “hard” asset management decisions, based on sound financials, across the whole capability lifecycle’.
• Defence had multiple repositories and versions of asset management information making it hard to assess whether Defence was aligned to best practice.
• Information sharing about asset management was inconsistent and fractured.

5.58 In 2011, DMO set about developing a Sustainment Business Model drawing on generically applicable asset management principles.¹⁷⁹ There is then little evidence of substantial progress of the project for some years. However, in 2013, a further report by external consultants stated that Defence needed an integrated, whole-of-life approach to sustainment and that Defence could realise significant benefits by modelling the management of their assets on the approach taken by commercial organisations—an asset management approach.

5.59 Defence paid $5.5 million over four years (2011–12 to 2014–15) for consultancy services related to its sustainment business model. Also, it incurred related staff costs of $3.1 million over six financial years (2011–12 to 2016–17). Defence informed the ANAO that it had used this work to reform Maritime asset management, inform its submissions to the First Principles Review, underpin the Systems Program Office reform guide and develop its Systems Program Office review methodology.

5.60 In May 2015, DMO released a sustainment policy stating that Defence assets would be managed in accordance with the international standard for asset management. However, a September 2015 assessment by external consultants of Capability and Sustainment Group policy and guidance against the requirements of the Asset Management Standards concluded¹⁸⁰:

While the organisation can demonstrate clear leadership and commitment to delivering and sustaining asset capabilities that meet its strategic business and operational intent, its ability to uniformly translate this intent into actions that demonstrate the consistent delivery of the business outcomes from its assets, is hindered by incomplete process documentation and [the absence of] an integrated Business Management System for its assets.¹⁸¹

5.61 The report noted that ‘it is impossible to compare different operational business processes within the organisation, in order to determine which process is the most effective for the entire organisation, until the organisation has determined the base policies (rules) by which it will conduct the comparisons’.¹⁸²

5.62 The same view was reflected in advice received by Capability and Sustainment Group from the Chief Defence Scientist, also in September 2015. The Defence Science and Technology Group’s Aerospace Division had concluded that a systematic and top-down approach such as enterprise-wide asset management was particularly suitable to address the challenges of reforming sustainment management in Capability and Sustainment Group. The Defence Science and Technology Group had been helping individual Systems Program Offices with asset management but believed its real potential was at the enterprise level. Therefore it was important to approach this from a Capability and Sustainment Group or Defence perspective.¹⁸³

5.63 The Sustainment Business Model project has not delivered all that DMO expected. In October 2015, some four years after the Sustainment Business Model project started, Defence’s First Principles Review Implementation Team decided that work on developing the Sustainment Business Model, along with other reform within Capability and Sustainment Group at that time, would cease and the ‘relevant elements’ were to be incorporated into the new Capability Life Cycle initiative. The documentation does not specify which elements remained relevant.

5.64 The current Capability Life Cycle manual and its detailed design document do not mention a business model for sustainment nor any overarching sustainment framework.

Has Defence developed an evaluation plan for the reform of Capability Acquisition and Sustainment Group?

5.65 The audit found limited evidence that Defence was preparing to evaluate the reforms to Capability Acquisition and Sustainment Group, its execution of the reforms or their outcome in terms of improvements to either acquisition or sustainment. Current proposals appear to be limited to an ‘independent health check’ on progress.¹⁸⁴ On the face of it, the scope of the health check is narrow, focused on closing recommendations and assessing progress. Given the substantial resources involved, both in the expenditure on sustainment and the reform process, Defence would benefit from assessing all aspects of these changes carefully.

Appendix 1 Defence’s Systems Program Offices

Table A.1: Key data for each of Defence’s Systems Program Offices: staff, assets and major contracts

Note a: Data is current as at 24 February 2017. No of contract staff as advised 2 June 2017.

Note b: Staff numbers are rounded to the nearest whole number.

Note c: Defence advised the ANAO that this amount ‘includes elements considered “acquisition” due to the impossibility of separating these items in this context’. SPOs with a preponderance of contractor activity in acquisition include Submarines and those SPOs designated ‘Project Offices’.”

Source: Department of Defence.

Appendix 2 Navy core Key Performance and Health Indicators for sustainment

Table A.2: Navy Materiel Sustainment Agreement Performance Framework – Core Key Performance Indicators

Source: Defence.

Table A.3: Navy Materiel Sustainment Agreement Performance Framework - Core Key Health Indicators

Note a: Navy Allowance (NAVALLOW) is the Navy logistics system.

Source: Defence.

Appendix 3 Reporting on the Armed Reconnaissance (Tiger) helicopter

The performance of the Armed Reconnaissance (Tiger) Helicopter is reported thus in the Quarterly Performance Report for April – June 2016, produced in August 2016:

The Final Acceptance Milestone under the Acquisition Contract with Airbus Group Australia Pacific (formerly Australian Aerospace) was declared achieved on 28 May 2013, approximately five years later than originally contracted. All 22 Tiger helicopters, simulators, ground support systems and contracted training devices have been accepted and introduced into service within project budget approvals. Final Operational Capability was declared with caveats by Chief of Army (CA) on 14 April 2016.

The Final Materiel Release (FMR) Report Approval Certificate has been signed by all stakeholders with caveats. The red traffic light for FMR is due to the late achievement against the MAA milestone. The last item of Support & Test Equipment requires codification before it can be formally accepted by the Commonwealth of Australia (CoA). Late delivery has minimal impact on the overall development of the Armed Reconnaissance Helicopter (ARH) capability.

[Materiel Acquisition Agreement] Closure and accompanying Product Schedule Change Proposal (PdCP) to transfer the delivery of the Deployable Aircraft Maintenance Rig capability to CA12 completed and awaiting Assets Under Control (AUC) balance adjustments before paperwork can be processed for signature.

The final Contract Acceptance milestone was achieved on 28 May 2013 within the approved acquisition budget envelope.

Final Operational Capability was declared in April 2016 and ministerial advice was noted on 3 May 2016. Closure of the Materiel Acquisition Agreement and project closure will be completed during 2016.

The Tiger fleet achieved a total of 312.0 flying hours in June against a planned 698 hours. This month closes out the Financial Year total at 3995.2 against the planned 5848 hours, a result of 68.32 percent. Improvements are still being seen in Rate of Effort (ROE) generation, since the implementation of the renegotiated ARH Sustainment contract arrangement. However, even with higher rates of availability, improved fleet serviceability and Repairable Item (RI) turnaround time, flying hours are still lower than expected.

Tiger is not currently deployed on operations.

Tiger sustainment is continuing to mature. Performance has been effective but unstable during the recent high demand collective training exercise series period that culminated in Exercise Hamel. This high tempo period of exercises has tested the robustness of the logistics system and deficiencies have been identified. Poor product reliability has contributed to weak serviceability at the 1st Aviation Regiment in Darwin.

The availability of the aircraft is either shown as ‘Green’ (acceptable performance) or ‘Amber’ (early signs of underperformance), depending on the part of the report examined.

The project has spent $133.966 million on sustainment in 2015–16 against a budget for the year of $133.679 million, and $1867 million of an acquisition budget of $2033 million.

In the Quarterly Performance Report, Defence does not identify the Tiger as an under-performing acquisition or sustainment project, even though the other sources suggest just that. For example, those sources state:

• that the aircraft has not met, and is unlikely ever to meet, the required level of capability expected by government. It was not expected to meet key sustainment metrics such as Reliability/Availability/Maintainability, Rate of Effort and cost of ownership, and should be replaced;
• the Chief of Army has place caveats on its use—effectively limiting what the aircraft can be used for, and where it can be used;
• Army did not deploy the helicopter in its engagements in Afghanistan or Iraq;
• the program has consistently underperformed;
• on average only 3.5 aircraft of the operational fleet of 16 aircraft were serviceable at 10 am on any given day in 2015, against a requirement of 12 aircraft;
• availability for the aircraft is predominantly limited by premature component level failures and is constrained by inadequate sustainment arrangements¹⁸⁵;
• despite the lower than expected hours flown, by June 2014 sustainment costs for the Tiger had exceeded the original sustainment contract (2004–19) value of $571 million, with five years still remaining; and sustainment costs to June 2016 total $921 million.
• in June 2016, the cost per flying hour for the Tiger fleet was $30 335, compared to a target of $20 000 (the long-term average in June 2016 was $39 472 per flying hour); and
• the project was approved in March 1999 with a budget of $1.58 billion (December 1999 prices). Since 2001, the budget has increased by $418.2 million from price index variation, and $121.5 million from exchange rate variation. The current approved budget is $2.03 billion.

The 2016 Houston Review also found that the full potential of the Australian Tiger program, when reached, will not be adequate to provide the robust, reliable and sustainable aviation reconnaissance and attack capability that Army requires.

Appendix 4 Sustainment activity public reporting 2015–16

Defence Portfolio Budget Statements 2015–16

• Table 6: Capability Sustainment Program lists estimates for nine elements of Defence’s Capability Sustainment Program (Navy Sustainment, Army Sustainment and so on) over the four financial years of the forward estimates period.¹⁸⁶ This table also includes estimates for items other than sustainment of military equipment.
• Table 94: Top 30 Sustainment Products by End of Financial Year Outcome 2015–16 sets out a list of the Top 30 sustainment products and the estimate, at Budget time, of expenditure on sustainment of each of those products over the 2015–16 financial year.¹⁸⁷ There is no reconciliation of the ‘total sustainment product funds available’ figure reported in this table with the ‘total sustainment’ estimate in Table 6 (discussed above).
• Under the heading ‘Top 30 Sustainment Product Descriptions’, Defence provides a description for each of the Top 30 sustainment products. Generally, this comprises a description of the asset and an account of the focus of Defence’s sustainment effort during the prospective year. This is potentially useful information though not always written in clear language. For example, the description for the Canberra Class ships states that during 2015–16, ‘the focus will be on embedding and fostering the critical enabling functions for the initial period of LHD and LLC operations’.¹⁸⁸ This conveys very little insight into what Defence is planning to achieve.
• The PBS includes other occasional references to sustainment. For example, a note to Table 21: Army Deliverables, which in relation to the Army’s Armed Reconnaissance (Tiger) Helicopter, states ‘while Tiger reliability, availability and maintainability continue to improve, rate of effort is affected by the low rate of availability of aircraft with fully serviceable mission equipment due to poor but improving contracted sustainment support and the requirement for unscheduled maintenance’.¹⁸⁹
• The PBS also contains broader commitments on sustainment management for example, ‘the DMO will continue the implementation of a more standardised Sustainment Model to promote better and more consistent practice’.¹⁹⁰

Defence Portfolio Additional Estimates 2015–16

• Table 10: Capability Sustainment Program corresponds to Table 6 in the PBS (see above). This table lists, for each of the nine elements of Defence’s Capability Sustainment Program: the 2015–16 budget estimate (as presented in the 2015–16 PBS); a revised estimate for 2015–16; and the variation.¹⁹¹
• Table 77: Top 30 Sustainment Products by End of Financial Year Outcome 2015–16 corresponds to Table 94 in the PBS (see above). This table lists the same Top 30 sustainment products presented in Table 94 of the PBS, in the same order. For each sustainment product, the table includes: the budget estimate for 2015–16 (as presented in the 2015–16 PBS); a revised estimate for 2015–16; variation; and an explanation of the variation plus a ‘project performance update’. Reasons for the variations in 2015–16 estimates vary, but changes in exchange rate and increased effort to support operations are among the more common. A notable feature of this particular table is that financial data and information about performance is contained in the same place, whereas it is presented in separate tables in both the PBS and Annual Report (see below).¹⁹²

Defence Annual Report 2015–16

The Defence Annual Report 2015–16 provides information relating to the PBS and PAES tables, referred to above, in online web tables. The web tables are accessible through the on-line version of the Annual Report on Defence’s website, specifically at the end of Chapter 8. The two most relevant tables for sustainment reporting are Web Table 8.11: Top 30 sustainment products by expenditure, as forecast in the Portfolio Budget Statements 2015–16, and Web Table 8.2: Top 30 sustainment products, performance summary as at 30 June 2016.¹⁹³

• Web Table 8.11 includes a list of the same Top 30 sustainment products reported on in the PBS and PAES. However, they are in a different order, which may reflect changes in organisation following the delisting of DMO. For example, the category listed in the PBS ‘Helicopters, Tactical Unmanned Aerial Systems and Guided Weapons’ no longer appears and the items are distributed among other categories. The table reports, for each Top 30 sustainment product: the budget estimate for 2015–16 (as reported in the PBS); the revised estimate for 2015–16 (as reported in the PAES); the actual expenditure for 2015–16; and the variation against the revised estimate. The table also reports the budget estimate for 2015–16, revised estimate for 2015–16, actual expenditure for 2015–16, and the variation against the revised estimate for ‘other approved sustainment products’ and ‘support to operations’.¹⁹⁴ The information reported in Web Table 8.11 is reproduced in Table A.4 below.
• Web Table 8.2 provides a one-paragraph ‘performance summary’ by product for each of the same Top 30 sustainment products.¹⁹⁵ In contrast to Web Table 8.11, Web Table 8.2 treats each product in the same order as the PBS and PAES.
• The 2015–16 Annual Report does not report the actual expenditure for the nine categories of estimated sustainment expenditure reported in Table 6 of the PBS and Table 10 of the PAES. Additionally, there is no reconciliation of the ‘total sustainment funding’ amounts reported in Web Table 8.11 of the 2015–16 Defence Annual Report and the Capability Sustainment Program in the PBS and PAES tables discussed above.

Table A.4: Web Table 8.11: Top 30 sustainment products by expenditure, as forecast in the Portfolio Budget Statements 2015–16

Source: Department of Defence, Annual Report 2015–16, Web Table 8.11 available from <http://www.defence.gov.au/annualreports/15-16/Chapters/chapter-8.asp> [accessed 20 April 2017].

Appendix 5 Lessons from Coles

Analysis based on 2012 Coles Report (Phase 2 and 3)

1. The 2012 Coles Report identified five root causes of problems with the sustainment of Collins Class Submarines, and made 25 recommendations.¹⁹⁶ Based on the ANAO’s analysis, the recommendations with potentially broader application across Defence sustainment are listed below.

2. Unclear sustainment requirements—Defence lacked a clear unclassified top-level operational requirement for the platform. A single defined and realistic requirement is critical to target setting, future planning, and the development of enterprise behaviours.

Recommendations with potential Defence-wide application:

• Set realistic performance targets and document them in the Materiel Sustainment Agreements.¹⁹⁷
• Define and document a clear, unclassified requirement for the sustainment program.

3. Lack of a performance based ethos—Defence had failed to define performance targets for individuals and organisations and this had been a key driver of inefficiency and sub-optimal behaviours that affected sustainment performance.

Recommendations with potential Defence-wide application:

• Design and implement sustainment arrangements that encourage performance-based behaviour.
• Defence needs to be an intelligent customer for sustainment.
• Create a collaborative framework for the management of sustainment without diluting individuals’ responsibilities. As part of the governance system, create a forum to bring together all suppliers to raise issues and identify opportunities.
• Develop and implement a contracting strategy to improve performance based contracting [Management Recommendation.
• Improve leadership skills, knowledge and experience.

4. Poor planning—Defence lacked a clearly stated long-term strategic plan. This prevents accurate lower level plans and targets (for example, for maintenance and operations) being established and achieved. The absence of asset management strategy resulted in poor obsolescence and a lack of reliability management and, consequently, a ‘bow wave’ of reliability defects and obsolescence.

Recommendations with potential Defence-wide application:

• Develop an Asset Management Strategy for sustainment.
• Performance measures in Materiel Sustainment Agreements must be realistic and must be consistent with the requirements in sustainment performance artefacts such as the sustainment contract.
• Develop a through-life capability management plan.
• Define and endorse an Asset Management Plan.

5. Unclear lines of responsibility—Many key sustainment roles and responsibilities were not clearly defined or understood, from both an individual and organisational perspective. This resulted in the various parties failing to act on, or enforce, their responsibilities within the sustainment space.

Recommendations with potential Defence-wide application:

• Develop clear lines of authority and responsibility for sustainment.
• Develop and implement a strategy to address skills shortages in sustainment role.

6. Defence lacked a single set of accurate information to inform decision making—The review found that, whilst significant levels of information and analysis exist across the Collins Class Sustainment Program, the basis for long-term decision making was not always consistent or accurate, with multiple systems and datasets in use for financial, maintenance and supply chain activities. In many cases, these are not linked resulting in data integrity issues. This affects financial planning, as it makes it extremely difficult to set an agreed baseline position and accurately link expenditure to outputs. Without a single version of the truth from a financial perspective, it is difficult to assess the impacts of variances in funding appropriations on sustainment objectives.

Recommendations with potential Defence-wide application:

• Develop enterprise-wide IT strategy and information management strategy.
• Develop cost baselines, models, and supporting processes for sustainment program.

Summary of lessons learned about sustainment from previous reviews (as summarised by Coles)

• Defence should give sustainment much higher attention and priority during the initial phases of the asset’s life cycle (‘Needs’, ‘Requirements’, and ‘Acquisition’). Failure to pay sufficient attention to sustainment early on will increase the cost of ownership to Defence. Retrofitting the necessary sustainment arrangements is more challenging than getting them right at the outset.
• Defence should base its selection of systems/equipment on systems sourced from credible/proven suppliers.
• Defence should give proper attention to asset management, logistic support and engineering requirements.
• The capability management role needs to be resourced appropriately to be able to take the lead roles of the asset owner and the sustainment customer.

Appendix 6 Earlier attempts in Defence to take a systematic approach to asset management

The Mortimer Review (2008)

The effectiveness of the reforms implemented following the 2003 Defence Procurement Review (the Kinnaird Review) were evaluated in 2008 by the Defence Procurement and Sustainment Review (the Mortimer Review).¹⁹⁸ Mortimer made specific recommendations to improve sustainment, including that Defence create ‘an independent Sustainment Efficiency Office to measure, benchmark, and find ways to improve the efficient delivery of sustainment’.

This could have provided, for the first time, a systematic approach to the organisation of sustainment activities across the Systems Program Offices. The government agreed to the recommendation but Defence did not implement it.¹⁹⁹ By December 2013, some five years later, Defence decided that it would not meet the recommendation’s intent. Defence then rewrote the Mortimer recommendation to read ‘implement a system²⁰⁰ to improve oversight of DMO sustainment performance’.²⁰¹ Thus Defence took a passive role, focusing on developing a system to monitor Systems Program Offices’ work rather than the more active analysis and management of sustainment across the organisation, envisaged by Mortimer.

The Sustainment Complexity Review (2010) found that diversity remained

A separate reform stream began in July 2010, when consultants engaged by DMO found its sustainment business model was strikingly different from those in the private sector. DMO’s approach was ‘extremely variable and inconsistent’, particularly when compared with most commercial and other government organisations:

It has inherited 50+ business models that existed before the DMO was formed in 2000. There is no evidence that specific merger integration activities have been undertaken to ensure consistent integrated operating models. From the perspective of the field review, there is some indication that the sustainment models continue to drift apart.²⁰²

The consultants attributed the diversity to ‘historical issues around original parent organisations, [and] individual service influences’. Their report made many recommendations generally aimed at reducing complexity in sustainment management, including greater use of performance-based contracts, a more consistent operating model across sustainment and improving commercial asset management expertise.

Defence attempted to develop a Sustainment Business Model

In 2011, DMO established a project to develop a Sustainment Business Model to address its ‘extremely variable’ approach to sustainment that was ‘inconsistent compared to most commercial and other government organisations’.²⁰³ DMO saw such an enterprise-wide approach to the management of sustainment as a critical component of one of the strategic recommendations of the Rizzo Review of naval sustainment, a review which had come about because of inadequate maintenance and sustainment practices.²⁰⁴

Appendix 7 The ‘Smart Buyer’ concept

Origins

The term ‘smart buyer’ is a term of art, with a specific meaning in public administration, relating to outsourcing, and which arose around 1990 following work in the USA by the GAO and the Brookings Institution.²⁰⁵ The key paragraph in an early GAO study states:

The government must have an active role in the decision-making process. Government officials must be the ones applying discretion and making value judgments throughout the process. A key criterion in determining whether consulting contracts would be appropriate is whether the government could maintain sufficient in-house capacity to be thoroughly in control of the policy and management functions of the agency. This includes the capacity to adequately direct, supervise, and monitor contracts. The government must be a “smart buyer” when purchasing assistance services and be able to make independent judgments about policy recommendations.²⁰⁶

Other, related GAO work has used the term in relation to the management of sustainment:

privatizing total support on new and future weapon systems can make it difficult for the organic depots [equivalent to Systems Program Offices] to acquire and sustain technical competence on new systems, leading edge technologies, and critical repair processes necessary to maintain future core capabilities, provide a credible competitive repair source, and be a smart buyer for those logistics activities that will be contracted out.²⁰⁷

Past use in Defence

With a substantial amount of its sustainment work contracted out, Defence has often emphasised its desire to have the internal expertise to be a smart buyer:

• In a submission to the Joint Committee of Public Accounts and Audit inquiry into contract management in the Australian Public Service in 2000, Defence, through the then Under Secretary for Defence Acquisition—almost a direct counterpart position to the current Deputy Secretary, Capability Acquisition and Sustainment Group —stated that: ‘Defence must see that it has available the technical, financial and managerial expertise to be a smart buyer’.²⁰⁸
• Similarly, in an international seminar on Defence acquisition in July 2000, an issue identified by Defence at the time was ‘the challenge to be a smart buyer’.
• The concept was also included in the initial Materiel Sustainment Agreements in 2005 between DMO and the Defence Service Chiefs.
• Defence refers to itself as a ‘smart buyer’ in the Defence Industry Policy Statements in 2007²⁰⁹ and in 2010.²¹⁰

Defence’s Defence Science and Technology Group has long been represented as providing the technical and scientific expertise to enable Defence to be a smart buyer. A 2016 Parliamentary inquiry into Capability of Defence’s physical science and engineering workforce noted that:

Several submissions and witnesses emphasised the need for Defence to have a sufficient [science and engineering] workforce to allow it to be a ‘smart’ buyer or an informed customer. For example, Dr Davies²¹¹ noted that in his experience, having in-house expertise from public service engineers who could examine bids for work from external firms made him ‘a much smarter buyer’.²¹²

The Committee recommended that Defence commit to maintaining its physical science and engineering workforce capabilities in key areas to allow it to be both a smart buyer and a technically proficient owner of materiel. The Government agreed to the recommendation.