Introduction

1. Materiel Sustainment Agreements (MSAs) have been used since 2005 as customer–supplier agreements formalising the relationship between the Department of Defence (Defence) and the Defence Materiel Organisation (DMO)¹ for the sustainment of specialist military equipment.² Through the agreements, the Defence Capability Manager³ undertakes to supply funding, and the DMO undertakes the sustainment of specific platforms (such as a ship or aircraft fleet), commodities (such as clothing or combat rations) and services (such as provision of maritime target ranges). Sustainment involves the provision of in-service support, including repair and maintenance, engineering, supplies, configuration management and disposal action. Effective sustainment of ships, aircraft and vehicles is necessary to maintain the preparedness of the Australian Defence Force (ADF) and enable the conduct of Defence operations.

2. In 2014–15, out of total funding of $35.781 billion, Defence budgeted $7.109 billion, or 20 per cent, for its Capability Sustainment Programme, with most of this funding transferred to the DMO.⁴ By way of context, the DMO’s sustainment expenditure has regularly exceeded its acquisition expenditure in the past decade.⁵

3. Sustainment activities are generally administered by the DMO’s Systems Program Offices (SPOs). The SPO serves as the single point of contact with industry, government entities, and other entities participating in the acquisition or sustainment of specialist military equipment. Generally, each major platform is managed by a single SPO, which may also manage the delivery of a commodity or a service.

Managing Defence’s sustainment function

4. The Australian Defence Organisation comprises the Department of Defence, the ADF and the DMO. The successful provision of Defence capability depends on the Defence Organisation as a whole collaborating effectively and the component parts meeting their respective functional responsibilities, including, for the DMO, acquisition and sustainment responsibilities. In September 2003, when the then Government approved the DMO’s transition to a prescribed agency under the Financial Management and Accountability Act 1997 (FMA Act)⁶, the intent was for the DMO to be more performance-oriented and business-like, so as to improve procurement and support practices, and establish a more transparent relationship between Defence and the DMO.⁷

5. The DMO’s February 2004 Business Model noted the primacy of Defence as the customer, with responsibility for setting requirements and determining priorities within agreed funding levels; and that the DMO would be funded and managed on the basis of agency agreements for the services it delivered. The agency agreements were to facilitate funding flows and delineate responsibilities and accountability between Defence and the DMO, and provide better visibility of the costs of procuring and sustaining Defence assets. The agreements were to be refined and improved over several annual cycles. The lowest tier of these agreements, for sustainment, would be the MSAs:

At the tactical level, the scope, price and time frame for specific services between Defence and DMO would be captured in simple agreements that describe the products and services flowing from DMO’s Outputs—projects, sustainment and policy advice and services. […] The products/services described in these agreements must be meaningful, manageable and measurable by both parties and facilitated by a principle of open books between the two with agreed underlying assumptions clearly stated and risk management measures built in.⁸

6. The implementation of these agreements led to MSAs consisting of two levels. The first level is the Heads of Agreement, which is an overarching document that covers a series of Product Schedules. The Heads of Agreement contain the high-level framework establishing the partnership between each Capability Manager and the DMO. The second level of each MSA is the Product Schedules. Each Product Schedule deals with the sustainment of a specific platform, commodity or service for the relevant Defence Service or Group. The Product Schedule defines: the supplies and services that will be provided by the DMO; the budget that is provided by the Capability Manager; and standards for matters such as responsiveness, availability levels, and maintenance timeframes.

7. Most of Defence’s $7.1 billion sustainment budget for 2014–15 is included in the DMO’s sustainment budget of $6.185 billion⁹, and of this, some $5.683 billion is to be expended through seven MSAs incorporating 116 Product Schedules. The main MSAs are those between the DMO and each of the three Services. For 2014–15, the Chief of Navy MSA includes 36 Product Schedules valued at $1.976 billion; the Chief of Air Force MSA comprises 28 Product Schedules valued at $1.976 billion; and the Chief of Army MSA includes 45 Product Schedules valued at $1.504 billion.¹⁰ Since 2012, the previous practice of complete annual revision of the MSAs has been replaced by partial revision as needed, particularly of the financial sections of Product Schedules.

Audit objective, criteria and scope

8. The audit objective was to examine Defence’s administration of Materiel Sustainment Agreements (MSAs) and the contribution made by MSAs to the effective sustainment of specialist military equipment.

9. Four high-level criteria were developed to assist in evaluating Defence’s administration of MSAs:

• MSAs constitute an effective arrangement formalising the relationship between Defence Services/Groups and the DMO for sustainment activities.
• MSA Product Schedules clearly identify costs, deliverables and appropriate Key Performance Indicators (KPIs).
• Risks, issues and constraints to achieving effective sustainment are identified in MSA Product Schedules, and appropriate management strategies are in place.
• Monitoring and reporting processes provide relevant and reliable information on sustainment activities.

10. The ANAO examined Defence’s MSA policy, procedures and systems, including reforms in these areas. The ANAO also examined Defence’s management of three Product Schedules. One Product Schedule was selected for each of the Navy, Army and Air Force:

• Navy’s FFH ANZAC frigates (Product Schedule CN02¹¹), as one of the first MSA Product Schedules to be revised after the Rizzo Report¹²;
• Army’s Protected Mobility Vehicle (Bushmaster) fleet (Product Schedule CA04), which has been heavily used in Iraq and Afghanistan; and
• Air Force’s AP-3C Orion fleet (Product Schedule CAF04), which was heavily involved in the search for Malaysian Airlines flight MH-370 in the Indian Ocean.

Overall conclusion

11. In 2014–15, Defence budgeted over $7.1 billion, or some 20 per cent of total Defence resourcing, for the sustainment of specialist military equipment operated by the ADF. The majority of Defence sustainment services are provided either directly or indirectly through the Defence Materiel Organisation (DMO), which applies some 50 per cent of its budget ($6.185 billion¹³) to sustainment. Defence has been using Materiel Sustainment Agreements (MSAs) since 2005 to formalise its requirements for sustainment services from the DMO, with the aim of facilitating effective and business-like relationships within the Defence Organisation. MSAs are contract-like arrangements that set out the level of performance and support required by Defence from the DMO, within an agreed price, as well as the Key Performance Indicators (KPIs) by which service delivery will be measured.

12. Over the past decade, Defence and the DMO have established and continued to refine a generally sound MSA framework to facilitate the management of sustainment activity for specialist military equipment. The framework has enabled the Defence Organisation to clearly identify roles and responsibilities at a functional level, and individual MSAs document funding, deliverables, risks and performance measures for each sustainment product. Further, the development and maintenance of the MSA framework has encouraged and facilitated collaboration between Defence and the DMO at both the management and operational levels. The MSA framework has evolved over time, in light of practical experience and the risk appetite of the parties to individual agreements, and there is an ongoing role for Defence senior leadership to shape the direction of the framework so as to realise its full potential. More generally, there remains scope for Defence to enhance its sustainment management through the implementation, use and refinement of newly developed performance measures.

13. As discussed, the MSA framework has continued to evolve since 2005, with relatively bureaucratic processes being replaced over time by simpler arrangements. In particular, when the DMO commenced an MSA reform process in 2012, Defence stakeholders observed that the practice of annually reviewing the entire suite of MSAs was long and tedious, with too little emphasis on sustainment planning and performance management to deliver the best outcome with available funding. The reform process resulted in revised procedures for the management of MSAs, including a move towards developing more enduring MSAs and a simplified process for amending them. Under the new process, the Services and the DMO collaboratively review sustainment progress at both the management and operational levels, focusing on capability issues and required funding changes.

14. A key strength of the MSA framework is the capacity for Capability Managers to adjust individual Product Schedules in light of assessed risks. Following Navy’s inability to supply vessels requested by the then Government to assist in the clean-up after Cyclone Yasi in 2011, Navy demonstrated a relatively conservative risk appetite which resulted in more detailed Product Schedules and a requirement to approve changes at higher management levels. While the preferred approach is a matter for Navy’s senior leadership, there is scope to review future settings, in light of delays experienced in processing changes to Product Schedules and the emergence of undocumented workarounds to manage those delays.

15. A key objective when introducing the MSA framework was to capture the scope, price and timeframe for the provision of specific services, and individual Product Schedules do so. Defence has also taken steps to enhance its sustainment performance management through the development of standardised suites of performance measures. Between 2012 and 2014, the DMO, Navy and Army developed new performance measurement frameworks, including measures of availability, cost, schedule, and materiel deficiencies, which are to be reported through a new DMO system. These performance measures will not be fully implemented until the DMO system is operational, and at the time of the audit, the first phase of the system rollout was scheduled for May 2015. While the new performance measures should provide a firmer basis for the evaluation and active management of sustainment performance and costs, their establishment remains at an early stage.

16. The delivery of ADF capability relies in large measure on effective collaboration between key elements of the Defence Organisation—including the three Services and the materiel sustainment arm. The MSA initiative introduced a structured framework for engagement on sustainment matters; an approach of continued benefit irrespective of specific organisational arrangements within Defence.¹⁴ The practical effectiveness of the MSA framework largely depends on active and timely management of identified risks by Capability Managers and the DMO, and in that respect, a robust MSA framework is an aid to management, not an end in itself. To build on the progress made to date through MSAs, the ANAO has made two recommendations focusing on: the review of change management processes for Navy Product Schedules and their level of detail, to support more flexible management of MSAs and avoid undocumented workarounds in their administration; and clarifying the internal treatment of acquisition and sustainment funding.

Key findings by chapter

Materiel Sustainment Agreements (Chapter 2)

17. In 2012, Defence stakeholders agreed that the MSA process was bureaucratic and inflexible. There was an overriding focus on the annual development and approval of MSAs, rather than on using the MSA framework to actively plan and manage sustainment funding and activities. The DMO led an MSA reform project and reached agreement with Defence Services and Groups on streamlined arrangements. There was a particular focus on treating MSAs as enduring documents rather than updating them annually, and delegating authority for updating sections of Product Schedules to line management. The November 2012 DMO Standard Procedure on MSAs formalised the new arrangements, and MSAs were generally revised in accordance with the Standard Procedure in a timely manner. Service Chiefs informed the ANAO that the MSA reforms helped strengthen collaboration between Defence and the DMO in sustaining specialist military equipment.

18. The ANAO reviewed the structure and content of a sample of MSAs as at 2014, based on criteria developed by the ANAO in 2010 for effective cross-entity agreements.¹⁵ For the most part, the MSAs examined by the ANAO met the key characteristics of well-structured cross-entity agreements. The MSAs outline governance arrangements, respective roles and responsibilities, sustainment deliverables, performance reporting and monitoring arrangements, sustainment issues and risks, and dispute resolution procedures. These features of the MSAs serve to clarify accountabilities, coordination arrangements and relevant processes. However, there remains scope to improve KPIs—well-structured cross-entity agreements will generally include reciprocal KPIs which recognise that one entity’s ability to perform work often depends on timely action by the partner entity¹⁶, whereas MSA Product Schedules tend to include performance measures related to only one party—the DMO. More generally, the DMO, Navy and Army have developed new sustainment performance measurement frameworks which are yet to be fully implemented.

Materiel Sustainment Agreements in Operation (Chapter 3)

19. Regular interaction between Defence and DMO personnel at both the management and operational levels is essential to developing a shared understanding of expectations and reaching agreement on how to effectively manage sustainment activity. During 2012–13, the Services introduced revised arrangements for Defence–DMO management review of sustainment, incorporating periodic strategic-level reviews, six-monthly multi-product reviews, and ongoing scrutiny at the working level. For the ANZAC ship, Orion aircraft and Bushmaster vehicle fleets, product and working-level reviews were effective in focusing the attention of management on capability planning and changes, and related changes in funding.

20. MSA Product Schedules are updated as necessary to formalise changes in sustainment arrangements and funding agreed between Defence and the DMO. Army and Air Force followed the DMO Standard Procedure in delegating authority for updating Product Schedule sections to line management, and their implementation of the revised process, discussed earlier, has been relatively smooth. Following Navy’s inability to supply vessels requested by the then Government to assist in the clean-up after Cyclone Yasi in 2011¹⁷, Navy required Product Schedule updates to be approved at higher management levels, and included additional financial and maintenance detail in the documents. Navy’s management of MSAs demonstrated a relatively conservative risk appetite, reflecting its assessment of risk, and resulted in a higher number of Product Schedule changes and delays in approving them. An unintended consequence of Navy’s approach was the emergence of undocumented workarounds to overcome delays in processing Product Schedule changes. While a matter for Navy’s senior leadership, there is scope to review the change management process for Navy Product Schedules and their level of detail.

21. The MSA framework recognises the importance of risk management in sustaining specialist military equipment, and provides a structured process for the identification, assessment and management of sustainment risks by the Services and the DMO. The ANAO’s examination of Defence’s management of the ANZAC, Orion and Bushmaster fleets indicated that senior leadership was kept up-to-date about the risks to the relevant capabilities, and that risk mitigation strategies were generally in place. However, the practical effectiveness of the MSA framework largely depends on active and timely management of identified risks by Capability Managers and the DMO, and in that respect, a robust MSA framework is an aid to management, not an end in itself.¹⁸

Sustainment Funding and Cost Estimates (Chapter 4)

22. The DMO’s acquisition and sustainment activities are presented as separate programs in the Portfolio Budget Statements (PBS), and the 2006 Defence–DMO Memorandum of Arrangements documents certain constraints on the transfer of funds within the DMO between acquisition and sustainment activities. However, while the PBS suggests that a relatively clear-cut distinction exists between acquisition and sustainment activities and funding, that distinction is not as clear-cut in the Memorandum of Arrangements, and experience indicates that the distinction is not hard and fast in practice. While acknowledging that there can be a period of transition between the acquisition and sustainment phases of a capability, the ANAO noted a number of instances of overlap in the use of acquisition and sustainment funding.¹⁹ To clarify the internal treatment of acquisition and sustainment funding, Defence should review relevant business rules and guidance.

23. From the establishment of the DMO as a prescribed agency²⁰ in 2005, control of sustainment funding rested with the CEO DMO, who was able to move funding between sustainment products as needs and priorities changed. Some years ago, Capability Managers were given renewed responsibility for controlling their sustainment budgets. Under this arrangement, transfers of funding between sustainment products can occur with the agreement of the relevant Capability Manager. The number of funding transfers increased from seven in 2011–12 to 55 in 2013–14, and the total value of these transfers across financial years increased from $170 million to $1.1 billion. Capability Managers informed the ANAO that they valued the ability to flexibly use sustainment resources according to operational and maintenance needs.

24. The three case studies examined by the ANAO indicated that there have been persistent inaccuracies in Defence’s sustainment cost estimates. Moreover, actual expenditure in 2013–14 for one third of all sustainment products (39 out of 118) varied from the budget estimate by over 25 per cent. Some departure from budget estimates can be expected due to flexible use of funding between sustainment products, and unforeseen factors such as the need to delay or bring forward maintenance work due to operational demands. However, variances of over 25 per cent are significant and suggest that there remains scope for the DMO to strengthen cost estimation techniques and understanding of cost variances. Improved cost estimation would strengthen the capacity of Defence’s Capability Managers to flexibly manage sustainment funding as informed purchasers of sustainment services.

Performance and Reporting (Chapter 5)

25. The 2008 Defence Procurement and Sustainment Review (the Mortimer Review) considered that sustainment performance would not improve unless it was measured, and reported that Defence did not have appropriate, quantifiable KPIs.²¹ In 2012, the DMO developed a sustainment performance measurement framework and commenced work on a new reporting system. In 2013 and 2014 Navy and Army developed revised sustainment performance measures which will be reported through the DMO reporting system. These include a suite of Navy KPIs and Key Health Indicators that assess sustainment performance and the state of each capability.²² However, the first stage of the rollout of the DMO’s new reporting system has been delayed from November 2014 until May 2015. It is only when the new measures have been reported for some time that their usefulness will be tested, and any need for refinement can be assessed.

26. Defence’s annual reporting on sustainment includes budget and expenditure data for the Top 30 sustainment products (representing some 77 per cent of current spending on sustainment), as well as an overview of the management of these products. While providing stakeholders with a basic summary of sustainment costs and activity, this information does not facilitate assessment of Defence and the DMO’s sustainment performance in terms of materiel availability, cost-effectiveness and key inputs such as inventory management, maintenance and configuration changes. Defence still has some way to go before it meets the intent of the recommendation of the Joint Standing Committee on Foreign Affairs, Defence and Trade for enhanced public reporting. Following a request by the Joint Committee of Public Accounts and Audit (JCPAA), in February 2015 the ANAO provided the JCPAA with options, developed in consultation with Defence, to enhance sustainment reporting to the Government and Parliament. The issue remains under consideration by the JCPAA at the time of preparation of this report.

Summary of entity response

27. Defence’s covering letter in response to the proposed audit report is reproduced at Appendix 1. Defence’s summary response to the proposed audit report is set out below:

Defence thanks the Australian National Audit Office (ANAO) for conducting the performance audit: Materiel Sustainment Agreements. The audit was conducted and completed in a positive and collegiate manner, with the ANAO and Defence staff working together to analyse performance of the selected Materiel Sustainment Agreements.

Defence is committed to the review of procedures around Materiel Sustainment Agreements and the internal treatment of acquisition and sustainment funding as noted in the recommendations. After the outcomes are known from the First Principles Review, Defence will be better positioned to meet the intent of, and implement, the recommendations from the report.

Recommendations

Background

1.1 Materiel Sustainment Agreements (MSAs) have been used since 2005 as customer–supplier agreements formalising the relationship between the Department of Defence (Defence) and the Defence Materiel Organisation (DMO)²³ for the sustainment of specialist military equipment.²⁴ Through the agreements, the Defence Capability Manager²⁵ undertakes to supply funding, and the DMO undertakes the sustainment of specific platforms (such as a ship or aircraft fleet), commodities (such as clothing or combat rations) and services (such as provision of maritime target ranges). Sustainment involves the provision of in-service support, including repair and maintenance, engineering, supplies, configuration management and disposal action. Effective sustainment of ships, aircraft and vehicles is necessary to maintain the preparedness of the Australian Defence Force (ADF) and enable the conduct of Defence operations.

1.2 In 2014–15, out of total funding of $35.781 billion, Defence budgeted $7.109 billion, or 20 per cent, for its Capability Sustainment Programme, with most of this funding transferred to the DMO, which either directly or indirectly provides the majority of Defence sustainment services.²⁶ The specialist military equipment being sustained was valued at $41.270 billion in 2013–14.²⁷

1.3 The DMO budget for 2014–15 comprises two major elements: $5.931 billion for acquisition projects, and $6.185 billion for its sustainment program. Figure 1.1 shows the DMO’s acquisition and sustainment expenditure between 2005–06 and 2013–14, as well as budgeted amounts for 2014–15 and the forward estimates. In five of the last nine financial years, sustainment expenditure exceeded acquisition expenditure, and budgeted sustainment expenditure for 2014–15 is at a record high level.

Figure 1.1: DMO acquisition and sustainment expenditure, 2005–14, and budgeted amounts for 2014–18

 

 

Source: ANAO analysis of expenditure and forward estimates for DMO Programme 1.1, Management of Capability Acquisition, and DMO Programme 1.2, Management of Capability Sustainment, in Defence Annual Reports and Portfolio Additional Estimates Statements 2014–15.

Note: The hollow columns represent the February 2015 Additional Estimates figures for 2014–15 and future years.

1.4 Sustainment activities are generally administered by the DMO’s Systems Program Offices (SPOs) and in particular through the Product Manager. The SPO serves as the single point of contact with industry, government entities, and other entities participating in the acquisition or sustainment of specialist military equipment. Generally, each major platform—such as an aircraft type or a class of ship (for example, F/A-18 Hornet aircraft, or the ANZAC Class frigates)—is managed by a single SPO, which may also manage the delivery of a commodity or service. On the customer side, a Capability Manager Representative liaises with the DMO on behalf of the relevant Defence Group or Service.

Managing Defence’s sustainment function

1.5 The Australian Defence Organisation comprises the Department of Defence, the ADF and the DMO. The successful provision of Defence capability depends on the Defence Organisation as a whole collaborating effectively and the component parts meeting their respective functional responsibilities, including, for the DMO, acquisition and sustainment responsibilities.

1.6 The then Minister for Defence approved the establishment of the DMO on 22 June 2000, through the amalgamation of the Department of Defence’s Defence Acquisition Organisation, Support Command Australia and part of its National Support Division. The objective of the amalgamation was to improve the delivery of equipment, systems and related goods and services to the ADF by integrating acquisition and through-life support into a whole-of-life capability management system. The DMO came into being on 1 July 2000, and related structural and organisational changes were made by December 2000.²⁸

1.7 In September 2003, when the then Government approved the DMO’s transition to a prescribed agency under the Financial Management and Accountability Act 1997 (FMA Act), the intent was for the DMO to be more performance-oriented and business-like, so as to improve procurement and support practices, and establish a more transparent relationship between Defence and the DMO.²⁹ The DMO became a prescribed agency under the FMA Act on 1 July 2005, with a separate Chief Executive Officer, financial accounts and annual reporting requirements, but staff provided by the wider Defence Organisation.³⁰

1.8 The DMO’s February 2004 Business Model noted the primacy of Defence as the customer, with responsibility for setting requirements and determining priorities within agreed funding levels; and that the DMO would be funded and managed on the basis of agency agreements for the services it delivered. The agency agreements were to facilitate funding flows and delineate responsibilities and accountability between Defence and the DMO, and provide better visibility of the costs of procuring and sustaining Defence assets. The agreements were to be refined and improved over several annual cycles.

1.9 Agreements between Defence and the DMO are not legally binding, because both organisations are part of the same legal entity, the Commonwealth of Australia.³¹ Defence and the DMO have an overarching Memorandum of Arrangements for customer–supplier agreements that establishes the framework for their other bilateral agreements. The Memorandum of Arrangements outlines commitments that both parties agree to meet, in order to procure and sustain materiel for Defence.

1.10 Underneath the Memorandum of Arrangements, Defence and the DMO have entered into contract-like agreements for the provision of sustainment services:

Materiel Sustainment Agreements are between the Capability Managers and the Chief Executive Officer of the Defence Materiel Organisation. These agreements cover the sustainment of current capability, including goods and services such as repairs, maintenance, fuel and explosive ordnance.³²

1.11 Each Materiel Sustainment Agreement (MSA) sets out the level of performance and support required by Defence from the DMO, within an agreed price, as well as the Key Performance Indicators (KPIs) by which service delivery will be measured. MSAs comprise a Heads of Agreement and Product Schedules for different platforms, commodities and services. Since 2012, the previous practice of complete annual revision of the MSAs has been replaced by partial revision as needed, particularly of the financial sections of Product Schedules. The content of the Heads of Agreement and Product Schedules is examined in Chapter 2.

1.12 As previously mentioned, the 2014–15 budget for DMO Programme 1.2, Management of Capability Sustainment, amounts to $6.185 billion³³, of which some $5.683 billion is to be expended through seven MSAs incorporating 116 Product Schedules (Table 1.1).

Table 1.1: Materiel Sustainment Agreements and Product Schedules, as at February 2015

Source: DMO, MSA Product Budgets as at February 2015.

Note: Includes baseline funding, operations funding ($315 million) and expected Net Personnel and Operating Costs ($85 million).

Audit objective and scope

1.13 The audit objective was to examine Defence’s administration of Materiel Sustainment Agreements (MSAs) and the contribution made by MSAs to the effective sustainment of specialist military equipment.

1.14 Four high-level criteria were developed to assist in evaluating Defence’s administration of MSAs:

• MSAs constitute an effective arrangement formalising the relationship between Defence Services/Groups and the DMO for sustainment activities.
• MSA Product Schedules clearly identify costs, deliverables, and appropriate KPIs.
• Risks, issues and constraints to achieving effective sustainment are identified in MSA Product Schedules, and appropriate management strategies are in place.
• Monitoring and reporting processes provide relevant and reliable information on sustainment activities.

1.15 The ANAO examined Defence’s MSA policy, procedures and systems, including reforms in these areas. The ANAO also examined Defence’s management of three Product Schedules. One Product Schedule was selected for each of the Navy, Army and Air Force:

• Navy’s FFH ANZAC frigates (Product Schedule CN02³⁴), as one of the first MSA Product Schedules to be revised after the Rizzo Report³⁵;
• Army’s Protected Mobility Vehicle (Bushmaster) fleet (Product Schedule CA04), which has been heavily used in Iraq and Afghanistan; and
• Air Force’s AP-3C Orion fleet (Product Schedule CAF04), which was heavily involved in the search for Malaysian Airlines flight MH370 in the Indian Ocean.

1.16 Assessing the overall effectiveness of the Product Schedules and their oversight involved engagement with: DMO staff responsible for the drafting and oversight of the Product Schedules; the project teams that carry out the sustainment of the selected military equipment; and staff assisting the relevant Capability Managers.

1.17 The 2010 ANAO performance audit on Effective Cross-Agency Agreements provided better-practice principles to assist in evaluating the MSAs, including discussion of consistency and clarity, guidelines, key provisions (such as achievable performance measures), and effective monitoring and review processes.³⁶

1.18 The audit was conducted in accordance with the ANAO’s auditing standards, at a cost to the ANAO of approximately $493 000.

Structure of this Audit Report

1.19 The remainder of the Audit Report is arranged as follows:

Table 1.2: Structure of this Audit Report

Introduction

2.1 Common drivers for formalising cross-entity arrangements include the need or desire to: promote a collaborative relationship between parties and demonstrate a commitment to joint work; establish a degree of control or assurance in relation to the activities or responsibility of another party; enhance accountability, transparency and efficiency; improve knowledge; and provide better services.³⁷

2.2 In September 2008, the report of the Defence Procurement and Sustainment Review (the Mortimer Review) found that ‘the MSAs are a workable mechanism for Capability Managers and DMO to use’. The review also found that ‘the intent for DMO to become more business-like is not yet adequately reflected in a mature customer-supplier relationship between Defence and DMO’. The review concluded that the DMO needed to focus on being a business-like supplier of products and services rather than trying to accommodate all that was asked, and Capability Managers needed to become more informed customers.³⁸

2.3 In this chapter, the ANAO examines:

• the Defence–DMO agency agreements framework;
• recent reforms of the MSA framework; and
• the structure and content of MSAs.

Defence–DMO agency agreements framework

2.4 A map of the formal relationships between Defence and the DMO, as mediated through an agency agreements framework, is shown in Figure 2.1.

Figure 2.1: Defence–DMO agency agreements framework

 

 

Source: DMO, Agreements Manual (draft as at January 2013), Chapter 2, p. 5.

2.5 Under the framework, an overarching Memorandum of Arrangements provides for five types of agency agreement between Defence and the DMO:

• Materiel Acquisition Agreements (MAAs);
• Materiel Sustainment Agreements (MSAs);
• Defence Services Agreements;
• Shared Services Agreements, covering services such as payroll, accommodation and banking services provided by Defence, and contracting policy and advice provided by the DMO; and
• Military Workforce Agreements, for posting of military personnel to the DMO.³⁹

2.6 Defence and the DMO entered into the Memorandum of Arrangements for Customer–Supplier Agreements on 15 June 2005, and it was last revised on 30 June 2006.⁴⁰ Under the Memorandum of Arrangements, in relation to sustainment:

• Defence is responsible for setting clear performance requirements and priorities for products and services, and preparing government submissions for additional funding; and
• the DMO is responsible for delivering the outputs agreed with Defence in the MSAs, controlling all resources, staffing and other inputs, providing appropriate evaluation and reporting, and setting policy instructions and governance arrangements for financial management of its funding and appropriations.

2.7 The Memorandum of Arrangements specifies that MSAs shall:

• include individual schedules identifying the price and deliverables for each sustainment product;
• separately identify baseline, supplemented and operations sustainment provided by the DMO to Defence⁴¹;
• provide a budget forecast for the following 10 years; and
• be renegotiated annually in conjunction with Defence’s financial planning process, known as the Defence Management and Financial Plan (DMFP).

2.8 In May 2009, in response to the Mortimer Review, the then Government formally committed to updating the Memorandum of Arrangements to clarify the respective authorities and responsibilities of Defence and the DMO.⁴² However, this has not yet occurred, and in respect of MSAs, the Memorandum of Arrangements has become increasingly obsolete. The outdated elements include:

• the requirement for annual renegotiation of MSAs—this requirement has been overridden by a subordinate document, the 2012 DMO Standard Procedure on MSAs;
• the discretion of the CEO DMO to move funds within the acquisition and sustainment areas—this discretion was restored to Defence Capability Managers some years ago;
• a reference to the DMO Service Fee—which has not existed since
  
  2009–10, when the DMO began to receive its own appropriation for workforce and operating expenses; and
• Product Schedules—the list of Product Schedules in the Memorandum of Arrangements is nine years out-of-date.⁴³

2.9 The DMO has attempted to update the Memorandum of Arrangements on many occasions, and prepared an updated document for approval: between 2007 and 2009; in 2011 and 2012; and again in 2013. However, on each occasion, momentum was lost and the revised Memorandum was not formally approved, reflecting difficulty in reaching agreement across a large number of Defence stakeholders.

2.10 The long delay in updating the Memorandum of Arrangements—which is the capstone of Defence’s agency agreements framework—has resulted in a subordinate document, the DMO Standard Procedure on MSAs, now filling a policy gap. That document has increasingly been relied on as the Defence policy and procedure for sustainment activities.

2.11 While the Defence Organisation has shown a capacity to adapt, the DMO Standard Procedure is not mandatory policy for any of the Services, and the DMO relies on the Services’ ongoing commitment to the Standard Procedure to help bring coherence to sustainment arrangements. In contrast, an updated Memorandum of Arrangements would achieve an authoritative and ongoing basis for MSA policy and procedure across Defence Groups and Services.

2.12 Defence informed the ANAO in December 2014 that it was likely to experience organisational change flowing from internal and external reviews, including the First Principles Review⁴⁴, and as a consequence it would be imprudent to update the Memorandum of Arrangements at this time. Defence further informed the ANAO that once the changes flowing from reviews are confirmed, Defence will shape an appropriate solution at the earliest possible opportunity.

Structure and content of Materiel Sustainment Agreements (MSAs)

2.13 As indicated in Figure 2.1, MSAs sit underneath the Memorandum of Arrangements in the Defence–DMO agency agreements framework. The first level of the MSA is the Heads of Agreement, which is an overarching document that covers a series of Product Schedules. The Heads of Agreement contain the high-level framework establishing the partnership between each Capability Manager and the DMO. Heads of Agreement are developed following a common template and can be tailored to the requirements of each Capability Manager. The main features of the Heads of Agreement template are set out in the text-box on page 4.

Source: DMO, Materiel Sustainment Agreement Heads of Agreement Template, November 2012.

2.14 The second level of each MSA is the Product Schedules, a list of which is attached to the Heads of Agreement. Each Product Schedule deals with the sustainment of a specific platform, commodity or service for the relevant Defence Service or Group. In February 2015, the seven Heads of Agreement incorporated 116 Product Schedules. The number of Product Schedules for each Capability Manager is shown in Table 1.1.⁴⁵

2.15 Product Schedules are where the specific sustainment requirements for each capability are found. The Product Schedule defines: the supplies and services that will be provided by the DMO; the budget that is provided by the Capability Manager; and standards for matters such as responsiveness, availability levels, and maintenance timeframes.

2.16 In 2013, the Product Schedules represented a significant body of paperwork, amounting to some 2600 pages for Navy, 1150 pages for Army, and 850 pages for Air Force. Since they are among the top planning documents for the expenditure of over $6 billion⁴⁶, it is important to maintain a balance between statements of principle and the level of detail in these documents.

2.17 The standard structure of the Product Schedules for all the Capability Managers (except Navy) is shown in Table 2.1.

Table 2.1: Structure of non-Navy Product Schedules

Source: DMO, Management of Materiel Sustainment Agreements including Product Schedules Standard Procedure, 6 November 2012, p. 4.

1. Army uses the term Module, and Air Force uses the term Annex, for the different parts of the Product Schedule.
2. On 2 May 2009 the then Government launched both the 2009 Defence White Paper and the SRP. Defence expected the SRP to improve accountability, planning and productivity and deliver savings of $20 billion over the decade 2009–10 to 2018–19, including $5.1 billion in sustainment savings through Smart Sustainment reform.

2.18 Navy Product Schedules are structured differently and are considerably longer than those of Army and Air Force. Their structure is shown in Table 2.2.

Table 2.2: Structure of Navy Product Schedules

Source: ANAO analysis of the ANZAC fleet Product Schedule as at July 2014, which was one of the models for subsequent Navy Product Schedules.

2.19 The nature of the products being sustained under the MSA framework is such that there is not necessarily a one-to-one relationship between a product and a Defence Service or Group, and a product may have relationships with other products. For example, the Bushmaster fleet, primarily ‘belonging’ to Army, has a small component of vehicles that are used by Air Force. Similarly, while the Bushmaster fleet is primarily sustained through Army’s CA04 Product Schedule, it could not operate without the fuel and lubricants supplied under another Product Schedule.⁴⁷

2.20 In 2011, the three Services signed a Memorandum of Understanding (MoU) to provide clear guidance on Capability Manager roles and responsibilities where there are multiple users of a product. In effect, the Capability Manager who is the major user of a capability takes on the Lead Capability Manager role for the Product Schedule, and coordinates other Service or Group requirements.⁴⁸

ANAO assessment of MSA quality

2.21 Key success factors for cross-entity arrangements include: clear roles, responsibilities and governance arrangements; a shared objective or outcome; clear funding arrangements; management of shared risks; and coordinated reporting and evaluation, with a clear focus on the shared objective as well as entity contributions.⁴⁹

2.22 Table 2.3 considers the extent to which the three Services’ MSAs (comprising both the Heads of Agreement and Product Schedules) have regard to better practice in agreement-making.

Table 2.3: ANAO assessment of MSA quality

Source: ANAO analysis of a random sample of six Product Schedules from each Service, based on criteria in ANAO Audit Report No.41 2009–10, Effective Cross-Agency Agreements, p. 54.

2.23 For the most part, the MSAs examined met the key characteristics of cross-entity agreements. The MSAs outline governance arrangements, respective roles and responsibilities, sustainment deliverables, performance reporting and monitoring arrangements, sustainment issues and risks, and dispute resolution procedures. These features of the MSAs serve to clarify accountabilities, coordination arrangements and relevant processes.

2.24 Some features of existing MSAs indicate scope for improvement in the MSAs. For example:

• Each of the MSAs refers to the Memorandum of Arrangements as the overarching document that authorises the MSAs. However, as discussed in the early part of this chapter, the Memorandum of Arrangements is increasingly out-of-date, and in practice it has been bypassed by the DMO Standard Procedure on MSAs.
• The establishment of shared objectives or outcomes as part of a cross-entity agreement assists in furthering individual entity outcomes, while focusing each entity on the overall intent and expected outcomes of the cross-entity initiative. The Navy MSA outlines its purpose and objectives, and supporting principles, whereas the other Services’ MSAs do not.⁵⁰
• While a bilateral agreement will generally include reciprocal KPIs which recognise that one entity’s ability to perform work often depends on timely action by the partner entity⁵¹, there is a tendency for MSA Product Schedules to include performance measures related to only one party—the DMO. More generally, the DMO, Navy and Army have developed new sustainment performance measurement frameworks which are yet to be fully implemented.
• The Army and Air Force MSAs have workable change mechanisms that allow them to be kept reasonably current. Navy has experienced a greater administrative burden in establishing change mechanisms and keeping the documents current. This is a result of both the higher level of detail present in the Navy MSA (resulting in more frequent changes), and Navy’s requirement for changes to be approved at higher levels than required by Air Force and Army (resulting in slower changes).

Reform of the MSA framework

2.25 The MSA framework was introduced in 2005 and has continued to evolve. A 2012–13 Lean Project ⁵² represented a significant turning-point in the design and administration of the MSA framework.

2012–13 Lean Project

2.26 In February 2012, the DMO informed the Services that it had initiated an MSA Lean Project ‘to improve the MSA and MSA change proposal (MSACP) processes to make them straight-forward, easy to use, faster and more flexible.’ By June 2012, the DMO had conducted the initial, data-gathering phase of the Lean Project, and reported that:

Overall, stakeholders reported that the MSA process was long and tedious with little apparent value-add. The process currently has too much emphasis on the approval process and too little on planning and performance management. The focus needs to be on delivering outcomes which are achievable within the funding available in order to obtain the best outcome for the money available. The initial workshop findings included:

• annual cycle to re-sign MSAs is not required and not supported;
• annual cycle is not adding value to the content of the MSAs;
• working level^[53] stakeholders require more clarity of Capability Manager requirements earlier in the development process;
• roles and responsibilities in the process are not clear;
• ownership of the document is not clear;
• current content and KPIs are not used for product management at any level;
• loss of visibility of the MSAs after leaving the working level during the endorsement and signature process engenders lack of ownership of the final document; and
• reporting on products is already a large burden.

2.27 In July 2012, the DMO organised five days of workshops to develop a streamlined MSA process with the intent to release capacity to focus on sustainment planning, management and performance. The DMO described the overall outcome to Capability Managers as follows:

We all agree that the current MSA process is too cumbersome, bureaucratic and inflexible. The good news is that the team conducting the Lean activity believe that it is possible to replace the annual cycle of MSA development stretched over months with a change process of less than five days. To achieve this level of reform will require us, as senior leaders, to consistently back this initiative at all levels of our organisations.⁵⁴

2.28 The workshops were the culmination of several years of attempts to develop an effective and uniform MSA policy and practice. Representatives of Defence Services and Groups reached a consensus on key reform initiatives, including:

• the Product Schedule being capability focused;
• all sections of the MSA and Product Schedule being enduring, with built-in performance review periods;
• a modularised Product Schedule template⁵⁵;
• introducing Product Schedule delegations to enable greater delegation aligned to line management accountability; and
• simplified/accelerated workflows for changing/updating/introducing Product Schedule sections under the proposed delegation framework.

2.29 The broad implementation parameters from the workshop also included the need to redesign the MSA framework to focus on sustainment outcomes and information required by the O5/O6 level, and to make the Product Schedules more concise and focused on the changeable aspects under management.

The DMO issued a Standard Procedure on MSAs in 2012

2.30 In November 2012, following on from the Lean Project workshops, the DMO issued the Defence Materiel Standard Procedure entitled Management of Materiel Sustainment Agreements including Product Schedules Standard Procedure.⁵⁶ The Standard Procedure addressed the key deliverables of the July 2012 workshops to develop a streamlined MSA framework. It sets out, essentially in four pages, the elements of the new approach to the MSA framework, and includes process flows for implementing scheduled financial decisions as well as other decisions.

2.31 The DMO asked Capability Managers to endorse the Standard Procedure as the ‘interim Defence and DMO policy’ for the management of MSAs, given the wide consultation involved in its development. This was seen as an expedient way of achieving Defence-wide commitment to the proposed MSA arrangements. Capability Managers endorsed the Standard Procedure before it was formally issued. The Standard Procedure included some exceptions from its requirements for Navy.

2.32 As a DMO policy, the Standard Procedure has no authority to direct the Services or other Defence Groups. It therefore explicitly states that it has been created ‘as an interim policy until a Defence Instruction (General) is considered by key stakeholders and released in early 2013.’⁵⁷ However, in June 2013, Defence’s System of Defence Instructions (SODI) administrators advised the DMO that the creation of a Defence Instruction (General) for MSA policy was not appropriate because MSAs are non-legally binding, and they do not pertain to everyone in Defence.

2.33 By August 2014, it was agreed that the Standard Procedure would be updated, and Capability Manager Representatives would continue to acknowledge the Standard Procedure as the agreed MSA protocol between the DMO, Defence Groups and the Services. Further, the DMO would investigate the inclusion of an MSA chapter in a sustainment manual being written by its Standardisation Office.

Implementation of the new MSA framework

2.34 Work on implementing the new MSA framework continued until the formal closure of the Lean Project in April 2014. The biggest task involved the development of enduring Heads of Agreement and Product Schedules based on new templates. The revised documents and related governance and monitoring arrangements reflect an intention to eliminate rework and excessive review, and focus efforts on the higher-value areas of sustainment planning, management and performance. The transition of different MSAs and Product Schedules to the new model is shown in Table 2.4.

Table 2.4: Transition of MSAs to the new model, 2013–14

Source: ANAO analysis of Defence records.

Notes: Taking a different approach, Navy had signed a new enduring Heads of Agreement on 9 August 2012, and completed its Product Schedule transition by 30 June 2013. Navy signed a revised Heads of Agreement on 20 January 2015.

 In a further round of revision, the Chief Information Officer Group and Army signed new Heads of Agreement with the DMO on 27 August 2014 and 3 September 2014 respectively.

As a result of the Rizzo Report, Navy took a different approach

2.35 While Navy representatives were involved in the DMO’s Lean Project, a different imperative drove the review and reform of the Navy–DMO MSA from 2011 onwards. In February 2011, Navy was unable to supply vessels requested by the then Government to assist in the clean-up after Cyclone Yasi. This was quickly followed by the early decommissioning of HMAS Manoora, the extended unavailability of HMAS Kanimbla and the temporary unavailability of HMAS Tobruk. These events resulted in the commissioning of the Rizzo Report.⁵⁸

2.36 In July 2011, the Rizzo Report found that the events mentioned above were ‘reflective of on‐going systemic failure’, and made several observations pertaining to the then Navy MSA and its Product Schedules, namely that:

• the MSA was critical for accountability, but was ‘currently poorly defined and weak’;
• the KPIs in the Product Schedules were inadequate, with no consequences for non-compliance; and
• the MSA should be used by the DMO to clearly define the obligations of Navy.⁵⁹

2.37 The Rizzo Report made two recommendations (out of 24) that directly related to MSAs:

Recommendation 11. Capture Mutual Obligations: The Navy MSA should be transformed into an active ‘contract’ that meaningfully captures the mutual obligations of Navy and DMO, supported by business-like performance measures.

Recommendation 12. More Effective Information Exchange: Navy and DMO must improve their internal reporting by capturing direct, timely and candid, document-based information that draws on a rigorous set of metrics.⁶⁰

2.38 Navy took the Rizzo recommendation for a more contract-like MSA to mean one that contained: a customer–supplier arrangement; clearly defined responsibilities, obligations, performance measures and deliverables; delegations of authorities and responsibilities; and appropriate management and reporting arrangements.

2.39 The Chief of Navy signed a new MSA with the DMO on 9 August 2012.⁶¹ This was three months before the DMO issued its Standard Procedure and new templates, in November 2012. The new MSA included a restructure of the 36 Navy Product Schedules, which were rewritten in the post-Rizzo format and approved progressively by 30 June 2013 as part of the Rizzo Reform Program.

2.40 In response to Recommendations 11 and 12 of the Rizzo Report, the Navy–DMO Heads of Agreement includes a section on the core obligations of Navy and the DMO, and requires a traffic-light system for monthly reporting in relation to both availability and price. Further, Navy’s new Product Schedules include a Statement of Work and a detailed breakdown of funding into line-items. The funding line-items are intended to give Navy visibility of where funding, below product level, is being spent.

2.41 The Standard Procedure on MSAs only requires details of overall baseline funding in Product Schedules; whereas Navy included 10 funding line-items in its post-Rizzo Product Schedules.⁶² Navy’s line-items were also not aligned with Defence’s corporate budgeting system, BORIS. These costs could therefore only be adjusted and reported through manual manipulation of data in spreadsheets.

2.42 In December 2014, Defence informed the ANAO that Navy and the DMO had agreed on a level of financial detail—for inclusion in Product Schedules—that could be budgeted and reported through BORIS. However, implementation of the solution had not yet occurred.

2.43 Another aspect of the Navy Product Schedules that differs from the DMO template is the inclusion of additional availability and maintenance information. Navy has responded to the non-availability of ships and the Rizzo Report by adopting a detailed approach:

• a Product Activity Plan details the number of Materiel Ready Days required for the next year, as well as expected dates of maintenance for the next three years; and
• the schedule for external (contractor) maintenance of each ship (where relevant) is set out for the following five years rather than just for the current financial year, as was previously the case.⁶³

2.44 The revised Navy MSA includes additional detail on mutual obligations, expenditure and maintenance scheduling, following significant failures in Navy which led to the unavailability of supply vessels. This approach reflects Navy’s relatively conservative risk appetite, and is intended to enable close oversight of the DMO’s management of sustainment activity. The design features of the Navy Product Schedules have led to very lengthy documents in comparison to other Services’ Product Schedules. For example, in 2013, the average Army Product Schedule was 28 pages, the average Air Force Product Schedule was 33 pages, and the average Navy Product Schedule was 70 pages.

Conclusion

2.45 For the most part, the MSAs examined by the ANAO met key characteristics of well-structured cross-entity agreements. The MSAs outline governance arrangements, respective roles and responsibilities, sustainment deliverables, performance reporting and monitoring arrangements, sustainment issues and risks, and dispute resolution procedures. These features of the MSAs serve to clarify accountabilities, coordination arrangements and relevant processes. The MSA framework has evolved over time, in light of practical experience and the risk appetite of the parties to individual agreements, and there is an ongoing role for Defence senior leadership to shape the direction of the framework so as to realise its full potential.

Introduction

3.1 Working across organisational boundaries presents many challenges, including harmonising different strategies and business processes to achieve the intended outcomes for government. The DMO–Defence customer–supplier relationship requires effective collaboration at both senior executive and operational levels, supported by efficient processes for MSA oversight, change management, and issues and risk management.

3.2 In this chapter, the ANAO examines management reviews of MSAs, the change management process for Product Schedules, and the management of sustainment issues and risks. The ANAO’s analysis draws on the three selected MSA Product Schedule case studies: ANZAC ships, Bushmaster Protected Mobility Vehicles and Orion aircraft.

Management reviews of MSAs

3.3 Although a system of MSA management reviews existed prior to 2012, the reviews were sometimes intermittent. When the Heads of Agreement template was released with the Standard Procedure on MSAs in November 2012, it included an indicative MSA review system. This consisted of three tiers of review: Strategic MSA Reviews held at least every two years at CEO DMO level; twice-yearly Strategic Product Schedule Reviews at DMO Division Head level; and monthly Product Schedule Meetings at working level.

3.4 During 2012–13, in renewing their MSAs, each of the Services also renewed the number and type of regular MSA reviews. Table 3.1 shows the tiers of MSA review that have been established by each of the Services.

Table 3.1: Tiers of MSA review

Source: ANAO analysis.

Periodic reviews—‘Strategic Review’/‘Deep Dive’

Navy

3.5 The 2012 Navy–DMO Heads of Agreement established an Annual Strategic Review between the Chief of Navy and the CEO DMO to consider current and future high-level management issues and the overall performance of Navy sustainment. These Reviews had been proposed as part of the implementation of Rizzo Recommendations 11 and 12.⁶⁴ However, to date, no meetings have taken place, and in January 2015 the frequency was changed to as-required.

Army

3.6 Army’s top level of review consists of ‘deep dives’ into Product Schedules (rather than strategic reviews), and these are formally known as Deputy Chief of Army (DCA) Product Schedule Reviews. The frequency of these deep dives varies with the level of risk identified for individual Product Schedules. Those deemed to present the highest risk are reviewed annually, with two and three-yearly reviews for those Product Schedules deemed to present medium and low levels of risk, respectively. These deep dives involve two to four hours of consideration of a single Product Schedule, and are intended to confirm the DMO’s management of Army materiel against the fleet management plan, Capability Manager priorities, budget constraints and the Product Schedule. The Product Schedule Reviews are empowered to approve funding priorities and transfers, and give in-principle agreement to inter-Service funds transfer for sustainment support. They can also authorise strategies to address particular sustainment issues.

Air Force

3.7 The first Air Force Strategic MSA Review occurred in June 2011, at Air Force’s initiative, with a view to establishing an annual forum of DMO and Air Force senior executives to discuss the broader strategic issues affecting the materiel sustainment of Air Force capability. At the time, Air Force was keen to use the MSA construct as a partnership rather than a customer–supplier relationship, acknowledging that it was only in recent times that Air Force had started to be involved at the appropriate depth of detailed understanding and engagement in the MSA and Product Schedule processes. During the June 2011 review, Air Force noted that its relationship with DMO was very good and continuing to mature, yet there were still opportunities for improvements, such as introducing a greater capability focus into Product Schedules, capability-driven KPIs and increased consistency in Air Force dealings across all DMO Divisions and SPOs. For its part, the DMO highlighted that in the past the MSAs had been ‘one way’, and that there was merit in pursuing a move to ‘two way’ arrangements and KPIs.

3.8 The 2013 Air Force–DMO Heads of Agreement provided for a Strategic MSA Review to be conducted at least once every two years, ‘to review the conduct of the MSA and the materiel sustainment relationship between the Air Force and the DMO’. No Strategic MSA Review has been held since 2011.

Six-monthly reviews—‘Fleet Screenings’

Navy

3.9 Navy conducts two levels of fleet screenings: Biannual MSA Reviews; and Force Element Biannual Progress Reviews, which consider the Product Schedules by Force Element Group.

3.10 The Biannual MSA Reviews form the most significant part of Navy’s MSA review system. These reviews consider the performance of all of the Navy Product Schedules for each DMO Division, and provide an opportunity for the Deputy Chief of Navy to manage sustainment funding between different Product Schedules, taking into account changing circumstances and operational needs. The reviews occur in February/March and September/October each year, to inform the development of the Commonwealth Budget. The reviews have generally been two-day events, and were well described in a recent Navy administrative instruction:

Cognisant of the findings of the Rizzo Report, the focus of the reviews has evolved beyond their original financial emphasis into a forum for DCN [Deputy Chief of Navy] and the relevant DMO Division/Group Head to consider how their respective organisations are meeting their obligations for whole of life management and sustainment of Navy capability. Financial planning and performance remains a key element of the reviews, and the composite view of Navy sustainment pressures provided by the biannual review activity enables DCN to make informed capability and resource allocation decisions.

3.11 The ANAO attended the October 2014 Maritime Systems Division fleet screening as an observer. The meeting demonstrated an open and collaborative relationship between Navy and the DMO, with attention given to the key issues affecting both parties.

Army

3.12 Army instituted its current system of biannual Product Schedule Screens in July 2012. The meetings provide an opportunity for Army to examine the current financial health of specific fleets; plan for the forward estimates; examine fleet and business management issues; move funding between products and assign priorities; and identify products to undergo a Product Schedule Review. The Product Schedule Screen is intended to occur over three days, and is chaired by the Deputy Chief of Army, who can adjust KPIs, approve funding priorities, and authorise management action plans to remediate issues.

Air Force

3.13 Air Force has the most intricate system of fleet screenings. Air Force’s Standing Instruction on MSAs notes that there are two review periods within each financial year: for the Defence Management and Financial Plan (DMFP) and the Mid-Year Review. The DMFP review period, between February and June each year, involves a strategic review of all materiel sustainment requirements to support Air Force capability. During the Mid-Year Review period, between September and December each year, Air Force and the DMO analyse the in-year sustainment and performance of all Air Force Product Schedules, and agree on the sustainment plan for the following financial year.

3.14 In each of the review periods, the same sequence of reviews occurs, namely Sustainment Assessment Reviews, Capability Sustainment Reviews, and Principals Meetings.⁶⁵ Sustainment Assessment Reviews are led by the relevant DMO SPO and chaired by the DMO Branch Head. These reviews examine sustainment requirements, risks, issues, and the cost of achieving Air Force’s capability requirements. The reviews develop Management Options related to these areas, to feed into Capability Sustainment Reviews conducted by Force Element Groups. The Capability Sustainment Reviews analyse funded and unfunded sustainment requirements, risks and issues, and select specific Management Options for higher-level approval.

3.15 After these two reviews, an Air Force Capability Sustainment Plan is developed, and is endorsed by the Air Command Board in the lead-up to the Principals Meeting.⁶⁶ In this way, Air Force is able to provide a

whole-of-sustainment proposal to the DMO from a capability-based perspective.

3.16 The Principals Meetings form the top tier of the six-monthly reviews. The DMFP Principals Meeting is chaired by the Deputy Chief of Air Force, and the Mid-Year Review Principals Meeting is chaired by the Air Commander Australia. The meetings consider and approve the Capability Sustainment Plan, omnibus Product Schedule Change Proposals⁶⁷ for financial adjustments, and Change Proposals for any non-financial adjustments to a Product Schedule.

Ongoing review—Working Level Meetings

Navy

3.17 Navy’s working level meetings are termed Operational Sustainment Management Meetings. These meetings occur monthly, and include a comprehensive overview of the current status of the relevant Navy Product. The chief participants are the SPO Director and the Capability Manager Representative.

Army

3.18 The Army–DMO Heads of Agreement provides for working groups to be formed to address identified issues and engage with stakeholders, with the findings to be referred to a suitable forum for decision.⁶⁸

3.19 In 2013, the DMO’s Head Land Systems Division initiated a new divisional level of review, referred to as Project and Product Review Boards. These Review Boards are intended to provide an opportunity for project and sustainment managers in various branches of the Division to inform the Head Land Systems about issues, and for him to provide direct input where necessary.

3.20 In August 2014, the DMO’s Land Systems Division informed the ANAO that:

There is a very strong (and regular) interaction between Land Systems Division and the Capability Manager that has created a much better shared understanding as to where each Product is at. This, linked with the DCA [Deputy Chief of Army] Reviews and new Head Land Systems Product Reviews, allows the twice-yearly screens to adopt a more strategic overview approach that focuses on the ‘big issues’. The layered approach, with multiple reviews and regular interaction, has allowed the six-monthly Product Screens to focus (as they should) on strategic 2-star issues: everything else gets picked up before these screens.

Air Force

3.21 The Air Force–DMO Heads of Agreement provides for Product Schedule Performance Meetings, to be conducted at least monthly, and to focus on the current and forecast performance of individual Product Schedules. The meetings are attended by SPO Directors and the relevant Air Force unit commanders and their teams.

Gate Reviews

3.22 In a significant step in the management and review of MSAs, Defence commenced Sustainment Gate Reviews⁶⁹ in early 2015 for Product Schedules in Materiel Sustainment Categories I–III.⁷⁰ Sustainment products will be subject to a Sustainment Gate Review once every three to five years in line with a major event (such as a re-contracting).⁷¹

3.23 During 2014, five Prototype Sustainment Gate Reviews were held, including one for the Bushmaster fleet in October 2014. The DMO advised the Gate Review Board that:

A simplified approach to reporting against this Product Schedule has been adopted by both DMO and Army, and refinement of the Product Schedule to rationalise performance measures and to address unfunded sustainment activities over the Forward Estimates and DMFP periods appears appropriate.

3.24 The ANAO has previously found that the DMO’s Gate Reviews for acquisition projects have the potential to improve the management of its projects.⁷² The reviews can also bring a valuable independent perspective to consideration of key issues and risks through the use of external members of Gate Review Boards.

Conclusion—MSA reviews

3.25 Overall, Defence has established a structured process for the review of each Service’s MSA and their Product Schedules, incorporating periodic strategic review, six-monthly reviews and ongoing scrutiny at the working level. However, strategic-level reviews of the MSA framework and sustainment performance, involving senior leadership, have not occurred as intended. On the other hand, the introduction of DMO Gate Reviews from early 2015 has the potential to bring a valuable independent perspective to the consideration of sustainment issues and risks.

Examples of recent Product Schedule reviews

3.26 This section examines the operation of the tiers of management review for three case studies: ANZAC ships, Bushmaster Protected Mobility Vehicles, and Orion aircraft. The analysis sheds light on whether the reviews provide a structured approach supporting the management of key sustainment issues.

ANZAC fleet

3.27 The most recent six-monthly fleet screening of Navy’s major platforms occurred in October 2014. In relation to the ANZAC fleet, the DMO was urgently seeking $2 million for a ship maintenance window of opportunity that was only three weeks away. This funding was necessary to keep the vessel seaworthy until its Anti-Ship Missile Defence (ASMD) upgrade, which had been postponed. As an emerging maintenance issue that required additional funding from Navy, this issue could not be dealt with at SPO/Capability Manager Representative level, and consideration of it was therefore elevated to the fleet screening.

3.28 The DMO made several bids for additional funding for the ANZAC fleet at the October 2014 fleet screening, amounting to $27.25 million, for items such as inventory, communications, ASMD, work health and safety and workforce augmentation. Further, the briefing material for the fleet screening gave detailed coverage of planned sustainment activities, risk management activities, achievement against KPIs and budgeting for the ANZAC fleet. Discussion took place during the fleet screening about delayed acquisition projects and their impact on ANZAC fleet sustainment more generally, with the DMO advising Navy that delays in three specific capability acquisition projects had led to DMO bids for $70.2 million in extra funding to maintain in-service capabilities until the new capabilities entered service.⁷³

3.29 The October 2014 fleet screening, a result of the MSA framework, in this case provided a useful vehicle for bringing capability and funding issues to the attention of senior Navy and DMO management and enabling discussion of them.

Figure 3.1: The eighth ANZAC frigate, HMAS

 

 

Source: Australian Defence Image Library.

3.30 In another type of review, nine Force Element Quarterly Reviews have been held since September 2011.⁷⁴ In relation to the ANZAC fleet, the June 2014 review discussed the need for a realistic schedule for the ASMD upgrade. The review also noted that movement of maintenance schedules had led to commercial pressures in the past, but maintenance staffing was currently high as a result of the attention paid to scheduling.

3.31 At the working level, by mid-2014, 28 Operational Sustainment Management Meetings had occurred between the ANZAC SPO and the relevant Capability Manager Representative, namely the Navy’s Frigate Group Capability Manager. Each meeting received a detailed report on ANZAC fleet sustainment activities, including performance against KPIs, engineering certification, finance, Product Schedule Change Proposals, risk management, and workplace health and safety. The regular management meetings and detailed reporting on sustainment activities support an effective Navy–DMO relationship at the working level.

Bushmaster fleet

3.32 Product Schedule Reviews of the Bushmaster fleet have occurred since 2012 at two-yearly intervals.⁷⁵ They provide a comprehensive overview of the status of the sustained fleet, including financial data, performance against the KPIs, and consideration of specific issues.

3.33 The most recent Product Schedule Review occurred in June 2014 and generated four action items. The action items considered: the number of additional capabilities being integrated into the Bushmasters; the extra sustainment funding that would be required due to an increased fleet size as a result of new acquisitions; the commencement of Mid-Life Review planning; and the introduction of equipment acquired for operational purposes into the Bushmaster baseline configuration.

3.34 Defence records show that Product Schedule Screens covering the Bushmaster fleet Product Schedule (among others) have occurred every six months since 2012, and have examined detailed financial performance; fleet management activities and capability upgrades; KPI performance; and the health of the sustainment system.

3.35 Further, an Integrated Project Team, with representatives from Army, the DMO and the principal sustainment contractor, meets monthly to discuss the management of the capability.

Orion fleet

3.36 The review process during the Mid-Year Review period in late 2013 provides a good overview of how Air Force and the DMO negotiated their sustainment plans for 2013–14 and 2014–15. In October 2013, the Sustainment Assessment Review noted that while Orion aircraft serviceability had not met the KPI target, capability had not been impacted and Air Force had met its tasking obligations. The increased Rate of Effort (flying hours) required by government for Operation Resolute⁷⁶ was being met within the existing funding allocation, which meant that no increase in funding was required for the operation. The Sustainment Assessment Review identified a requirement for an equity injection into the Orion fleet Product Schedule to take account of the transfer of responsibility for multi-platform avionics from another Product Schedule.

3.37 The Capability Sustainment Review conducted a few weeks later by Air Force’s Surveillance and Response Group considered the Sustainment Assessment Reviews of six Product Schedules. It endorsed the injection of funds for avionics into the Orion fleet Product Schedule, as well as the return of $4.3 million from Orion engine sustainment to Air Force due to the retirement of the C-130H fleet (which used the same engine).

3.38 A Capability Sustainment Plan was then drawn up by Air Command Board, covering all of Air Force’s Product Schedules and effecting a $20 million cut in Air Force sustainment costs during 2013–14. The plan noted that core support functions for each product remained as planned and ‘opportunities to delay activities, or their payment, into FY 2014–15 are the chosen mechanism to achieve an acceptable financial risk level in FY 2013–14’. A two-hour Principals Meeting in December 2013, involving the Air Commander Australia, Deputy Chief of Air Force and Head Aerospace Systems Division, endorsed the Capability Sustainment Plan.

3.39 A significant change had taken place for the Orion fleet by the next review period, April–June 2014, with the approval of an accelerated drawdown plan for most of the Orion fleet (changed from 2020 to 2019), changes to the asset life of major ground systems⁷⁷, and a requirement to transfer $316 million from the Orion fleet Product Schedule budget so as to support the new Poseidon fleet.⁷⁸

3.40 The June 2014 Capability Sustainment Plan noted that Air Force was significantly better placed to meet needs in 2014–15 and 2015–16, but also stated that a balanced sustainment budget could not be achieved in the following years, with the net shortfall exceeding $50 million in 2016–17, $70 million in 2017–18 and $75 million in 2018–19. For the Orion fleet Product Schedule, the plan noted that savings from a new contract for engine sustainment would be used to fund an extension to the life of the last Orion aircraft in the fleet.

Conclusion—MSA reviews for case studies

3.41 For the ANZAC ship, Orion aircraft and Bushmaster vehicle fleets, product and working-level reviews were effective in focusing the attention of management on capability planning and changes, and related changes in funding.

Change management

3.42 Product Schedules are a key component of the MSA framework because they constitute the core plans for sustainment of particular platforms, commodities and services. This section of the audit report discusses arrangements for revising MSA Product Schedules, firstly in terms of delegations authorising particular office-holders to effect a revision, and secondly in terms of the Change Proposal process set up by the DMO Standard Procedure on MSAs.

Delegations

3.43 In 2008, the Mortimer Review found that the process for revising MSAs was unnecessarily time-consuming, requiring Division Head (2-Star) approval for any changes to the agreements. The Mortimer Review observed that:

such a high level of control is at odds with commercial practice. In the private sector, authority and day-to-day management responsibility tend to be much more closely aligned. This should become the norm in DMO.⁷⁹

3.44 The Mortimer Review recommended that:

Systems Program Office Directors should be empowered through greater delegation to deliver the performance levels set in Materiel Sustainment Agreements and, where necessary, to negotiate changes with Defence.⁸⁰

3.45 One of the objectives of the DMO’s 2012 Lean Project was to improve the MSA Change Proposal process through appropriate delegations for updating Product Schedules. As part of the Lean Project, a delegation framework and revised Change Proposal process were agreed by stakeholders and incorporated into the Standard Procedure on MSAs. The Standard Procedure aptly describes the goals and limitations of the delegation framework:

Core to the success of a manageable Product Schedule is the balance between control and the delegation of authority and responsibility. The objective of the delegation of authority is to provide flexibility, speed of response within the bounds of the risk appetite and level of visibility sought by senior management.

[…]

It is accepted that [Capability Managers] and the DMO may operate differing authorities and levels of delegations as suits their respective organisations but both levels of authority must be clear to both organisations and the responsiveness must be achieved.

3.46 In following the Standard Procedure on MSAs, Army and Air Force now have almost identical delegation settings for changes to Product Schedules. The 2013 Army and Air Force Heads of Agreement contained a table of delegations by rank⁸¹, and each Army and Air Force Product Schedule includes a similar table detailing the specific office holder who has the delegation for approving different types of change for that Product Schedule. The types of change being delegated are expressed in generic rather than dollar terms—such as establishing a new Product Schedule, prioritising unfunded items, or incorporating the agreed outcomes of higher-level reviews. The delegations range from the Senior Executive Service Band 2/2-Star level down to the SPO Director/Capability Manager Representative level, with material changes dealt with at the Band

1/1-Star level and above. Minor changes that update financial arrangements, roles and responsibilities, and risks/issues/constraints can be effected at the working level.

3.47 The delegations for amending Army and Air Force Product Schedules reflect the wider body of applicable Defence delegations, such as the expenditure delegation of up to $5 million to persons at SPO Director or Capability Manager Representative level, or the DMO’s $20 million procurement delegation to SPO Directors.

3.48 Navy, by contrast, requires changes to be approved at higher levels under the Navy–DMO MSA. This approach reflects Navy’s relatively conservative risk appetite following the events of 2011, when Navy ships were unavailable for a requested deployment, and the subsequent Rizzo Report. The biggest difference in Navy’s approach compared to Army and Air Force is in the area of financial changes. For the ANZAC fleet, the SPO Director and Capability Manager Representative have been delegated authority to approve budget changes of up to $1 million. Changes between $1 million and $5 million require 1-Star approval, and changes over $5 million require 2-Star approval. This stands in contrast to the more general delegation levels in Defence outlined in paragraph 3.47.

3.49 As a result of having a higher level of delegation and including much more financial detail (line-items) in the Product Schedule, Navy has a greater administrative burden in managing its Product Schedules. Further, Navy Product Schedules are being used to control expenditure at a much lower level than is the case with other Capability Managers.

3.50 Since early 2013, the ANZAC SPO has made two attempts to have the level of financial delegation for the SPO Director and the Capability Manager Representative raised to $3 million, and to clarify the delegation for maintenance schedule changes. In January 2014, an email from ANZAC SPO advised Navy that:

The financial delegations impose an administrative burden on 1-Stars and above for financial changes. Presently any value between $1m and $5m requires 1-Star approval and above $5m 2-Star approval. [ANZAC fleet] budget is $250m this FY. $1m represents 0.4% of the entire budget. In terms of risk management, clearly there is no appetite for risk, and administratively this imposes serious constraints on business. The SPO Director and Capability Manager [Representative] should have more flexibility in their delegations to better manage the business. […] The constraints placed on the management and administration of Product Schedules [are] making the entire process burdensome and not what Mr Rizzo intended.

3.51 In July 2014, a two-week maintenance schedule change was approved by the ANZAC SPO Director and Capability Manager Representative, even though they did not have a clear delegation to make the change.

3.52 In mid-2014, Navy reviewed its MSA delegation framework and proposed to authorise 1-Star Force Commanders to make Product Schedule changes that would not affect the capability output delivered to government, to a value of $5 million. A further review of delegations for changes that could affect the capability output delivered to government was initiated at the Navy’s October 2014 fleet screening, with a view to establishing a graduated delegation framework within the Product Schedule. The review of Navy’s MSA delegations framework is expected to be completed in 2015. Navy informed the ANAO that:

There is disagreement regarding the interpretation of the [ANZAC fleet] Section 6 delegations table and the correct level of delegate approval for changes such as [scheduled maintenance periods]. The [Capability Manager Representative and SPO Director] consider they are entitled to approve such changes on the basis that they are minor changes to the product activity program whereas [Navy Strategic Command] has taken the view that changes that impact the number of [Materiel Ready Days] to be delivered under the CN02 ‘contract’ are effectively changing a key performance measure between Navy and DMO and the changes therefore require higher delegate approval/acknowledgment. This is an important issue given [Materiel Ready Days] underpin Unit Availability Days (UADs) that are reported to Government. […] Navy has agreed to review current delegation arrangements in this area to ensure there is an appropriate balance between the need to respond to maintenance change requirements at the waterfront in a timely manner and the need to ensure appropriate oversight of key performance measures at the corporate level which underpin Navy’s obligations to Government.

3.53 Mortimer recommended that responsibility be devolved to the extent possible, and this has generally been implemented by Defence. However, Navy’s risk appetite is lower as a consequence of the non-availability of Navy ships in early 2011 and the subsequent Rizzo Report. There continues to be an active debate within Defence around the appropriate balance to be struck in the delegation structure to streamline administration while maintaining appropriate senior leadership oversight by Navy.

Product Schedule Change Proposals

3.54 Product Schedules require amendment whenever acquisition projects, maintenance issues or operational activities, for example, affect funding requirements or other elements of the Product Schedule. Before 2012, 40 manual steps were involved in reviewing a Product Schedule and implementing a Change Proposal. As a result, it took an average of 3–4 months for Change Proposals to be finalised, with the result that some decisions were not communicated to stakeholders and documents were lost.

3.55 A key goal of the DMO’s Lean Project in 2012 was to make the process for revising MSAs ‘straight-forward, easy to use, faster and more flexible’. One of the key deliverables of the Lean workshops was a consensus on ‘simplified/accelerated workflows for changing/updating/introducing Product Schedule sections under the proposed delegation framework’. The Standard Procedure on MSAs outlines the process and its benefits:

Changes to the [Product Schedule] may be proposed by any participant at any time and must use the agreed [Product Schedule] Change process and Product Schedule Change Proposal (PdS-CP) form … Navy [Product Schedule] changes are to be actioned using the PdS-CP. However, the detailed process employed to achieve the change will be determined on a case by case basis. Changes only take effect from signature of both [Capability Manager] and DMO delegates. Amendments to the [Product Schedule] may be initiated as a result of a scheduled or unscheduled review or as a result of the financial / budget cycle. Each amendment proposed is to be incorporated into a new draft of the area affected by the change (i.e. [Product Schedule] Annex). Upon approval of the amendment the newly drafted [Product Schedule] annex is to be inserted into the master version of the [Product Schedule]. This process ensures that a single, up to date version of each [Product Schedule] is available for use by all stakeholders.

3.56 Table 3.2 shows the number and type of Change Proposals for the audit case study Product Schedules since they transitioned to the new MSA framework, as well as the net financial impact of all budget changes for a 10year period. The net financial impact is effected by Product Schedule Change Proposals, releases of funding for new capabilities, and Budget adjustments. The proportion of Army and Air Force Change Proposals approved at the 2Star level and above is a result of these Services’ preparing an omnibus Change Proposal to reflect all decisions at six-monthly reviews regardless of lower-level authority to update Product Schedules. Navy’s change process is discussed below.

Table 3.2: Analysis of Change Proposals for case study Product Schedules

Source: ANAO analysis of Defence records.

1. From February 2014, the ANZAC SPO Director and the Capability Manager Representative did not raise Change Proposals for amounts less than $1 million; by September 2014, 15 internal transfers of funding between line-items, amounting to $4.98 million, had been approved under this arrangement. Another two transfers totalling $1.44 million were made in January 2015. See discussion below at paragraph 3.60.

ANZAC SPO raised numerous issues about Change Proposals during 2014

3.57 During 2014, the ANZAC SPO and the Capability Manager Representative for the ANZAC fleet were attempting to handle the administrative complexity of keeping the 88-page ANZAC fleet Product Schedule up-to-date.⁸² These complexities included a Change Proposal processing backlog, the level of financial detail included in the Product Schedule, and the method of implementing decisions taken at the six-monthly review point. These issues are examined in the paragraphs below.

3.58 In February 2014, the ANZAC SPO advised the Capability Manager Representative that the administrative and management processes applicable to Product Schedules had become too complex, and were presenting risks to capability that were having to be managed by the SPO. The SPO was awaiting sign-off of an omnibus Change Proposal from November 2013, delaying the progress of three subsequent Change Proposals, two of which were to secure funding for the installation of new capabilities. In the interim, the SPO was providing funding of some $618 000 to fund these capability upgrades. This presented risk to the SPO’s ability to fund other commitments if the Change Proposals remained delayed for much longer. The SPO noted that Change Proposals have to be processed in the order in which they are submitted, which maintains the accuracy of Navy’s 10 financial line-items in Product Schedules.

3.59 The minutes of the June 2014 Force Element Quarterly Review noted that:

Overachievement in [Materiel Ready Days], operational expenditure, product schedule changes and issues with the process of [Change Proposals] are affecting the performance of the FFH [ANZAC] Fleet.

3.60 Further, the inclusion of 10 separate financial line-items within Navy Product Schedules means that any internal movement of funds by ANZAC SPO, such as from ‘Planned Maintenance’ to ‘Engineering Change’, nominally requires a Change Proposal, no matter how small the amount.⁸³ In January and February 2014, by exchange of emails, the SPO Director and Capability Manager Representative agreed not to raise Change Proposals for transfers of funding between line-items amounting to less than $1 million (the level of their delegation under the ANZAC fleet Product Schedule), and to instead include a summary report each month to the Operational Sustainment Management Meeting so that the transfers could be endorsed there. The SPO Director commented that this was a practical approach that retained clear audit trails of decision making. By January 2015, 17 internal funding transfers amounting to $6.42 million had been approved in this way, including transfers from, for example, Materiel Support to Maintenance (Planned), or from Maintenance (Unplanned) to Maintenance (Planned).⁸⁴

3.61 In January 2015, Navy updated its Heads of Agreement, including the Change Proposal process. The revised Heads of Agreement stipulates that all Product Schedules will be ‘comprehensively reviewed and updated as necessary during the financial year, to ensure the currency and accuracy of their content for ongoing application’. An Administrative Instruction outlining this process and the timetable for completing the annual review has also been published. Any Product Schedule changes instituted through the annual review process will not be approved through a Change Proposal.

3.62 Navy’s annual review process represents a departure from the modular approach to Product Schedules established by the Standard Procedure on MSAs, and reintroduces the pre2012 arrangements involving annual review. The revised approach may increase the difficulty of keeping up-to-date a large number of Product Schedules, all of which are now significantly longer post-Rizzo.

Conclusion—MSA Change Proposals

3.63 Implementation of the Standard Procedure’s revised change process has been relatively smooth in Army and Air Force. Navy’s management of MSAs demonstrated a relatively conservative risk appetite, reflecting its assessment of risk, and resulted in a higher number of Product Schedule changes and delays in approving them. An unintended consequence of Navy’s approach was the emergence of undocumented workarounds to overcome delays in processing Product Schedule changes. While a matter for Navy’s senior leadership, there is scope to review the change management process for Navy Product Schedules and their level of detail, to support more flexible management of MSAs and avoid undocumented workarounds in their administration.

Recommendation No.1

3.64 The ANAO recommends that Navy and the DMO review change management processes for Navy Product Schedules, and the level of detail in the Schedules, to support more flexible management of the Navy Materiel Sustainment Agreement.

Defence response: Agreed.

Risk management

3.65 Risk management has been a feature of MSA administration since 2005, when the first Product Schedules included a section on ‘Risks and Mitigation Strategies’. The MSA framework now includes a number of levels of risk management and reporting:

• The 2012 Standard Procedure on MSAs provides for Annex E of Product Schedules to address the ‘Issues, risks and constraints’ affecting that product.⁸⁵ Product Schedules are to outline the current management action for existing issues, and the risk mitigation strategy and contingency plans for identified risks.
• The Army and Air Force Heads of Agreement similarly require their subordinate Product Schedules to contain an evaluation of ‘Issues, risks and constraints’. The Navy Heads of Agreement lists the types of interdependent materiel risks that the MSA seeks to address, and commits Navy and the DMO to ‘work towards a mutually accessible and integrated Enterprise Risk Management System’.
• The regular fleet screening reviews receive a narrative outline of current risk management activity, and briefing material for the reviews includes lists of current risks. Further, monthly meetings between DMO SPOs and the relevant Service receive and discuss risk reports.
• Through the DMO’s Monthly Reporting System, DMO Product Managers and Service Capability Manager Representatives provide information on materiel availability and operational matters to senior leadership.

3.66 This section focuses on risk management changes within Navy, and the treatment of issues and risks for the three case studies examined by the ANAO: ANZAC ships, Bushmaster Protected Mobility Vehicles and Orion aircraft.

Navy and the ANZAC fleet

3.67 By way of background, the June 2010 Product Schedule for HMAS Manoora and HMAS Kanimbla (two of the ships whose materiel condition prompted the Rizzo Report) had identified as almost certain the risk that these ships were operating beyond their service life, with severe consequences for performance. The Product Schedule also noted that ship configuration and maintenance data was inadequate to comply with the RAN Maintenance System, and that the materiel support regime for the two vessels was almost wholly reactive, with the bulk of maintenance being effected in response to breakdown events.

3.68 The DMO and Navy did not effectively mitigate the known sustainment risks for HMAS Kanimbla and HMAS Manoora, and consequently were unable to meet government requirements for use of the vessels, prompting the commissioning of the Rizzo Report. The 2012–13 ANAO Major Projects Report noted that risk management became a major focus on the sustainment side of the DMO’s business in response to the Rizzo Report.⁸⁶ As part of the Rizzo Reform Program, the DMO and Navy have developed the Integrated Mission Management System (IMMS) to address the need for an integrated Navy/DMO risk management system, and introduce greater accountability in relation to risk management.⁸⁷

3.69 IMMS was endorsed by the then Chief of Navy in June 2011 as a mechanism for him to have visibility of the Product Schedules, their KPIs and the interdependent risks shared by Navy and the DMO. The system is intended to record sustainment information on all of the products across Navy, in particular highlighting information that might be more readily apparent to Navy than the DMO, or vice versa. IMMS sits above existing DMO and Navy reporting tools and provides senior decision makers with management information. Defence informed the ANAO in March 2015 that the Deputy Chief of Navy has directed that IMMS be used at fleet screenings, and the Guided Missile Frigate SPO (FFG SPO) is to pilot the use of IMMS at the October 2015 Fleet Screening.

3.70 In relation to reviews of the ANZAC fleet, the briefing material for the Navy’s October 2014 fleet screening contained a detailed five-page analysis of the activities under way to manage risks identified in the Product Schedule, including risks related to obsolescence, supportability, communications systems and maintenance.

3.71 At the operational level, each monthly Operational Sustainment Management Meeting receives an update of the ANZAC SPO Strategic Risk Register and the Navy’s Frigate Group CN02 Strategic Risk Register, and discussion of progress is a standing agenda item. By May 2014, these Strategic Risk Registers identified seven major risks as having been escalated by being transferred to the ANZAC fleet Product Register within IMMS.

Army and the Bushmaster fleet

3.72 The Bushmaster Product Schedule identified five issues affecting the product in 2013–14 and 2014–15, including: engine obsolescence; absence of sustainment funding to support a Bushmaster fleet numbering more than 737 (see the text-box on pages 74–75)⁸⁸; and power pack serviceability. In 2014–15, the Lead Capability Manager provided an assessment of the management action being undertaken to address each issue identified that year.

3.73 Similarly, the Bushmaster Product Schedule identified a total of five risks for 2013–14 and 2014–15, including: labour shortfalls and late delivery of services by the Support Services Contractor; and attrition rates affecting the post-Afghanistan size of the fleet. In 2014–15, three risks were assessed by the Lead Capability Manager as being ‘low’, and two as ‘medium’.

Figure 3.2: Bushmaster Protected Mobility Vehicle (PMV)—Troop variant

 

 

Source: Thales, by permission.

Note a: The Hon. Stephen Smith MP, Minister for Defence, More Bushmasters for Army, Hawkei development proceeds, media release, 2 July 2012.

Note b: By July 2014, Defence was planning for a Bushmaster fleet of 856 vehicles under Plan Beersheba.

Air Force and the Orion fleet

3.74 The Orion Product Schedule identified 11 issues affecting the product over the last two financial years, including: fleet and mission system availability, arising from spares shortage, obsolescence, ageing systems and supportability challenges; corrosion; and the expiration of the engine maintenance contract. Each issue in the Product Schedule has been accompanied by a detailed narrative of the management action being undertaken to address the issue, including the area responsible for the action.

3.75 The Orion Product Schedule also identified 11 risks over the last two financial years, including: structural ageing-aircraft issues; airframe fatigue life; spares supportability concerns; and hydraulics maintenance pending the signing of a new maintenance contract. In 2014–15, one risk was assessed as being likely, four risks were assessed as being possible, and the remainder were assessed as unlikely or remote. Each risk was accompanied by an outline of the mitigation strategy and contingency plan.

3.76 In March and April 2014, aircraft from the Orion fleet took part in the search for Malaysian Airlines flight MH370, under Operation Southern Indian Ocean. The ANAO sought DMO advice on any risk assessments made as a result of the operation, including the potential impact of the operation on the sustainment of the Orion fleet. The DMO informed the ANAO that, for the period of the operation, no additional risk or expenditure was noted outside that allowed for within capability requirements. The DMO also informed the ANAO that, although the Orion fleet flew 62 per cent more than the planned monthly flying hours for the period of the operation:

• the total rate of effort for 2013–14 remained unchanged;
• there was no effect upon Deeper Maintenance servicing inductions or the Planned Withdrawal Date of the Orion fleet; and
• the operation established a new and enduring capability to detect Flight Recorders and Locator Beacons.

Conclusion—risk management

3.77 The MSA framework recognises the importance of risk management for sustainment activities, and provides a structured process for identification, assessment and management of sustainment risks by the Services and the DMO. The ANAO’s examination of the case study Product Schedules indicates that senior leadership was kept up-to-date about the risks to the relevant capabilities. However, the Navy’s experience relating to HMAS Manoora and Kanimbla, discussed at paragraphs 3.67 to 3.68, highlights that the practical effectiveness of the MSA framework largely depends on active and timely management of identified risks by Capability Managers and the DMO, and in that respect, a robust MSA framework is an aid to management, not an end in itself.

Introduction

4.1 Once it has received its annual appropriation, Defence has managed its acquisition and sustainment activities by transferring funding to the DMO. For management and reporting purposes, acquisition and sustainment funding is treated separately: DMO Programme 1.1 relates to Management of Capability Acquisition, and DMO Programme 1.2 relates to Management of Capability Sustainment. Sustainment funding is transferred to MSA Products according to their approved budgets.

4.2 Accurate estimation of Product Schedule sustainment costs is important because of the large sums and the long planning timeframes involved. A small percentage difference in costs, for example, can quickly amount to millions of dollars. Reliable cost estimates enable Defence’s Capability Managers to confidently use their current funding across different platforms, commodities and services as ‘informed purchasers’, and to move funding where it is most needed while still achieving required sustainment outcomes elsewhere.

4.3 In this chapter, the ANAO examines:

• the provision of Defence funding for the DMO’s sustainment activities, and the application of the funding across capabilities; and
• the accuracy of cost estimates for individual Product Schedules, focusing on the three case study Product Schedules: ANZAC ships, Bushmaster Protected Mobility Vehicles and Orion aircraft.

Defence funding for the DMO

4.4 To pay for the services to be provided by the DMO for acquisition or sustainment, Defence has made a prepayment to the DMO at the beginning of the financial year. The annual up-front payment—covering not only sustainment but also Major and Minor Capital Investment (that is, acquisition activities)—has been placed into a DMO Special Account.⁸⁹ In 2014–15, the initial Defence prepayment amounted to $10.686 billion, of which $5.532 billion was for sustainment.⁹⁰ Following the prepayment, the sustainment funding was distributed to the MSA sustainment products according to their approved budgets.

4.5 During the financial year, Defence’s prepayment to the DMO has been adjusted as needed, for reasons such as operational requirements or closure of acquisition projects and the transfer of unused funding to sustainment. During 2013–14, for example, the prepayment was adjusted five times. For sustainment, the changes included adjustments ranging from $4 million to $55 million. The final result was that the prepayment for 2013–14 relating to sustainment amounted to $5.064 billion.⁹¹

Distinguishing between acquisition and sustainment activity

4.6 As noted above, the DMO’s acquisition and sustainment activities are presented as separate programs in the Portfolio Budget Statements (PBS)—with Programme 1.1 covering the management of capability acquisition, while Programme 1.2 covers the management of capability sustainment.⁹² The DMO program structure broadly reflects the DMO Business Model approved by the Minister for Defence in September 2004, which identified three DMO outputs: acquisition projects; sustainment of ADF equipment fleets; and the provision of policy and advice services to Defence and government. The Business Model, and the related business rules agreed by the Defence Committee in October 2004, were both referred to in the 2006 Defence–DMO Memorandum of Arrangements, which documents certain constraints on the transfer of funds within the DMO between acquisition and sustainment activities.

4.7 However, while the PBS suggests that a relatively clear-cut distinction exists between acquisition and sustainment activities and funding, that distinction is not as clear-cut in the Memorandum of Arrangements. Further, experience indicates that the distinction is not hard and fast in practice, and that there is generally a transition period (illustrated in Figure 4.1), between Initial Operational Capability (IOC) and Final Operational Capability (FOC).⁹³ The transition period may last for several years, and can present practical challenges for the internal management of acquisition and sustainment funding and expenditures.

Figure 4.1: Defence’s Capability Systems Life Cycle

 

 

Source: Extract from DMO Submission to Joint Committee of Public Accounts and Audit, Review of the 2011–12 Defence Materiel Organisation Major Projects Report, Submission 3, 5 April 2013.

Examples of acquisition–sustainment funding transfers

4.8 The DMO informed the ANAO that there were four transfers of funding from acquisition to sustainment in 2011–12, two in 2012–13 and one in 2013–14. Table 4.1 summarises such transfers from 2011–12 to 2013–14.⁹⁴

Table 4.1: Detail of funding transfers from acquisition projects to sustainment products,

July 2011–June 2014

Source: ANAO analysis of DMO records.

4.9 In this audit, the ANAO noted a number of instances of overlap between acquisition and sustainment for the three case study Product Schedules:

• in 2008–09, sustainment funding was used for the completion of an ANZAC ship;
• in 2011–12, Army was uncertain whether anti-mine equipment and a gun mount required for Bushmaster operations in Afghanistan would be funded through Army Minors⁹⁵ acquisition funding or through sustainment funding⁹⁶;
• in 2013–14, Army Minors acquisition project funding was used to provide extra cladding for Bushmaster vehicles being used on operations (ADF operations are normally funded through MSA Product Schedules);
• in February 2014, Navy transferred $7.972 million from the Navy Minor Capital Equipment Program to the ANZAC fleet Product Schedule, for replacement of an obsolete Combat Management System at West Head Gunnery Range; and
• the ANZAC fleet Anti-Ship Missile Defence (ASMD) refit and upgrade program is treated by Defence as composed of acquisition-funded (SEA 1448) and sustainment-funded (CN02) elements, but there are a number of indications that at least one sustainment-funded element, renewal of the Operations Room layout, was an integral part of SEA 1448, and should have been funded as such.⁹⁷

4.10 In a similar example of project expenditure being classified as sustainment, the 2012 ANAO performance audit report on the M113 upgrade project reported that the preparation and extension of M113 hulls for upgrade incurred the bulk of the costs of the M113 Commercial Support Program, which was a sustainment program. An estimated $32.43 million of sustainment funding was expended over four years on initial work to upgrade some 80 per cent of the M113 fleet (350 of 431 vehicles). The DMO at one stage requested that Army spend more sustainment money so as to mitigate potential delays to the upgrade project.⁹⁸

4.11 Defence informed the ANAO in March 2015 that:

There is a clear distinction regarding the constraints on the transfer of funds within the DMO between acquisition and sustainment activities, which is also reflected in the PBS as separate Programmes. Funding received from Defence under Programme 1.1 Management of Capability Acquisition is clearly linked to the Materiel Acquisition Agreements (MAAs) and deliverables that have been expressly stated under these documents. Funding received from Defence under Programme 1.2 Management of Capability Sustainment is again clearly linked under the Materiel Sustainment Agreements (MSAs). DMO is not able to utilise funding received from Defence for MAAs for MSA activities or vice versa. There are instances, however, where as part of a Project closure process there may be remaining activities that were in scope under the MAA and are transferred to a relevant Product Schedule under an MSA. In these instances it is clearly documented and agreed by all signatories to the MAA and MSA Product Schedule Change Proposal that Defence is agreeing that the activity funded initially through the MAA can now be transferred to the MSA.

Joint Committee of Public Accounts and Audit concerns

4.12 In May 2014, the Joint Committee of Public Accounts and Audit (JCPAA) expressed doubt about whether there is always financial and budgetary separation of acquisition and sustainment.⁹⁹

4.13 As discussed, the clear distinction between acquisition and sustainment activities and funding shown in the DMO PBS is not as clear-cut in the Memorandum of Arrangements, and the underlying Business Model and business rules. In this light, there is scope to review current settings for clarity, recognising that there is often a transition period between the acquisition and sustainment phases of a capability. The timing of such a review will depend in part on the outcomes of the current First Principles Review of Defence.¹⁰⁰

Recommendation No.2

4.14 To clarify the internal treatment of acquisition and sustainment funding, the ANAO recommends that Defence review relevant business rules and guidance.

Defence response: Agreed.

Funding transfers between sustainment products

4.15 From the establishment of the DMO in 2005, control of sustainment funding rested with the CEO DMO, who was able to move funding between sustainment products as needs and priorities changed. In 2009, the Government Response to the Mortimer Review proposed to strengthen the role of Capability Managers¹⁰¹ and some years ago Capability Managers were given renewed responsibility for controlling their own sustainment budgets. Under this arrangement, transfers of funding between sustainment products can occur with the agreement of the relevant Capability Manager.¹⁰²

4.16 Table 4.2 shows that sustainment funding has been regularly transferred from one MSA product to another over the last three financial years, and that the number of transfers has increased significantly, reflecting increased use of the flexibilities available in the system. During the audit, Capability Managers informed the ANAO that they valued the ability to flexibly use sustainment resources according to operational and maintenance needs, which they considered a key strength of the sustainment framework.

Table 4.2: Funding transfers between sustainment products, July 2011–June 2014

Source: DMO Finance.

Product Schedule cost estimates

4.17 Every year since the inception of the MSA framework in mid-2005, Defence and the DMO have agreed on the cost estimates for each MSA Product Schedule for the following 10 financial years—that is, the period covered by the Defence Management and Financial Plan (DMFP).

4.18 This section reviews the Product Schedule cost estimates for the three case studies examined by the ANAO. The focus is on the four-year period covered by the Commonwealth Budget, including the budget year and three years of forward estimates. A greater degree of accuracy can be expected for this period than for the full 10-year estimation timeframe included in MSA Product Schedules. The four-year estimates are compared to the actual spending that eventually occurred. The ANAO also examined the accuracy of estimates made just prior to, and during the budget year, for each year.

Navy

4.19 Figure 4.2 compares baseline cost estimates (that is, excluding operations funding) against expenditure for the ANZAC fleet Product Schedule, for each financial year from 2005–06 to the present. More specifically, Figure 4.2 illustrates:

• the range of sustainment cost estimates for each financial year (the grey columns);
• the estimate made at the beginning of each financial year (the blue circle);
• the actual expenditure incurred in the financial year (the blue line); and
• the range of estimates for the three financial years covered by the current forward estimates (2015–16, 2016–17 and 2017–18—the blue column/bars).¹⁰³

4.20 In the past seven financial years, estimates made at the beginning of the financial year were well below actual expenditure for three years (2008–09, 2012–13 and 2013–14), while estimates for three years (2007–08, 2009–10 and 2010–11) were well above. For one year (2011–12), the estimate made at the beginning of the financial year was accurate. The figure also shows marked variation in actual expenditure from year to year.

Figure 4.2: ANZAC fleet sustainment cost estimates and outcomes for budget year and forward estimates, 2005–06 onwards

 

 

Source: ANAO analysis of CN02 Product Schedules, 2005–14.

Note: Defence was not able to supply an actual cost for 2005–06. The actual cost for 2006–07 is sourced from a hard-copy Sustainment Overview Report, rather than from Defence’s financial systems, and may reflect a slightly different cost basis.

4.21 Figure 4.3 shows the original and updated budget-year cost estimates for the ANZAC fleet Product Schedule for the last seven financial years. Two estimates are shown: for the annual Portfolio Budget Statement (in March); and Additional Estimates (in December). These estimates are shown as columns, and the actual expenditure for each financial year is shown by the blue line.

Figure 4.3: Budget-year ANZAC fleet cost estimates, and actual expenditure, 2007–08 to 2013–14

 

 

Source: DMO Finance, August 2014.

Notes: PBS—Portfolio Budget Statements (March); PAES—Portfolio Additional Estimates Statements (December).

 The blue column in this figure reflects an earlier estimate than the blue dot for the estimate made at the beginning of the financial year in Figure 4.2.

4.22 The figure shows that, in the last two financial years, actual spending has exceeded the March budget-year estimates, by $21 million (10 per cent) in 2012–13 and $39 million (17 per cent) in 2013–14.

4.23 The DMO informed the ANAO in October 2014 that:

• The high actual costs in 2008–09 were a result of engineering changes to the ANZAC Class to meet the HMAS Perth build specification. SEA 1348 Phase 2 (Ship Build Project) did not have sufficient funding available to rollout all the progressive changes made during the ship build program, and consequently some changes (related to system issues or early obsolescence) had to be funded from the ANZAC fleet sustainment budget.¹⁰⁴ Further, a logistics shortfall was identified to the then Government, and an additional $1 billion of funding was approved for the sustainment of the ANZAC Class.¹⁰⁵
• The low actual costs in 2010–11 were a result of the Strategic Reform Program (SRP) being initiated by Defence.¹⁰⁶ In response to the SRP, the ANZAC SPO prioritised work and cancelled or reduced the value of contracts. Navy also put two platforms into extended readiness (that is, it would take longer to make them ready for service).

4.24 As indicated previously (paragraphs 4.6 to 4.14), there is a lack of clarity surrounding the financial and budgetary distinction between acquisition and sustainment. It is unclear why the use of sustainment funding for SEA 1348 Phase 2 was reported as actual expenditure against the MSA Product Schedule when it was used for acquisition rather than sustainment purposes.

4.25 As part of the Rizzo Reform Program, Navy developed a Whole of Life Cost Plan for each of its MSA products.¹⁰⁷ Cost studies were still being undertaken when the revised ANZAC fleet Product Schedule was first agreed in August 2012. In late 2013, the Product Schedule included a provisional total Whole of Life Cost of $3.05 billion, somewhat less than the provisional approved funding of $3.17 billion. In July 2014, the Whole of Life Cost increased to match the approved funding, at $3.19 billion.

Army

4.26 Figure 4.4 compares baseline cost estimates (that is, excluding operations funding) against expenditure for the Bushmaster Product Schedule, for each financial year from 2005–06 to the present. More specifically, Figure 4.4 illustrates:

• the range of the sustainment cost estimates for each financial year (the grey columns);
• the estimate made at the beginning of each financial year (the blue circle);
• the actual expenditure incurred in the financial year (the blue line); and
• the range of estimates for the three financial years covered by the current forward estimates (2015–16, 2016–17 and 2017–18—the blue column/bars).¹⁰⁸

4.27 In five out of the past seven financial years, actual sustainment expenditure has exceeded the range of estimates for the year, three times by a wide margin (2008–09, 2011–12 and 2012–13).¹⁰⁹ In 2011–12, actual expenditure was more than twice the estimate made at the beginning of the financial year, and in 2012–13 it was 45 per cent higher. Actual expenditure was close to the estimate made at the beginning of the financial year in 2009–10 and 2013–14 only. The figure also shows marked variation in actual sustainment expenditure from year to year.

Figure 4.4: Bushmaster sustainment cost estimates and outcomes for budget year and forward estimates, 2005–06 onwards

 

 

Source: ANAO analysis of CA04 Product Schedules, 2005–14.

Note: Defence was not able to supply an actual cost for 2005–06. The actual cost for 2006–07 is sourced from a hard-copy Sustainment Overview Report, rather than from Defence’s financial systems, and may reflect a slightly different cost basis.

 At Additional Estimates in February 2015, the 2014–15 baseline estimate (that is, excluding operations funding) for sustainment of the Bushmaster fleet increased from $35.6 million to $47.8 million.

Budget-year cost estimates

4.28 Figure 4.5 shows the original and updated budget-year cost estimates for the Bushmaster fleet Product Schedule for the last seven financial years. Two estimates are shown: for the annual Portfolio Budget Statement (in March); and Additional Estimates (in December). These estimates are shown as columns, and the actual expenditure for each financial year is shown by the blue line.

Figure 4.5: Budget-year Bushmaster cost estimates, and actual expenditure, 2007–08 to 2013–14

 

 

Source: DMO Finance, August 2014, and further Defence advice, December 2014.

Notes: The blue column in this figure reflects an earlier estimate than the blue dot for the estimate made at the beginning of the financial year in Figure 4.4.

 In 2010–11, the original budget of $59.5 million was made in error, and was corrected to $17 million at Additional Estimates; the figure shows the corrected amount.¹¹⁰

4.29 The figure shows that actual expenditure has generally exceeded the original budget-year estimate as well as the mid-year revision. In 2012–13, the estimate increased markedly during the financial year to more closely align with actual expenditure. In contrast to earlier years, both cost estimates for 2013–14 were accurate.

4.30 The DMO informed the ANAO in June 2014 that:

• 2010–11: The variance was primarily due to the cost of supporting Protected Mobility Vehicles procured by Land 116 Phase 3. Army accepted the financial liability of introducing the additional vehicles into service prior to the triggering of sustainment funding for the vehicles. Further, in 2010–11, the sustainment budget was managed by the DMO with Army on a ‘whole-of-Army’ basis, and not on an individual Product basis.
• 2011–12: The variance was primarily due to SPARK II Mine Roller and Platt MR550 Ring Mount procurement activities.¹¹¹ It was initially expected that Army would fund the SPARK II requirement using Minor Project funding controlled by the Land Capability Battleworthiness Board (LCBB), and the Platt MR550 Ring Mount using
  
  no-win/no-loss funding.¹¹² Changes were made around this time regarding the application and eligibility for no-win/no-loss funding. Ultimately, both procurements were funded through Army baseline sustainment funds, with the variance managed on a whole-of-Army basis.¹¹³

4.31 As indicated previously (paragraphs 4.6 to 4.14), there is a lack of clarity surrounding the financial and budgetary distinction between acquisition and sustainment. It is unclear why acquisition funding was initially expected to fund the Spark II Mine Roller, which was an operational procurement.

Air Force

4.32 Figure 4.6 compares baseline cost estimates (that is, excluding operations funding) against expenditure for the Orion Product Schedule, for each financial year from 2005–06 to the present. More specifically, Figure 4.6 illustrates:

• the range of the sustainment cost estimates for each financial year (the grey columns);
• the estimate made at the beginning of each financial year (the blue circle);
• the actual expenditure incurred in the financial year (the blue line); and
• the range of estimates for the three financial years covered by the current forward estimates (2015–16, 2016–17 and 2017–18—the blue columns/bar).¹¹⁴

Figure 4.6: Orion sustainment cost estimates and outcomes for budget year and forward estimates, 2005–06 onwards

 

 

Source: ANAO analysis of CAF04 Product Schedules, 2005–14.

Note: The actual costs for 2005–06 and 2006–07 are sourced from a hard-copy Sustainment Overview Report, rather than from Defence’s financial systems, and may reflect a slightly different cost basis.

4.33 From 2007–08 to 2010–11, and in 2013–14, actual sustainment expenditure was within the range of the annual estimates.¹¹⁵ In three years (2006–07, 2010–11 and 2013–14) actual expenditure was close to the estimate made at the beginning of the financial year. In 2011–12 and 2012–13, actual expenditure significantly exceeded the estimate made at the beginning of the financial year, by 37 per cent and 17 per cent respectively.

Budget-year cost estimates

4.34 Figure 4.7 shows the original and updated budget-year cost estimates for the Orion fleet Product Schedule for the last seven financial years. Two estimates are shown: for the annual Portfolio Budget Statement (in March); and Additional Estimates (in December). These estimates are shown as columns, and the actual expenditure for each financial year is shown by the blue line.

Figure 4.7: Budget-year Orion cost estimates, and actual expenditure, 2007–08 to 2013–14

 

 

Source: DMO Finance, August 2014.

Note: The blue column in this figure reflects an earlier estimate than the blue dot for the estimate made at the beginning of the financial year in Figure 4.6.

4.35 The figure shows that actual expenditure has often been close to the original budget estimate, as well as to the mid-year revision. In 2011–12 and 2012–13, the estimate increased markedly during the financial year to more closely match actual expenditure.

4.36 The DMO informed the ANAO in October 2014 that:

• the high actual expenditure in 2011–12 (the year with the most variance from estimates) was primarily due to the purchase of sonobuoys and imaging systems, and recovery of a maintenance backlog; and
• in 2012–13, responsibility for multi-platform avionics support was transferred from another Product Schedule to the Orion Product Schedule, and a funding transfer was approved at the Mid-Year Review Principals Meeting.

Conclusion—case study cost estimates

4.37 In summary, the three case studies examined by the ANAO show persistent inaccuracy in Defence’s sustainment cost estimates. The ANZAC fleet estimates widely straddled actual expenditure, the Bushmaster fleet estimates were mostly below actual expenditure, and while the Orion fleet estimates were more accurate, they were still on occasions well below actual expenditure. Defence provided an explanation of variances in specific instances, including unanticipated changes in the allocation of funding across projects, products and sustainment activities. Defence also informed the ANAO that foreign exchange fluctuations and changes in the timing of contractual payments affect the accuracy of cost estimates. However, in no instances did Defence cite unexpected deployments or major materiel failure as a cause of inaccuracy in the estimates.

Sustainment cost estimates across all Product Schedules

4.38 The cost-estimation inaccuracy encountered for the case study Product Schedules is also evident for other sustainment products. Figure 4.8 shows that, out of 118 sustainment products in 2013–14 (the blue dots):

• for 18 products (15 per cent), actual expenditure was over 25 per cent below the original budget cost estimates; and
• for 21 products (18 per cent), actual expenditure was over 25 per cent higher than the original budget cost estimates.

Figure 4.8: Accuracy of Product Schedule cost estimates for 2013–14

 

 

Source: ANAO analysis of Defence data.

Notes: The order of the Product Schedules is different in each series. A blue dot may not represent the same Product Schedule as the adjacent grey dot.

 The figure does not include a number of variances that were off the scale:

• expenditure for five Product Schedules exceeded the PBS estimate by more than 100 per cent (105, 199, 335, 643 and 2636 per cent respectively);

• expenditure for three Product Schedules exceeded the PAES estimate by more than 100 per cent (110, 218 and 3613 per cent respectively).

4.39 Actual expenditure for one third of sustainment products (39 out of 118) varied from the original budget cost estimates by over 25 per cent. For another quarter of sustainment products (28 out of 118), actual expenditure varied from the original budget cost estimates by 10 to 24 per cent.

4.40 In relation to the mid-point of the financial year (Additional Estimates—the grey dots), actual expenditure for 39 products varied from the revised cost estimates by 10 per cent or more.

4.41 Figure 4.9 shows that, for the Top 30 sustainment products¹¹⁶, actual expenditure for seven products varied by 10–24 per cent from the PBS estimate (the blue dots), and for four products there was a variance of over 25 per cent. In relation to Additional Estimates (the grey dots), actual expenditure for seven of the Top 30 products varied from cost estimates by 10 per cent or more.

Figure 4.9: Accuracy of Top 30 Product Schedule cost estimates for 2013-14

 

 

Source: ANAO analysis of Defence data.

Notes: The order of the Product Schedules is different in each series. A blue dot may not represent the same Product Schedule as the adjacent grey dot.

 Defence reporting of the Top 30 sustainment products in 2013–14 combined certain cross-Service activities; for example, ‘Fuel & Lubricants – Air Force, Army and Navy’ was actually covered by three Product Schedules, and similarly, ‘Guided Weapons’ and ‘Explosive Ordnance’ (together covered by six Product Schedules), were shown as one line in the PBS and as two lines in the online supplement to the Annual Report. The ANAO analysis follows Defence’s practice: the chart shows 29 data points, and notes that the 30th (MRH90) is off the scale.

4.42 In response to a request about recent steps to analyse sustainment cost estimates or improve their accuracy, Defence informed the ANAO that in 2014 it:

• began to classify Product Schedule Change Proposals by category, which should aid identification of reasons for cost variances¹¹⁷; and
• released the first edition of the Defence Cost and Schedule Estimation Manual. The Manual prescribes policy and procedures for the development, validation and approval of cost and schedule estimates for all Defence resources (personnel, materiel, facilities, services and systems). The policy and procedures are to be applied for each stage of the capability life cycle, from identification of needs to disposal.

4.43 The Defence Cost and Schedule Estimation Manual was developed in response to Recommendation 1.2 of the 2008 Mortimer Review, which sought more disciplined cost, schedule and risk information for a project’s entry into the Defence Capability Plan. However, the May 2014 edition of the manual included only Chapter 1, containing the policy statement, roles and responsibilities, and definitions. While the manual was due to be completed by December 2014, no further sections had been published by February 2015.¹¹⁸

4.44 Despite the inherent uncertainty of financial forecasting, and the ambiguity introduced by the Capability Managers’ discretion to move funding between products, the extent of the variances in Product Schedule cost estimates indicates that Defence could do more to improve these estimates. This would in turn improve the capacity of Defence’s Capability Managers and the DMO to flexibly manage sustainment funding to achieve agreed outcomes.¹¹⁹

Conclusion

4.45 The DMO’s acquisition and sustainment activities are presented as separate programs in the Portfolio Budget Statements (PBS), and the 2006 Defence–DMO Memorandum of Arrangements, which documents certain constraints on the transfer of funds within the DMO between acquisition and sustainment activities. However, while the PBS suggests that a relatively clear-cut distinction exists between acquisition and sustainment activities and funding, that distinction is not as clear-cut in the Memorandum of Arrangements, and experience indicates that the distinction is not hard and fast in practice. While acknowledging that there can be a period of transition between the acquisition and sustainment phases of a capability, the ANAO noted a number of instances of overlap in the use of acquisition and sustainment funding. To clarify the internal treatment of acquisition and sustainment funding, Defence should review relevant business rules and guidance.

4.46 The three case studies examined by the ANAO indicated that there have been persistent inaccuracies in Defence’s sustainment cost estimates. Some departure from budget estimates can be expected due to flexible use of funding between sustainment products, and unforeseen factors such as the need to delay or bring forward maintenance work due to operational demands. However, variances of over 25 per cent are significant and suggest that there remains scope for the DMO to strengthen cost estimation techniques and understanding of cost variances. Improved cost estimation would strengthen the capacity of Defence’s Capability Managers to flexibly manage sustainment funding as informed purchasers of sustainment services.

Introduction

5.1 The 2008 Mortimer Review found that the ‘efficiency and effectiveness of DMO sustainment performance will not improve unless it is measured’, and that Defence did not have appropriate, quantifiable KPIs, nor did it accurately record sustainment costs. The Mortimer Review therefore recommended that:

DMO and Defence need to further develop the key performance indicators in Materiel Sustainment Agreements and the systems needed to record sustainment performance and costs.¹²⁰

5.2 In this chapter, the ANAO examines Defence’s:

• sustainment performance reporting; and
• public reporting on sustainment activities.

Sustainment performance reporting

Defence currently reports sustainment performance through its Monthly Reporting System

5.3 Under the Memorandum of Arrangements and individual MSAs, the DMO is required to report monthly against all of the performance measures contained in Product Schedules. The DMO meets its sustainment reporting obligations through its Monthly Reporting System (MRS). There are two series of DMO sustainment reports produced each month:

• The Sustainment Performance Report (SPR) covers the Top 30 sustainment products as listed in the PBS. The SPR is amended by DMO SPOs each month and cleared and endorsed by the respective DMO Branch and Division Heads.
• The Navy Sustainment Performance Report (NSPR) is developed from the cleared DMO Division Head SPR and includes additional information to cover all Navy sustainment products. The NSPR was developed in response to the 2011 Rizzo Report, and is intended to provide a high level of visibility on sustainment performance across all Navy products.

5.4 For each sustainment product, the reports include a cost update (budget and year-to-date expenditure), an availability update (red/green/amber), and commentary on current performance by the relevant DMO Product Manager, Branch Head and Division Head. They also include advice on anything that has become known or has occurred in the reporting month that will either actually or potentially affect a product’s materiel readiness. This advice is intended to identify any issues similar to those experienced by Navy in 2011, when the supply ships HMAS Manoora and HMAS Kanimbla were unavailable for service.

5.5 The reports on sustainment product performance are reviewed by the CEO DMO, the Deputy CEO DMO, DMO General Managers, and the DMO Chief Finance Officer. The reports are also provided to the Government and central agencies.

5.6 Defence preparedness—the ultimate goal of Defence sustainment—is the subject of classified reporting to government:

Each quarter a Defence Preparedness Assessment (CDPAS) is undertaken, the report from which is considered by the Chief of Defence Force and the Secretary at their Strategic Command Group. […] The outcomes and key judgements from this assessment process are provided to the Minister as the classified Preparedness and Concurrency Ministerial Submission. This process has been refined considerably over the last four years, and provides a high level of assurance to Government as to the capability of Defence to meet current commitments and conduct future operations.¹²¹

Defence is developing a new sustainment performance reporting system

5.7 The MRS was originally designed for acquisition reporting, and has limitations in relation to sustainment reporting. In particular, the MRS can only report performance using relatively simple measures, and the system cannot present trend analysis. The Directorate of Sustainment Measurement and Analysis was created within the DMO Standardisation Office in July 2011, to work with DMO Divisions and Capability Managers, to recommend and implement an improved sustainment performance management system. The new system has now been designed and includes two components: a performance measurement framework—the Sustainment Performance Management Framework (SPMF); and a reporting system—the Sustainment Performance Management System (SPMS).

Sustainment Performance Management Framework (SPMF)

5.8 During the first half of 2012, the DMO reviewed the performance measures used for aviation and maritime sustainment products, and developed a framework involving four types of performance measures:

• Outcome KPIs (for example, Mission Success Rate, Rate of Effort Achievement, or Mission Capable Days Achievement);
• Output KPIs (for example, Aircraft Serviceability, Aircraft Availability or Materiel Ready Days Achievement);
• Key Health Indicators (KHIs) to enable the DMO and Capability Managers to review a range of information to assess both the current and future health of a product (for example, maintenance schedule changes); and
• Strategic Sustainment Analytics (SSA)—high-level health indicators used for cross-product comparison of performance (for example, DMO expenditure per Materiel Ready Day).

Sustainment Performance Management System (SPMS)

5.9 The SPMS has been developed since September 2012, with a total of $2.85 million in funding allocated for system development by February 2015. The SPMS is to generate dashboards for each sustainment product and reporting cycle, based on current and historical KPI, KHI and SSA data. Separate dashboards are to be designed for three levels of the Defence Organisation—System (DMO CEO/General Manager and Service Chief/Group Head), Division (Division Head/Fleet Commander and Service Deputy Chief), and Product (SPO/Product Operator). The dashboards are intended to assist Defence and the DMO to collaboratively manage current and future capability and sustainment risks.¹²²

5.10 Pilot projects of the SPMF/SPMS were conducted in the maritime and aviation domains during late 2012–early 2013. Defence documentation indicated that, as at July 2014, the system build, training and documentation development were on track to support rollout to the Navy and the DMO’s Maritime Systems Division in November 2014. However, by January 2015, initial training and rollout had been delayed to April–May 2015. Rollout to the remaining Services and DMO Divisions is planned to occur over the following year. Further, Defence intends to progressively link SPMS to existing Defence systems. The aim is to enable complex analysis of financial and other data to identify risks.¹²³

Navy’s new suite of performance measures

5.11 In September 2013, Navy established an MSA Performance Framework Project, with the objective of instituting a standard suite of KPIs and KHIs for its Product Schedules in response to Rizzo Recommendation 12.¹²⁴ This work was intended to take advantage of the DMO’s development of the SPMS. The new framework and standardised performance measures were endorsed by the Navy Capability Committee on 4 March 2014, and by the Chief of Navy Senior Advisory Committee (CNSAC) on 8 May 2014. Eighty-nine performance measures across the 37 Navy Product Schedules were to be replaced with the standard suite of measures.

5.12 Navy’s standard measures include six KPIs and 15 KHIs (Table 5.1). For each KPI/KHI, Navy sets a tolerance, in terms of an acceptable difference between targeted and actual performance. Most of the measures will be used for each Navy platform. The services and commodities supported by Navy Product Schedules will generally be covered by two of the KPIs, and three or four of the KHIs.

5.13 The KPIs were introduced for Navy Product Schedules from 1 July 2014, and the KHIs will be implemented when the SPMS is rolled out to Navy and the DMO’s Maritime Systems Division, which is now planned for April–May 2015. The KPI data were recorded in the MRS, pending the transition to the SPMS.

Table 5.1: Navy performance measures

Source: Navy, Navy MSA Performance Framework: Implementation, 26 May 2014, p. 11.

1. A Materiel Ready Day is any programmed day where a platform is not in an External Maintenance period, undergoing defect repair, in an Extended Readiness, or subject to a Priority 1 Urgent Defect report that because of its nature prevents the ship from achieving its current tasking.
2. In relation to Cost per Materiel Ready Day achieved, Navy’s intent is that an annual target should be set and trend data captured. Navy also intends to pursue international benchmarking, with Navy/DMO developing skills to better understand cost drivers and performance in comparison to international benchmarks.

ANAO assessment of Navy performance measures

5.14 In 2012–13, the ANAO developed a set of three criteria to evaluate the appropriateness of entity KPIs.¹²⁵ The criteria examine whether KPIs are relevant (focused and understandable), reliable (measurable and free from bias) and complete (balanced and collective). While devised in the context of the Australian Government’s Outcomes and Programs Framework, the criteria may also usefully inform an assessment of other KPIs. The ANAO applied the criteria to Navy’s performance measures.

5.15 Overall, Navy’s KPIs/KHIs are appropriately designed and measurable. The KPIs provide balanced and usable coverage of Navy’s sustainment products in terms of availability, cost, schedule, and materiel deficiencies. The KPIs addressing Cost per Materiel Ready Day and Price Reliability are particularly noteworthy, and should generate better information on the DMO’s financial performance. Further, Navy’s KHIs address key factors affecting sustainment performance, including staffing levels, inventory management, maintenance performance and configuration changes.

5.16 While Navy’s performance measures are a step forward, there remain areas for improvement. None of Navy’s KPIs address outcomes, which is one of the four types of measures included in the DMO’s guidance. Measures such as Materiel Ready Days Achievement identify availability of platforms, but they do not indicate whether platforms were available when needed for operations. Further, some of the measures are not necessarily free from bias—that is, allowing for clear interpretation of results. The KPIs for Materiel Ready Days Achievement and Service Level Achievement are expressed in percentage terms against a plan or target, and changes in plans or targets may mask deteriorating performance. In presenting performance using these KPIs, Navy should be transparent about any changes in plans or targets.

5.17 Since the transition to the standardised system of KPIs in 2014, there has only been one KPI addressing Navy’s performance—Conformance to Operating Intent. In contrast, there were as many as seven KPIs of Navy’s contribution to sustainment in 2006, which addressed matters such as timely delivery of platforms to the DMO, timely notification of defects or damage to assets, and completion of mandatory organic-level maintenance. Some of these measures are now addressed by KHIs rather than KPIs.

ANZAC fleet KPIs

5.18 The ANZAC fleet Product Schedule transitioned to the new Navy KPI framework in July 2014. There are five KPIs addressing the DMO’s sustainment performance: Materiel Ready Days Achievement, External Maintenance Period Milestone Achievement, Cost Per Materiel Ready Day Achieved, Price Reliability and Priority 1 Materiel Deficiency Reports Raised.

5.19 Materiel Ready Days Achievement is an output measure. It is measured as a percentage achieved against planned days, and reflects the reliability of the Mission System and Support System. Figure 5.1 shows that the ANZAC fleet has been close to achieving its target for Materiel Ready Days over the last three financial years. However, Navy has varied the targeted number of Materiel Ready Days for the ANZAC fleet, and the KPI does not shed light on this reduction. The target was reduced by 20 per cent in 2012–13 and another 13 per cent in 2013–14, and then increased by 3 per cent in 2014–15.

Figure 5.1: ANZAC fleet achievement of Materiel Ready Days, as a percentage of target, June 2011 to January 2015

 

 

Source: ANZAC SPO.

Note: As at January 2015, a delay in HMAS Parramatta’s ASMD upgrade was inflating performance beyond planned levels, because the ship was materiel ready for longer than originally expected.

Army’s new Measures of Success framework

5.20 Army has independently developed a trial performance framework known as Measures of Success, and an associated reporting tool, which uses the same software as the DMO’s SPMS.¹²⁶ The Measures of Success Framework is intended to address two primary goals: outline the required outcomes or performance for a specific aspect of a Product Schedule; and generate information to shape future actions relating to Product Schedule management.

5.21 The Measures of Success Framework is intended to provide Army Product Schedule drafters with a standardised suite of performance measures from which to select appropriate measures for a particular product. Once implemented, it will establish a hierarchy of measures, as shown in Table 5.2. The framework outlines specific measurement techniques for each measure.

Table 5.2: Army’s Measures of Success

Source: Army, Army Product Schedule—Measures of Success, version 0.2, March 2013.

Note: The 41 measures (10 Defence Output Measures and 31 Health Indicators) that support the Critical Success Factors comprise 21 that address DMO performance, 16 that address Army performance, and four that address shared performance.

5.22 Army’s different types of measures address outcomes, outputs and inputs. However, they are presented in a way that differs from the DMO’s guidance; for example, Army’s ‘Health Indicators’ are more akin to outcome measures. Ideally, the Services’ KPI frameworks would apply similar terminology to support Defence senior leadership and ministerial review.

5.23 From July 2013 to June 2014, Army conducted a pilot program of the Measures of Success Framework and the associated reporting tool. The primary objective of the pilot was to develop a management and reporting knowledge domain to monitor, control and improve how the Lead Capability Manager, Supported Capability Managers and the DMO collaboratively manage a Product Schedule. The pilot was conducted across seven Product Schedules in four DMO Divisions. Defence informed the ANAO in March 2015 that Army had developed a formal report on the pilot for submission to the Chief of Army and formal endorsement.

5.24 Army’s intent is to apply the Measures of Success Framework to all Product Schedules when the SPMS is implemented. In the meantime, Army’s existing KPIs, with minor amendments, will continue to be used. During 2013–14, Army and the DMO were working to bring together the separate development streams of Army’s Measures of Success Framework and reporting tool, and the DMO’s SPMS. The DMO intends to roll out the SPMS to Army late in 2015 incorporating the Measures of Success Framework.

Bushmaster fleet KPIs

5.25 Until January 2014, the main Bushmaster sustainment KPI was Operational Availability. Performance against a targeted level of Operational Availability was assessed using a traffic-light system: set at 70 per cent or above (Green), 60–70 per cent (Amber), and less than 60 per cent (Red). From February 2014, this KPI measure was changed to the ‘more appropriate’ metric of Functional Equipment—which measures a combination of Fully Functional and Restricted Use.¹²⁷ The new measure, in the DMO’s view, better accounts for assets that remain taskworthy, albeit within certain operational parameters or restrictions.

5.26 The largest of the four Bushmaster fleets belongs to Army’s Forces Command, and Figure 5.2 shows the achievement of the main Bushmaster KPI for Forces Command since July 2011. The figure indicates better performance following the change in KPI from Operational Availability to Functional Equipment. This again highlights the importance of KPI measurement techniques and maintaining transparency about these to inform the pursuit of better performance and lower costs over time.

Figure 5.2: Operational Availability/Functional Equipment KPI for Bushmasters, Army’s Forces Command, 2011–14

 

 

Source: ANAO analysis of DMO data.

5.27 Since 2010, no KPIs have addressed Army’s performance in relation to sustainment of the Bushmaster fleet. The Measures of Success framework that will be applied to the fleet includes some Army-related measures.

Air Force

5.28 The Air Force Capability Management Manual (2012) provides guidance for developing performance measures for Air Force Product Schedules. The manual notes that the Capability Manager Representative should consider three distinct sets of performance measures, which are related to organisational responsibility: DMO only; Air Force only; and shared.

5.29 The Manual suggests that KPIs cover at least three of the following criteria: availability; reliability; sustainability; and maintainability. The Manual advises that the KPIs should: cover all the required outcomes; be limited in number (three to five being viewed as an appropriate amount); be objective rather than subjective measures where appropriate; strike the right balance between being tailored to the Product and standardised across Air Force; and be specific, measurable, achievable, relevant and time-based (SMART). The Manual also discusses risks associated with the use of common KPIs, and mitigation strategies.¹²⁸ As indicated in paragraph 5.10, the SPMS is expected to be rolled out to Air Force in 2015–16.

Orion fleet KPIs

5.30 The Orion Product Schedule contains seven KPIs that address rate of effort and the DMO’s performance in relation to the availability of the aircraft, and the simulators and analytical equipment that support them. There is also one shared KPI—a measure of aircraft serviceability. Three of the KPIs are discussed below, highlighting the potential value in a set of appropriate KPIs, in terms of exploring sustainment performance and identifying issues.

5.31 Rate of Effort has been identified as a critical performance metric in several iterations of the Orion Product Schedule. Figure 5.4 shows the forecast annual Rate of Effort versus the actual Rate of Effort achieved. The figure shows that actual flying hours have generally been close to forecast flying hours, with a low of 91.02 per cent of forecast hours flown in 2005–06, and a high of 103.64 per cent of forecast hours flown in 2010–11.

Figure 5.3: An AP-3C Orion conducts Air Sea Rescue Kit training off the South Australian coast, October 2013

 

 

Source: Australian Defence Image Library.

Figure 5.4: Orion fleet rate of effort (forecast and actual), 2004–14

 

 

Source: ANAO analysis of Defence data.

Note: The DMO informed the ANAO that the forecast of 7900 flying hours for 2012–13 was revised to 7100 at the Mid-Year Review, due to a substantial reduction in funding.

5.32 Aircraft availability is a measure of the number of aircraft that have completed their SPO-level maintenance and have been given to the Air Force Wing that is operating them. Figure 5.5 shows the actual availability of the Orion fleet as a percentage of the targeted availability. In the three financial years from July 2011, aircraft availability as a percentage of the target averaged 97.05 per cent.

Figure 5.5: Orion fleet availability, as a percentage of target, 2011–14

 

 

Source: ANAO analysis of Defence data.

5.33 Serviceable aircraft are aircraft that have been made available to the Operational Maintenance Unit at Air Force’s 92 Wing to be rendered capable of flying missions. The target number of serviceable aircraft for this KPI has not been met since July 2011, even though the target was reduced in 2014. However, Defence informed the ANAO that the Orion aircraft are meeting their tasking requirements despite the serviceability target not being met.

Progress of sustainment performance measures

5.34 In terms of activities and processes, Defence has in recent years made a significant effort to establish new sustainment KPI frameworks and performance management systems, with relevant activity occurring across the Services and the DMO. Further, Defence records show that Navy has commenced reporting against its new set of KPIs, and Army has conducted a pilot of its Measures of Success.

5.35 On the whole, the new performance frameworks provide a firmer basis for the evaluation of Defence’s sustainment performance. However, there remain some inconsistencies in the definitions of measures adopted by the different Services. For example, while Navy has adopted Price Reliability as a KPI, Army has adopted a similar measure as a KHI.

5.36 The establishment of new DMO, Navy and Army performance measurement frameworks remains at an early stage. It is only when new measures have been reported against for some time that their usefulness will be tested, and any need for refinement can be assessed.

Transparency of external reporting

5.37 There is strong parliamentary interest in Defence’s public reporting of its sustainment performance, and two parliamentary committees have recently made recommendations and sought improvements in this area.¹²⁹ The level of parliamentary interest reflects the importance of the state of readiness of Australia’s military capabilities, and the size of Defence’s sustainment budget, at over $7.1 billion. The ANAO reviewed Defence’s public sustainment reporting in light of other countries’ reporting on sustainment activities.

5.38 The DMO Portfolio Budget Statements (PBS) now include a brief description of sustainment arrangements for the Top 30 sustainment products by cost (Top 20 in 2012–13 and earlier years), and an outline of the focus of sustainment efforts for the budget year.

5.39 In its 2014–15 PBS, the DMO stated that its KPIs for Programme 1.2, Management of Capability Sustainment, vary with each sustainment product and are specified in the MSAs.¹³⁰ However, the MSAs are not public documents. In the Defence PBS, each of the Services describes its deliverables related to capability performance, including Navy Unit Availability Days and Army and Air Force aircraft rate of effort. These measures are then reported against in Defence’s Annual Report.¹³¹

5.40 Defence also reports financial information for the Top 30 sustainment products in its Annual Report, namely the Budget estimate; the revised estimate; actual expenditure; the variation between the revised estimate and the actual expenditure; and an explanation of the variation. Table 5.3 shows three examples of the financial reporting on sustainment from Defence’s latest Annual Report.

Table 5.3: Sustainment financial reporting

Source: Department of Defence, Annual Report 2013–14, supplementary information for Chapter 6 [Internet], available from <http://www.defence.gov.au/AnnualReports/13-14/Supplementary Chapter6.pdf> [accessed November 2014], pp. 32–3.

Note: The Bushmaster is not among the Top 30 sustainment products, and so is not included in the table.

5.41 Defence also presents descriptive information on sustainment performance for the Top 30 sustainment products.¹³² Table 5.4 shows three examples of the descriptive reporting on sustainment from Defence’s latest Annual Report.

Table 5.4: Sustainment performance reporting

Source: Department of Defence, Annual Report 2013–14, vol. 1, pp. 100, 105.

Note: The Bushmaster is not among the Top 30 sustainment products, and so is not included in the table.

5.42 The information in Defence’s Annual Report provides stakeholders with a basic summary of sustainment costs and activity. However, it does not facilitate any systematic assessment or analysis of Defence’s sustainment performance, and there may also be scope for additional standardisation and improved data consistency.

5.43 Defence informed the ANAO in March 2015 that, overall, reporting in the PBS and the Defence Annual Report needs to be carefully managed to avoid the disclosure of classified or other information.

Overseas public reporting of sustainment expenditure

5.44 In reviewing Defence’s public reporting on sustainment, the ANAO considered the extent of reporting on sustainment expenditure in Canada, New Zealand, the United Kingdom and the United States (Table 5.5). The table only considers sustainment-related reporting, and it does not address the overall quality of the countries’ reporting on defence.

Table 5.5: Overseas budgeting and reporting of defence sustainment expenditure

Source: ANAO analysis.

Note a: A separate appropriation is made for the New Zealand Ministry of Defence, which has primary accountability for defining what capability is needed to meet government objectives, and purchasing, replacing or upgrading major defence equipment. The Ministry of Defence has its own annual report.

Note b: The Defence Equipment and Support organisation became a bespoke trading entity within the Ministry of Defence from 1 April 2014, and will in future publish its own annual report, including a detailed financial statement. Its annual accounts will be consolidated with those of the Ministry of Defence.

Note c: National Audit Office, The Major Projects Report 2013 [Internet], London, 13 February 2014, available from http://www.nao.org.uk/report/ministry-of-defence-the-major-projects-report-2013/ [accessed December 2014].

Note d: The other major Defense appropriations are: Military Personnel; Procurement; Research, Development, Test and Evaluation (RDTE); Military Construction; and Family Housing. Overseas operations are appropriated separately.

Note e: The Budget Activity most relevant to sustainment is Operating Forces, with 116 line-items across Army, Navy and Air Force in Fiscal Year 2015. However, the other three Operation and Maintenance Budget Activities—Mobilization (15 line-items), Training and Recruiting (50 line-items) and Administration and Service-wide Activities (89 line-items)—also contain elements of sustainment funding, such as Depot Maintenance (under Air Force Mobilization and Air Force Training and Recruiting) and Planning, Engineering and Design (under Navy Administration and Service-wide Support).

5.45 The United Kingdom publishes very little detail on its defence sustainment expenditure, and the United Kingdom National Audit Office has observed that the equipment support budget has not received the same degree of scrutiny as the procurement budget. The United States Government spends some US$272 billion on ‘Operations, Readiness & Support’, compared to Defence’s $6 billion expenditure on sustainment. The operating context of the two countries also differs markedly. That said, the United States reports sustainment expenditure down to detailed line-items such as Ship Maintenance costs, as well as maintenance schedules for individual vessels. Australia has only a small number of platforms, and specific details on maintenance of any platform could potentially reveal details of capability.

Joint Committee of Public Accounts and Audit concerns

5.46 In March 2013, the Joint Committee of Public Accounts and Audit (JCPAA) questioned the DMO about sustainment reporting, and asked whether there were any reasons why an annual consolidated report on sustainment could not be compiled, in a similar vein to the annual Major Projects Report on significant Defence acquisition projects. The DMO responded that a consolidated sustainment report would be classified:

Where we run into highly sensitive matters in the sustainment area is that most of the performance metrics that we have for sustainment are against the [Chief of the Defence Force’s] preparedness directive, which is classified. If we were to do something similar in sustainment in terms of assessing performance against measures in a public fashion, it would be classified. […] If we were to put a consolidated report together of how we are performing against all the sustainment activities, that would give you a public indication of preparedness, which is something we cannot do because it is classified.¹³³

5.47 In May 2013, the DMO undertook to increase its annual reporting to cover the Top 30 sustainment products (around 77 per cent of current spending on sustainment). This is consistent with the number of projects examined in the Major Projects Report and the Top 30 Major Capital Projects disclosed in the Defence Portfolio Budget Statement and Annual Report.¹³⁴ The JCPAA welcomed the move to increase coverage, but noted that, given the increasingly large sum of public money devoted to sustainment, more needed to be done to increase the depth of sustainment reporting, not just the number of products covered.¹³⁵

5.48 In its May 2014 review of the 2012–13 Major Projects Report, the JCPAA observed that the total budget for sustainment (over the next 10 years, the period covered by the Defence Management and Financial Plan) was not published, depriving the Major Projects Report of necessary context. The JCPAA recommended that the DMO should prepare, within six months, a suitable and separate methodology for reporting sustainment activity and expenditure. As discussed in paragraph 4.12, the JCPAA also expressed concern about the financial and budgetary separation between acquisition and sustainment.¹³⁶

5.49 Defence responded to the JCPAA’s May 2014 request for a separate sustainment reporting methodology in December 2014, advising the Parliament that:

Defence’s position is that the current arrangements of Portfolio Budget Statements (PBS) and Defence Annual Report (DAR) reporting to Parliament, and Preparedness reporting to Government, balance effectively the obligation to allow Parliamentary scrutiny of the expenditure of Commonwealth funds on sustainment efforts, while protecting the classified information on capability readiness and availability which is associated with those sustainment efforts and which is separately provided to Government.¹³⁷

5.50 In September 2014, the JCPAA also requested that the ANAO develop an options paper on sustainment reporting, and review other international work in this area. The ANAO provided an options paper to the JCPAA in February 2015, developed in consultation with Defence, and identifying four options for increasing the transparency of sustainment expenditure.¹³⁸ The four options, which could be considered in isolation or in combination by the JCPAA, were:

• Option 1: an annual in camera briefing for the JCPAA on Defence sustainment;
• Option 2: expansion of sustainment reporting in the Defence Annual Report;
• Option 3: expansion of the Major Projects Report to include further sustainment reporting; and
• Option 4: development of a new sustainment report and limited assurance review.

5.51 In relation to the Defence Annual Report, the options paper observed that:

In addition to the reporting currently appearing in the Defence Annual Report, the new suites of sustainment Key Performance Indicators being developed by Defence for Materiel Sustainment Agreements offer opportunities for improved reporting. There may be merit in Defence disclosing against a small number of Key Performance Indicators for the Top 30 sustainment products, through measures that do not pose a [security] classification risk, e.g. measures of financial performance against estimates, performance as a percentage of a target, or Cost per Materiel [Ready] Day Achieved. The DMO could also report on other financial information in the Defence Annual Report, including measures such as cost growth across the capability life-cycle.¹³⁹

5.52 The issue remains under consideration by the JCPAA at the time of preparation of this report.

Joint Foreign Affairs, Defence and Trade Committee concerns

5.53 The Joint Standing Committee on Foreign Affairs, Defence and Trade has also examined Defence’s annual reporting. In June 2013, the Standing Committee recommended that Defence enhance its public reporting by developing a more precise method for reporting capability acquisition and sustainment performance, including specific performance targets, how performance is assessed in relation to these targets, and the specific reasons why targets are, or are not, achieved.¹⁴⁰

5.54 In October 2014, Defence agreed in principle to this recommendation, and advised the Standing Committee that it was expanding its reporting to cover the Top 30 sustainment products.¹⁴¹ However, the descriptive nature of Defence’s sustainment reporting (see Table 5.4) shows that Defence still has some way to go before it meets the intent of the recommendation.

Conclusion

5.55 Between 2012 and 2014, the DMO, Navy and Army developed new performance measurement frameworks, including measures of availability, cost, schedule, and materiel deficiencies, which are to be reported through a new DMO system.¹⁴² These performance measures will not be fully implemented until the DMO system is operational, and at the time of the audit, the first phase of the system rollout was scheduled for May 2015. While the new performance measures should provide a firmer basis for the evaluation and active management of sustainment performance and costs, their establishment remains at an early stage.

5.56 Defence’s annual reporting on sustainment includes budget and expenditure data for the Top 30 sustainment products (representing some 77 per cent of current spending on sustainment), as well as an overview of the management of these products. While providing stakeholders with a basic summary of sustainment costs and activity, this information does not facilitate assessment of Defence and the DMO’s sustainment performance in terms of materiel availability, cost-effectiveness and key inputs such as inventory management, maintenance and configuration changes. Defence still has some way to go before it meets the intent of the Joint Standing Committee on Foreign Affairs, Defence and Trade’s recommendation for enhanced public reporting. Following a request by the JCPAA, in February 2015 the ANAO provided the JCPAA with options, developed in consultation with Defence, to enhance sustainment reporting to the Government and Parliament. The issue remains under consideration by the JCPAA at the time of preparation of this report.

Appendices

Please refer to the attached PDF for the Appendices:

• Appendix 1: Entity Response
• Appendix 2: Sustainment Products and Budgets

Abbreviations