Introduction

1. The Defence Materiel Organisation (DMO) is a large organisation within the Department of Defence which buys and maintains equipment for the Australian Defence Force (ADF). DMO currently manages about 180 major capital acquisition projects, many of which are technically complex, high-cost endeavours conducted over many years.

2. Gate Reviews are an internal DMO assurance process for these projects. They are not a replication of the Australian Government’s Gateway Review process.¹ DMO’s objective in conducting Gate Reviews is to supplement regular management processes by: providing a degree of external scrutiny of projects; facilitating frank communication among stakeholders; and enabling DMO senior management to intervene with projects at critical times, and report to government where necessary.²

3. Gate Reviews involve a periodic assessment of a project, at key milestones during a project’s lifecycle, by a DMO-appointed Gate Review Assurance Board. This board comprises senior DMO management and up to two external members. The centrepiece of each Gate Review is a meeting attended by the project team, key stakeholders and the board. Before this meeting, the board is informed by a preliminary analysis of the project, undertaken by someone not involved with the project, which describes the project’s history, current status and the major issues it faces.

4. Gate Review meetings provide an opportunity for senior DMO management to seek insight into a project’s progress and for project staff to discuss difficult issues with senior management and seek their guidance. Following the meeting, the board makes a recommendation regarding the progress of the project and develops a list of action items to address identified issues. DMO’s own analysis of Gate Reviews it has conducted to date indicate that common issues leading to project problems are unrealistic schedules and poorly-defined project scope.

5. DMO first publicly discussed its intention to implement Gate Reviews in July 2008.³ It held its first two Gate Reviews in August and September 2008 but did not conduct any more until September 2009. DMO approved its administrative policy for Gate Reviews in December 2009 and held Gate Reviews in accordance with this policy through 2010 and early 2011, though with a varying degree of rigour. Between September 2009 and July 2011, some Gate Reviews were centrally managed by DMO’s Major Program Control Branch and some were managed by the DMO division in which the project resided.

6. The program was reinvigorated following a ministerial announcement in May 2011, and Gate Reviews are now DMO’s most prominent project assurance activity. In July 2011, DMO established its Independent Project Performance Office (IPPO) and centralised management for Gate Reviews within this office. IPPO holds records for 81 of the 115 Gate Reviews conducted between July 2008 and July 2011 and holds records for all but one of the 48 Gate Reviews conducted between July 2011 and December 2011.

7. As a mechanism for providing additional management focus on project risk, DMO’s Gate Reviews are developing an increasingly high profile. The Ministers for Defence and Defence Materiel have stated that they value the assurance they provide and have, on occasions, either requested a Gate Review be conducted or announced that one has been held.

Audit objectives and scope

8. The objective of this audit was to examine the effectiveness of DMO’s implementation of its Gate Review process for major Defence capital acquisition projects.

9. The audit examined how DMO developed, has implemented and is monitoring its Gate Review process. First, the audit examined how DMO developed its administrative policy for conducting Gate Reviews; second, it analysed records of Gate Reviews held between July 2008 and 31 December 2011 against objectively testable characteristics; and third, it examined how DMO monitors and reviews the effectiveness of its Gate Review program, including how it follows up action items recommended by Gate Review Assurance Boards and monitors the cost of running the program.

Overall conclusion

10. Major defence capital acquisitions contribute to the overall capability of the ADF. Building and maintaining the ADF’s capability is important to the defence of Australia and in enabling the ADF to fulfil Australia’s international obligations. Major defence capital acquisition projects require the allocation of considerable public resources which, if not well-managed, are not readily or easily replaced by a new allocation.

11. It is likely that most members of the Australian public have little perception of the size of the DMO project portfolio and the number of capital acquisition projects under DMO management at any given time—about 180 major projects (each costing over $20 million) and about 100 minor projects. Not only is it self-evidently important that such major projects are completed as effectively and efficiently as possible, but it can also become a matter of considerable public interest and debate if projects fail to deliver. At worst, such failures can give rise to gaps in defence capability.

12. DMO has identified the failure of the Super Seasprite helicopter acquisition project as motivating it to introduce Gate Reviews to mitigate project risks and to obtain assurance of a project’s continued viability.⁴ This type of review is not novel and is a widely used project assurance technique in both the public and private sectors. Since first recognising the need for this approach, DMO has taken several years to implement the program and, during the period July 2008 to July 2011, the quality and rigour of the Gate Reviews conducted varied, as did the number being undertaken. However, DMO has now developed an approach that is providing better assurance, with potential to improve the management of its projects.

13. DMO has improved the effectiveness with which it implements its Gate Review program since establishing its Independent Project Performance Office (IPPO) in July 2011, and centralising the management of all Gate Reviews within IPPO. As a result of these changes the program has strengthened and is better regulated. However, there remain opportunities for further improvement and rigour, especially in ensuring the follow-up of Gate Review outcomes.

14. From the outset DMO intended to build the program to undertake one Gate Review each year for each major project. This could require around 200–220 Gate Reviews a year and includes Gate Reviews for major projects under direct DMO management as well as Gate Reviews of an additional 20–40 projects in which DMO has an involvement but are under the direct management of other Defence groups. DMO informed the ANAO that it plans to reduce the total number of Gate Reviews required by excluding those projects close to completion where the benefits of a Gate Review are estimated to be low.

15. DMO’s capacity to execute the number of Gate Reviews it wishes to undertake is constrained by the small pool of skilled resources (including the group from whom it selects the external members to place on Gate Review Boards) and the substantial involvement of senior management in Gate Reviews. This introduces a degree of fragility into the arrangements, an aspect which warrants monitoring by senior management to ensure that the expected benefits can be delivered by this program.

16. DMO has not yet established a reliable monitoring system to ensure that actions flowing from a Gate Review are completed by those assigned responsibility for them. This means that there is no guarantee the insights gained will be recognised and improvements required of projects will be implemented. This could detract from the overall effectiveness of the program. In addition, it inhibits the ability to measure the effectiveness of the program. DMO has acknowledged this shortcoming and has advised that it plans to address it.

17. DMO’s development and implementation of its Gate Review program has been slower than hoped for at the outset and the program has clearly been maturing to this point. The additional impetus and resources DMO has devoted since July 2011 are now starting to bear fruit. However, there would be merit in DMO taking steps to reinforce within the organisation those aspects of the arrangements that are now yielding value. This includes the relative independence of the management of Gate Reviews by IPPO from DMO’s operational divisions, IPPO’s authority to manage the program in accordance with DMO’s administrative policy, and communicating the issues—which Gate Reviews highlight—that lead to project problems. Further, it is necessary to ensure that regular project management and oversight processes continue to improve. Gate Reviews are a useful supplement, but no substitute, for effective day-to-day management of projects.

18. The ANAO has made three recommendations, relating to:

• ensuring that a control mechanism is deployed to monitor the status and completion of actions arising from Gate Reviews;
• planning for and, in due course, evaluating the Gate Review program; and
• implementing a mechanism to identify from the Gate Review program the risks that arise in the management of capital acquisition projects.

Key findings by chapter

Implementation of Gate Reviews (Chapter 2)

19. DMO’s Gate Reviews are an internal assurance process which include elements of external scrutiny and independence. This process is not a replication of the Commonwealth-wide Gateway Review Process, a wholly external review process. Rather, DMO advises its methods are partly derived from Gateway and partly from practices observed by DMO executives in private organisations.

20. In July 2008, DMO senior managers advised a Parliamentary Committee that the expected benefits of the Gate Review program would include: external scrutiny of projects; detailed, analytical executive-level review; risk mitigation; and improved advice to government as to the health and outlook of projects.

21. DMO held its first two Gate Reviews in August and September 2008, shortly after giving this advice. They took place before the development of any administrative policy prescribing how they were to be conducted. No further Gate Reviews were held until September 2009.

22. After the publication of the Going to the Next Level report (Mortimer Review) in September 2008, Gate Reviews became associated with that review although none of the Mortimer Review’s formal recommendations refer to Gate Reviews. However, the report did note that DMO was implementing Gate Reviews and suggested that they be expanded.⁵

23. In August 2009, DMO assigned responsibility for the Gate Review program to its Major Program Control Branch and, during the remainder of 2009, held 18 Gate Reviews. On 16 December 2009, the then CEO DMO approved DMO’s Gate Review policy, which was formally released as a Defence Materiel Instruction. The frequency of Gate Reviews increased steadily during 2010, with most being coordinated by the DMO’s six divisions until July 2011.

24. DMO’s Gate Review program was reinvigorated following a ministerial announcement on 6 May 2011. In response to this announcement and consistent with a recommendation of the Mortimer Review, DMO established its Independent Project Performance Office (IPPO) in July 2011, and centralised management of all Gate Reviews within this office. Under IPPO’s management, standardised practices and better recordkeeping have become evident. However, DMO’s target of holding Gate Reviews at a rate of at least one Gate Review per major project per year is not yet attainable unless it can increase its current effort. It is likely that DMO will conduct a total of 125 Gate Reviews in 2011–12 from approximately 200–220 eligible projects within its purview.⁶

25. Since the original Gate Review administrative policy was approved, the program has evolved and IPPO has attempted to incorporate developing practice by updating the policy in two separate draft amendments. Towards the end of the audit, in late April and early May 2012, two further amendments to the policy were approved.⁷ DMO’s compliance with the administrative policy reflected in these latest amendments has not been specifically assessed in this audit.

Conduct of Gate Reviews (Chapter 3)

26. ANAO obtained records from DMO for 163 Gate Reviews held between July 2008 and 31 December 2011. Of these records, 35 (or 21 per cent) were incomplete. ANAO analysed the conduct of the remaining 128 Gate Reviews, for which complete records were available, to determine the level of compliance with the following essential characteristics:

• the presence of a documented preliminary analysis of the project, conducted before the Gate Review, by someone not involved in the project;
• the presence of external members on the Gate Review Assurance Board; and
• that the chair was sufficiently senior and was not directly responsible or accountable for the project under review.

27. ANAO found that, between August 2009 and July 2011, the conduct of Gate Reviews varied according to the acquisition category (ACAT level—broadly, the scale and complexity) of the project being reviewed. During this period there was, in effect, a two-tiered Gate Review program. Most Gate Reviews of the largest and most complex projects—ACAT I and II—were coordinated by DMO’s Major Program Control Branch (IPPO’s predecessor), and most Gate Reviews of the smaller and less complex ACAT III and IV projects were coordinated by the divisions in which the projects resided.

28. During this period, Gate Reviews of ACAT I and II projects were generally chaired by a General Manager with no responsibility for the project under review, informed by a documented preliminary analysis conducted by someone not involved with the project, and considered by a Gate Review Assurance Board that included at least one external member. In comparison, Gate Reviews of ACAT III and IV projects were generally chaired by the Division Head responsible for the project under review, not informed by a documented preliminary analysis, but were considered by a Gate Review Assurance Board that included one external member. Most Gate Reviews for which DMO’s records are incomplete were reviews of ACAT III and IV projects held during this period.

29. In July 2011, DMO centralised the management of all Gate Reviews within IPPO. Under IPPO’s management, the variation has decreased between Gate Reviews for ACAT I and II projects and ACAT III and IV projects, and the administration of the program has improved: DMO had complete records for all but one of Gate Reviews held since IPPO assumed responsibility. Most of these Gate Reviews were informed by a preliminary analysis conducted by someone not involved with the project and were considered by a Gate Review Assurance Board that included two external members. However, for more than half of all Gate Reviews held during this period, the chair had some responsibility or authority for the project under review.

30. ANAO attended and observed eight Gate Reviews between September and November 2011. All the Gate Reviews attended were supported by a comprehensive preliminary analysis. Most were well attended with an appropriate level of representation from all the relevant stakeholder groups within Defence (including, for example, representatives of the Capability Manager and Capability Development Group). The discussion at each Gate Review appeared to be open and frank and aimed at resolving problems without blame.

31. Interviews with individuals involved in Gate Reviews (including external members, chairs and members of the executive) indicate a generally positive view of the program. All interviewees agreed that the preliminary analysis conducted by someone external to the project was the most important aspect of the process. Other positive views expressed related to the benefit of external members and centralising the management of Gate Reviews within IPPO. Negative comments referred to the absence of a mechanism to follow up action items, the risk of developing an over-optimistic perception that the Gate Review program will identify and address all problems, and the resource impact on project staff and senior management.

Monitoring and Reviewing Gate Reviews (Chapter 4)

32. Assessing the success of any initiative, and determining whether it is performing as expected, requires a means of measuring performance, preferably using objective indicators. DMO’s positive assessment of the success of Gate Reviews has hitherto been based on the professional judgement of those involved. Although relevant DMO staff have met to consider and evaluate the program on several occasions, they have not been able to base their assessment on any objective data. While DMO has recently started collecting data against a range of metrics, it will take several years to aggregate systematic information on the effectiveness of Gate Reviews in reducing project risk and improving project outcomes.

33. Gate Reviews are generating a large number of action items aimed at resolving problems and improving outcomes for the project under review. DMO has not yet established a reliable monitoring system to ensure that these actions are completed by those assigned responsibility for them. This could detract from the overall effectiveness of the program. In addition, it inhibits the ability to measure the effectiveness of the program. DMO has acknowledged this shortcoming and plans to address it.

34. There have been some incidental consequences of the Gate Review program. Some are positive, such as increased stakeholder engagement across Defence in projects, and some have been negative, such as variable use of the term ‘Gate Review’ both in DMO and in the broader Defence organisation, which risks devaluing the program. There is an opportunity for a consistent understanding of DMO’s Gate Review program to develop throughout DMO, now that it has an approved policy that reflects current practice.

35. Gate Reviews are a resource intensive activity. Sustaining the program with the current level of rigour and increasing the rate to one Gate Review per major project per year will be challenging. Using figures provided by DMO, ANAO estimates the cost of the program at some $6 million annually. This is a substantial figure, representing an additional investment in project management of 0.1 per cent of the total acquisition budget managed by DMO. To date, DMO has not conducted a formal cost benefit analysis of the Gate Review program.

Agency response

36. Defence provided the following response to the report:

Defence welcomes the ANAO audit report on Gate Reviews for Defence Capital Acquisition Projects and accepts all three recommendations. Defence acknowledges that the report provides opportunities to build on the progress already made in the delivery and assurance of capital acquisition projects. The ANAO acknowledge that the Gate Reviews are resource intensive. Implementing the Recommendations will require, in particular, additional resources to provide a more rigorous systematic program evaluation to measure the effectiveness of Gate Reviews as well as to include Gate Review risk analysis.

Defence notes that effecting fundamental change in an organisation as large and complex as Defence is a time consuming and challenging process. To ensure that the full benefits of effective Gate Reviews can be achieved, cultural change is required. The project management improvements flowing from the Gate Reviews need to be embedded into Defence practice. Defence has taken a measured approach to ensure the process is implemented effectively, incorporating lessons learned along the way. This measured approach to the introduction of Gate Reviews is now delivering the transformational behaviour that Defence had anticipated when introducing the Gate Review program.

Although this program introduction may have been deemed ‘slower than hoped’ by the ANAO, Defence notes that the ANAO has also acknowledged that the ‘additional impetus and resources DMO has devoted since July 2011 are now starting to bear fruit’.

Footnotes

[1]   The Australian Government’s Gateway Review Process is managed by the Department of Finance and Deregulation. It is a proprietary process developed by the UK Office of Government Commerce (OGC) which involves short, intensive reviews of projects by a team of reviewers not associated with the project. See ANAO Audit Report No. 22 2011–12, Administration of the Gateway Review Process, February 2012.

[2]   The formal objective of a Gate Review varies with the point in the project lifecycle at which it takes place. The current milestones and objectives are set out in an attachment to the DMO administrative policy for Gate Reviews and Appendix 1 of this report.

[3]   Joint Standing Committee on Foreign Affairs, Defence and Trade, Review of the Defence Annual Report 2006–07, Committee Hansard, 10 July 2008, p. 10.

[4]   The Super Seasprite project ran for more than 12 years, delivered no capability and, at cancellation, cost over $1.4 billion. See ANAO Audit Report No.41 2008–09, The Super Seasprite, 17 June 2009.

[5]   Mortimer, David, Going to the Next Level: the report of the Defence Procurement and Sustainment Review, 2008, p. 35.

[6]   This includes about 180 major projects under direct DMO control as well as 20 to 40 projects in the pre-second pass phase of capability development.

[7]   The current policy, approved in May 2012 differs significantly from the original approved policy and the two draft amendments which were the focus of this audit. The major changes in the latest approved amendment are outlined in Chapter 2.