Introduction

1. The Gateway Review Process is a project assurance methodology developed in the United Kingdom to improve the delivery of major projects. It involves short, sharp and confidential reviews conducted by reviewers not associated with the project at six key stages of the project lifecycle, also known as Gates. At each Gate the review is aimed at:

• assessing the project against its specified objectives at a particular stage in the project’s lifecycle;
• providing an early identification of the areas that may require corrective action; and
• providing validation that a project is ready to progress successfully to the next stage.¹

2. Gateway is not an audit, a detailed technical review or an inquiry. The intention is that Gateway reviews will identify and focus on issues that are most important to a project, so that the project team’s effort is directed to those aspects that will help make the project successful.²

3. Gateway reviews are coordinated by the Gateway Unit in the Department of Finance and Deregulation (Finance). This includes: verifying the risk scores of projects meeting the financial thresholds; the selection, training, appointment and coordination of the Gateway Review Teams; the development of guidance to support agencies; and the dissemination of lessons learned from Gateway reports.³

4. The Australian Government phased in Gateway over three years from the 2006–07 Budget. After initially commencing with a small number of high risk projects, Gateway was expanded to cover all projects of $10 million and over for IT projects, and $20 million and over for other procurement and infrastructure projects, assessed as representing a high risk in Financial Management and Accountability Act 1997 (FMA Act) agencies.

5. At the conclusion of a Gateway review, and following a briefing by the Review Team on the review’s conclusions and recommendations, a report is provided to the Senior Responsible Official (SRO) within the project’s administering agency on the last day of the onsite review activity.⁴ A copy of the final review report is also provided to the Gateway Unit. A red/amber/green (RAG) rating is used to indicate the overall assessment of delivery confidence for the project. Until mid 2009, the traffic light system related to the urgency of addressing report findings. This has now changed to relate to the confidence in delivering the project outcomes. Table S.1 outlines the previous and current definitions for these ratings.

Table S.1: Gateway review report rating definitions

Source: ANAO analysis of Finance publications.

6. Over the first five years of operation (to 30 June 2011) of the Gateway review process in the Australian Government, a total of 150 Gateway reviews were completed. These reviews related to 46 ‘high risk’ projects estimated to cost in aggregate in excess of $17.6 billion at the time the projects commenced. The projects that have been subject to Gateway reviews have been predominately information communications and technology (ICT) projects (74 per cent of all projects). Gateway has also been applied to a number of infrastructure projects (22 per cent of all projects) and procurement projects (four per cent of all projects).

7. Gateway reviews were forerunners to a range of measures being taken by government to provide greater assurance that government policies could be implemented in line with government expectations and agency plans. The Finance Minister recently announced further measures designed to provide independent assurance on how well practical delivery issues are being addressed in preparing for implementation in relation to certain high risk proposals.⁵

Audit objective and criteria

8. The objective of the audit was to examine the effectiveness of the administration of the Gateway review process by Finance and FMA Act agencies. The audit also examined the extent to which those Gateway reviews that have been conducted have contributed to improvements in the delivery of major projects undertaken by FMA Act agencies.

9. The audit examined the operations of the Gateway Unit within Finance as it relates to the design, implementation and ongoing administration of Gateway. All aspects of the Gateway Unit’s operations were examined including: the selection and management of the Review Teams; support and guidance provided to agencies; the dissemination of lessons learned across the Australian Public Service (APS); and the continual improvement of Gateway. The audit examined all projects that had been the subject of Gateway reviews (to 30 June 2011) across 23 agencies.

Overall conclusion

10. Overall, the Gateway review process has been effectively implemented within the Australian Government. There has been a focus on high risk, high value projects with 46 projects valued at more than $17 billion across 23 agencies examined in the first five years of Gateway’s application. In the first three years, about one in every five reviews identified that there were significant issues that needed to be addressed before the project proceeded further. In the last two years⁶, there have been no reviews that have identified major issues requiring urgent action.⁷ However, participation in the Gateway review process does not guarantee success in meeting specified project objectives. At least three of the nine projects that have completed the full suite of Gateway reviews up to 30 June 2011 were not completed on-time and on-budget and/or did not deliver the outcomes expected when funding was approved.⁸

11. Because Gateway is a licensed process used with permission, there has been a relatively high degree of methodological standardisation across jurisdictions, including within the Australian Government. Nevertheless, the Gateway review methodology has been tailored to the Australian Government context. This has included recent changes to the basis for allocating risk ratings in review reports and, since late 2008, escalation processes to provide added visibility at senior levels within the agency. In certain circumstances, the relevant Minister is informed where the Gateway review rating indicates a project is experiencing difficulties.

12. While the processes used to determine which projects are to be subject to Gateway reviews were generally effective, several projects that met the criteria were not subject to Gateway reviews due to the timing of their risk assessments.⁹ In addition, there would be benefits in Finance periodically examining the outcomes of those projects excluded from Gateway on the basis of their assessed level of inherent risk so as to be assured that the risk assessment process is working as intended. Accordingly, the ANAO has made two recommendations aimed at enhancing the screening processes for targeting Gateway reviews at all high value, high risk projects.

13. A key factor that influences the extent to which Gateway reviews are able to contribute to improved on-time and on-budget delivery of projects relates to the timing of reviews. There are often significant delays between Gate reviews, including as a result of reviews being rescheduled by the sponsoring agency. In light of steps taken or being considered in other jurisdictions that apply Gateway to avoid extended periods between Gates, the ANAO has recommended that there is a role for Finance in monitoring extended periods between Gateway reviews.

14. Further, as noted at paragraph 10, performance by completed projects that have been subject to Gateway has been mixed. However, at present, the extent of any delays, cost overruns and shortfalls in delivery of outcomes in projects subject to Gateway is not quantified by Finance, and is not currently considered in any assessment of the overall contribution of Gateway to improving project planning and implementation by agencies. Accordingly, the ANAO has recommended that greater attention be paid to assessing whether the Gateway review process is contributing to improved project delivery by agencies.

15. The contribution that Gateway makes to improved project delivery depends, to a significant extent, on agencies promptly progressing the issues raised in reviews. However, it is common for agencies to not fully implement review recommendations in a timely manner.¹⁰ Of particular note in this respect is that demonstrable action in response to a Gateway review report often does not occur until shortly before the next review is undertaken.¹¹ By rescheduling a review, there is less visibility over any delay with the agency taking action in response to the prior review report. Accordingly, and without detracting from the responsibility of individual agencies to manage and deliver their projects, the ANAO has recommended that Finance promote a greater focus on the timely and effective implementation of Gateway review report recommendations by agencies.

Key findings by Chapter

Gateway project selection (Chapter 2)

16. The Gateway review process focuses on higher value projects where there are high risks. A risk assessment tool is completed by individual agencies, and verified by the Gateway Unit, to decide which projects will be subject to Gateway. For the majority of the 198 projects with an estimated value above the relevant financial threshold (134 projects or 68 per cent), neither the agency’s initial risk assessment nor the Gateway Unit’s final risk assessment required that the project be subject to Gateway reviews.

17. Gateway Unit moderation of agency assessments has promoted consistency in risk assessment. In this context, the Gateway Unit has changed the agency’s indicative rating in respect of one in every five projects. Most commonly, the Gateway Unit has increased the risk rating. Of the 64 projects with a final assessment rating of ‘high’ and thereby requiring their participation in the Gateway process, 27 (42 per cent) were only included in Gateway as a result of Finance increasing the indicative self-assessed rating of the project’s sponsoring agency.

18. Whilst providing a clear basis for the selection of projects for Gateway reviews, there is one aspect of the screening process that could be improved. Currently, projects are not being subject to Gateway review if the relevant agency has not submitted a risk assessment to Finance prior to seeking government funding through a New Policy Proposal. There would be benefits in options being examined that would ensure that any decisions to exclude a project from Gateway are being taken by relevant Ministers.

Reviewer selection and engagement (Chapter 3)

19. The success of the Gateway review process depends to a significant extent on the qualities of the persons engaged to lead and undertake individual reviews. In the majority of cases, Gateway reviewers from both the public and private sector have been selected from a list of suitably qualified and experienced persons recorded in a database developed and maintained by the Gateway Unit. Potential reviewers from the private sector are able to apply at any time to be included on the database, and APS staff are also able to apply to be registered.

20. Training of Gateway reviewers is provided through an accredited training provider, with the course material based on that developed for the Victorian Government. As at 30 June 2011, some 85 per cent of reviewers on the Gateway Unit database (375 of 440) were recorded as having participated in this training, with almost all participants in recent years rating this training as ‘very effective’.

21. A high quality pool of suitably qualified and trained Review Team Leaders and Review Team Members is integral to the success of Gateway. In selecting the composition of individual Gateway review teams, including the team leader, the Gateway Unit balances a range of appropriate considerations, including the experience and expertise of potential candidates and the views of the sponsoring agency for the project. Agencies (through the SRO) have reported to Finance a high level of satisfaction with the quality of the review teams and leaders.

22. To date, there has been significant reliance on private sector reviewers, with targets for participation by APS staff not being met.¹² In addition, to date, only one review has been led by a member of the APS, and there has been a high degree of reliance on a small number of private sector participants to lead individual reviews. In recent years, Finance has revised its targets for APS participation on three occasions, and taken steps to increase the number of APS staff registered on the database as potential Gateway reviewers. APS staff participation on Gateway reviews increased in both 2009–10 and 2010–11, although to a level still below the current target.

Conduct of Gateway reviews (Chapter 4)

23. Funding is provided to Finance to meet the cost of Gateway reviews. Factors such as the timing of government decisions and individual project progress mean that actual expenditure will rarely match the amount of funding made available to Finance at the beginning of each year. Finance is nevertheless required to commission all reviews as and when they are required. In recent years, the direct costs of Gateway reviews have been significantly less than that budgeted, due to fewer reviews being undertaken than had been forecast and the cost of each review being less than that expected. In addition to the direct costs of Gateway reviews (for which Finance receives specific funding), Finance also has to meet the costs of oversighting and coordinating the Gateway review process. In this context, Finance advised the ANAO that it had contributed around $1.5 million in general departmental internal funding to Gateway over the years 2005–06 to 2010–11, to supplement government appropriations so as to ensure that the objectives of Gateway are met.

24. The scheduling of reviews is determined in consultation between the Gateway Unit and the sponsoring agency. It has become common for planned reviews to be rescheduled, with about one-third of all planned reviews being rescheduled. Some reviews were scheduled and rescheduled up to six times. There have also been long delays between reviews. For example, nine of the 29 ongoing projects as at 30 June 2011 had not been the subject of a review for at least 12 months, including one project that was last reviewed more than three years ago. In addition, the duration between reviews has tended to increase as a project progresses through the sequence of Gates. Accordingly, to provide assurance that the high risk projects subject to Gateway are being progressed in accordance with the stated objectives and expected time, cost and quality parameters approved by government, there may be benefits in annual Gateway reviews being conducted for projects where there would otherwise be an extended delay between reviews.

25. Compared with other jurisdictions that have adopted Gateway, fewer Australian Government projects receive an amber or red rating. The rating approach was changed in mid-2009 from relating to the urgency with which report findings and recommendations were to be addressed, to be an assessment of the review team’s confidence that the project will be delivered in a timely and effective manner. Since that change was made, the proportion of reports receiving a green rating has doubled to over 60 per cent and there have been no reports allocated a red rating (previously about one in every five reports received a red rating). Figure S.1 shows the percentage of reports issued each year that received red, amber and green ratings respectively.

Figure S.1: Gateway Report RAG Ratings for the years 2006–07 to 2010–11

Source: Source: ANAO analysis of Finance data.

26. This trend could be seen as indicating that the management of ICT and infrastructure projects by FMA Act agencies has improved in recent years, and that FMA Act agencies are better at project management than agencies in other jurisdictions applying the Gateway review methodology. However:

• such a trend is not consistent with the results and recommendations of the Gershon review¹³, the APS Reform Blueprint¹⁴ or successive State of the Service reports¹⁵;
• Finance has advised the ANAO that it would not be advisable for Gateway review results to be extrapolated to all FMA Act agency projects as Gateway reports are specific to a single project, and its circumstances at a point-in-time; and
• at least three of the nine projects that have completed the full suite of Gateway reviews up to 30 June 2011 were not completed on-time and on-budget and did not deliver the outcomes expected when funding was approved.

27. In this latter respect, of the nine projects that had completed the full suite of Gateway reviews as at 30 June 2011, Finance’s assessment was that six projects were on-time, on-budget and delivered the expected outcomes. One project was completed ahead of schedule, within the revised budget but delivered reduced outcomes compared with those expected when funding had been approved.¹⁶ Two projects were delivered late and one of the late projects provided only partial delivery of the expected outcomes.¹⁷ The extent of the delays, cost overruns and shortfalls in delivery of outcomes in these projects was not quantified by Finance, and is not currently considered in any assessment of the overall contribution of Gateway to improving project planning and implementation by agencies. In this context, there would be benefits in Finance tracking and periodically reporting on the impact that the Gateway review process has had on improving the outcomes of completed projects.

Agency actioning of Gateway review reports (Chapter 5)

28. Gateway was one of two processes introduced in the latter half of the last decade aimed at providing improvements in cost certainty and to facilitate greater scrutiny of projects.¹⁸ In this context, Gateway reviews are expected to provide an arm’s length assessment of a project at critical stages of the project’s lifecycle so as to:

• assist with the on-time and on-budget delivery of projects in accordance with their stated objectives; and
• strengthen project oversight and governance.

29. Consistent with the philosophy underpinning the development of Gateway, Finance’s implementation of the Gateway Review process has endeavoured to reinforce the responsibility of individual agencies for the management and delivery of their projects. Accordingly, in general, agencies are provided with considerable discretion in how they administer projects subject to Gateway and the extent to which Review findings and recommendations are made known outside of those with management responsibility for the project. However, there are two scenarios in which added attention is expected to be brought to bear but in neither circumstance has administration of the relevant arrangements been sufficiently effective. Specifically:

• a small number of projects (12 of the 46 projects subject to Gateway up until 30 June 2011) have been categorised as ‘mission critical’.¹⁹ Whilst certain minimum governance and reporting arrangements are expected to apply to these projects, it has been relatively common for these arrangements to not be implemented; and
• where the Gateway Review rating²⁰ indicates a project is experiencing difficulties, escalation processes have been in place since late 2008 to provide added visibility over project progress at senior levels within the agency and, in certain circumstances, the relevant Minister.²¹ As at June 2011, the Enhanced Notification process had been triggered for ten projects. Whilst the process has largely been followed and subsequent ratings have generally improved, the process also appears to have impacted on the way in which the Gateway Review process is being administered. In particular:
• the number of projects subject to the Enhanced Notification process has been reduced through the practice adopted in some projects of repeating a gate (sometimes on more than one occasion) thereby obtaining an improved rating²²; and
• although the Gateway review process is designed to provide an external challenge to the robustness of plans and processes on a ‘going forward’ basis so as not to unduly impact project schedules, it has become common for scheduled reviews to be deferred until such time as the agency is confident that the review rating will not result in the Enhanced Notification process being triggered. Prior to its introduction, about one in 20 Gateway reviews was deferred at the request of the relevant agency. Since November 2008, almost half of all reviews have been deferred at least once.

30. The Enhanced Notification processes were intended to provide greater visibility over projects experiencing problems as well as to promote early remedial intervention. In most instances, the escalation processes are only triggered by consecutive red or amber ratings. Accordingly, the design of the Enhanced Notification processes did not envisage a high frequency of gates being repeated or of scheduled reviews being deferred. More broadly, Gateway is best able to provide assurance and improve project management through the timely conduct of Gate reviews combined with prompt agency action being taken prior to the next review.

31. Based on the feedback provided by agency SROs, there is a high level of satisfaction with the quality of the review teams that are undertaking Gateway reviews. In addition, the significant majority of SROs have reported to Finance that they find Gateway reviews beneficial and consider that the recommendations made in the Gateway report will enable them to achieve improvements in project outcomes. That said, it is common for agencies to not fully implement Review recommendations in a timely manner.

Summary of agency responses

32. A number of agencies provided summary comments on the audit report, as follows.

Department of Finance and Deregulation

Well targeted and applied assurance through Gateway Reviews plays an important role in assisting agencies deliver government initiatives of high value and risk, as demonstrated through the consistently positive Senior Responsible Official feedback.

The report notes that the Department of Finance and Deregulation (Finance) has been active and progressive in managing the Gateway Review Process.²³ This includes enhancing Gateway’s effectiveness through improving the methodology, its integration with other assurance processes and its administration (for example by increasing the participation of Australian Public Service senior executives in Gateway reviews). The costs of Gateway have also been reduced.

Finance supports the recommendations.

Australian Taxation Office

The Australian Taxation Office (ATO) supports the conclusions and recommendations made in the Australian National Audit Office’s Performance Audit of the Gateway Review process.

The ATO agrees that the Gateway Review process has been effectively implemented and believes that the improvements recommended in the audit will further enhance the Gateway Review process.

ComSuper

Comsuper has had one project, the Administration Platform Modernisation (APM) project, which has been subject to the Gateway Review Process. ComSuper welcomes the ANAO report and supports all recommendations aimed at enhancing the Gateway Review Process.

Department of Education, Employment and Workplace Relations

The report provides an informative analysis of the Gateway Review Process and the department supports the outcomes expressed to improve current practices as outlined in the recommendations. The department’s participation in this audit has provided an opportunity for beneficial internal discussion on future enhancements to Gateway Reviews and the Gateway Review Process.

Department of Industry, Innovation, Science, Research and Tertiary Education

While none of the recommendations of the report require specific action from the Department of Industry, Innovation, Science, Research and Tertiary Education, we would welcome working with the Department of Finance and Deregulation in streamlining these processes.

Department of Regional Australia, Local Government, Arts and Sport

The department agrees with the five recommendations aimed at improving the risk assessment process as outlined in the report, noting that they are to be actioned by the Department of Finance and Deregulation. The Department of Regional Australia, Local Government, Arts and Sport acknowledge the Gateway Review Process as an important mechanism for monitoring and assurance of major Government projects and support its continued refinement.

Department of the Treasury

The Treasury strongly supports the Gateway Review Process and notes that it has been particularly useful in assisting the implementation of Standard Business Reporting. The Treasury supports the audit report’s recommendations, which we consider will further enhance the Gateway Review Process.

Department of Veterans’ Affairs

The Department of Veterans’ Affairs (DVA) acknowledges the value of the Gateway process as a project assurance methodology, noting that it does not purport to be an audit nor a detailed technical review. The Gateway reviews conducted on DVA’s Choice and Maintainability in Veterans’ Services (CMVS) Program in 2010 were constructive in (1) affirming the program’s continuing strategic fit with the department’s longer term strategic objectives and those of the Australian Government, (2) the need for an additional on-line channel to support our client base, and (3) acknowledging our Minister’s understanding and support for the program. The reviews also assisted in cementing Executive support for the program, with interviews of several Executive level officers below the Senior Responsible Official (SRO), and they confirmed the soundness of the business need and the robustness of the business case for the program.

DVA also considers that the reviews did assist in providing an additional management focus on specific program elements, which enabled timely risk mitigation activities to be exercised. For example, the need for increasing resourcing and a focus on a detailed final ICT enterprise architecture for the program, both of which were subsequently addressed as program priorities.

In general, DVA agrees with each recommendation stemming from the review, noting that, whilst they are principally directed to the Department of Finance and Deregulation, they have potential impacts on agencies in terms of increasing the number of reviews through an annual review framework, and greater project reporting by agencies to enable central agency measurement of project outcomes against original project objectives.

Footnotes

[1] Department of Finance and Deregulation, FMG 20, Guidance on the Gateway Review Process – A Project Assurance Methodology for the Australian Government, August 2006.

[2]   Department of Finance and Deregulation, Gateway Review Process – Overview for the Senior Responsible Official, November 2009.

[3]   During 2010–11 the Gateway Unit had an average of about seven full-time equivalent staff. In July 2011, the Gateway Unit was renamed the Assurance Reviews Branch.

[4]   Onsite review activity typically takes less than five days to be completed.

[5]   Speech by the Minister for Finance and Deregulation, Better Government: Improving Program Implementation and Delivery, Institute of Public Administration Australia Forum, Canberra, 15 November 2011. One measure is Implementation Readiness Assessments, which focus strategically on key areas in determining the capability and preparedness of the agency in planning to implement a proposal and, in doing so, provide agencies with the opportunity to gain independent assurance on how well practical delivery issues are being addressed in preparing for implementation. The process draws from the methodology contained in the Better Practice Guide on Implementation of Programme and Policy Initiatives: Making Implementation Matter (Australian National Audit Office and the Department of the Prime Minister and Cabinet, October 2006) and the United Kingdom’s Office of Government Commerce Gateway Review Process.

[6]   The rating approach was changed in mid-2009.

[7]   Gateway reports are specific to a single project and its circumstances at a point-in-time. Accordingly, Finance advised the ANAO that it would not be advisable for Gateway review results to be extrapolated to all FMA Act agency projects.

[8]   Each of these three projects received two or more Gateway review reports with an amber rating, and one project received a report with a red rating during the course of the project’s implementation. A further eight projects are expected to complete their involvement in Gateway during 2011–12.

[9]   The Gateway risk assessment is intended to be completed prior to a New Policy Proposal (NPP) being considered by government, with projects above the financial thresholds that are assessed as high risk expected to be subject to the Gateway Review Process. However, where a risk assessment has not been completed and provided to Finance prior to government consideration of the NPP, the project has not been subject to Gateway reviews.

[10]   Similarly, a recent investigation undertaken by the Victorian Ombudsman in consultation with the Victorian Auditor-General concluded that the effectiveness of the Gateway review process in that State was limited by its reliance on agencies engaging in and being supportive of the process, which often was not the case. The concerns identified included that whilst some review recommendations were addressed, others were ignored and that some agencies regularly continued to progress projects despite receiving red or precautionary amber lights. See further in Victorian Ombudsman, in consultation with the Victorian Auditor-General, Own motion investigation into ICT-enabled projects, November 2011.

[11]   Notwithstanding, in respect to 106 instances up to June 2011 where an earlier Gate review had been conducted, the review team reported in 72 per cent of cases that the recommendations from the earlier Gate had only been partially implemented.

[12]   Participation of reviewers from both the public and private sectors was expected to enhance the prospect that an adequate range of skills and experience would be brought to reviews, with the participation of APS staff also expected to contribute to the development of project management capability across FMA Act agencies.

[13]   A Review of the Australian Government’s use and management of ICT, Sir Peter Gershon, October 2008.

[14]   Advisory Group on Reform of Australian Public Administration, Ahead of the Game: Blueprint for the Reform of the Australian Public Service, March 2010.

[15]   Australian Public Service Commission, State of the Service Report, various years.

[16]   This project had received two amber reports, had cancelled one scheduled review (that was to repeat a Gate) and deferred another review (that repeated a Gate). Two reviews noted that previous review recommendations had only been partially implemented.

[17]   The first project received four amber reports, repeated one Gate and deferred another Gate review. Four reviews noted that previous recommendations had only been partially implemented. The second project received one red and eight amber reports, repeated one Gate once and another Gate three times, and deferred one review for a Gate five times. Five reviews noted that previous review recommendations had only been partially implemented.

[18]   The other was the two‐stage Cabinet approval process for capital works. See Finance advice to the ANAO reflected in ANAO Audit Report No. 43 2008–09, Construction of the Christmas Island Immigration Detention Centre, p.32. A two-stage process for ICT investment was also introduced in September 2009.

[19]   Defined as a project deemed essential to the Australian Government’s ability to successfully deliver a major legislative requirement or a major policy initiative; or a project for which failure would have catastrophic implications for the delivery of a key public service, national security or the internal operation of an FMA Act agency. The Gateway Unit advises agencies if their project is mission critical at the time they are notified that the project is subject to Gateway.

[20]   Changes were made to the RAG rating process in mid-2009 to, amongst other things, reduce the number of projects that would be subject to the Enhanced Notification process.

[21]   There are three escalation stages in the Enhanced Notification process. Where one red or successive amber reports are received, the Secretary of Finance writes to the agency’s Chief Executive Officer. At the second level, if further red or amber reports are received, agencies are required to prepare an action plan and inform their Minister, the Secretary of Finance and the Secretary of the Department of the Prime Minister and Cabinet of the problems and proposed solutions. The third level requires the agency to conduct an independent external project review.

[22]   In this respect, 13 projects (18 reviews) have involved repeated Gates. In 15 cases, the Gate review was repeated following the receipt of a red or amber rating for the initial review at that Gate level. In nine instances, the repeated gate resulted in an improved rating which, in five of those instances, enabled a project to avoid being subject to the Enhanced Notification process.

[23]   The overall conclusion formed by the ANAO is reflected in paragraph 10.