Introduction

The Australian Government uses a wide range of business systems, typically to support program delivery and for internal management purposes. Because of new or changed programs, or to improve efficiency, government agencies often need to develop new business systems, or change existing systems. This is done by planning, approving and implementing a business system project.

These projects typically involve several steps. The project objectives and system requirements are researched and agreed. Next, the Information Communications and Technology (ICT) software for the business system is obtained by either purchase - perhaps with subsequent tailoring - or by development. Then the system is implemented, which involves preparing new procedures, installing any ICT equipment required, providing training to the people who use or are affected by the new system, and transferring or entering all the data needed. Business benefits are more likely to be achieved if both ICT and business activities are effectively planned and carried out.

Significant expenditure is involved - for example, in 2007–08 Australian Government agencies subject to the Financial Management and Accountability Act 1997 spent an estimated $1 billion on the creation of new ICT capabilities.

Audit scope and objective

The objective of this audit was to assess whether selected agencies effectively managed the initial planning and approval of small and medium sized business system projects.

The focus of the audit was on:

• projects with a business objective, such as implementing a new program, rather than a technology focus, such as replacing an agency's desktop computers;
• the approval stage of business system projects as this is a significant contributor to project success¹; and
• agencies which generally undertake ICT projects valued at under $10 million each, as these small and medium value projects are generally not the subject of other reviews.

The three Australian Government agencies involved in this audit were the former Department of Education, Science and Training², the Department of Health and Ageing, and the Department of Veterans' Affairs.

To address the audit objective, in each of the audited agencies the ANAO:

• reviewed the arrangements for business system approval (such as policies and governance structures) in comparison with good practice;
• reviewed the operation of the arrangements for business system approval in practice (by examining a selection of business cases, and governing committee minutes); and
• identified the results for some projects in comparison to the approval.

Following the conduct of audit fieldwork, each audited agency was provided with a management report setting out audit findings, conclusions and recommendations for improvement.

In examining the arrangements and approvals, the ANAO considered there are three key aspects to the effective initial planning and approval of a business system project:

• the decision-maker being provided with appropriate information on which to make the decision - typically in a project business case, or investment proposal;
• the project being defined with a focus on business results, rather than on ICT systems; and
• decisions being clear, and documented.

Conclusion

Business systems are used by the Australian Government to support program delivery and for internal management. Each year government agencies spend an estimated $1 billion on new ICT capabilities. This audit reviewed the approval of small and medium sized business system projects. Projects of this size often support targeted Government initiatives and improvements to internal efficiency that tend not to be subject to the same level of review as larger projects.

Overall, the ANAO concluded that the three agencies subject to audit were in most cases planning and approving the ICT component of business system projects adequately. However there was scope to improve the planning and definition of the business component of projects and to undertake better quality assurance of proposals. In addition, improving the clarity of project approval decisions, particularly in setting out the planned business benefits, would increase the focus on achieving business benefits during implementation and subsequent operation.

The three agencies had each approved governance arrangements for business system projects which, with minor exceptions, were soundly designed. They had also periodically reviewed and improved their arrangements. The governance arrangements were usually followed, although several projects inappropriately bypassed the approved processes.

Agencies had clearly established the reasons for the 62 projects examined during this audit, with the planned business benefits described in most cases. However, the benefits were often described in general terms without specific targets set, and without a specific plan to measure the benefits. In most cases the ICT aspects of the projects were appropriately defined. However, in many cases, business activities needed to achieve the identified benefits were not included in the project scope. Addressing these issues early in planning, by setting indicative targets for benefits and checking the scope of work, would assist with decisions on which proposals go to the next stage of planning.

In many business cases, there was inadequate information for a 'reality check' of the proposed approach, planned benefits, and the likely cost and timetable. For example, information about project assumptions, the analysis of options, addressing identified risks and stakeholder issues, and lessons from similar projects was often incomplete or missing. Greater emphasis given to gaining assurance around these critical elements of the business case is likely to improve the rate of project success. Accordingly, a priority for improvement during detailed planning is the validation of the proposal, so decisions can be made using reliable information.

The ANAO examined the outcomes for 13 projects. Most of these projects delivered the intended ICT systems and functions at close to the forecast cost. A number of projects successfully met challenging schedules. However, half of the projects took 50 per cent or more longer than planned.

Notwithstanding the ICT successes, in many cases the forecast business benefits - such as cost saving or improved service - had not been demonstrated. Contributing factors to this include the absence of approved targets for benefits, and a lack of checking during implementation of progress toward planned benefits. Accordingly, recording the project approval concisely, with targets for interim and long term benefits, would help focus the attention of those proposing, approving and implementing the project on its fundamental purpose.

The ANAO made 3 recommendations designed to improve agencies' approval arrangements for business system projects.

Key findings by chapter

Governance arrangements (Chapter 2)

All three agencies had developed policies, procedures, guidance material and committee structures to support the initial planning and approval of business system projects. Typically there was a committee focused on strategic issues related to information, technology and business investment. This committee was supported by operational committees focused on developing strategic options and assessing and making recommendations on business technology investments.

Overall the three agencies had arrangements for business system approval that were designed appropriately, with scope to improve the focus on achieving business benefits. The ANAO found that the agencies had:

• established their governance arrangements for business system approval following reasonable research and review;
• designed the processes and committee structures of the approval arrangements reasonably, although in two agencies there was insufficient coordination with their annual business planning;
• identified key issues for consideration at initiation reasonably, with scope for improvements to guidance material - particularly for business benefits; and
• provided adequate practical support for the approval arrangements.

The processes and committees generally operated as designed. However, in several cases, projects bypassed the expected approval processes.

Developing and approving business cases (Chapter 3)

The three agencies had considered several hundred project proposals over the period 2004¬–05 to 2007–08. The ANAO examined the initiation of 62 projects which had a business system - rather than ICT infrastructure - focus. These 62 projects were estimated to cost $152 million.

The ANAO assessed the information provided to decision-makers and reviewers when they were asked to approve business system projects. The ANAO assessed the presence and quality of 29 elements of information relevant to sound approval, such as the project scope, the business benefits, the identification of risks, and cost benefit analysis. The findings on these 29 elements are summarised in Table 1 under five broad headings, namely information about the project's reason, specification, implementation planning, validation, and authorisation.

Table 1 Areas of initiation, and overall summary assessment for the 62 projects examined

Source: ANAO analysis.

Satisfactory levels of information were provided on the reason and implementation planning of the projects. There was adequate authorisation of the projects.

There were less satisfactory findings on the quality of the specification and validation of the projects. The more significant opportunities for improvement were:

• Most projects did not include arrangements to measure and monitor planned business benefits; with the risk these benefits would not be achieved.
• Some projects had not described the processes to be automated clearly enough to allow firm costing, and many did not describe other important requirements, such as work volumes, expected system life and security, that also affect the project feasibility and cost.
• Many projects had not clearly identified the key project assumptions, leaving a risk that the project was underpinned by invalid assumptions.
• Most project proposals did not report research on similar projects to help provide a 'reality check' on the project timetable, cost and benefits.
• While most projects properly identified risks and potential stakeholder issues, they had not made sufficient allowance to address them

Project results (Chapter 4)

The ANAO assessed the results of 13 of the 62 projects in comparison to the approved budget, schedule, and objectives.

Most of these projects were completed with costs close to the planned budget, and several were under budget. However, one project cost more than double the planned budget. In terms of schedule, half of the projects took 50 per cent or more longer than planned. Nevertheless, projects to deliver new programs to the public generally provided the most important features on time - often to challenging schedules.

Project steering committees in most instances monitored implementation progress, such as the development and testing of software modules to provide specific functions. However, they generally did not assess progress toward approved business benefits, such as 'faster reporting' or 'reduced processing costs'.

In terms of objectives, the projects generally delivered their planned ICT systems and functions. In most cases, business benefits were broadly identified, but were not set on a basis that would allow definite assessment of project success. Moreover, agencies had not assessed the business benefits at the end of most projects.

Recommendations

The report makes three recommendations based on findings from fieldwork at the audited agencies. These are likely to be relevant to all APS agencies. Therefore, all APS agencies should assess the benefits of implementing the recommendations in light of their own circumstances, including the extent that each recommendation, or part thereof, is addressed by practices already in place.

Summary of agencies' responses

Each of the audited agencies, together with the Department of Finance and Deregulation, agreed with the three recommendations.

The Department of Education, Employment and Workplace Relations advised:

The Department of Education, Employment and Workplace Relations (DEEWR) accepts the findings presented in the proposed report.

The Department notes that overall the ANAO considered that the governance framework at DEST (upon which the current DEEWR framework is based) for the initiation of business system projects was soundly based.

We accept the need for continued efforts to improve IT governance frameworks, noting especially the recommendation to improve the focus on intended business benefits.

We are currently reviewing and improving our IT Investment framework for the start of the 2009–10 investment round and the recommendations from this audit will provide a valuable input into this process.

The Department of Health and Ageing advised:

Considerable improvement has been made by the Department of Health and Ageing in project planning and approval in recent years. Implementing the recommendations made in this report, particularly the strengthening of governance and quality assurance processes for business case validation and approval, provide the impetus for further improvement and increased project success.

The Department of Veterans' Affairs advised:

The Department of Veterans' Affairs agrees with the overall findings and recommendations of the ANAO report, in particular, that the arrangements for business system approval were designed appropriately, with scope for some improvements to make them more effective. Since establishing the Project Management Office in 2006, DVA has taken positive steps to strengthen the project governance and management framework. The report's findings highlight areas where the Department's ongoing commitment to strengthening the governance framework is improving the quality of project governance in the initiation and implementation stages of projects, as well as identifying better practice in project governance throughout the project life cycle.

Department of Finance and Deregulation

The Department of Finance and Deregulation (Finance) supports the report and endorses its recommendations. As noted in the report, Finance has prepared extensive guidance on the Government's ICT investment framework. This includes a guide to developing business cases for ICT proposals. A key part of the ICT investment framework is the ICT Two Pass Review process, agreed by Cabinet in April 2008.

The ICT Two Pass Review process applies to ICT-enabled proposals that: have a total cost estimated to be $30 million or more, including ICT costs of at least $10 million; and involve high risk.

Footnotes

^[1] The audit has not reviewed post-approval activities such as project management, and financial management.

^[2] The audit fieldwork was undertaken at the former Department of Education, Science and Training (DEST), which became part of the new Department of Education, Employment and Workplace Relations in December 2007.