Background

Edge was a joint project between the Australian Government Department of Family and Community Services (FaCS) and Centrelink to develop an expert system for the Family Assistance Office (FAO). Edge was a processing application, for the administration of claims and payments for people applying for entitlement to family-related payments. SoftLaw, a private sector company, was the successful tenderer that supplied software and expertise for the development.

Development of Edge (then called the Life Events Expert System, LEES) commenced in March 2000. In June 2002, pilots of the system were assessed as successful, and progressive roll out of the system started in July 2002.

However, in August 2003, FaCS and Centrelink jointly commissioned a review of the project to assess whether there was a viable business case for further development of the Edge system. In November 2003, the report of that review recommended discontinuing development of the project. FaCS and Centrelink accepted the recommendation.

The development, from an initial trial in 1997 to the end of 2003, had taken some six years and come at significant cost—for example contractual payments to SoftLaw of around $30 million, and involving up to 150 staff from three organisations; FaCS, Centrelink, and SoftLaw. Information provided by FaCS and Centrelink, confirmed to the extent possible by the ANAO, estimated the total expenditure on the Edge project to be around $64.4 million, for the almost four years from contract signing to termination.

A timeline for Edge, together with legislative changes made to the relevant families assistance legislation during the project, is included at the end of this brochure.

FaCS and Centrelink

During the period of the Edge project, the Family and Community Services Portfolio was responsible for providing advice on a broad range of social policy issues affecting Australian society and the living standards of Australian families, communities and individuals. FaCS was the principal policy formulating and advising body in the portfolio. Centrelink was the service delivery agency in the portfolio, delivering a range of Commonwealth services, such as pensions, benefits and allowances to the Australian community.

Since the termination of Edge, in October 2004, as part of machinery of government changes affecting several departments and agencies, FaCS' responsibilities have changed. In addition, Centrelink now resides within the Human Services Portfolio.

The fieldwork for this audit was completed before the agencies were restructured. The relationship between FaCS and Centrelink was a significant issue during the project, and is discussed in some detail in this report. In particular they were unable to resolve disagreements in funding Edge. While the two agencies will still have a business relationship under the new administrative arrangements, the nature of the ongoing relationship is not yet clear. The ANAO has, therefore, made no specific recommendations on that relationship. However, we have suggested that both agencies consider their processes for resolving disputes with other agencies.

The environment

Payments made by Centrelink are subject to increasingly complex, and frequently changing, rules. These changes follow from revised government policies and from new and revised legislation. At the start of the Edge project it was estimated there were 8000 such rules for Family Assistance payments. It was, therefore, appropriate that Centrelink, and its partner agencies, explore and research more sophisticated solutions to delivering the required services, with a view to reducing the administrative costs of delivery, while increasing the accuracy of advice and payments to recipients of the services. Expert systems promised both cost reductions, and greater accuracy of advice and payments.

Therefore, the ANAO considers the question for the Edge project is not whether it should have been attempted, but whether management of the project, and decisions made during the project, were in accordance with better practice.

There were tensions between FaCS and Centrelink at all levels during the project. It would have been surprising if there were not, as the two agencies had somewhat differing needs from the project. Generally, work on the project progressed despite the tensions. The tensions were greater at more senior levels, where funding could not be agreed, and effective high-level governance of the project was not evident.

Audit objectives

The objectives of the audit were to determine whether FaCS and Centrelink had:

• a valid Business Case for the Edge project, as revised from time to time, including estimated costs, actual costs, and expected benefits;
•  effective governance of the project, including reviews at critical points in the project and subsequent decisions to continue or, in the final analysis, to discontinue;
•  an appropriate contract with SoftLaw, which was adequately managed;
• delivered appropriate advice on progress, project viability, and acceptable solutions to technical issues to the Executives of FaCS and Centrelink during the project; and
•  valid reasons for discontinuing the project.

The ANAO began this audit in March 2004, four months after the Edge project was terminated, following the Auditor-General's agreement to a suggestion by the Joint Committee of Public Accounts and Audit that the project was a suitable subject for audit.

Key audit findings

The Business Case for the project (Chapter 2)

The ANAO found that the original 1999 Business Case for the project, approved by the FaCS Executive Board and the Centrelink Board of Management, addressed most of the basic elements expected of a Business Case. These basic elements include costs, benefits, a risk analysis, a timeline, as well as a number of other items.

This 1999 Business Case estimated the total cost of the project over seven years to be $59.26 million (using a 6.33 per cent discount rate, the project team estimated the Net Present Value [NPV] of the cost to be $52.03 million). The Business Case estimated a cumulative return of $76.99 million (NPV $54.69 million) over seven years from the investment in the Edge project.

However, that Business Case did not include a methodology for measuring whether the project was a success or not, and did not identify measures to assess the achievement of objectives.

The ANAO also found that a significant risk identified in the Business Case, that is, the risk of delays in connecting Edge with the mainframe system and ISIS¹ , had inappropriate and ineffective mitigation strategies. In the event, this risk proved to be a significant factor in the delay to the project.

The ANAO found that an internal review of the Business Case in late 2002, while making some reductions to the expected benefits, did not re-examine the basic assumptions on which the benefits were based. Instead, the review team deferred that assessment until such time that the Edge system was fully implemented and ‘bedded down', and benefits could be assessed in actual operation. The review revised estimated savings downwards from a ‘conservative'² annual figure of $27.78 million to $23.18 million.

This 2002 review considered the original business aims still relevant and achievable. There was no suggestion, in that review, of Edge not meeting the requirements of the Families program, unlike the review conducted 12 months later.

The ANAO noted that the 2002 review stated that expenditure for the first three years of the project had increased from the original business case estimate of $35.62 million to $43.76 million, and the five-year estimate increased from $47.28 million to $64.94 million.

The 2002 review also showed the project to be 19 months behind the target date at one point, before being rolled out 10 months behind target.

The ANAO considers that it would have been appropriate to conduct a major independent review of the project, in July 2002, before roll out began, as this was a critical point in the project.

The procurement process (Chapter 3)

FaCS and Centrelink were unable to provide the ANAO with a copy of a procurement risk assessment, or any evidence of identifying and implementing any risk mitigation strategies. This does not meet better practice in procurement planning as outlined in relevant Department of Finance and Administration guidance on competitive tendering and contracting³ .

In examining the Request for Tender (RFT), the ANAO found that the information provided for bidders in the RFT was adequate. However, the short timeframe imposed for comments on the draft contract issued with the tender document was not in accordance with the specifications of the relevant Commonwealth Procurement Guidelines (CPGs)⁴ . FaCS and Centrelink did mitigate this situation by extending the RFT closure date by two weeks, as well as by holding an open forum to assist bidders with questions about the final document.

FaCS and Centrelink developed a Tender Evaluation Plan. However, this was finalised 16 days after the evaluation process had begun, which is inconsistent with better practice, and did not comply with advice from the probity advisor. However, the ANAO found no evidence to indicate that this impacted adversely on the RFT outcome.

A standard template was not used when scoring the RFT requirements. The ANAO conducted a random crosscheck of scores transcribed from the raw evaluation forms to the tender evaluation report, and found errors. Centrelink staff were unable to explain the discrepancies.

FaCS and Centrelink records indicate that privacy requirements were met and sensitive information was secured appropriately. The ANAO found that potential conflict of interest issues were dealt with adequately.

The ANAO noted that several important documents, decisions and associated reasons relating to the project were not filed or otherwise captured as a matter of record.

The ANAO concluded that the Edge project procurement process was generally conducted in accordance with the relevant CPGs. The ANAO considered that there was adequate evidence to show that the preferred tenderer offered best value for money.

However, in the ANAO's view, the procurement process was not as transparent as it could have been, in that:

• a probity plan was not developed nor, despite a variety of probity advice being provided, was a probity audit undertaken to ensure the tender evaluation process was conducted to the appropriate standards of probity and confidentiality;
•  the Tender Evaluation Plan was approved 16 days after tenders closed, and after the tender evaluation had begun;
• the project's procurement records did not consistently contain reasons for decisions and information such as dates, author, and addressee, and comments indicating the status of a document were frequently omitted from records; and
•  a standard template was not used when scoring the RFT requirements, and there were transcription errors in transferring scores from the raw forms to the tender evaluation report, albeit it did not affect the outcome in terms of the selection of the tenderer.

The ANAO concluded that the agencies had an appropriate contract with SoftLaw, and that the contract was managed adequately.

Governance of the project (Chapter 4)

The ANAO found that regular advice on the progress of the Edge project was provided to the FaCS Executive Board and the Centrelink Board of Management. However, the ANAO could find no evidence that the Boards were appropriately informed of the lack of progress on agreeing a Memorandum of Understanding (MOU) between the two agencies.

The ANAO found that the appropriate governance bodies at Centrelink, the Business Improvement Committee (BIC) (responsible for funding decisions), the Information and Technology (I&T) Committee, and the Guiding Coalition (comprising all Senior Executives in Centrelink), were appropriately informed of progress on the project. However, Centrelink advised the ANAO that advice to the Centrelink BIC of the expenditure on the Edge project was overstated by an amount of $3 million (of $58.95 million reported to BIC).

The ANAO found that the two FaCS committees that the ANAO expected to be involved in the project, the Business Planning and Resource Allocation Committee (BARAC) (responsible for funding decisions) and the Information and Communications Technology Committee (ICTC) (responsible for IT projects), were not involved in managing the Edge project. FaCS advised the ANAO that ‘there was no requirement for FaCS areas to report to the BARAC as this was an allocation committee only'.

The ANAO found that the Audit Committees of both agencies, and the internal audit function of both agencies had appropriately reviewed the project. The relevant agency audit committee reviewed the findings of the audits and a timeline was set for implementation of audit recommendations.

The ANAO found that the joint FaCS–Centrelink Steering Committee for the project met regularly, at intervals of one to two months, in 1998 and 1999, once in 2000, and four times in 2001, but never as a full committee after November 2001, although the Edge project continued until November 2003. The Committee did not meet even though the Centrelink BIC referred Edge funding submissions to BIC, on to the Steering Committee for advice. Papers were circulated to members of the joint FaCS–Centrelink Steering Committee. No response was taken as no objection, and therefore as approval.

The ANAO found that, with the exception of the joint FaCS–Centrelink Steering Committee, day-to-day management of the project was in accordance with the Development Contract.

The ANAO concluded, from findings in this and other chapters, that governance of the project was not as effective as it should have been, in that:

• predictions given to the agencies' Executives of the number of customers that could be processed through the system were optimistic, and never met;
•  advice that 70 per cent of claims processed through ISIS (the mainframe system) had errors of which 74 per cent could have been avoided using Edge, was optimistic and potentially misleading.
•  the FaCS governance committee with responsibility for IT was not involved in the project;
•  it was not clear that the FaCS Executive Board and Centrelink Board of Management were informed of the lack of progress on agreeing the MOU;
•  the joint FaCS–Centrelink Steering Committee did not meet during the latter two years of the project;
•  responsibility for the project was split between the two agencies, with no Senior Responsible Owner identified;
•  an MOU between FaCS and Centrelink was never agreed, and hence funding and savings were never agreed;
•  advice given to the Centrelink committee with responsibility for funding, regarding expenditure on Edge, was overstated by $3 million; and
•  the project plan was not maintained, and there was no formal development methodology.

In March 2005, Centrelink provided the following comment regarding the second point above:

Since July 2002, Centrelink uses random sample surveys to assure the correctness and accuracy of social security outlays. The target for correctness is 95 per cent and Centrelink's payment correctness figures have always exceeded this figure.

Financial management of the project (Chapter 5)

The ANAO found that the financial aspects of the Development Contract were managed appropriately. Changes made to the contract were approved by the three parties involved, and at the appropriate level. The procedure for accepting deliverables was in accordance with the contract, and payments were made accurately against the contract.

An MOU was intended to formalise funding arrangements for the project between FaCS and Centrelink. Disagreements over the responsibility for costs, and the calculation of savings, occurred throughout the life of the project, and the MOU was never signed. Centrelink considered that FaCS owed it $2.79 million at the conclusion of the project. FaCS declined to pay this amount due to cancellation of the project, and as savings were not generated by the project.

There was no internal reporting of finances by the FaCS Edge team to FaCS' BARAC. As mentioned earlier, FaCS advised the ANAO that ‘there was no requirement for FaCS areas to report to the BARAC as this was an allocation committee only'.

The Centrelink Edge team kept Centrelink's BIC informed of the financial status of the project, via expenditure updates provided in quarterly reports. However, as mentioned previously, Centrelink advised the ANAO that the expenditure figure, reported to BIC in these reports, was overstated by $3 million over the four-year period of the project. Centrelink advised the ANAO that the overstated figure was due to a misunderstanding, by the Edge project team, of the role of accrual accounting on finances.

Information provided by FaCS and Centrelink, confirmed to the extent possible by the ANAO, estimated that the total expenditure on the project was $64.4 million. Centrelink capitalised $25 million of the costs as internally developed software. Centrelink wrote off $15.15 million of this in 2003–04 for the Edge project. This consisted of $12.36 million for software unable to be used by other projects, and $2.79 million for costs that Centrelink considered FaCS owed to it.

Progress of the project (Chapter 6)

The ANAO found that the Edge project had no project plan current during development of Edge, and had no formal development methodology.

During the development, there were 129 formal changes to the contract with SoftLaw, 34 versions of the contract, and the number of ‘deliverables' specified in the contract increased from 205 to 365.

The ANAO found that predictions of available functionality were optimistic. The 2002 review, mentioned previously, stated the expectation that the March 2003 release would be capable of processing 90 per cent of customer claims. However, an exercise in that month showed less than 50 per cent of customer claims could be processed.

The ANAO found that development of the system included testing at a user acceptance facility before each release, testing of pilots, and a progressive roll out. This accorded with better practice, and allowed development to be paused at any time with minimum disruption to Centrelink as a whole.

In November 2001, comparison by Centrelink of accuracy between claims processed through ISIS and claims processed using Edge claimed that the former had an error rate of 70 per cent, of which 74 per cent could have been avoided using Edge. This result was used to support the continued development of Edge. Centrelink advised the ANAO of significant work being undertaken during the latter part of 2001 to define and improve payment correctness and accuracy for all systems. Centrelink further advised the ANAO that ‘the definitions and methodology used in the November 2001 Edge accuracy check do not, and could not, have reflected the definitions and methodologies finally agreed between FaCS and Centrelink, and which are now being used in Centrelink and ANAO audits'. A further comparison, in early 2003, found little difference in accuracy between Edge and ISIS.

During the Edge development, a number of differences between Edge and ISIS were found, raising questions about the processing of ISIS. As a result, corrections were made to ISIS in a small number of cases.

Termination of the project (Chapter 7)

In August 2003, FaCS and Centrelink jointly commissioned a review of the Edge project, led by an independent consultant, to assess whether there was a viable business case for further development of the Edge system. The review recommended that the project be terminated. The agencies accepted the recommendation of the review and terminated the Edge project in November 2003.

The 2003 review gave four reasons for recommending termination, as follows: Edge in its planned form was no longer properly aligned with the business needs of the Families program; the operation of Edge in parallel with ISIS was unsustainable; changes to the Families program meant Edge could have only limited effect on a key driver—improvement in accuracy; and the level of anticipated benefits were unlikely to be realised, leading to a negative return on investment.

The review recommended a feasibility study be conducted, to explore an option for the targeted use of expert systems to leverage the work undertaken on the Edge project. That feasibility study was not finalised at the time of completing this report.

An independent consultant from the private sector led the review team. That consultant's organisation provided the methodology for the review. The ANAO found that the FaCS and Centrelink officers included in the review team had, for the most part, little to do with the Edge project, and could, therefore, be considered independent. The ANAO expected there would be no pressure placed on the review team to make a particular finding, and found this to be the case.

The ANAO found that the Edge project was terminated in November 2003, when it appeared that it would have been completed, according to the contract, in December 2003. However, there was no guarantee that Edge would be able to replace the equivalent part of ISIS at that time, or into the foreseeable future.

In addition to the four reasons stated in the 2003 review report, a number of other issues affected the project, as follows.

FaCS–Centrelink relationship

The funding and savings for the project were to be defined in an MOU between the two agencies. The MOU was never agreed, and the relative responsibilities for the cost of the project were in dispute. Progress in a project where the owners are in dispute over costs is difficult. There was also a difference of views between FaCS and Centrelink as to what constituted work on Edge, as opposed to work that was needed for Edge, but which was also more generally applicable across a number of Centrelink projects.

Funding

As funding could not be agreed between the two agencies, it fell to Centrelink to provide the bulk of funds until such time as savings occurred. This meant repeated requests for additional internal funding within Centrelink, and the eventual reluctance of Centrelink to continue funding a project that had no clear finalisation.

Centrelink would have to continue funding both Edge and ISIS until such time as Edge was able to duplicate equivalent ISIS functions. It was unclear when this might be achieved.

The 2003 Business Case Review stated that one of the reasons for recommending termination was that the level of benefits anticipated was unlikely to be realised, leading to a negative return on investment.

The business arrangement with FaCS meant any savings were to be shared between the two agencies. There was little incentive for Centrelink to conclude the project, at which point they would have to give up savings to FaCS, potentially upwards of $11 million annually, unless they could clearly obtain those savings. Assessment of the actual savings had been deferred until the Edge system was implemented.

Technical issues

The ANAO is of the view that one of the reasons the Edge project was unsuccessful, was not due to any inherent defects in expert systems, but because Centrelink had difficulties in successfully integrating the expert system into its current IT environment. This was due both to the constraints of that environment, and the complexities of the solution developed for integrating Edge with the mainframe. The solution was being developed concurrently with Edge. Connectivity with the mainframe was a major source of delays to the project. Edge had more functionality (that is: ability to process customer claims) than it was able to process through to Centrelink's mainframe. The communications and data matching needs of interfacing Edge with the mainframe had been identified as a high risk in the original Edge Business Case. However, the mitigation strategies were inappropriate and proved to be ineffective.

Changing business requirements

When the Family Tax Benefit application was selected for development as an expert system it was reasonably stable. Legislation in 1999, including formation of the Family Assistance Office, meant that legislative and other changes were frequent, requiring changes to Edge. In addition, while the 1999 Edge Business Case recognised the requirement for annual reconciliation of Family Tax Benefits, the agencies did not fully appreciate the implications of fewer client interviews and more posted forms and call centre work, for which Edge was not originally designed. The introduction of the More Choice for Families initiative in 2002 resulted in a requirement for the ability to continuously adjust the Edge rulebase. This was not envisaged in the original design and would require some redesign. These issues led the 2003 Business Case Review team to conclude that the system no longer met the requirements of the business.

Accuracy

A key aim of the project was to optimise the accuracy, consistency, and completeness of assessment decisions. A comparison with ISIS in 2001 indicated significant improvements were potentially achievable by using Edge. However, the annual reconciliation process reduced the potential benefits of Edge, and a further comparison with ISIS in 2003 indicated little, if any, difference between the two systems. Hence a key driver for Edge no longer applied.

Functionality

Predictions of the percentage of customer claims that could be processed through Edge were optimistic. At the time of roll out to the Centrelink network in July 2002 around 38 per cent of claims could be processed. Although this percentage gradually improved over the next 15 months, Customer Service Officers (CSOs) were placed in the position of trying to use a system that may not be able to process the claim, and they would have to resort to using ISIS. Hence, Edge gained a poor reputation. Centrelink staff preferred to use ISIS upon which they could rely. Although documents indicate that Edge would be completed according to contract by December 2003, there was no guarantee that it would be able to fully replace ISIS at that time, or in the foreseeable future.

The ANAO concluded that the FaCS and Centrelink decision to bring the project to a close, and to explore other ways to use the products and knowledge developed during the Edge project, was appropriate.

Overall audit conclusion

In short, the project was over time, over budget, and terminated before completion. Direct financial savings from the project were not realised and the project was unsuccessful when assessed against its aims. There were deficiencies in the project, particularly in the governance of the project, from which lessons for the future can be learnt. While FaCS and Centrelink advised the ANAO that they did gain some benefits from the project, nevertheless it was appropriate for the agencies to terminate the project.

Recommendations

The ANAO made two recommendations aimed at improving FaCS' and Centrelink's project Business Cases, and governance of projects. Both agencies responded positively to the recommendations.

Agencies' responses

FaCS

The Secretary of FaCS advised the ANAO in March 2005 that he agreed with the report's two recommendations. FaCS' specific comments in relation to the recommendations are included directly following the recommendations in the body of the report. No additional comments were provided for attachment to the report.

Centrelink

The Chief Executive Officer of Centrelink also advised the ANAO that he welcomed the report and agreed with the recommendations. Centrelink's specific comments in relation to the two recommendations also follow the recommendations in the body of the report. No additional comments were provided for attachment to the report.

SoftLaw

In addition, given SoftLaw's role in the Edge project, the Auditor-General invited the company to provide comments on relevant extracts of the proposed report. SoftLaw's comments were considered in the finalisation of the audit report.

Footnotes

1 ISIS (Income Security Integrated System) is a suite of systems for recording customer claims, and processing Centrelink payments. It operates on the Centrelink central computer (generally called the mainframe).

2 The project team originally estimated that the benefits likely to be realised from the project ranged between $27.78 million (conservative) and $40.62 million (optimistic) each year from 2002–03 onwards.

3 Department of Finance and Administration, Competitive Tendering and Contracting, Guidance for Managers, March 1998, p.14.

4 Department of Finance and Administration, Commonwealth Procurement Guidelines: Core Policies and Principles, March 1998.