Background

1. Defence recognised the requirement for an integrated personnel management system in 1997, as an outcome of the Defence Efficiency Review. The Personnel Management Key Solution (PMKeyS) Project (the Project) was a significant and complex human resource business process change in Defence, and involved moving military and civilian staff off purpose-built long running human resource legacy systems to a common platform. Implemented between September 1997 and December 2002, the PMKeyS system has become Defence's core management information system for personnel management. PMKeyS is the authoritative management record for all Defence personnel in the areas of: administration and leave; development and training; career management; organisational structure; workforce planning; and recruitment. PMKeyS also manages payments to Defence's 19 028 civilian staff as at 30 June 2004. In excess of 10 000 staff are certified as PMKeyS users, who perform transactions on PMKeyS as part of their normal work function.

2. In September 1997, the Project was initiated with the establishment of the Project Working Group that was tasked with defining the Defence personnel management functional requirement, the selection of an implementation partner, and the recommendation of a preferred software vendor solution. The Project's planned procurement activities were very complicated, extended across five phases over two years and, by Project closure, had included the provision of a diverse range of services and products by over 40 contractors.

3. In respect of the Defence Financial Statements 2003–04, the ANAO reported that it was unable to express an opinion as to whether the financial statements were true and fair.¹ The ANAO identified a $1.23 billion audit limitation of scope in respect of the Australian Defence Force (ADF) employee leave provision, which related to historical leave records deficiencies and PMKeyS system data integrity failings. The Defence Financial Statement Audit reported issues with PMKeyS relating to, inter alia, security administration; operator classifications; system training; leave and civilian payroll processing; and military data inconsistencies.

Audit approach

4. The objective of the audit was to assess the Project's planning and approval processes and the Project's contract and project management. The audit addresses the scope of the delivered system, the expectations of end-users, and the system's ability to meet their capability requirements.

5. The ANAO's ability to audit Project expenditure was limited by an absence of reliable financial records and documentation. As a result, the ANAO is unable to provide a high level of assurance over Defence's own reporting of aggregate expenditure of $63.4 million for this Project. Defence was however able to validate and attribute vendor expenditure across a limited sample of Resource and Output Management and Accounting Network (ROMAN)² transactions, selected at random by the ANAO. The Project's ROMAN financial record has not been used to validate Project expenditure reporting throughout the Project.

Overall audit conclusion

6. The Project, when completed, was to deliver a single integrated system that would consolidate both the delivery of personnel management functions ³ and the number and disparate nature of the systems that performed those functions. The Project was to deliver these outcomes by June 2000. The Project suffered from extensive schedule slippage, with Phase 1 delayed by 39 weeks and Phase 2 components rolled out between 75 and 158 weeks late. When the Project closed in December 2002, major outcomes under Phases 3 and 4 had not been delivered (see Figure 1).

7. The Project was also to facilitate significant savings of $100 million per annum, which had been identified in the 1997 Defence Efficiency Review. The Defence Personnel Executive (DPE) reported in May 2003, nearly six months after Project closure, that PMKeyS was yet to demonstrate a return on investment and that although savings had been achieved through decommissioning the legacy systems, those savings fell well short of the costs so far expended.

8. The ANAO found that the Project was not approved in accordance with Government requirements, including failure to obtain Cabinet approval. The Project met Defence's own criteria for classification as a Major Capital Equipment project. The ANAO notes that the Project was not treated as a Major Capital Equipment project within the applicable guidance available at the time of initiation. ⁴

9. The Project was found to have exceeded its notional budget of $25.0 million by $38.4 million, representing an increase of more than 150 per cent. In addition to Project expenditure, Defence reportedly incurred additional costs for Infrastructure ($26.3 million) and Production Support ($41.2 million) between July 1998 and June 2003. The total cost to Defence to bring PMKeyS into service, including the Production Support costs during the Project rollout period, is estimated to be at least $131 million. This cost exceeded Defence's 1998 estimate of $103.5 million to maintain its legacy personnel systems for five years by more than $26 million.⁵

10. When the Software Vendor was unable to commit in August 2000 to a fixed price for Phase 2, after nearly 18 months of work on Phase 2, Defence did not have effective contractual control over Project costs and outcomes. Extensive Project delays, along with a significant underestimation of the training requirement, resulted in higher costs in the delivery of training to end-users. When rolled out, the Project training was inadequate in that it often did not reflect the delivered functionality. Initial Phase 2 training, which was delivered 14 months prior to the system's rollout to Army, was inappropriately timed.

Figure 1: Project delivery to end-users

Note: (a) Phase 4 was to include website enablement and employee self service functionality.
Source: ANAO analysis of Defence records

Key findings

Project planning and approval (Chapter 2)

11. The ANAO found that the Project Office, as part of its planning processes, did not develop, complete and update a number of the key project planning and management documents, as mandated by Defence procurement policy. The absence of this documentation contributed to the Project's inadequate financial management, as well as to further procedural and procurement policy non-compliance.

12. The ANAO found that the DPE did not follow Defence management guidance on at least two occasions. In July 1997, the then Vice Chief of the Defence Force (VCDF) stipulated that a Minor Capital Submission would need to be prepared for Defence Executive consideration. In September 1997, at the time of a $4.0 million initial funding allocation, the Defence Management Committee requested validation of the scope, cost, schedule and achievability of the Project prior to further funding allocation. Defence could not substantiate that these internal governance obligations were met.

13. The Project was not managed as a strategic procurement activity,⁶ nor was it managed as a Major Capital Equipment project. Initial Project funding of $25.3 million, allocated between September 1997 and December 1998, exceeded the Defence threshold requiring management as a Major Capital Equipment project and also exceeded the then funding threshold requiring Cabinet approval of the procurement activity. Defence advised the ANAO in June 2005 that it was not Defence practice at that time to treat corporate IT systems as a Major Capital Equipment project. The ANAO found that Defence did not seek, nor obtain Cabinet approval for the Project as was required.

14. Additional departmental funding of $20.3 million, $9.0 million and $4.2 million, in support of cost increases, was allocated to the Project between September 1999 and June 2003. The ANAO found that Defence did not seek, nor obtain approvals from Cabinet, or the Minister for Defence and Minister for Finance and Administration for any of these cost increases in accordance with Defence procurement policy. Allocated funding to Project activities totalled $58.8 million, which was $4.6 million less than the actual Project expenditure reported by Defence of $63.4 million.

Contractual arrangements (Chapter 3)

15. In the Project planning phase, Defence identified a requirement for a Review Partner and selected SMS Consulting Group Pty Ltd from a shortlist of two interested parties in October 1997. The total contract price and the price of individual contract deliverables were not fixed across the service period of October 1997 to June 1998. Payments were determined by the application of agreed fixed hourly or daily rates for specified personnel to the number of post approved work hours incurred toward each contract deliverable on a monthly basis (known as a ‘time and material' contract).

16. Defence and the Review Partner undertook a selection process to identify a software vendor from the then Office of Government Information Technology shortlist of human resource management system providers. In July 1998, Defence announced that PeopleSoft Australia Pty Ltd had been selected as the Project's Software Vendor. The Software Vendor was contracted on a fixed price basis to implement the Project between July 1998 and June 2000 at a cost of $13.46 million, which included a licence fee of $3.5 million. Under this contract, which was known as the Official Order, Defence was also required to pay an annual maintenance fee of $0.7 million to the Software Vendor.⁷ In July 1998, Defence advised the then Minister for Defence that the contract value was $16.5 million.

17. The Official Order required that prices for the Project's phases were to be fixed in advance. However, Defence did not always approve prices in advance and prices were often determined retrospectively. In 1999, the Software Vendor and Training Contractor ⁸ were operating for an extended period without contractual coverage for the work that they were undertaking.

18. The ANAO has assessed that expenditure of at least $27.6 million, or 44 per cent of Defence's estimated total Project costs of $63.4 million, was incurred on contract terms akin to ‘time and material' terms. Figure 2 illustrates the breakdown of the Project's total costs by the underlying contractual terms, principally fixed cost or time and material.

19. Following the completion of the software and vendor selection process in June 1998, SMS Consulting Group was re-contracted as the Project Manager on the same time and material payment terms. In this role, the Project Manager was accountable to the Project Director who was a Defence employee. Defence have advised that for a period between 1999 and 2001, the Project did not have a Project Director and employees of the contracted Project Manager fulfilled the Project Director's role.

Figure 2: Breakdown of Project costs of $63.4 million by contractual terms

Note: Non-attributed costs have not been attributed by the ANAO as either fixed or time and material costs.
Source: ANAO analysis of Defence records

20. The cost of software customisations and technical support provided by the Software Vendor, as well as the costs of the Project Manager, were effectively determined by the period of time that contractor resources were applied against those tasks. The time and material nature of these services meant that Defence bore the cost risk associated with schedule delays.

21. The Office of Government Information Technology contract with the Software Vendor required that services provided under the Official Order had to be formally accepted by Defence by way of a Certificate of Acceptance. The ANAO found limited evidence of Certificates of Acceptance and Defence have advised that they had adopted less specific acceptance criteria. In the absence of Certificates of Acceptance, most of the payments made to the Software Vendor under the Official Order were not made in accordance with the contract terms.

22. The Official Order was not compliant with Defence procurement guidance in that it did not include specific provisions for liquidated damages and performance incentives. Rebates were to become payable under the Official Order in the event of missed contract milestone dates. Notwithstanding that the Project missed all milestone dates, no rebates were paid to Defence.

Cost Performance (Chapter 4)

23. The Project Office put forward a notional budget of $25.0 million in March 1998. The budget proposal was not formally endorsed and, even though the Project had not received approval by an appropriate authority, initial funding of $25.3 million was allocated to the Project. The Project was closed down in December 2002 and Defence have reported that total Project expenditure to June 2003 was $63.4 million. However, as at May 2005, Defence was unable to identify the source of budget funding approval for Project expenditure in excess of the total allocation of $58.8 million.

24. Defence was found to have spent $15.76 million on contracted project management services, which was a thirteen-fold increase over the initial budget estimate of $1.22 million (see Figure 3). The Project Manager was paid $14.58 million or 92 per cent of the costs related to Project Management.

Figure 3: Comparison of forecast and actual Project expenditure

Notes:

(a) Software includes the core software, licence and maintenance fees payable to the Software Vendor from within the Project's budget.

(b) Other includes, inter alia, infrastructure, civilian salary, travel and other contractor expenditure.

Source: ANAO analysis of Defence records

25. The Project Office underestimated the cost of providing training to PMKeyS end-users, which increased nearly fourteen fold, from $0.35 million to a total contractor cost of $4.79 million. Following Project rollout, responsibility for training was devolved across Defence, producing a disparate ongoing training environment that was reported to adversely effect personnel data quality. This situation was addressed through the establishment of a central training authority in early 2004, more than four years after the first stage of PMKeyS rollout.

Schedule performance (Chapter 5)

26. As the Project was not formally approved, the Project's scope and schedule were not formally approved, nor were they examined for their achievability, as requested by the Defence Management Committee in September 1997.

27. The Project did not deliver significant elements of the scope for which it was initially funded, such as ADF Payroll. Career Management functionality, a major element of PMKeyS Phase 2, is not fully utilised by Army when planning its annual posting cycle.

28. Major Project outcomes, including payroll functionality for military staff (Phase 3) and website enablement and employee self service (Phase 4), were not delivered. Total Project costs however increased by $38.4 million.

29. Upon rollout of the delivered Project phases, end-users suffered problems with: network performance; losses of functionality compared to the legacy systems; and the quantity and quality of migrated data. Navy temporarily restarted their NPEMS ⁹ system when Career Management data was not transferred adequately into PMKeyS. Civilian staff lost the retrospective functionality of their previous personnel system, which has necessarily resulted in the development of a number of manual workarounds.

30. With the Project's closure in December 2002, DPE reported that it would delay the planned upgrade to give greater priority to correcting serious deficiencies in the current system that had resulted in the increased stress, lost productivity and lower morale of staff. The cost of the related system remediation activities to May 2005 has been reported by DPE to be $6.0 million.

31. Defence submitted the First Pass Business Case for the PMKeyS Upgrade Project to the Defence Capability Committee in December 2004. The submission identified strategies for the upgrade of PMKeyS including the transition to an alternate software platform, which was rejected by Defence as being high risk and too expensive. The Defence Capability Committee endorsed the Upgrade Project's preferred strategy to sole source the PeopleSoft Version 8 application.

32. Defence effectively committed to the sole source strategy in June 2004 when additional funding of $5.248 million was approved by DPE to complete the purchase of the Software Vendor's seven-year enterprise licence at a total cost of $6.4 million.¹⁰ In doing so, the DPE did not consider the impact that this decision would have on subsequent evaluation of Upgrade Project options. In December 2004, as part of its Upgrade Project Business Case submission, Defence stated that prior investment in the Software Vendor's product had been a factor in DPE's decision to recommend sole source selection of the software platform. Defence advised the ANAO in June 2005 that:

… the purchase of the enterprise licence was not the driver for ‘sole source' of the PeopleSoft Version 8 software application … the Upgrade (Project) Business Case makes it clear that staying with PeopleSoft for the next software generation delivers best value for money.

33. At the time of ANAO fieldwork, the PMKeyS Upgrade Project, which is planned to be funded under Joint Project 2080 Phase 2B, was not approved. Joint Project 2080 Phase 2B received Government First Pass approval in July 2005.

Lessons learnt

34. The lessons to be learnt from the Project include:

• The need for project approval processes for IT systems to comply with Government and departmental requirements. To ensure improved project governance, future management information system projects should be approved in accordance with applicable Government and departmental procurement policies.
• Defence incurred significant project and infrastructure related expenditure in excess of the original funding allocation. To improve relative project cost and schedule outcomes, future management information system projects should be based on realistic estimates of project costs and system infrastructure requirements that have been subject to close analysis and review, prior to project approval. This will be facilitated by the strengthened two-pass project approvals process, which comprises:
  • First pass approval – Government considers alternatives and approves a capability development option(s) to proceed to more detailed analysis and costing, with a view to subsequent approval of a specific capability; and
  • Second pass approval – Government agrees to fund the acquisition of a specific capability system with a well-defined budget and schedule.
• The need for a structured process of periodic management review, following the awarding of contracts, to provide additional assurance on the robustness of schedule, cost and performance outcomes being achieved in material projects.
• Project management business processes should accord with sound management practice for contractual and financial management, and for the retention of appropriate records, to ensure legislative compliance and that project outcomes meet with end-user needs.
• To improve Defence training outcomes, training delivered to end-users as part of new management information system implementations should be appropriately timed and reflect the functionality of the delivered system. Where this does not occur, follow-up or revised training programs need to be implemented.
• Meaningful and measurable key performance indicators should be implemented to assist Defence in the monitoring of the effectiveness of management information system remediation initiatives.

Agency response

35. Defence accepts the thrust of the ANAO report and agrees with the recommendations and ANAO's lessons to be learnt from the Project.

Footnotes

1 Defence Annual Report 2003–04 and ANAO Report No.21, 2004–05, Audits of the Financial Statements of Australian Government Entities for the Period Ended 30 June 2004.

2 The ROMAN system is Defence's core financial management information system.

3 The Working Group focused the needs analysis and functional fit assessments for the PMKeyS software solution in the areas of Career Management, Organisational Structures and Workforce Planning, Personnel Development and Training, Personnel Administration and Leave, and Recruitment.

4 The ANAO's audit of the SDSS Upgrade Project (Audit Report No.5, 2004–05, Management of the Standard Defence Supply System Upgrade) also found that business management information systems were not treated as Major Capital Equipment procurement activities.

5 The comparison of cost between the legacy systems and the new system does not recognise any quality and functionality improvements that may have occurred over time.

6 Strategic procurement activities are focused on delivering outcomes that are critical to Defence's ability to meet its core objectives.

7 Defence exercised a contractual option to upgrade the database upon which PMKeyS was to operate. The additional cost of this option was $1.26 million and included a $0.70 million increase in the Licence Fee and a $0.14 million increase in the Annual Maintenance Fee for four years.

8 DA Consulting Group was contracted by Defence in October 1999 to develop training materials and to deliver training to Phase 1 end-users.

9 The Navy Personnel and Establishment Management System or NPEMS was Navy's major legacy personnel system.

10 Defence had previously purchased two components of the enterprise licence at a combined cost of $1.165 million. The contract price of $5.248 million, which was paid on 24 June 2004, was observed to be non-cancellable, non-refundable and non-contingent.