Background

The Job Network is an Australia-wide network of around 109 community-based and private organis¬ations. It provides public employment services to the unemployed under contract to the Department of Employment and Workplace Relations (DEWR). In other words, DEWR makes payments to the Job Network to help unemployed people find paid work, preferably long-term work. Since the Job Network's inception in 1998, there have been three contracts for these services. The latest, known as the ‘Employment Services Contract 2003–06' (the third employment services contract, usually abbreviated to ‘ESC3'), started on 1 July 2003 and goes for three years.

The Government sought to use the opportunity of a new employment services contract to improve employment service delivery and outcomes by providing simpler arrangements and a service guarantee to job seekers, with easier access to a wider range of job opportunities; better targeting and more timely provision of services to those most in need; a stronger culture of active job search participation and mutual obligation, and more effective incentives for providers to invest in support that achieves outcomes for all job seekers, especially those most at risk of long-term unemployment.

ESC3 brought major change to the operation of the Job Network by initiating the Active Participation Model (APM). The APM introduces a continuum of services to the unemployed. They receive those services from a single Job Network member for the duration of their unemployment, during which the services are intended to increase in intensity.

Under ESC3, as with the earlier contracts, Centrelink provides certain essential, related services to the unemployed. DEWR manages this through a Business Partnership Arrangement (BPA) with Centrelink. DEWR itself does not provide services direct to unemployed people, but it is ultimately accountable for the services provided by the Job Network and Centrelink. Centrelink, a statutory agency within the Human Services portfolio, is also separately accountable for its activities through its minister.

The introduction of ESC3 and the APM depended on the use of innovative computer technology. DEWR developed a new computer application, Employment Assistant 3000 (EA 3000) for ESC3 and it is accessed by the department, Centrelink and Job Network providers in daily Job Network operations.

Audit objective and scope

The objective of this audit was to assess whether DEWR had implemented ESC3 and its computer system EA3000, efficiently and effectively. The primary focus is on the period of change from the previous employment services contract to ESC3 and the first full year of its operation, 2003–04.

The scope of the audit was limited to the implementation of Job Network services under ESC3, the introduction of the supporting computer application, EA3000, and DEWR's use of modelling to estimate the effects of the APM. The audit did not test the effectiveness of the APM. DEWR has a plan to evaluate the new model.

A separate, concurrent ANAO audit assessed DEWR's oversight of Job Network services to job seekers.

Key findings

Consistency of administrative framework with policy directions (Chapter 2)

DEWR first implemented the new employment services market—the Job Network—in 1998. The primary objective of the Government's reforms in establishing the Job Network was to ensure that labour market assistance would have a clear focus on job outcomes and would genuinely make a difference to those assisted. ¹

Because legislation providing for case management services for the unemployed, the Employment Services Act 1994 (ES Act), had not been repealed, the Government implemented the new arrangements under executive power.

In 1998, the ANAO undertook a performance audit of the Job Network implementation and examined the steps the department took to minimise the risk of legal challenges arising from the use of existing legislation and administrative authority. The existing legislation defines and specifies the operation of the case management system in providing employment services. One significant source of risk flowed from the need for any new service to be sufficiently different from case management to withstand legal challenge. In that earlier audit, the ANAO concluded that the department had addressed those risks particularly by increasing the differences between the new service types and case management, as defined under the ES Act.

The ANAO has found that, in designing ESC3, DEWR had considered the risk of a legal challenge arising from the further changes to the nature of Job Network services.

Key policy elements of the APM are for each job seeker to be assisted by a single Job Network provider and for them to have an opportunity to make an informed choice of provider. One of DEWR's requirements under its BPA with Centrelink is that ‘Centrelink will obtain job seeker preference for a JNM'. The ANAO found that the BPA does not include an obligation for Centrelink to advise the unemployed of this and other DEWR requirements governing their attachment to Job Network providers. Seminars designed to provide job seekers with information to help them to choose a provider are only held ‘where practical'. For these reasons, the ANAO concluded that informed choice does not take place in all cases. Further, the intention to have each job seeker assisted by only one provider throughout their period of unemployment is intended to be a cornerstone of the APM. This policy is intended to minimise the reprocessing of job seekers among providers. During the course of the audit, DEWR updated its Complaint Management guidelines to provide criteria to help DEWR customer service officers to approve or refuse requests to transfer a job seeker to another provider.

Modelling and funding (Chapter 3)

DEWR's purchase of employment services for the unemployed from the Job Network had cost around $1 billion each year. For this reason and because of the scale of the changes introduced with ESC3, DEWR modelled the operation to help predict the financial consequences. DEWR recognised in advance that transition to a very different model of employment services would pose a challenge for estimating expenditure. The department sought to:

• keep within its budgeted forward estimates of expenditure;
• ensure that there would not be an unacceptable financial effect on the employment services industry; and
• increase the proportion of funds attaching to outcomes as compared to service fees.

DEWR made a considerable effort to improve its predictive capacity, engaging an expert consultant to help with its modelling. The use of both DEWR staff and the consultant's staff to develop the model allowed greater flexibility in undertaking the work, which was appropriate given the short time available for the work to be completed.

DEWR has depended on its consultant for assurance about the integrity and robustness of the model. Because the department prepared no documentation and required none, this left DEWR in a weaker position for future development. A project with the Department of Finance and Administration (DoFA) to redesign the model for future estimates calculations provides an opportunity to address these shortcomings.

DEWR was helpful in providing information and advice to the industry association to help it with its own modelling, which was structured more towards use by individual Job Network members.

DEWR reported an overspend of $8 million on the Job Network against a revised 2003–04 budget of $991 million. DEWR has tracked and reported its expenditure in detail internally since implementation of ESC3. This shows that, in the first year of ESC3, the department used a greater proportion of funds to pay intensive support service fees ($486 million) than to pay outcome payments for job placements ($171 million). However, DEWR reported that the proportion of funds going to outcome fees had risen to around 50 per cent by the end of the second year of the contract.

Job Network outcome payments and service fees are elements of Job Network services which, in turn, comprises part of the Job Network programme. The current level of aggregation of financial information does not allow external stakeholders to identify the contribution that outcome payments and service fees make to Job Network expenditure and any trends that reflect DEWR's success in making the programme outcome-focused.

IT support for ESC3 (Chapter 4)

Implementing ESC3 involved substantial systems development for DEWR and Centrelink, including the move to web services technology. To a significant extent, the ESC3 initiative relied upon a major DEWR IT release, Employment Assistant 3000 (EA3000). EA3000 is an application developed for use by external service providers and departmental staff to manage the operation of Job Network services and to enable the department to monitor and regulate job seeker flows. Centrelink, as ‘gateway' to the Job Network, developed the capability for exchange and updating of job seeker information between EA3000 and Centrelink systems.

The first release of EA3000 was on 14 April 2003. The second and major release was on 1 July 2003, when ESC3 commenced.

Centrelink and Job Network providers were dependent upon the timely implementation and proper functioning of EA3000. Centrelink staff used an interface between Centrelink and EA3000, particularly web services, to make its primary contribution to employment services, that is, to check job seeker registration details and to make diary appointments with Job Network providers for job seekers. These initial steps were essential to a successful transition to ESC3. Developing vocational profiles (VPs) for both existing job seekers (the stock) and those referred during the transition period (the initial part of the flow) was a prerequisite to providing those job seekers with Job Network services after 1 July 2003 and ensuring a smooth start to ESC3.

From the first release of EA3000, major implementation problems were reported by Job Network providers and Centrelink staff. The risks associated with implementing a large, sophisticated and crucial system with new technology were heightened by a tightly restricted time frame. Although a systematic planning approach was developed by DEWR, no significant load testing was performed before the 1 July 2003 implementation date.

The consequences of IT system instability and intermittent unavailability, particularly with DEWR's systems, leading up to 1 July 2003, were that Centrelink was severely constrained in its ability to complete job seeker registrations, perform job seeker assessments, and book or confirm appointments for VPs. When EA3000 access was unacceptably slow or unavailable, Centrelink resorted to manual workarounds. This contributed to delays in referrals and affected the rate at which job seekers attended Job Network appointments. DEWR later paid Centrelink for the additional work undertaken to deal with the backlogs.

Achieving operational stability remained a challenge for DEWR until August–September 2003. The department later stated that delivering its major releases on time has ‘often been at the cost of compromised quality as testing and training timeframes are compressed to compensate for overruns in earlier project phases'. DEWR has also advised that IT for ESC3 was not fully operational from the outset. It said that the system took five weeks to be stabilised, and stated that this is ‘well ahead of industry standards for such a large project'.

DEWR has reported that it delivered the system on time and within budget. However, it is not possible to verify this statement. Although DEWR delivered baseline requirements, it did not clearly state its expectations about the functionality and likely cost of EA3000 in advance. Over several months after implementation, DEWR substantially improved the performance of the system. While Job Network providers had encountered difficulties with the implementation of the system, the majority agreed that problems had been mostly resolved.

The ANAO acknowledges that since system implementation, DEWR's Employment Systems Group (ESG) has recognised opportunities for improvement for IT processes. Particularly in more recent times and with the assistance of external consultants, ESG has improved its IT general controls environment and has gradually introduced changes to bring about a more mature IT operating environment.

Management of the transition to ESC3 (Chapter 5)

DEWR had undertaken extensive risk assessment and planning for the transition to ESC3. DEWR's intention was to ensure that all job seekers eligible for Job Network services were attached to ongoing Job Network providers well ahead of the commencement of ESC3. The department described this as ‘one of the principal aims of transition management'. Specifically, the department expected that 60 per cent or about 450 000 vocational profile (VPs) would be completed by 30 June 2003 with the balance, about 300 000, completed by September 2003.

DEWR had also taken into account previous experience, that is the transition from ESC1 to ESC2, and set four ministerially-endorsed objectives to be met during transition. These were to:

• minimise any disruption to services for job seekers and employers;
• minimise any reduction in outcomes achieved during the transition period;
• have all eligible job seekers referred to Job Network members contracted under ESC3 as quickly as possible; and
• provide, in consultation with the industry, a consistent, manageable flow of job seekers to Job Network members, which maintained appropriate cash flows.

Two major issues arose during transition. These were difficulties with DEWR's computer system, EA3000, particularly for Centrelink (as explained above), and low job seeker attendance at VP appointments. When considering these issues, it is important to bear in mind that the APM is a continuum of service. Therefore, any issue impeding initial VP atten¬dance was likely to have subsequent consequences on services for job seekers, potentially delaying long-term job outcomes and lowering cash flow to Job Network providers. The ANAO examined DEWR's management of these two issues and concluded that it took action to address them.

The ANAO's assessment against each of DEWR's transition objectives shows that:

• complaints recorded by DEWR during transition to ESC3 exceeded that experienced in the previous transition period;
• reported outcomes dipped more markedly in the transition from ESC2 to ESC3 than in the previous transition, though they recovered after transition;
• by 27 June 2003, (only a few days before the start date for ESC3, 1 July 2003) nearly 451 000 VP appointments had been scheduled to take place by that date but only 184 200 VP attendances had been recorded; and
• because the flow of job seekers to the Job Network was much lower than expected, Job Network members raised concerns about cash flows. These were addressed by two changes in payment arrangements. The first change, in July 2003, involved $100 million advance payments for services that DEWR had expected to be provided and the purchase of $33 million of additional services. The second change, in September 2003, allowed payments to be made quarterly, in advance, to Job Network providers for all Intensive Support Customised Assistance contacts with job seekers, for expected appointments.

On the evidence available, the ANAO concluded that DEWR largely did not meet the transition objectives. However, following implementation, the IT system was stabilised and job placements and long-term job numbers recovered within a few months, with job placements and long-term (13-week) job rates exceeding those recorded at a similar point in ESC2.

During the implementation of ESC3, DEWR did not explicitly monitor and report in terms of the four transition objectives endorsed by the Minister. DEWR's Annual Report 2003–04, which reports the successful implementation of the APM, does not refer to the department's performance in terms of these four objectives.

DEWR did advise the ANAO that an additional factor in reviewing the low attendance during transition was DEWR's expectation that the compliance effect of the APM would identify some income support recipients who would cease to receive those payments. In early 2002, this was expected to yield annual savings of $21.7 million to the Commonwealth. There has been no report against this expectation. It is difficult to discern whether, where unemployed people failed to attend appointments, this should have been interpreted as a compliance effect or by concluding that the transition process had not engaged the unemployed into the new arrangements as effectively as intended.

Performance information (Chapter 6)

DEWR's Outcome 1 in the period under review was ‘An effectively functioning labour market'. A range of different factors, only some which are under the control of or subject to the influence of DEWR, is likely to bear upon the effective functioning of the Australian labour market. This means that DEWR needs to specify its contribution to that effective functioning.

DEWR told the ANAO that:

It can be inferred from Connolly and Cunningham [2004] and Connolly, Herd and Neo (2002) that it is very difficult to determine quantitatively the effect of Departmental programmes (including Job Network) on the average duration of unemployment and the rate and incidence of long-term unemployment—most of the explanatory variables reported in these two studies are largely outside the Department's influence and Job Network spending accounts for well under half of a per cent of GDP. This means that it is not feasible, at this stage, to link Departmental programme outcomes with the average duration of unemployment in a formal, empirical manner. Instead, the judgement of Departmental staff (informed by empirical work including that by Connolly and Cunningham 2004) is currently used to monitor and report on the influence of Departmental programme outcomes on the effectiveness indicators [emphasis added].

Under the Performance Management Principles, output performance information relates to the quality, quantity and price of agency outputs. In the DEWR Annual Report 2003–04, the department provides tables setting out performance indicators and actual performance by output, together with related commentary on each. The ANAO found that DEWR would more clearly satisfy annual reporting requirements if it provided an explanation, reconciling the final result with the original estimates in the corresponding portfolio budget statement (PBS).

DEWR has recently re-established an indicator to enable it to assess the efficiency of the Job Network. This will provide a ‘cost per employment outcome' for those who are employed or studying three months after leaving Job Network assistance. It would improve transparency if, where DEWR reports performance in these terms, it makes it clear that outcomes for all job seekers who have received Job Network assistance are included, not just those who attracted a specific outcome payment from DEWR.

The most frequent and prominent report on Job Network activity is DEWR's Job Network Performance Profile. However, the ANAO concluded that DEWR could use better measures of Job Network Performance than those currently reported in this document, which would:

• exclude results attributable to external factors (such as economic trends) or which were achieved by job seekers without the direct help of DEWR's providers; and
• give due emphasis to the Government's primary target group, income-tested income support recipients.

Provider performance management (Chapter 7)

DEWR does not directly measure and report in terms of the key performance indicators (KPIs) set out in the contract with Job Network providers. Instead, it uses a star ratings system to encourage performance improvement.

DEWR has used star ratings successfully as an incentive to improve performance. However, the system is not meeting all of the objectives set for it by DEWR; in particular, the available evidence shows that few job seekers make use of them to choose a provider. DEWR could also make it easier for Job Network providers to monitor their performance by making its use of weightings more transparent.

DEWR uses the star ratings approach to reallocate business based on the relative performance of Job Network providers, not their individual performance against agreed standards. The lack of specific performance standards in the contract is a potential weakness in the business reallocation mechanism. The department has accepted the risk and continues business reallocation among Job Network providers by its existing method.

Overall conclusion

The implementation of ESC3 was especially challenging because of the coincidence of substantial new policy, a simultaneous procurement process to select Job Network providers and a new IT system built on a new technology to a deadline.

ESC3 implementation on 1 July 2003 critically depended on a new IT system, EA3000. The new system was intermittently unavailable for several months until August-September 2003. This adversely affected the completion of job seeker registrations and referral of job seekers to Job Network appointments. Because the APM is a continuum, any delay at the start of the process had ensuing consequences for the flow of job seekers through the new system.

A low attendance rate by job seekers at appointments with Job Network providers also impeded the transition to the new model. The low attendance contributed to cash flow problems among Job Network providers. DEWR data shows that services to job seekers were disrupted and placement of the unemployed—which had previously displayed a long-term upward trend—dipped temporarily, as did the number of long-term jobs being achieved by the unemployed through the Job Network.

As explained earlier, DEWR set four ministerially-endorsed objectives to be met during transition. Because DEWR did not explicitly monitor and report against these four transition objectives it is difficult to assess objectively the extent to which they were achieved. On the evidence available, the ANAO concluded that DEWR largely did not meet the transition objectives. However, following implementation, the IT system was stabilised and job placements and long-term job numbers recovered within a few months.

DEWR expected ESC3 to be increasingly outcome rather than process-focused and yield savings in employment services expenditure. In its first year, the greater part of the expenditure went to service fees, not outcome payments. When DEWR sought additional funds, it attributed this to higher outcome rates whereas service fees required the increase.

During the audit DEWR undertook to consider updating aspects of its performance information in the context of ‘overall evaluation activities' including: identifying the department's contribution to reported outcomes; achieving a ‘clear read' from PBS to the related annual report; and updating its data series on the average cost of Job Network outcomes. These changes would help stakeholders to understand better the figures reported by the department and to show whether it has achieved expected performance levels.

DEWR response

The full DEWR response to the draft audit report is reproduced at Appendix 4. DEWR's summary response was:

DEWR welcomes the audit of the implementation of ESC3, particularly the ANAO's acknowledgement of the substantial policy and operational reforms heralded by the Active Participation Model and its delivery in time for the new Job Network contracts in July 2003.

DEWR welcomes evidence in the Report that issues such as job seeker attendance and IT systems were promptly addressed and settled shortly after the start of the contract. DEWR is disappointed however that the Report does not make it explicitly clear that Job Network not only moved quickly beyond transition but very significantly increased performance over past levels, including for long term unemployed and other disadvantaged job seekers. This needs to be taken into account in considering the success of the ESC3 transition.

In Financial Year 2004–05. the number of Job Network placements was 29 per cent higher than the new record that was set in 2003–04, the first year of service delivery under the Active Participation Model. In 2004–05, Job Network also earned more in outcome payments than service fees. This is a testament to Job Network members' success in getting the neediest job seekers into sustainable employment.

The Report makes six recommendations. DEWR has agreed in part with a majority of these, noting that recommended measures are in many cases already in place.

ANAO comment

This audit focused upon the implementation of the third employment services contract, ESC3, and the key findings present the issues arising and the overall audit conclusion.

The department's summary response (above) and some responses to specific recommendations (set out in the body of the report) raise issues outside the scope of the audit, including the recent reported performance of the programme. Accordingly, no audit opinion is presented on these issues or any externalities which bear on the recent performance of the programme reported by the department.

This performance audit was conducted concurrently with another audit focusing on the Job Network, DEWR's Oversight of Job Network Services to Job Seekers, (Audit Report No.51 2004–05). Apart from the reports' recommendations directed at improving performance, these reports, taken together, are intended to inform the Parliament and the public of the department's administration of the Job Network programme which has been designed to improve employment service delivery and outcomes.

Centrelink response

Centrelink, noting that the recommendations of this report are directed at DEWR but that the overall report contains many references to the DEWR–Centrelink relationship, provided the following response:

Centrelink notes the ANAO's views concerning the nature of the DEWR–Centrelink relationship as it affects the provision of services to Job Network providers and job seekers. Centrelink welcomes the ANAO's acknowledgement of the complexities involved in implementing large, sophisticated and crucial information technology systems within tight, restricted time frames. Centrelink agrees to discuss this report and its recommendations with DEWR in the context of the next Business Partnership Arrangement.

Centrelink has no specific comment in relation to any individual recommendation.

Footnotes

1 Minister for Employment, Education, Training and Youth Affairs 1996, Reforming Employment Assistance, Ministerial Statement, August.

2 ANAO 1998, Management of the Implementation of the New Employment Services Market, Audit Report No.7 1998–99.