Introduction

1. Many Australians experience barriers to economic and/or social participation due to disadvantage, which can arise from: low income; joblessness; health problems; low educational achievement; personal safety issues at home; or not having friends or family who can assist them in times of crisis.¹ Around five per cent of working age Australians experience multiple disadvantages—that is, three or more of these factors. People experiencing disadvantage, where eligible, may access relevant payments and services administered by the Department of Human Services (Human Services), which delivered $149.4 billion in health, welfare and social payments on behalf of the Australian Government in 2012–13.

2. The 2011–12 Budget provided $92.4 million over four years for Human Services to conduct two separate trials of intensive service models aimed at improving services to disadvantaged customers by delivering more tailored and intensive support. The two trials were:

• Case Coordination—focused on assisting disadvantaged customers on all benefit types; and
• Local Connections to Work—focused on assisting disadvantaged job seekers with complex needs.

3. The trials were components of a wider Service Delivery Reform (SDR) agenda announced in December 2009, which foreshadowed major structural and service delivery reforms within the Human Services portfolio, to be implemented over 10 years.²

The trials of intensive service delivery

4. The Case Coordination and Local Connections to Work (LCTW) trials were targeted at disadvantaged customers with complex needs and barriers to economic and/or social participation. The trials were intended to assist customers to more effectively access local services that could address their needs. Research³ which informed the SDR agenda found that customers with the highest needs, including those with mental illness and other disabilities, were required to navigate, by themselves, a complex range of services from government and non‑government service providers.

5. The trials were designed to adopt a very different approach to customer service as compared to the department’s established ‘transactional’ model. The traditional model tends to focus on the immediate reasons prompting a customer to attend a service centre, whereas the trials aimed to offer a more individualised approach focusing on the coordination of local services to address clients’ longer term needs and the barriers to overcoming disadvantage.

6. An important difference between the two trials was that that there was no defined customer group targeted in Case Coordination, and participation was voluntary. This trial had a broad social inclusion focus, aimed at assisting disadvantaged customers on all benefit types. LCTW, on the other hand, assisted long term unemployed job seekers and unemployed youth who could be readily identified. Participation in LCTW was generally compulsory and also had a clear employment‑related objective.

7. In other ways the trials had key similarities:

• they were delivered by trained customer service advisers, at times assisted by specialist staff such as social workers, and focused on harnessing a customer’s positive attributes rather than on ‘fixing’ an individual’s deficits ⁴; and
• both relied on staff having a good knowledge of, and relationships with, local service providers to help customers access the services that may assist in addressing their needs. The range of local services was similar in both trials and included health, counselling, housing, emergency support, drug and alcohol and employment service providers (noting that employment service providers had a more central role in LCTW).

8. At the end of February 2014, 142 339 customers had participated in the Case Coordination trial and 38 Case Coordination trial sites had been established. Over the same period, 13 198 job seekers had participated in the LCTW trial and 17 LCTW trial sites had been established. The learnings and evaluations from the trials were intended to inform the development of options for an intensive services ‘offer’, for consideration by the Australian Government in 2014–15.

9. On 15 January 2014 the Minister for Human Services agreed to the department’s proposal for the early closure of the two trials effective from early 2014–15 (instead of the end of the financial year), to assist with achieving savings associated with the Efficiency Dividend.^5,6 The department informed the ANAO that it has put in place preparations for the closure of the trials.⁷

Audit objectives, criteria and scope

10. The audit objective was to assess the effectiveness of the Department of Human Services’ management of the trials of intensive service delivery for customers with complex needs.

11. To form a conclusion against the audit objective, the ANAO adopted the following high‑level criteria:

• the department had arrangements to support the effective planning and design of the trials;
• the department had effective mechanisms in place to identify, document and transfer learnings across the trial sites on a regular basis to continuously improve implementation over the life of the trials; and
• the department had evaluation strategies in place, including information/data gathering and monitoring systems, to allow for the formal assessment and documentation of the effectiveness of the trials.

12. The ANAO examined the department’s management of the trials since 2011–12. The audit did not examine: the earlier management of the LCTW trials; and the department’s management of intensive service initiatives under the Building Australia’s Future Workforce package ‘Helping Young Parents’ and ‘Jobless Families‘.⁸

Overall conclusion

13. It has been estimated that over 750 000 working‑age Australians face multiple disadvantages—such as a combination of joblessness, poor health and low educational achievement—and that most of these people are in receipt of payments and services delivered by the Department of Human Services.⁹ The 2011–12 Budget provided $92.4 million for the department to conduct trials of two separate intensive service delivery models aimed at providing more tailored and intensive support to disadvantaged customers. At the end of February 2014 over 150 000 customers had participated in the trials at a cost of $44.7 million with 55 trial sites established. While the trials were initially funded for four years¹⁰, the Minister for Human Services agreed to the department’s proposal for the early closure of the two trials after three years of operation, to assist the department to achieve savings associated with the Efficiency Dividend.

14. The Department of Human Services’ administration of the actual Case Coordination and Local Connections to Work trials of intensive service delivery has been generally effective. A sound governance framework was established to oversee the planning process, and the trials were developed having regard to the available evidence base, including related trials and overseas experience. A suite of performance measures, feedback mechanisms and evaluation strategies were also established, to track progress, inform the conduct of the trials and capture lessons learned. However, there was scope for improvement in key aspects of the Case Coordination trial’s planning and monitoring, to strengthen the cost‑benefit of the initiative, as the case for a trial on this scale¹¹ was not assessed and delivery targets were not adjusted in light of lower than expected client take‑up of services available under the trial .

15. The planning and design of the trials were guided by sound governance structures and informed by a solid evidence base which included the views of relevant stakeholders, research, international experience and the results of previous trials. Output and outcome performance indicators were identified, along with measurable annual targets, to enable continuous tracking and measurement of progress against expected outcomes. The department also established a range of mechanisms in the Case Coordination trial to identify, analyse, transfer and capture learnings. Fewer mechanisms were adopted for the LCTW trial and for learnings across the two trials, which meant that the potential for learning may not have been fully realised.

16. Evaluation and monitoring strategies were developed during the design phase, consistent with a better practice approach, and the trials have been subject to continuous monitoring and periodic formal evaluations. Both qualitative and quantitative information was collected from a range of stakeholders—including customers, staff and service providers—and there was a focus on monitoring and evaluating performance against outputs and outcomes. Notwithstanding this review activity, the department did not brief the Minister or relevant stakeholders, such as the Department of Employment, on the findings of the interim evaluations—an oversight given the Government’s substantial investment.

17. The findings of the review activity undertaken to date for the Case Coordination trial have been generally positive, with indications that the trial improved relationships between the department and local service providers. The majority of customers participating in the trial indicated that Case Coordination assisted them in achieving their social and/or economic goals. The longer term outcomes from the trial, for example, whether participants achieved greater self‑reliance¹², have not yet been evaluated. Review findings for the LCTW trial have been mixed in relation to whether the trial was achieving the expected employment outcomes and of the primacy of employment versus social outcomes. The most recent review activity suggests that the participants in the LCTW trial have not achieved ‘off‑welfare’ outcomes¹³ at a higher rate than similarly disadvantaged job seekers not participating in the trial. However, positive social benefits to job seekers participating in the trial were reported as were community benefits such as significantly improved relationships between the department and local service providers.

18. The Case Coordination trial was planned to involve 44 sites over four years with 38 established; a substantial initiative¹⁴ which would have benefited from careful assessment of the advantages and disadvantages of a large‑scale trial. However, the planning phase did not include a systematic analysis of options for the size of the trial, including the cost‑benefit of including additional trial sites. Further, an ambitious target¹⁵ was originally set for the number of customers to be assisted through the Case Coordination trial. While the target was selected in the absence of reliable information¹⁶, the trial indicated that around 20 per cent of customers declined the invitation to participate in case coordination, and internal analysis and an interim evaluation suggested that the target warranted review.¹⁷ Nonetheless, the target was not revised and the department instead focussed intensively on increasing customer numbers to meet the target; an approach with potential implications for the cost‑effectiveness and direction of the trial, as it risked servicing a wider group of clients than originally intended.

19. As noted, the department will close the trials early in 2014–15 rather than at the end of the financial year, to assist it to achieve Efficiency Dividend savings.¹⁸ In the absence of agreement by Cabinet or senior Ministers, it is unusual that an agency would truncate a program funded as a new policy measure to deliver savings to contribute to meeting the Efficiency Dividend. Such a course is not consistent with the purpose of the new policy measure or the Efficiency Dividend. To garner the benefits of the approximately $45 million and the three years already expended on the trials, the department will need to consider the best way to capture the lessons learned. Realising these benefits would involve the conduct of final evaluations of each trial to synthesise learnings and review findings to date; and utilising the baseline data for the Case Coordination trial to assess outcomes. It will also be important to achieve an orderly closure of the trial sites, with the timely provision of information and advice to customers and local service providers on future service delivery arrangements. The department has advised that it is putting in place processes to manage the early closure of the trials and has plans to undertake internal final evaluations of both trials.

20. The ANAO has made one general recommendation aimed at the benefits of evaluating and documenting the outcomes achieved from such lengthy and costly trials, to realise an appropriate return from their conduct and to inform any future initiatives relating to intensive service delivery.

Key findings by chapter

Planning and Design of the Trials (Chapter 2)

21. Effective planning of trials is a significant contributor to their successful implementation. At the outset, the department invested in the planning and design of both the Case Coordination trial and the LCTW trial. Governance structures used in the design phase of the trials included representation from relevant government agencies. Officials had sufficient opportunity to consider key aspects of the trials’ design and established linkages with related government policies. Key stakeholders were also consulted, allowing the final trial designs to incorporate external views and experience. A solid evidence base was established drawing on relevant research, evaluations of previous trials and overseas experience. Evaluation strategies and program logic were also developed prior to implementation, consistent with a better practice approach.

22. The scope and duration of the trials were considered in the design and planning phase. Determining the number of trial sites required an assessment of the appropriate balance between: the financial commitment; testing key concepts and variables; and measuring the benefits obtained. Given the barriers facing some of the target participants and the nature of the social and economic outcomes sought, large‑scale and longer‑term trials may have been justified. However, departmental records do not indicate that the department assessed the cost‑benefit of different options to inform the Government’s decisions about the scale of the trials. The Case Coordination trial, in particular, was large but not adequately assessed, raising reasonable questions about the value for money of a trial on this scale.

Implementation and Continuous Learning (Chapter 3)

23. Establishing arrangements for continuous learning is critical to maximising the benefits of trials, building an evidence base and reducing risks. An important prerequisite to systematic learning is identifying performance measures for the desired outputs and outcomes of trials. The performance measures adopted for the trials had a number of positive features—being relevant, specific and measureable within particular timeframes. Both output and outcomes indicators were defined, and targets were set for each performance measure. For the Case Coordination trial, some targets were revised in light of experience gained as the trial progressed.

24. However, the target for one key performance measure—the number of customers assisted through the Case Coordination trial—was not revised, notwithstanding departmental analysis and the findings of an interim evaluation which suggested that the target warranted review. For instance, experience from the trial indicated that around 20 per cent of customers declined to participate in Case Coordination, but no allowance had been made for this in the original assumptions of customer demand which underpinned the target. In addition, initial customer demand estimates had assumed that all sites would be operational from the beginning of each financial year, but the implementation of new sites was in fact staggered over the term of the trial; further contributing to lower customer numbers than originally estimated. The significant difference between actual customer numbers and the target in the first and second years of the trial¹⁹ resulted in an increasing departmental focus on strategies and processes to actively identify customers for referral to Case Coordination. While this was a reasonable interim response to the shortfall in customer numbers—given that one of the objectives of the trial was being able to accurately identify customers who could benefit from the Case Coordination trial—in parallel, the department could also have usefully reviewed the original assumptions of customer demand on which the target was based. The department’s decision not to reconsider the target had potential implications for the cost‑effectiveness and direction of the trial, as it risked servicing a wider group of clients than originally intended.

25. A range of mechanisms were used in the Case Coordination trial to identify, analyse and transfer learnings. Effective mechanisms included: implementation guidance; training; on the ground support; a helpdesk; periodic stocktakes and monthly reports; wiki pages; and mini‑trials. However, a different approach was adopted for capturing learnings from the LCTW trial, based on the department’s view that most of the necessary testing and refining of the service model had been conducted previously. While there is evidence that the LCTW trial evolved somewhat to reflect lessons learned, it was not on a scale that would have been expected for a long term trial.

26. A limited range of mechanisms were established to identify cross‑trial learnings, notwithstanding recognition, in the design phase, of the potential benefit of concurrent trials for learnings. As a consequence, some key cross‑trial learnings—apparent in interviews with staff and service providers in the ANAO’s sample of trial sites—were not highlighted in the lessons learned analyses from the department.

Evaluation of the Trials (Chapter 4)

27. Formal evaluations of trials, at appropriate intervals, assist in: consolidating and documenting learnings; assessing progress; and informing policy direction. The usefulness of evaluations was demonstrated by the use of evaluation findings from previous trials in the evidence base that informed the design of the Case Coordination and LCTW trials. Both trials have been evaluated, employing aspects of better practice: the department advised that a baseline dataset was established for Case Coordination; qualitative and quantitative information was gathered from a range of stakeholders (including customers, staff and service providers); there was a focus on outputs and outcomes; initial cost effectiveness calculations were undertaken for the LCTW trial; and comparative analysis was employed with ‘control’ populations.

28. The findings of the review activity undertaken to date for the Case Coordination trial have been generally positive, with indications that the trial improved relationships between the department and local service providers. The majority of customers who participated in the trial indicated that Case Coordination assisted them in achieving their social and/or economic goals. The tentative conclusion from the interim evaluation suggested that the Case Coordination trial responded to customers’ needs and assisted many customers, however, it was not generally an ‘individually transformational program’²⁰. The department advised that a base line data set for each Case Coordination trial site has been established including customer vulnerability measures but whether the trial has had an impact at this broad outcome level has yet to be assessed.²¹

29. Review findings for the LCTW trial have been mixed in relation to whether the trial was achieving the expected employment outcomes and of the primacy of employment versus social outcomes. The most recent review activity suggested that the participants in the LCTW trial had not achieved ‘off‑welfare’ outcomes²² at a higher rate than similarly disadvantaged job seekers not participating in the trial. However, positive social benefits to job seekers who had participated in the trial were reported as were community benefits such as significantly improved relationships between the department and local service providers.

30. Despite establishing continuous learning mechanisms and conducting periodic formal trial evaluations, the department did not advise the Minister or other key stakeholders, such as the Department of Employment, of trial findings—an omission given the substantial time, resources and policy expectations invested in the trials.

31. The early closure of the trials, with the agreement of the responsible Minister, is to contribute to the savings required for the department’s Efficiency Dividend obligations.²³ Truncating a program funded as a new policy measure to deliver savings, to contribute to meeting the Efficiency Dividend, is a course not consistent with the purpose of the new policy measure or the Efficiency Dividend.

32. Given the early closure, however, it will be important for the department to achieve an appropriate return on the approximately $45 million and three years already expended on the trials. This would involve conducting final evaluations of each trial to synthesise and document learnings to date, and utilising the baseline data for the Case Coordination trial to assess the social and economic outcomes. It will also be important to achieve an orderly closure of the trial sites.

Summary of agency responses

33. The proposed audit report was provided to the Department of Human Services and an extract was provided to the Department of Employment. The departments’ summary responses to the audit are provided below. The formal departmental responses are included at Appendix 1.

Human Services

34. The Department of Human Services’ summary response to the proposed audit report:

The Department of Human Services (the department) welcomes this report, and considers that implementation of its recommendations will further enhance the department’s ability to realise the benefits of the Trials of Intensive Service Delivery.

The department agrees with the ANAO’s recommendation. The department will complete an internal evaluation of the trials by December 2014 utilising the existing comprehensive evidence base in order to document learnings and considerations for future service delivery models. Outcomes of the evaluations will be reported to the Secretary.

The department also notes the Audit findings that the administration of the Case Coordination and Local Connections to Work trials of intensive service delivery have been generally effective, that the planning and design of the trials were guided by sound governance structures and that evaluation and monitoring strategies developed during the design phase, were consistent with a better practice approach.

Employment

35. The Department of Employment’s summary response to the proposed audit report:

The Department of Employment has not seen the full report and notes ANAO’s advice that none of the recommendations in the proposed report relate to the Department. As a key stakeholder in both trials, the Department stands ready to support the implementation of Recommendation 1 by continuing to provide the Department of Human Services with data relating to Case Coordination and Local Connections to Work.

Recommendations

Footnotes

[1] Australian Social Inclusion Board (2012) Social Inclusion in Australia – How Australia is faring, p.24.

[2] SDR had three objectives: to make people’s dealings with government easier through better service delivery and coordination of services; to achieve more effective service delivery outcomes for government by contributing to the achievement of government policy objectives; and to improve the efficiency of service delivery by integrating and automating service delivery and creating a flexible and agile system. Department of Human Services, Service Delivery Reform:Transforming government service delivery, Department of Human Services, Canberra, 2011, p. 5.

[3] Research was undertaken by the Boston Consulting Group in 2009 for the Department of Human Services to inform the development of the department’s Service Delivery Reform agenda.

[4] This ‘strengths‑based’ approach is intended to focus on the positive attributes and underdeveloped capabilities of people who have been in some way compromised in their abilities or are seeking help for problems. It is an alternative to problem or deficit based approaches which are characterised by a focus on what is ‘wrong’ with a person and practitioner‑driven interventions. Australian Social Inclusion Board, Strengths‑based approaches to service delivery, June 2011, p.1.

[5] The Efficiency Dividend, first introduced in the 1987–88 Budget, is an annual reduction in agencies’ eligible appropriations. The Efficiency Dividend is intended to act as an incentive for agencies to find efficiencies and provides a visible return from the efficiencies to the Budget. See JCPAA (2008) The efficiency dividend and small agencies: Size does matter.

[6] The 2011–12 Budget provided $92.4 million over four years for the trials. Total expenditure on the trials to the end of February 2014 was $44.7 million.

[7] The department advised that the preparations have included: the suspension of further planning for the final external evaluation of the Case Coordination trial; limitations on new customers being referred to the trials; and no new sites have been established for either trial since the end of 2013.

[8] The Building Australia’s Future Workforce package, funded in the 2011–12 Budget, was comprised of 39 measures including the ‘Helping Young Parents’ and Jobless Families’ measures and was designed to provide greater opportunities for skills development, training and employment. The LCTW trial was also a component of the package as well as being a measure in SDR.

[9] Australian Social Inclusion Board (2012) Social Inclusion in Australia – How Australia is faring.

[10] The intention was to build the evidence base and demonstrate proof‑of‑concept, with a view to informing options for an intensive services ‘offer’ for government consideration in 2014–15.

[11] The department planned to implement 44 sites over the four years of the trial. Thirty eight had been established at the end of February 2014.

[12] Becoming more self‑reliant could include, for example, a customer making fewer requests for advance payments or, for a customer on an activity‑tested payment being less reliant on welfare payments or moving off welfare payments altogether.

[13] ‘Off‑welfare’ outcomes occur when job seekers are no longer reliant on welfare payments.

[14] The trial was large compared to other trials conducted in the social policy area: the 2008 Centrelink Place Based Services trial involved seven sites; the Helping Young Parents and Jobless Families initiatives involved 10 local government areas; and the LCTW trial involved 24 sites established over four years.

[15] Over the three years to 2013–14 the target for customers assisted by Case Coordination was 294,386.

[16] In the absence of a pre‑defined population group for Case Coordination, a key objective from the trial was to reliably identify customers who would benefit from this type of intensive service model.

[17] The shortfall between the target for the number of customers assisted by Case Coordination and the actual numbers assisted was significant in the first two years of the trial: the 2011–12 target was 35,066 while the result was 9181 customers assisted; and the 2012–13 target was 113,160 whereas the result was 41,993 customers assisted.

[18] The 2011–12 Budget provided $92.4 million over four years for the trials. Total expenditure on the trials to the end of February 2014 was $44.7 million, based on information provided by the department.

[19] See footnote 17.

[20] Report to the Department of Human Services from the Australian Institute of Family Studies and the Australian National University (2013) Evaluation of the Case Coordination Trial – Interim Report, p.167.

[21] Baseline data to be collected from customer records for each trial site include vulnerability indicators (such as drug and alcohol addiction, and homelessness), repeat requests for advance payments, urgent payments (related to crises), use of Centrepay, and breaches/participation failures and demographic data.

[22] ‘Off‑welfare’ outcomes occur when job seekers are no longer reliant on welfare payments.

[23] The 2011–12 Budget provided $92.4 million over four years for the trials. Total expenditure on the trials to the end of February 2014 was $44.7 million based on information provided by the department.