Introduction

1. Australia’s vocational education and training (VET) system is designed to help individuals develop or enhance workplace skills and knowledge. Supporting people to develop workplace skills is a key factor in improving productivity and workforce participation, and plays a central role in strengthening economic and social wellbeing.¹

2. Apprenticeships and traineeships—hereafter referred to as Australian Apprenticeships—are an important component of Australia’s VET system. An Australian Apprenticeship is an employment arrangement that combines paid work with a structured program of ‘on-the-job’ and ‘off-the-job’ training. Australian Apprenticeships enable employees to gain experience, develop practical skills and acquire nationally-recognised qualifications. As at 30 June 2014, there were approximately 351 000 apprentices and trainees ‘in-training’.² Of these, approximately 55 per cent were classified as technicians and trades workers.³

3. The Australian Government has a number of programs in place to support Australia’s VET system, and Australian Apprenticeships in particular. The largest component of the Australian Government’s financial support for Australian Apprenticeships is the Australian Apprenticeships Incentives Program (AAIP).

Australian Apprenticeships Incentives Program

4. The objective of the AAIP, which commenced in 1998⁴, is to contribute to the development of a highly skilled and relevant Australian workforce that supports economic sustainability and competitiveness. The Australian Government aims to achieve this objective by:

• providing genuine opportunities for skills-based training and development of employees by providing incentives to employers of eligible apprentices; and
• encouraging people to enter into skills-based training through an Australian Apprenticeship by providing personal benefits.⁵

5. A key part of the AAIP is the delivery, under contract to the Australian Government, of Australian Apprenticeships Support Services (Support Services) by Australian Apprenticeships Centres (AACs). Since the AAIP commenced, there have been five separate contract rounds for the delivery of Support Services—the current round commenced on 1 July 2012 and is scheduled to end on 30 June 2015.⁶ A broad range of Support Services are required to be provided in the current contract round, including: assessing the eligibility of applications and claims for financial assistance under the AAIP; providing information and advice to potential applicants; and marketing and promoting Australian Apprenticeships. The AACs are paid a fee for each Australian Apprenticeship that they administer as part of delivering the Support Services.

6. In September 2014, the then Minister for Industry—now the Minister for Industry and Science—announced the establishment of the Australian Apprenticeships Support Network (Support Network) to replace the current round of Support Services from 1 July 2015. At the time, the then Department of Industry’s website noted that:

the Support Network [is designed to] make it easier for employers to recruit, train and retain apprentices and better support individuals in a proven earning and learning pathway, helping improve completion rates.⁷

7. To the end of December 2014, total AAIP expenditure in the current Support Services contract round has been approximately $2.8 billion—$2.3 billion in financial assistance and $0.5 billion in fees paid to AACs.⁸

8. The key steps involved in the life of an Australian Apprenticeship are shown in Figure S.1.

Figure S.1: Key steps in an Australian Apprenticeship

 

 

Source: ANAO, adapted from A shared responsibility. Apprenticeships for the 21st Century, Final Report of the Expert Panel. January 2011.

Administrative arrangements

9. In December 2011, responsibility for the administration of the AAIP was transferred from the former Department of Education, Employment and Workplace Relations (DEEWR) to the then Department of Industry, Innovation, Science, Research and Tertiary Education (DIISRTE). As part of further changes to its responsibilities, the department was renamed twice during 2013: in March 2013, to the Department of Industry, Innovation, Science, Climate Change, Research and Tertiary Education (DIICCSRTE); and in September 2013, to the Department of Industry (Industry). On 23 December 2014, the Australian Government transferred responsibility for the administration of the AAIP from Industry to the newly established Department of Education and Training (Education and Training).⁹ At the same time, Industry was renamed the Department of Industry and Science.

10. Apart from references to activity prior to December 2011, the discussion of audit findings in this audit report refers to Industry—the administering department’s name at the time of the audit. However, the audit recommendation and other suggestions for improvement made in the report have been addressed to Education and Training, the responsible department at the time of preparing the audit report.

Audit objectives, criteria and scope

11. The audit objective was to assess the effectiveness of Industry’s administration of the AAIP. To form a conclusion against the audit objective, the ANAO adopted the following high-level criteria:

• an effective governance framework for the AAIP has been implemented and Industry provided suitable guidance, processes and tools to support the AACs to effectively deliver AAIP services;
• suitable contractual arrangements, including sound contract management practices, were in place to support service delivery by the AACs and overall program management by Industry; and
• appropriate program management, performance monitoring and reporting structures were in place and used to inform the administration of the AAIP.

12. The audit focussed on arrangements in place since 1 July 2012—the commencement of the current round of Support Services contracts. The ANAO examined relevant AAIP documentation, including a sample of contracts between the Australian Government and the AACs, and conducted interviews with key staff from Industry and a selection of AACs. The ANAO also analysed data from the Training and Youth Internet Management System (TYIMS).¹⁰

13. The audit did not examine the management of individual Australian Apprenticeships or of the AACs. In addition, the audit did not examine the procurement undertaken by the then DEEWR in 2011–12 to establish the current round of contracts, nor the procurement undertaken by Industry to establish the Support Network.

14. The AAIP was previously audited by the ANAO in Audit Report No.9 2007–08, Australian Apprenticeships. That audit assessed the effectiveness of the former Department of Education, Science and Training’s (DEST) administration of its role in Australian Apprenticeships. The audit found that the program was appropriately used by employers, that financial assistance payments to employers were accurate, and that contract management practices were sound.¹¹ The audit did however identify an opportunity to improve performance monitoring and evaluation activities and recommended (Recommendation No.2) that DEST:

• analyse program usage by employers of apprentices and trainees in occupations in national demand; and
• perform a sensitivity analysis of incentives payments to employers compared with Australian Apprenticeships completions.¹²

15. The current audit examined whether the two parts of the recommendation have been implemented.

Overall conclusion

16. Administered by the Department of Industry (Industry) at the time of the audit¹³, the Australian Apprenticeships Incentives Program (AAIP) aims to contribute to the development of a highly skilled and relevant workforce by providing financial incentives to support employers and apprentices undertake Australian Apprenticeships. A key component of the program is the delivery of Australian Apprenticeships Support Services (Support Services) under contract by Australian Apprenticeships Centres (AACs). The current contract round commenced on 1 July 2012, and is scheduled to end on 30 June 2015. To the end of December 2014, total AAIP expenditure in the current contract round has been approximately $2.8 billion—comprising $2.3 billion in financial assistance grants and $0.5 billion in fees paid to the AACs.

17. Overall, Industry’s administration of the AAIP in the current Support Services contract round was generally effective.¹⁴ Appropriate contract and program management arrangements were largely in place, and operating as intended. Notably, Industry’s processes for monitoring the performance of the contracted service providers (the AACs), including arrangements for assessing AACs’ compliance with contractual requirements, were soundly-based and well-targeted. The majority of AACs were performing reasonably well against most performance measures, and over the two completed years of the current contract round, the combined total of reported employers and apprentices assisted by the AAIP has been in-line with Industry’s targets.¹⁵ Generally, AAIP payments made from 1 July 2012 to 30 June 2014 examined by the ANAO accorded with the program’s eligibility criteria and related policy conditions. Nonetheless, the audit highlighted some opportunities to further strengthen the AAIP’s management and oversight arrangements. This includes more regular data analysis to help assess the integrity of apprenticeship records and identify incorrect payments, and the development of a structured evaluation framework to help assess the program’s performance against its policy objective and intended outcomes.

18. In the current contract round, Industry implemented relevant contract and program management arrangements and practices, which contributed to the overall effectiveness of the delivery of the AAIP and the Support Services. These arrangements included:

• fit-for-purpose contracts with the AACs, including clear service delivery requirements and performance measurement arrangements;
• a multi-faceted and well-targeted approach to monitoring the activities of the AACs, including assessing compliance against contractual requirements. A central part of these monitoring activities was a structured approach to assessing the validity of AACs’ eligibility assessments;
• generally well-designed risk management arrangements, including processes for assessing and managing the risks associated with fraud and conflicts of interest; and
• well-founded and instructive internal management reporting arrangements.

19. Industry’s internal performance measurement and reporting arrangements provided departmental management with a range of useful and relevant information. In particular, the program’s performance measures included a useful mix of intermediate and proxy targets. However, these measures did not enable a complete assessment of the extent to which the AAIP is achieving its objective or intended outcomes. Further, Industry did not put in place an evaluation plan for the AAIP, and had not, at the time of the program’s transfer to Education and Training, conducted a formal evaluation of the effectiveness or performance of the AAIP during the current contract round. A program of structured evaluation activity would assist Education and Training to better assess the AAIP’s performance.

20. The ANAO’s examination of AAIP-related records in the Training and Youth Internet Management System (TYIMS) identified shortcomings in record-keeping arrangements, as well as some 960 incorrect financial assistance payments with a value of approximately $970 000.¹⁶ Although the incorrect payments identified are financially immaterial—representing less than 1 per cent of total program expenditure in the period 1 July 2012 to 30 June 2014—the results suggest that there would be benefit in regularising data analysis activities and upgrading data matching and validation functions.

21. The ANAO has made one recommendation to Education and Training aimed at improving the evaluation framework for AAIP, noting that well-designed monitoring and review arrangements can assist entities, and other interested stakeholders, make informed assessments about a program’s progress and relative contribution towards its intended outcomes and expected benefits.

Key findings by chapter

Program Management (Chapter 2)

22. The risk assessments maintained by Industry relating to the administration of the AAIP covered a broad range of risks and controls. Most risks were described in sufficient detail so as to be informative, and for the most part, the controls and treatments listed in the assessments were suitable and appropriate. However, a particular shortcoming was that no controls were documented for one of the identified risk events—which was focussed on false information provided in relation to eligibility criteria—and there would be benefit in doing so.¹⁷ The accuracy and suitability of the risk assessments related to the AAIP could be further enhanced by strengthening arrangements for monitoring and review. A structured program of review can help confirm the ongoing accuracy of the identified risks, and contributes to an assessment of whether risk mitigation measures are well-targeted and operating as intended.

23. Under their contracts with the Australian Government, the AACs are required to develop and maintain a risk management plan (RMP) and a conflict of interest management plan (COIMP). The ANAO’s analysis of a sample of COIMPs showed that most contained practical strategies and relevant processes—including clear accountabilities—for dealing with conflict of interest situations. However, the ANAO identified some variation in the quality and relevance of the AACs’ RMPs. In particular, many of the plans examined only outlined the AAC’s risk management policy (without identifying specific risks) or identified risks not tailored to the services being delivered. Education and Training advised that it would encourage greater consistency in the form and content of the RMPs into the future, by providing guidance to the providers engaged in the delivery of the Support Network. There would also be merit in Education and Training examining ways to gain assurance of the continued effectiveness and operation of the service providers’ RMPs.

24. Industry did not assess whether the AAIP was affected by amendments to the former Financial Management and Accountability Regulations in May 2013, which altered the definition of a grant. Following the ANAO’s enquiries, Industry established that the AAIP was a grants program, according to the new definition, and then acted in a timely manner to consider the application of the Australian Government’s grants administration framework¹⁸ to the program.

25. Industry had a range of well-designed and informative guidance and support arrangements in place to assist staff and the AACs administer the program. In addition, the department had well–established internal performance monitoring and reporting arrangements for the AAIP, providing managers with access to a range of useful and relevant information to assist their oversight of the program. In large part, however, the AAIP’s Key Performance Indicators (KPIs) are designed to measure the program’s outputs, and not the program’s outcomes. In this context, a number of the performance measures are akin to intermediate¹⁹ and proxy²⁰ performance measures. While these measures afford some useful insights about the performance of the program they do not enable the administering department, or other interested stakeholders to assess the extent to which the AAIP is achieving its overall objectives.

26. Industry did not have an evaluation plan in place for the AAIP, and had not, at the time of this audit, conducted a formal evaluation of the AAIP in the current contract round.²¹ To enable greater insights into the effectiveness of the AAIP, including the performance of the program against its objective, Education and Training would benefit from implementing a program of evaluation activities for the AAIP. The development of a structured evaluation program in the context of Education and Training’s new approach to service delivery—the Support Network—would be particularly timely.

27. Industry and its predecessor departments largely addressed the intent of Recommendation No.2 from the ANAO‘s 2007 audit of the AAIP (see paragraph 14). Industry advised the ANAO that in light of changes to employer incentive rules in 2011 and 2012, it had decided to discontinue conducting sensitivity analysis of incentive payments and apprenticeship completions—which was related to the second part of the recommendation in the 2007 ANAO report.

Managing Delivery of Australian Apprenticeships Support Services (Chapter 3)

28. The contracts with the AACs are fit-for-purpose and contain terms and conditions consistent with better practice.²² In particular, the key service requirements are clear, and the five key performance indicators (KPIs) and 15 performance measures are useful to help assess the AACs’ performance against those requirements. However, most of the benchmarks associated with two of the five KPIs are based on information dating from 31 December 2010. The moderate (and growing) time-difference between the date of the data used to set the benchmarks, and the dates of the actual assessments of the AACs’ performance, affects the usefulness and reliability of these measures.

29. Industry adopted a multi-faceted and well-targeted approach to monitoring the performance of the AACs, which examined key areas of the AACs’ service delivery policies, practices and decision-making. In this context, Industry’s performance monitoring included arrangements for assessing the AACs’ compliance with contractual requirements, including requirements relating to the assessment of eligibility. The ANAO’s analysis indicated that, for the most part, Industry’s performance monitoring processes were operating as intended.

30. Monitoring undertaken by Industry indicated that the majority of AACs were performing reasonably well against most performance targets. In particular, the results indicated that there had been a general improvement in the quality of the AACs’ administration. In addition, the majority of AACs were meeting targets associated with the majority of measures relating to apprenticeship commencement and completion measures—measures relevant for analysing achievement against the program’s outcomes. On the other hand, the majority of AACs were not meeting benchmarks for most measures relating to apprenticeship retention rates. Industry advised that a number of external factors were likely to have affected performance in relation to retention rates, including a downturn in economic conditions and the associated tighter labour market.

31. The ANAO’s analysis indicated that the controls in place to protect the confidentiality and integrity of data in TYIMS were largely operating as intended. Furthermore, the fee-for-service (FFS) and financial assistance expenditure during the period 1 July 2012 to 30 June 2014 examined by the ANAO was materially correct. However, as outlined in Table S.1, the ANAO’s analysis of nearly 920 000 financial assistance payment records, worth approximately $1.2 billion, identified some 960 incorrect payments, albeit with a relatively low value (almost $970 000) compared to overall program expenditure.

Table S.1: Incorrect financial assistance payments during 1 July 2012 and 30 June 2014 identified by ANAO analysis

Source: ANAO analysis.

Note A: The value of the actual payment errors identified by the ANAO is financially immaterial—representing less than 1 per cent of total financial assistance expenditure recorded in TYIMS for the period 1 July 2012 to 30 June 2014. At the time of this report, Education and Training was investigating a further four financial assistance payments with a value of $6 700 that were also potentially paid incorrectly.

32. The ANAO identified a number of further inaccuracies among the data recorded in TYIMS. Most significantly, the ANAO found:

• more than 27 600 apprentice records that are potentially duplicated; and
• around 2 500 payment records containing invalid Australian Business Numbers (ABNs).

33. During the audit, Industry advised that it was examining ways of ‘cleaning’ the data in TYIMS, including merging duplicate records, prior to migrating data from TYIMS to the proposed replacement system—the Australian Apprenticeship Management System (AAMS).²³ Further, Industry advised that the data matching and validation functionality in AAMS would be more robust than TYIMS. In addition, to further strengthen the department’s monitoring and compliance activities and help assure the continued accuracy and integrity of program expenditure, there would be merit in Education and Training examining options to regularise data analysis and data mining activities in relation to the AAIP. The design of a program of data analysis could focus on areas of higher risk and exposure to fraud.

Summary of entity responses

34. Education and Training, and Industry and Science each provided responses to the proposed audit report. A summary of these responses is below, with the full responses provided in Appendix 1.

Education and Training

The Department of Education and Training (the department) is committed to providing targeted and relevant investment in training to ensure the development of a more skilled Australian workforce that delivers long−term benefits for our nation and our international competitiveness.

The rigorous administration of the Australian Apprenticeships Incentives Program by the department has been acknowledged by the ANAO in its identification of optimal or “better practice” measures in contractual terms and conditions including performance monitoring as well as operating guidance materials.

The department will implement a program of structured evaluation activity as recommended.

The department has already implemented, or is intending to implement, many of the suggestions contained in this report including those relating to risk assessments at the program and provider levels.

The department is committed to better practice in the delivery of the Australian Apprenticeships Incentives Program.

Industry and Science

I am pleased that the ANAO has found that the overall administration of the AAIP has been generally effective. The audit finding that appropriate contract and program management arrangements were in place and that performance monitoring processes were soundly-based and well-targeted is positive.

Recommendations

Introduction

A skilled and flexible workforce … will be critical to Australia’s future standard of living.²⁴

1.1 Australia’s vocational education and training (VET) system is designed to help individuals develop or enhance workplace skills and knowledge. Supporting people to develop workplace skills is a key factor in improving productivity and workforce participation, and plays a central role in strengthening economic and social wellbeing.²⁵

1.2 Apprenticeships and traineeships²⁶, hereafter referred to as Australian Apprenticeships, are an important component of Australia’s VET system. In June 2013, approximately 25 per cent of students enrolled in the VET system were associated with an Australian Apprenticeship.²⁷ An Australian Apprenticeship is an employment arrangement that combines paid work with a structured program of ‘on-the-job’ and ‘off-the-job’ training. Australian Apprenticeships enable employees to gain experience, develop practical skills and acquire nationally-recognised qualifications. As at 30 June 2014, there were approximately 351 000 apprentices and trainees ‘in-training’.²⁸ Of these, approximately 55 per cent were classified as technicians and trades workers.²⁹

1.3 Australian Apprenticeships are managed under the terms of a training agreement between an employer and the apprentice. Among other things, training agreements are required to set out the employer’s obligations relating to training, and the qualifications to be obtained by the apprentice. The effectiveness of an Australian Apprenticeship is shaped by the employer’s and the apprentice’s level of commitment to the success of, and the outcomes from, the agreement. In this regard, the Australian Government has published a ‘Code of good practice for Australian Apprenticeships’ to help employers and apprentices better understand their obligations and expectations.³⁰

1.4 The Australian Government has a number of programs in place to support Australia’s VET system, and Australian Apprenticeships in particular. The largest component of the Australian Government’s financial support for Australian Apprenticeships is the Australian Apprenticeships Incentives Program (AAIP).

Australian Apprenticeships Incentives Program

1.5 The AAIP commenced in 1998—at that time it was known as the Commonwealth Incentives Programme, before being renamed the New Apprenticeships Incentives Programme. Since 1998 there have been a number of adjustments and refinements to the range of incentives available in relation to Australian Apprenticeships. Fundamentally, however, the aims and structure of the AAIP have broadly remained unchanged.³¹

1.6 The objective of the AAIP is to contribute to the development of a highly skilled and relevant Australian workforce that supports economic sustainability and competitiveness. The Australian Government aims to achieve this objective by:

• providing genuine opportunities for skills-based training and development of employees by providing incentives to employers of eligible apprentices; and
• encouraging people to enter into skills-based training through an Australian Apprenticeship by providing personal benefits.³²

Australian Apprenticeships Support Services

1.7 A key part of the AAIP is the delivery of a broad range of services relating to Australian Apprenticeships—known as the Australian Apprenticeships Support Services (Support Services). The Support Services are delivered under contract by a cohort of organisations, known as Australian Apprenticeships Centres (AACs). Since 1998, there have been five separate contract rounds for the delivery of Support Services:

• Round one—1 May 1998 to 30 November 1999;
• Round two—1 December 1999 to 30 June 2003;
• Round three—1 July 2003 to 30 June 2006;
• Round four—1 July 2006 to 30 June 2012; and
• Round five—1 July 2012 to 30 June 2015.³³

1.8 For the fifth contract round, the Australian Government entered into 72 contracts with 23 different organisations to provide services at around 300 sites across Australia.³⁴ During the current contract round, approximately 1 000 people were engaged by the AACs to deliver Support Services. In the fifth contract round, the Support Services include:

• providing information, advice and support to employers and apprentices;
• working towards improving apprenticeship participation rates in the key priority groups—Indigenous Australians; people with a disability; school-based apprentices; mature age workers; and occupations identified as having a ‘skill need’;
• undertaking a range of promotion and marketing activities;
• maintaining effective working relationships with key stakeholders (such as State Training Authorities, Registered Training Organisations and Group Training Organisations);
• assessing the eligibility of applications and claims for incentives and personal benefits; and
• processing applications and claims for financial assistance in the Training and Youth Internet Management System (TYIMS) accurately and in a timely manner.³⁵

1.9 In September 2014, the then Minister for Industry—now the Minister for Industry and Science—announced the establishment of the Australian Apprenticeships Support Network (Support Network) to replace the Support Services from 1 July 2015. On 21 October 2014, the former Department of Industry released a Request for Tender (RFT) for the establishment of the Support Network. The Australian Government’s online Tender System, AusTender, stated that:

The new [Support Network] arrangement aims to:

• simplify and improve user access to and engagement with the Australian Apprenticeships system by establishing Network Providers as hubs for the delivery of quality end-to-end advice and Support Services to Australia’s apprentices and their employers;
• improve apprenticeship completion and satisfaction rates through the provision of new services designed to deliver integrated, targeted support to apprentices and employers prior to commencement and while they are in-training;
• provide services to assist individuals to find the right VET or employment pathway for them; and
• reduce red tape and the administrative burden on providers, stakeholders and system users, particularly employers.³⁶

1.10 At the time of preparing this report, the successful tenderers had not been announced.

Types of financial assistance

1.11 Table 1.1 shows the categories of financial assistance available to eligible employers and apprentices under the AAIP in the current contract round.

Table 1.1: Categories of financial assistance available under the AAIP

Source: ANAO analysis.

Note A: Tools for Your Trade payments were replaced by the Trade Support Loans scheme from 1 July 2014.

Note B: The Australian Government announced in the 2014–15 Mid-Year Economic and Fiscal Outlook Statement that the financial assistance to apprentices under Support for Adult Australian Apprentices would cease from 1 July 2015.

Eligibility criteria

1.12 The provision of the incentives and personal benefits available under the program is subject to employers and apprentices satisfying the AAIP’s eligibility criteria. As shown in Table 1.2, there are two broad categories of eligibility criteria—the primary and targeted criteria. The primary criteria are common to all incentives and personal benefits, while the targeted criteria determine the level of certain payments.

Table 1.2: AAIP primary and targeted eligibility criteria

Source: ANAO analysis, based on the AAIP Program Guidelines.

Note A: Over the course of the current contract round, there have been a number of policy changes that have affected the AAIP’s eligibility criteria. During the audit, Industry advised the ANAO that these policy changes represented efforts by the Australian Government to rationalise the level and type of support available to the Australian Apprenticeships system from the AAIP, and in particular, to better target and prioritise the funding available to more effectively support Australia’s skills needs. Appendix 3 contains a summary of these policy changes.

Note B: The NSNL was developed by the former Department of Education, Employment, and Workplace Relations to help ensure that incentives were directed towards occupations experiencing persistent skill shortages.

1.13 The key steps in the life of an Australian Apprenticeship under the AAIP are shown in Figure 1.1.

Figure 1.1: Key steps in an Australian Apprenticeship

 

 

Source: ANAO, adapted from A shared responsibility. Apprenticeships for the 21st Century, Final Report of the Expert Panel. January 2011.

AAIP financial and performance information

1.14 To the end of December 2014, total AAIP expenditure in the current Support Services contract round has been approximately $2.8 billion—$2.3 billion in financial assistance and $0.5 billion in fees paid to AACs. A breakdown of estimated and actual AAIP expenditure in each year of the current contract round is shown in Appendix 2.

Administrative arrangements

1.15 In December 2011, responsibility for the administration of the AAIP was transferred from the former Department of Education, Employment and Workplace Relations (DEEWR) to the former Department of Industry, Innovation, Science, Research and Tertiary Education (DIISRTE). As part of further changes to its responsibilities, the department was renamed twice during 2013: in March 2013, to the Department of Industry, Innovation, Science, Climate Change, Research and Tertiary Education (DIICCSRTE); and in September 2013, to the Department of Industry (Industry). On 23 December 2014, the Australian Government transferred responsibility for the administration of the AAIP from Industry to the newly established Department of Education and Training (Education and Training).³⁷ At the same time, Industry was renamed the Department of Industry and Science.

1.16 Apart from references to activity prior to December 2011, the discussion of audit findings in this audit report refers to Industry—the administering department’s name at the time of the audit. However, the audit recommendation and other suggestions for improvement made in the report have been addressed to Education and Training, the responsible department at the time of preparing the audit report.

Previous audit activity

1.17 The AAIP was previously audited by the ANAO in Audit Report No.9 2007–08, Australian Apprenticeships. The earlier audit assessed the effectiveness of the former Department of Education, Science and Training’s (DEST) administration of its role in Australian Apprenticeships, including, whether it:

• monitored whether Australian Apprenticeships was achieving its objectives; and
• effectively managed the AAIP, including the contracts with the AACs.

1.18 The audit found the program was appropriately used by employers, payments were accurate, and contract management practices were sound. However, the audit did identify some limitations in relation to performance monitoring and evaluation activities. The audit report made two recommendations, one of which (Recommendation No.2) is relevant to the current audit—that the administering department improve performance monitoring and reporting processes by:

• analysing program usage by employers of apprentices and trainees in occupations in national demand; and
• performing sensitivity analysis of incentives payments to employers compared with Australian Apprenticeships completions.

Audit objective, criteria, scope and methodology

Audit objective and criteria

1.19 The objective of the audit was to assess the effectiveness of Industry’s administration of the AAIP. To form a conclusion against the audit objective, the ANAO adopted the following high-level criteria:

• an effective governance framework for the AAIP had been implemented and Industry provided suitable guidance, processes and tools to support AACs to effectively deliver AAIP services;
• suitable contractual arrangements, including sound contract management practices, were in place to support service delivery by AACs and overall program management by the department; and
• appropriate program management, performance monitoring and reporting structures were in place and used to inform the administration of the AAIP.

Audit scope

1.20 The audit focused on arrangements in place since 1 July 2012—the commencement of the current Support Services contract round. In particular, the audit examined:

• arrangements for governing the delivery of the AAIP;
• processes for oversighting the AACs;
• guidance, practices and assurance mechanisms in place to support the:
  • assessment of employers’ and apprentices’ eligibility for incentives and personal benefits; and
  • accurate and complete payment of those incentives and personal benefits;
• performance management and reporting practices, including whether Industry captured information about the success of the program in terms of its objective; and
• the extent of the implementation of Recommendation No.2 in ANAO Audit Report No.9 2007–08.

Out of scope

1.21 The following aspects are not in scope for the current audit:

• an assessment of individual training agreements between apprentices and employers;
• an examination of the procurement exercise conducted by the then DEEWR in 2011–12 to set up the current round of contracts with the AACs;
• an examination of the procurement exercise conducted in 2014–15 to set up the Support Network round of contracts;
• an examination of the AACs’ operations—other than through review of Industry’s records, including copies of performance reports provided to Industry; and
• the administration of the Trade Support Loans (TSL) scheme.

Audit methodology

1.22 The audit methodology included: the examination of relevant AAIP policy and operational documents; an analysis of the results of performance monitoring activities for the current contract round of Support Services; an analysis of data extracted from TYIMS; and an assessment against relevant requirements of the Australian Government’s financial management framework. In addition, the ANAO held discussions with staff in Industry’s national office as well as three of Industry’s state contract managers; as well as with representatives from six AACs.

1.23 The audit was conducted in accordance with the ANAO’s Auditing Standards at a cost to the ANAO of approximately $449 000.

Structure of the Audit Report

1.24 The structure of the audit report is shown in Table 1.3.

Table 1.3: Structure of the Audit Report

Source: ANAO.

Introduction

2.1 Delivering the Australian Government’s programs and services is one of the key responsibilities of government entities. In achieving the Government’s intended outcomes, public sector entities have to deal with a range of complex challenges. Some of the contemporary challenges include: the need to continually innovate and manage risks; engaging constructively with interested stakeholders and, in some cases, the broader community; and, as necessary, collaborating with other entities, including across jurisdictional and sectoral boundaries. Within this context, effective program management arrangements and practices contribute to sound, sustainable and accountable program and service delivery performance.³⁸

2.2 Typically, effective program management arrangements include: robust risk assessment and monitoring arrangements; clear planning and program documentation; well-targeted. operating procedures; accurate and informative guidance material; and ongoing monitoring and reporting of performance.

2.3 To evaluate the effectiveness of the Australian Apprenticeships Incentives Program’s (AAIP) program management arrangements, the ANAO examined whether Industry had:

• structured processes in place to identify and manage risks, including the risk of fraud and the risks associated with conflicts of interest;
• established suitable program guidance and support arrangements, including well-designed program guidelines;
• met key financial framework requirements relevant to the administration of the AAIP; and
• developed (and reported against) measures to provide insights into the AAIP’s performance, including performance against its objectives and outcomes, as well as key aspects of the administration of the AAIP.

Risk management

2.4 Risk management is an integral part of effective public administration and plays an important part in securing value with public money.³⁹ Entities should adopt a structured approach to identify, assess, treat and monitor potential threats or risks to the effective administration of government programs or the achievement of policy objectives.

2.5 The Public Governance, Performance and Accountability Act 2013 (PGPA Act)—which took effect from 1 July 2014—establishes a new framework for the use and management of public resources by Commonwealth entities and encourages managers to take a risk-based approach. In particular, under section 16 of the Act, the ‘accountable authority’ of a Commonwealth entity⁴⁰ must ensure that appropriate systems and processes are in place relating to risk and control.

AAIP-related risk management arrangements

2.6 During the current Australian Apprenticeships Support Services (Support Services) contract round⁴¹ Industry undertook two risk assessments that are pertinent to the AAIP. These assessments relate to the:

• delivery and management of the Support Services; and
• the operation of the Training and Youth Internet Management System (TYIMS).

2.7 Overall, the ANAO’s analysis indicated that the risk assessments were well-designed While the two assessments were not integrated—having been developed at different times by different areas of the department—the ANAO’s analysis showed that the information in the plans was complementary.

2.8 Together, the two risk assessments covered a broad range of relevant risks. Most of the risks were described in sufficient detail and considered to be appropriate. Nevertheless, there are a few additional risks that the Department of Education and Training (Education and Training)—the administering entity from 23 December 2014—may wish to consider for inclusion in future risk assessments for the AAIP. This includes risks arising from:

• implementing significant changes to the program, including the eligibility criteria, at short notice;
• disruptions to the availability of AACs’ premises or staff; and
• loss of institutional knowledge about the program.

2.9 For the most part, the controls and treatments listed in the risk assessments were suitable and appropriate. However, the accuracy and usefulness of the Support Services risk assessment would be enhanced by addressing the following shortcomings:

• no controls were documented for risk 24 (false information provided in relation to eligibility criteria). That said, the ANAO observed that Industry did have a variety of controls and processes designed to help mitigate the risks around fraudulent claiming of financial assistance; and
• the description of the identified controls did not clearly align with the risk event in a number of cases, including: risk seven (AACs do not have appropriate financial management measures); risk 12 (AACs management and administration of contract is ineffective); and risk 18 (failure to act on agreed terms in the contract).

2.10 In addition, the following controls listed against risks 21 and 22 (relating to fraud) in the Support Services risk assessment were not operating as intended during the current contract round:

• ‘Audit and Investigations Group undertake performance audits of AACs’;
• ‘targeted monitoring resulting from trend analysis’; and
• ‘Audit and Investigations Group undertake data analysis’.

2.11 At the time of the audit, Industry’s entity-wide Fraud Risk Assessment (FRA) also contained information on AAIP-specific fraud risks and controls.⁴² The ANAO’s analysis indicated that one of the controls listed in the FRA—namely, ‘data mining to examine claiming trends’—was not operating as intended during the current contract round. Industry advised the ANAO that it did not undertake any structured data analysis or data-mining activities. However, it advised that ad-hoc enquiries would be undertaken in response to issues identified, including allegations of potential errors or fraud.

2.12 The ANAO also observed that the Support Services risk assessment did not include details of monitoring strategies for the identified risks or controls. Industry advised that risk assessments were required to be reviewed at least annually and also if there were significant changes to either the program or the Support Services contracts. However, the ANAO observed that formal reviews of the Support Services and TYIMS risk assessments did not occur as often as might be expected during the current contract round. Specifically, both assessments were only reviewed once during the current contract round—the Support Services plan in July 2013; and the TYIMS risk assessment in August 2012.

2.13 Education and Training would benefit from strengthening arrangements for monitoring and reviewing risk assessments associated with the AAIP. Monitoring and review can be particularly valuable in helping: assess the accuracy of the identified risks and the attendant risk controls and treatments; provide insights about new and emerging risks; and confirm the ongoing suitability and continued operational effectiveness of risk treatments and controls. In the case of the AAIP, regular monitoring of risks is important in light of the ongoing changes to aspects of the program’s design and fluctuations in the economy and the labour market, which can affect the take-up of apprenticeships and, in turn, the operation of the AACs.

AACs’ approach to risk management

2.14 Under their contracts with the Australian Government, the AACs are required to have a risk management plan (RMP). Each year, as part of their reporting to the department, the AACs are required to provide a copy of the plan, together with details of any actions taken under the plan.⁴³

2.15 While each of the reports examined by the ANAO contained a copy of the relevant AAC’s RMP, none of the reports examined contained supporting commentary or analysis about the RMP, including a description of recent, current or proposed activity in relation to the RMP. In this context, the ANAO reviewed the RMPs of a selection of AACs and observed significant variation in terms of completeness, quality and focus on Support Services risks. In particular, only four of the reviewed plans had an explicit focus on the Support Services. The other plans reviewed either only outlined the AAC’s risk management policy (without identifying specific risks) or identified risks largely from their own business, rather than a Support Services perspective.

2.16 Education and Training could usefully clarify expectations about the RMPs by providing advice to the AACs on key areas of focus, such as the core service delivery risks. Such advice should encourage AACs to give greater attention to the identification and management of shared program risks. Education and Training advised the ANAO that guidance on the content of RMPs will be included in the Operating Guidelines issued under the terms of the new Australian Apprenticeships Support Network (Support Network) contractual arrangements.⁴⁴ Further, there would be merit in Education and Training examining ways to gain assurance of the continued effectiveness and operation of the service providers’ RMPs.

Managing conflicts of interest

2.17 The effective management of conflicts of interest⁴⁵ is an important aspect of the administration of Australian Government services. The potential for conflicts of interest is a noteworthy risk in the administration of the AAIP. For instance, departmental records showed that the majority of organisations contracted to deliver Support Services are also involved in the delivery of other services relevant to Australian Apprenticeships. Specifically, 16 of the organisations are also either a Registered Training Organisation or a Group Training Organisation or, in some cases, both. A number are also involved in the provision of other employment services, such as recruitment.

2.18 Industry had a number of processes in place to manage conflicts of interest risks associated with the delivery of the AAIP. These arrangements included processes for promoting awareness about, and obtaining up to date information on, conflicts of interest. Specifically, Industry’s ‘Code of Conduct in Contracting’ set out the expectations of staff involved with the department’s business partners, and provided practical guidance on potential conflict situations.

2.19 The AACs’ contracts with the Australian Government contained a number of provisions relating to the management of potential conflicts of interest. Among other things, the AACs are required to:

• report details of any conflicts that arise, including the steps taken to resolve the conflict⁴⁶; and
• maintain a Conflict of Interest Management Plan (COIMP)— each year, as part of their contractual reporting obligations, the AACs are required to confirm whether the strategies identified in the COIMP for managing actual or perceived conflicts of interest have been effective.

2.20 The ANAO examined a sample of AACs’ COIMPs and observed that most of the plans indicated that the AACs had structured processes in place for managing conflicts of interest. In particular, many of the plans outlined practical strategies and identified clear accountabilities for dealing with conflict of interest situations.

Program guidance and support

2.21 The ANAO examined the design and content of key program guidance and support material, including: the AAIP Program Guidelines; the Support Services Operating Guidelines; and application and claim forms. In addition, the ANAO examined other arrangements for supporting the AACs, including, Industry’s processes for keeping them informed about relevant policy changes.

Program Guidelines

2.22 The design of the Program Guidelines established for a grant program can play a central role in effective and accountable grants administration. Specifically, under the Australian Government’s grants policy framework, officials must develop, and make publicly available, guidelines for all new grant programs.⁴⁷

2.23 The appropriate design and the content of grant guidelines will vary depending on the size, scope and nature of the grant program. However, an underlying principle is that grant guidelines support effective grants administration by addressing those matters necessary to promote transparent and equitable access to the grant program. A number of common information elements should nonetheless be addressed in grant guidelines, including information on the:

• program’s purpose, scope and objectives;
• roles and responsibilities of those involved in administering the program;
• methods of applying for funding under the program; and
• processing of applications or claims, including assessments against the program’s eligibility criteria.⁴⁸

2.24 The ANAO assessed the extent to which the design of the AAIP Program Guidelines reflected the better practice guidance on information elements contained in the ANAO’s grants administration Better Practice Guide and in the 2013 Commonwealth Grant Guidelines (CGGs). Overall, as shown in Table 2.1, the Program Guidelines largely addressed these elements.

Table 2.1: ANAO assessment of the content of the AAIP Program Guidelines

Source: ANAO analysis.

Note A: The nature of the AAIP as a grants program is not disclosed.

Note B: The total limit of program funding was not disclosed in the guidelines. However, the amounts payable for each individual incentive and personal benefit are outlined.

2.25 Overall, the ANAO’s analysis indicated that the AAIP Program Guidelines are designed to give effect to the policy intent of the program. Representatives of each of the six AACs interviewed by the ANAO indicated that the information in the Program Guidelines was useful and informative. However, a common view was that the design of the guidelines could be made more user-friendly. For instance, several of the AACs observed that interpreting technical elements, especially around eligibility, can be confusing.

Operating Guidelines

2.26 The Operating Guidelines outlined the administrative and procedural arrangements between the department and the AACs. Among other things, the Operating Guidelines:

• described the department’s approach to monitoring the performance of the AACs; and
• contained procedural and administrative detail on claim processing, reporting and other contractual requirements, including the rules relating to determining the AACs’ fees.

2.27 Overall, the ANAO’s analysis indicated that the nature and level of the information contained in the Operating Guidelines was useful and well-targeted. In particular, the design of the Operating Guidelines can be expected to: help support consistency in the delivery and management of the Support Services; and encourage effective and efficient claim management and decision-making by the AACs.

Application and claim forms

2.28 To support consistent and effective claim processes, Industry used a suite of standard application and claim forms for the different types of AAIP payments. Overall, the ANAO’s analysis indicated that the application and claim forms were well-designed. In particular, the forms:

• were sensibly structured and clearly written;
• included guidance and key messages that were consistent with the Program Guidelines;
• reflected the evidence requirements of the Program Guidelines;
• included appropriate and relevant requests for information and supporting evidence; and
• contributed to efficient and effective claims administration. In particular, the forms were designed to be largely auto-populated from information in TYIMS, minimising the need for applicants to enter information previously provided.

2.29 The ANAO observed that the relative brevity of the application and claim forms is consistent with the contractual requirements on the AACs to provide support to employers and apprentices. Given the complexity of AAIP policy, including the myriad of eligibility conditions, the provision of support by the AACs is a more effective approach to supporting claimants than adding more explanatory material to the forms.

Other support mechanisms

2.30 Industry had a number of other processes designed to support the AACs keep abreast of program developments and support their administration of the Support Services.

Advice on policy changes

2.31 Industry’s national office was responsible for ensuring that the AAIP’s policy changes were reflected in relevant reference material. The national office notified the department’s state contract managers (SCMs) when policy changes occurred. The SCMs, in turn, notified the AACs in their sphere of influence about the changes.

2.32 As outlined in chapter 1, a number of policy amendments were made to the AAIP by the Australian Government during the current contract round. The ANAO’s analysis showed that details of each of these policy changes had been accurately incorporated into the Program Guidelines; and notified to SCMs and to the AACs, in a timely manner.

2.33 The ANAO’s examination of Industry’s arrangements for dealing with issues raised by the AACs—such as seeking clarification of AAIP policy—indicated that the department’s approach was functioning appropriately. SCMs and their staff advised the ANAO that they usually respond directly to queries from the AACs, and only sought clarification from national office where they considered the query had national implications. While this approach is reasonable, to help minimise the risk of inconsistent advice being provided to AACs, there would be merit in Education and Training developing a protocol for dealing with AACs’ queries, including outlining the characteristics of queries that should be escalated to the department’s national office.

Training

2.34 As part of the transition from the fourth to the fifth Support Services contract round, Industry provided a structured program of training to all AACs. This training was delivered by SCMs and their staff using material prepared by Industry’s national office. Training on the use of TYIMS was also provided, particularly aimed at new AACs. The ANAO observed that the training materials were clear and covered key aspects of the Support Services arrangements.

2.35 The ANAO’s interviews with SCMs and representatives of the AACs indicated that further training during the current contract period had typically occurred on an informal and ad-hoc basis. For instance, the ANAO was advised that informal training sessions were provided at the time of policy changes and also following six-monthly file monitoring exercises. One AAC also advised the ANAO that the department had assisted in the provision of training to a number of new starters in one of its processing teams.

2.36 To complement the training offered by Industry, the AACs are also required to provide training and staff development activities. The ANAO’s examination of a sample of AACs’ annual Operational reports during the current contract round indicated that most AACs had provided Industry with details of training relevant to the AAIP. Among other things, this included training on: the Program Guidelines; the Operational Guidelines; TYIMS; customer service; and cultural development. An example of good practice observed in one case, was that Industry’s fraud team was scheduled to provide fraud awareness training to a selection of the AAC’s staff.

Key financial framework requirements

2.37 Industry’s administration of the AAIP was subject to the requirements of the Australian Government’s financial framework established by the former Financial Management and Accountability Act 1997 (FMA Act) and the associated FMA Regulations.⁴⁹ Specifically, the ANAO examined whether Industry met the following financial framework requirements:

• approval of spending proposals—former FMA Regulation 9; and
• acting in accordance with the former CGGs.

Approving spending proposals

2.38 The former FMA Regulation 9 prohibited the approval of a spending proposal unless the approver was satisfied, after making reasonable enquiries, that the spending proposal would be a proper use of Commonwealth resources.⁵⁰ Industry had separate processes for obtaining approval of spending proposals for each of the key components of AAIP expenditure—fee-for-service (FFS) and financial assistance (employer incentives and personal benefits).

2.39 The ANAO’s analysis indicated that the necessary approvals under FMA Regulation 9 were obtained for the FFS expenditure associated with the current round of Support Services contracts—both the initial two-year term from 1 July 2012; and the 12 months extension of the term to 30 June 2015. In relation, to financial assistance expenditure, Industry advised that approval (under FMA Regulation 9) was obtained each time the Program Guidelines were updated. The ANAO’s analysis showed that the necessary approvals were obtained for each new version of the Program Guidelines in the current contract round from February 2013 onwards. However, while each of earlier versions of the guidelines in the current contract round—namely July 2012; August 2012; and October 2012—stated that approval under FMA Regulation 9 had been given, Industry was unable to provide evidence of such approvals.

The Commonwealth Grant Guidelines

2.40 The CGGs⁵¹—and from 1 July 2014, the Commonwealth Grants Rules and Guidelines (Grants Rules)⁵²—set out the Australian Government’s requirements relating to the administration of grant programs. In particular, the CGGs and the Grants Rules outline seven key better practice principles to be adopted in granting activities.

2.41 Following the release of the CGGs in July 2009, the then Department of Education, Employment and Workplace Relations (DEEWR) assessed that the AAIP was not a grants program and, as such, did not fall within the scope of the CGGs. The department’s view was based on the following exemption from the CGGs listed in the then FMA Regulations:

a payment of benefit to a person, including a payment of an entitlement established by legislation or by a government program [emphasis added]. ⁵³

2.42 Amendments to the FMA regulations in May 2013⁵⁴ altered the definition of grants. However, following the re-release of the CGGs in June 2013, Industry did not reassess if the financial assistance provided by the AAIP aligned with the updated definition of a grant in the FMA Regulations.⁵⁵

2.43 The ANAO assessed that the AAIP met the revised definition of a grants program in the amended FMA Regulations and the 2013 CGGs (and now the Grant Rules). As such, Industry was required, for the period 1 June 2013 to 30 June 2014, to comply with the requirements of the 2013 CGGs, and from 1 July 2014, to comply with the requirements of the Grant Rules.

2.44 Following the ANAO’s enquiries, Industry acted in a timely manner to consider the application of the grants administration framework to the AAIP. Industry advised the ANAO that it established that the AAIP was a grants program according to the revised definition and indicated that:

• details of AAIP’s grants expected to be paid in 2014–15 had been reported on the department’s website;
• a breach of the reporting requirements of the Grants Rules has been recorded in the department’s Certificate of Compliance (CoC) register for the 2014–15 financial year; and
• the department’s CoC return for the 2013–14 financial year would be updated to reflect breaches of the former CGGs.

2.45 Industry’s experience in this case is a reminder that the administration of its programs needs to be consistent with the Australian Government’s financial management framework requirements—and revisions to that framework which occur from time to time—in particular, requirements around grants administration. The ANAO also identified issues relating to Industry’s understanding and application of the grants administration framework in its recent performance audit of the (then) Commercialisation Australia Program.⁵⁶

Measuring program performance

2.46 Well-designed performance measurement and reporting processes can assist managers by providing:

• a solid basis against which to measure progress and performance;
• an understanding of the drivers of progress and performance; and
• insights into whether the program is being administered effectively and is delivering the outcomes expected by government.

2.47 In particular, regular and ongoing monitoring and review of a program’s performance—using, among other things, well-timed reviews and evaluations, and key performance indicators—assists entities to assess the program’s progress towards its intended outcomes and expected benefits.

Management reporting

2.48 Entities should have processes in place to disseminate relevant information about a program’s performance, including the effectiveness and efficiency of its administration. The ANAO considered whether Industry analysed and reported information to help provide insights into the AAIP’s operations, as well as its performance.

2.49 Industry had well–established internal reporting arrangements in place for assisting managers to maintain oversight of the AAIP. Among other things, the key management reports examined by the ANAO provided:

• quantitative data on Australian Apprenticeships activity levels;
• analysis of AAIP expenditure levels and trends;
• details of the performance of the AACs against their contractual requirements; and
• details on the number and amount of Special Claims.⁵⁷

2.50 Overall, the ANAO considers that internal departmental reporting on the AAIP was satisfactory with reports generally containing sufficient and appropriate information. Managers had access to useful and relevant information to maintain effective oversight of the program, and to gain timely insights into trends and performance. Table 2.2 summarises the ANAO’s analysis of the usefulness of the suite of internal AAIP reports.

Table 2.2: Assessment of AAIP internal management reports

Source: ANAO analysis.

Key: M = met; P = partly met.

Key Performance Indicators

2.51 Entities should have a balanced set of key performance indicators (KPIs) in order to provide management, and other interested stakeholders, with a mix of perspectives about a program’s performance. An appropriate set of KPIs will provide information concerning the outputs delivered, the outcomes achieved, as well as information about the effectiveness and efficiency of the way the program is administered.⁵⁸ In particular, when considered together, the KPIs should present a view as to whether a program is meeting its stated objective.⁵⁹

2.52 As shown in Table 2.3, Industry had two sets of KPIs relevant to measuring the progress and performance of the AAIP during the current contract round. These were the:

• four measures⁶⁰ set out in relevant Portfolio Budget Statements (PBS); and
• five indicators used to assess the performance of the contracted service providers in delivering the Support Services.

Table 2.3: Relevant AAIP performance measures

Source: Industry’s Portfolio Budget Statements 2012–13 and 2013–14, and the Australian Apprenticeships Support Services Operating Guidelines.

Note A: In 2012–13 and 2013–14, these measures were described as ‘deliverables’. In 2014–15, the measures are described as ‘contributing component performance measures linked to an overarching performance measure’, namely: ‘Increased participation in apprenticeships and increased skills in the workforce’.

Note B: The design and operation of the five KPIs relating to measuring the performance of the AACs is examined in chapter 3.

2.53 The ANAO assessed the design of the AAIP’s performance measures against the elements of better practice outlined in ANAO Report No.28
2012–13 Pilot Project to Audit Key Performance Indicators.⁶¹ The results of the ANAO’s analysis are shown in Table 2.4.

Table 2.4: ANAO assessment of AAIP performance measures

Source: ANAO analysis.

2.54 As shown in Table 2.4, the ANAO’s analysis indicated that the AAIP’s KPIs only partly meet the better practice criteria. In large part, the AAIP’s KPIs are designed to measure the program’s ‘outputs’, and not the extent of achievement against the program’s outcomes. In this context, a number of the AAIP’s KPIs are akin to intermediate performance measures—for instance: the numbers of employers and apprentices assisted; and the level of satisfaction with AACs’ services. Intermediate performance objectives or targets are important in cases where overall outcomes can only be achieved over the longer-term. In particular, intermediate performance measures can help demonstrate progress towards, and promote better understanding of the factors contributing to the achievement of longer-term outcomes.⁶²

2.55 Importantly, two of the AAIP’s measures—the measures relating to apprenticeship participation rates (KPIs 2 and 3)—are useful proxy measures.⁶³ Such measures are valuable where overall program objectives are difficult to measure because, for instance, the objectives have been set at a high level or their achievement is dependent on a number of external factors.⁶⁴ In particular, these two measures provide insights into apprenticeship trends and, more broadly, the AAIP’s contribution to ‘developing a highly skilled workforce’.

2.56 Overall, while the suite of performance measures developed by Industry offers some useful insights, it is not possible to determine from these measures alone the extent that the program is achieving its overall objective. The relatively limited insights about the AAIP’s overall performance that can be obtained from the KPIs, underlines the importance of well-designed evaluation activity to support broader assessments of the program’s overall performance.

Performance against the targets in the PBS

2.57 For each of the two completed years of the current Support Services contract round, the actual level of performance for the three KPIs contained in the PBS has been reported in Industry’s annual reports. As shown in Table 2.5, Industry’s annual reports for 2012–13 and 2013–14 show that while the actual number of employers assisted in 2012–13 was significantly below target, the combined totals for employers and apprentices assisted through the AAIP in these years exceeded the combined targets. In particular, the number of apprentices assisted in both years is significantly higher than expected.

Table 2.5: Annual reporting of AAIP performance

Source: ANAO analysis.

Note A: The department’s 2012–13 annual report stated that the number of employers assisted in 2012–13 fell below expectations following changes to the program that removed eligibility for some payments.

Evaluations

2.58 Program evaluations are an important part of maintaining an outcomes orientation—one of the key grants administration principles set out in the Grants Rules.⁶⁵ Evaluations involve consideration of the appropriateness, effectiveness, efficiency and achievements of an initiative. By incorporating both qualitative and quantitative analysis, well-designed evaluation activity can facilitate a thorough analysis of program design and performance issues. Evaluations can also provide insights into the success of the program against its objective, and inform its future direction.

2.59 Industry did not have an evaluation plan for the AAIP, and had not, at the time of the audit, conducted a formal evaluation of the AAIP during the current contract round. Industry advised the ANAO that it monitored relevant external research activity, such as the work of the National Centre for Vocational Education Research (NCVER), to assist it to gauge the effectiveness of the AAIP.⁶⁶

2.60 In 2011, the then DEEWR engaged an external reviewer to assess the efficiency and effectiveness of the range of financial assistance available under the program.⁶⁷ The review report offered Industry useful insights to help it to better target the types of financial assistance provided by the AAIP. In particular, the ANAO’s analysis indicated that Industry drew on the findings of the review report in recommending many of the policy amendments adopted by government during the current contract round.

2.61 Given the economic and financial materiality of the program, Education and Training would benefit from implementing a targeted program of evaluation activities for the AAIP. A structured evaluation program would ideally include a longer–term focus—this is particularly important in the context of the new service delivery contract round starting on 1 July 2015. An overarching evaluation plan would facilitate consideration of the appropriate timing and focus of evaluation activities.

Recommendation No.1

2.62 To assist the Department of Education and Training better assess the efficiency and effectiveness of the Australian Apprenticeships Incentives Program, including performance against the program’s policy objective, the ANAO recommends that the department implement a program of structured evaluation activity.

Education and Training’s response:

2.63 Agreed.

The ANAO’s previous recommendation

2.64 The ANAO examined the extent that Recommendation No.2 in the ANAO’s 2007 performance audit of Australian Apprenticeships had been implemented.⁶⁸ Recommendation No.2 was that the then Department of Education, Science and Training (DEST):

… strengthen its analysis of Incentives Programme data to improve performance monitoring and reporting, and policy analysis development by:

• analysing programme usage by employers of apprentices and trainees in occupations in national demand; and
• performing a sensitivity analysis of incentive payments to employers compared with Australian Apprenticeship completions.

2.65 Overall, the ANAO considers that Industry and its predecessor departments had largely implemented Recommendation No.2 from the ANAO’s 2007 audit report. Specifically, during the current audit, the ANAO observed that Industry had processes in place to capture details about, and to monitor, Australian Apprentices in occupations in national demand—the first part of the recommendation. However, there was no evidence that Industry had undertaken any sensitivity analysis of incentive payments to employers—the second part of the recommendation.

2.66 Industry advised the ANAO that following the 2007 audit, DEST commenced a program of quarterly sensitivity analysis of incentive payments focussed on the practice of ‘churning’.⁶⁹ Following changes to the employers’ incentives regime⁷⁰, Industry advised that it had assessed the risks involved with ‘churning’ to be less prevalent and decided that further regular analysis was no longer necessary.

2.67 As well as gaining insights into the issues associated with ‘churning’, the ANAO’s 2007 report also highlighted other potential advantages of analysing incentives payments, such as measuring program performance and assessing the program’s impacts. In particular, the report stated…

… performing sensitivity analysis of incentive payments to employers compared to Australian Apprenticeships completions, is likely to assist DEST to better understand the influence of any potential policy variation on the cost-effectiveness of the … program.⁷¹

2.68 In this context, there would be benefit in Education and Training periodically undertaking sensitivity analyses of employer incentive payments. Ideally, such analysis would be built into any structured evaluation activity (see Recommendation 1 at paragraph 2.62).

Conclusion

2.69 Industry’s risk assessments relating to the AAIP’s administration covered a broad range of risks and controls. Most risks were described in sufficient detail so as to be informative, and for the most part, the controls and treatments found in the risk assessments were suitable and appropriate. However, no controls were documented in the Support Services risk assessment for one of the identified risk events (risk 24, false information being provided in relation to eligibility criteria), and there would be benefit in doing so. The risk assessments could be further enhanced by strengthening arrangements for monitoring and review. A structured program of monitoring and review will help: confirm the accuracy of the identified risks and the attendant risk controls and treatments; assess whether the risk mitigation measures are well targeted and operating as intended; and provide insights about new and emerging risks.

2.70 Each of the AACs is required to develop and maintain a RMP and a COIMP. ANAO analysis of a sample of COIMPs indicated that most contained practical strategies and relevant processes for dealing with conflict of interest situations. However, many of the RMPs examined only outlined the AAC’s risk management policy (without identifying specific risks) or identified risks not tailored to the services being delivered. Education and Training advised that it would provide guidance on the form and content of risk management plans to the providers engaged in the delivery of the new Support Network. In addition, there would be merit in Education and Training examining ways to gain assurance on the continued effectiveness and operation of the service providers’ RMPs.

2.71 Industry did not assess whether the AAIP was affected by amendments to the former FMA Regulations in May 2013 and the re-release of the former CGGs in June 2013 that altered the definition of a grants program. Following the ANAO’s enquiries, Industry acted in a timely manner to consider the application of the grants administration framework to the AAIP, and established that the AAIP was a grants program according to the new definition.

2.72 Industry had in place a range of useful and informative support and guidance arrangements to assist staff and the AACs administer the program. In particular, key program documentation—the AAIP Program Guidelines and the Support Services Operating Guidelines—were well-designed. The department also had well–established internal performance monitoring and reporting arrangements for the AAIP, providing managers with access to a range of relevant and useful information.

2.73 In large part, the AAIP’s KPIs are designed to measure the program’s outputs, rather than the program’s outcomes. In this context, a number of Industry’s performance measures are akin to intermediate and proxy performance measures. While these intermediate and proxy measures afford some useful insights about the performance of the program—for instance, Industry’s annual reports for 2012–13 and 2013–14 show that the combined total of employers and apprentices assisted through the AAIP in these years exceeded the combined targets—the measures do not enable the administering department, or other interested stakeholders, to assess the extent to which the AAIP is achieving its overall objectives.

2.74 Industry did not have an evaluation plan in place for the AAIP, and had not, at the time of this audit, conducted a formal evaluation of the AAIP in the current contract round. Education and Training would benefit from implementing a program of evaluation activities for the AAIP, so as to enable greater insights into the effectiveness of the program, including the performance of the program against its objective. The development of a structured evaluation program in the context of Education and Training’s new approach to service delivery—the Support Network—would be particularly timely.

2.75 Industry and its predecessor departments largely addressed the intent of Recommendation No.2 from the ANAO‘s 2007 audit of the AAIP. In relation to the conduct of sensitivity analysis—the second part of the recommendation—Industry advised that it had decided to discontinue the analysis of incentive payments and apprenticeship completions following changes to employer incentive rules in 2011 and 2012.

Introduction

3.1 Contracts for the delivery of services on behalf of the Australian Government need to be actively and consistently managed. Among other things, the management of contracted service providers should be designed to ensure that:

• contract requirements and obligations are being met;
• the performance of the service providers meets required standards or benchmarks; and
• value for money outcomes are achieved, within agreed timeframes.⁷²

3.2 The effective management of the delivery of contracted services is facilitated by well-designed and informative contracts. The form and content of contracts should be commensurate with the scale and the risks associated with the services to be delivered.

3.3 To evaluate the effectiveness of Industry’s management of the contracts for the delivery of services associated with the Australian Apprenticeships Incentives Program (AAIP), the ANAO examined whether the:

• design of contracts provides a sound foundation for the delivery of the services and the active management of those services by the department;
• performance of the contracted service providers, including their achievement against contract deliverables, is regularly and effectively monitored; and
• mechanisms in place for dealing with complaints about the program and for reviewing decisions on eligibility were operating effectively.

3.4 The ANAO also examined the accuracy and integrity of the data contained in the Training and Youth Internet Management System (TYIMS)⁷³, including records of payments made during the period 1 July 2012 to 30 June 2014.

Contracts for the Australian Apprenticeships Support Services

3.5 The Australian Government contracts with a cohort of organisations, known as Australian Apprenticeships Centres (AACs), to provide Australian Apprenticeships Support Services (Support Services).⁷⁴ The management of the delivery of the Support Services is a key part of Industry’s administration of the AAIP.

3.6 The AACs are paid a fee for each Australian Apprenticeship that they administer—hereafter referred to as a fee-for-service (FFS) payment. FFS payments are payable in instalments depending on the progress of the Australian Apprenticeship and, in some cases, the milestone achieved by the AAC. In the fifth contract round, the amount of FFS payable to the AACs is dependent on:

• the zone⁷⁵ that the AAC operates in; and
• whether the apprentice is categorised as a Tier one or Tier two employee—the Tier one rate is payable for apprentices defined as existing workers and for apprentices undertaking Australian Qualification Framework level I or II qualifications. Tier two relates to all other apprentices.

3.7 To assess the sufficiency and appropriateness of the Support Services contracts, the ANAO examined whether the contracts contained a set of desirable provisions.⁷⁶ As outlined in Table 3.1, the ANAO’s analysis indicated that the contracts contained terms and conditions consistent with better practice.

Table 3.1: Examination of the Support Services contract terms and conditions

Source: ANAO analysis.

Note A: Each of three major variations to the Support Services contract during the current contract term was properly approved, and an examination of a selection of individual contracts indicated that the variations were appropriately processed and executed.

Note B: The ANAO examined a sample of contracts—one contract for each AAC—and confirmed that the contracts had been signed by appropriate representatives of both the department and the AAC prior to the start of the contract period on 1 July 2012.

Monitoring the performance of the AACs

3.8 The contract management approach adopted by an entity, including the level or type of resources devoted to the task, should reflect the purpose, complexity, duration and value of the contracts. In this context, the contractual arrangements with AACs, and the attendant risks, are significant. Specifically, in the current contract round:

• the total amount of FFS payments to the AACs is estimated to be around $210 million per annum;
• the AACs are responsible for processing—including assessing the eligibility of—applicants’ claims for incentives and benefits estimated to total some $2.4 billion;
• the services are delivered over a three year period by 23 organisations—under 72 individual contracts—at more than 300 sites in Australia⁷⁷; and
• the range of services required to be delivered by the AACs is broad.

Performance and compliance monitoring arrangements

3.9 As shown in Table 3.2, Industry had a multi-faceted approach to monitoring the performance of AACs in delivering the Support Services.

Table 3.2: Key performance monitoring components

Source: ANAO analysis, based on the Support Services Operating Guidelines.

Note A: At the time of the audit, four rounds of file monitoring had occurred: Round one—covering the period 1 July 2012 to 31 March 2013; Round two—covering the period 1 April 2013 to 30 September 2013; Round three—covering the period 1 October 2013 to 31 March 2014; and Round four—covering the period 1 April 2014 to 30 September 2014. A further round of file monitoring was planned—covering the period 1 October 2014 to 31 March 2015.

Note B: The timing and focus of targeted monitoring was determined by Industry—for instance, based on observations or issues arising from other monitoring activity. At the time of the audit, the department had only completed two formal targeted monitoring activities during the current contract round.

3.10 Industry established five key performance indicators (KPIs) to help it assess the performance of the AACs:

• KPI 1—performance against Business Case;
• KPI 2—increase in apprenticeships retention and completion rates;
• KPI 3—increase in apprenticeships participation rates (commencement, retention and completion) for five priority groups;
• KPI 4—employers’ and apprentices’ satisfaction levels; and
• KPI 5—administrative quality.

3.11 Industry grouped these KPIs into two categories:

• level one—those that the AACs can directly control (comprises KPIs 1, 4, and 5); and
• level two—those that the AACs only have limited capacity to influence (comprises KPIs 2 and 3).

3.12 Across these five KPIs, Industry developed 15 specific measures and 25 different benchmarks against which the AACs’ actual performance was compared—see Table 3.3.

Table 3.3: Measures and benchmarks used to measure AACs’ performance

Source: Support Services Operating Guidelines.

Note A: The national average benchmark rates for the measures comprising KPIs 2 and 3 are based on each of the AAC’s benchmark rates weighted to reflect the amount of each AAC’s Support Services business allocation. An AAC’s Support Services business allocation is the indicative number of Australian Apprenticeships that an AAC is contracted to provide services to in a region.

Note B: Overall quality of administration is measured using a composite score that incorporates elements from each of the other KPI 5 measures. The calculation of the score is weighted towards those factors that pose the greatest risk to the program’s integrity.

3.13 Overall, the ANAO’s analysis indicated that Industry’s performance monitoring arrangements were soundly-based and well-targeted, and for the most part, operating as intended. The arrangements were designed to elicit information and analysis about key aspects of the quality and accuracy of the AACs’ policies, practices and decision-making.

3.14 Industry designed its performance monitoring and compliance framework to measure the AACs’ compliance with contractual requirements. Among other things, the framework included processes to:

• assess the validity of a sample of financial assistance eligibility assessments made by the AACs;
• examine the integrity of a sample of fee-for-service payments made to the AACs; and
• measure the quality of the services required to be provided by the AACs, through a survey of employers and apprentices.

3.15 Case Studies 1 and 2 illustrate two well-designed components of Industry’s performance monitoring arrangements—approaches to file monitoring and guidance for Business Case reporting.

Addressing the risk of fraud as part of performance and compliance monitoring

3.16 Industry had in place a variety of controls and processes to help mitigate the risks around fraudulent claiming of financial assistance, including the processes discussed in paragraph 3.14.⁷⁸ In particular, Industry advised that staff undertaking file monitoring checks are expected to remain alert to indicators of potential fraudulent activity. Further, the ANAO’s interviews with a selection of Industry’s state contract mangers (SCMs) and AAC representatives indicated that they had a level of awareness and understanding around addressing fraud risks.

3.17 To further strengthen the department’s monitoring and compliance activities and help build assurance about the accuracy and integrity of the program’s financial assistance payments⁷⁹, there would be value in the Department of Education and Training (Education and Training)—the administering department since 23 December 2014—examining options to regularise data analysis and data mining activities for the AAIP.⁸⁰ In particular, a more structured approach to data analysis would enhance the department’s ability to identify irregular and potentially fraudulent transactions.

3.18 The design of any program of data analysis could focus on areas of higher risk and exposure to fraud. In this context, regular analysis that assists the department confirm the integrity of selected apprenticeship records would be beneficial. Reviewing opportunities to further enhance or refine compliance and monitoring activities would be timely given the commencement of the Australian Apprenticeships Support Network (Support Network) arrangements from 1 July 2015.⁸¹

Quality of the performance measurement benchmarks

3.19 The SCMs and AAC representatives interviewed by the ANAO generally considered the KPI 5 benchmarks to be well-targeted and provide a creditable indication of how an AAC is performing. However, several AACs observed that meeting Industry’s evidentiary requirements was time consuming and impacted on their ability to provide more valuable or outcome focussed services. In this context, there is a balance to be struck in maintaining a level of documentation that helps the administering department gain sufficient and appropriate assurance about the quality of the AAC’s administration. The introduction of the Support Network, from 1 July 2015, also provides an opportunity for Education and Training to re-assess the nature and level of evidentiary requirements for service providers.⁸²

3.20 The ANAO’s review of program benchmarks identified some shortcomings in the usefulness, currency and reliability of the benchmarks for KPIs 2 and 3. In particular:

• with the exception of the measure relating to the ‘skills needs occupations’ priority group (Measure E of KPI 3), the benchmarks are based on data recorded in TYIMS in the period 1 July 2009 to 31 December 2010—meaning there is a moderate time-difference between the date of the data used to set the benchmarks, and the dates of the actual assessments of the AACs’ performance;
• the various participation benchmarks have not been updated during the current contract round⁸³—for instance, to reflect changes in market conditions and also the policy changes that occurred during the current contract round; and
• there is a wide divergence among the benchmarks relating to the ‘skills needs occupations’ priority group. The diversity of benchmarks made it difficult to compare the AACs’ relative performances.

Performance monitoring results

3.21 The records of performance monitoring undertaken by Industry during the current Support Services contract round indicated that the majority of AACs had achieved most of the 25 benchmarks associated with the 15 performance measures. More specifically, over the course of the completed monitoring rounds:

• most AACs, in most regions, have met four of the five benchmarks relating to the quality of administration (KPI 5).⁸⁴ In particular, as shown in Figure 3.1, the overall national average composite scores relating to the AACs’ quality of administration have exceeded the benchmark in each monitoring round;
• employers’ and apprentices’ satisfaction levels (KPI 4)—as reported in the 2014 survey⁸⁵—were generally higher than in the previous survey conducted in 2011. Specifically, more employers and apprentices indicated that they were ‘very satisfied’ with the AACs’ services or that they ‘strongly agreed’ with the descriptions of the AACs’ services contained in the survey; and
• the majority of AACs had achieved most of the 11 benchmarks relating to apprenticeship commencements and completions (part of KPIs 2 and 3).

Figure 3.1: Overall national average results for the quality of the AACs’ administration (Measure A of KPI 5)

 

 

Source: ANAO analysis, based on departmental records.

3.22 On a less positive note, the majority of AACs had not achieved most of the six benchmarks associated with apprenticeship retention rates (part of KPIs 2 and 3). Further, the national average results for apprenticeship retentions have generally been below benchmarks during the current contract round.⁸⁶

Analysis of overpayments identified by Industry

3.23 An important part of the file monitoring work undertaken by Industry was the assessment of whether any financial assistance or FFS payments have been made incorrectly and should be recovered—Industry described these instances as ‘overpayments’.

3.24 Details of the total overpayments identified by Industry during the four file monitoring exercises completed at the time of the audit are shown in Table 3.4.

Table 3.4: Overpayments identified by Industry during file monitoring activities in the current contract round^A

Source: ANAO analysis of information in ‘KPI 5 National Monitoring Results’.

Note A: Industry advised that the number (and value) of the overpayments identified during file monitoring activities typically reduces over time as the AACs provide additional evidence to address the concerns raised.

3.25 Using the overpayment rates reported from Industry’s file monitoring work, as shown in Table 3.4, the ANAO estimates that overpayments over the full contract round could approximate $41.7 million. This amount is approximately 1.4 per cent of the total estimated AAIP expenditure in the current contract round—an immaterial proportion of total expenditure. Nevertheless, there would be merit in Education and Training examining whether there are further opportunities to reduce the incidence of incorrect payments and to identify and recover any incorrect payments that are made. As discussed in paragraph 3.17, there would be value in examining options for data analysis. The department could also consider assessing the cost effectiveness of undertaking further targeted monitoring of AAC activities to help identify instances of AACs not adhering to contractual requirements, including making invalid or inappropriate eligibility assessments.

Complaints and review mechanisms

3.26 Effective mechanisms for the handling of complaints and the review of decisions are important to help identify and redress instances of incorrect or unfair treatment experienced by members of the public. In addition, the information and insights obtained from review mechanisms can help identify program weaknesses or service delivery faults. The ANAO examined Industry’s arrangements for the:

• management of complaints;
• conduct of reviews of AACs’ eligibility decisions; and
• conduct of investigations by the Commonwealth Ombudsman.⁸⁷

Complaints handling

3.27 Section 26 of the AACs’ contracts requires the AACs to have processes for dealing with complaints, and to maintain a complaints register. These registers are required to set out details of: complaints received; who made the complaint; and the action taken in response to that complaint. Each year, the department required the AACs to provide it with a copy of their complaints register.

3.28 The ANAO reviewed a sample of AACs’ complaints registers held by the department. The information in the registers examined indicated that the AACs had structured processes and procedures in place to record and respond to complaints received. In addition, for the most part, the complaints that were examined were promptly followed up and resolved.

3.29 The ANAO’s analysis also showed that information in the AACs’ registers indicated that complaints of a similar nature, for instance complaints about AAIP payments and eligibility queries, were being handled in a generally consistent manner.

Reviews of eligibility decisions

3.30 Program documentation stated that Industry is committed to procedural fairness in the administration of the AAIP. Procedural fairness means that all decisions are consistent, equitable and transparent. The AAIP’s decision review process is shown in Figure 3.2.

Figure 3.2: AAIP decision review process

 

 

Source: ANAO, based on the AAIP Program Guidelines.

Reviews conducted by Industry

3.31 Industry’s reviews of AACs’ decisions were undertaken in the department’s state and territory office network. The ANAO’s examination of a sample of the reviews conducted by Industry showed that the escalation processes set out in the Program Guidelines were being followed. The ANAO also observed that the:

• reviews were undertaken in a timely manner;
• recommendations made to the departmental delegate, and the decisions ultimately made, were appropriately documented; and
• claimant was advised of the outcomes.

Investigations by the Commonwealth Ombudsman

3.32 Industry had structured procedures in place for responding to complaints made to the Ombudsman. These included:

• registering the formal notification received from the Ombudsman about the complaint⁸⁸;
• preparing a detailed response about the matter for the Ombudsman, including providing copies of relevant supporting evidence; and
• registering the formal advice from the Ombudsman that the matter has been resolved.

3.33 The ANAO’s analysis has shown that Industry’s procedures for handling matters escalated to the Ombudsman were largely operating as intended.

Data integrity

3.34 The ANAO examined the accuracy and integrity of a range of AAIP information recorded in TYIMS in the period 1 July 2012 to 30 June 2014. In particular, the ANAO analysed whether:

• FFS payments made to the AACs were consistent with the approved unit fees;
• processes to introduce, change or remove financial assistance eligibility rules in TYIMS were operating effectively;
• controls to help support the ongoing confidentiality and integrity of information were in place and operating as intended;
• payments of financial assistance were being made to eligible claimants and at the correct amounts; and
• there were any unusual, irregular or anomalous records.

FFS payments

3.35 The ANAO’s examination indicated that FFS payments recorded in TYIMS during the period 1 July 2012 to 30 June 2014 have been made in accordance with approved amounts. The ANAO’s results were complemented by analysis undertaken by Industry in July 2014, which assessed trends in the movement of FFS expenditure over the course of the current contract round. Overall, the department concluded that amounts of FFS expenditure were broadly consistent with variations in the number of apprenticeship commencements⁸⁹, and reflected changes to the AAIP’s eligibility rules, as well as, ‘wider economic influences’.

Amending TYIMS for policy changes

3.36 The ANAO’s analysis indicated that Industry’s processes for introducing, changing and removing incentive and personal benefit eligibility rules into TYIMS during the period 1 June 2012 to 30 June 2014 were generally operating effectively. However, the ANAO observed that Industry had not documented the processes involved. In light of the Australian Government’s intention to replace the TYIMS system from 1 July 2015, it is timely that Education and Training prepare, and have endorsed, documentation of the processes associated with making updates to the eligibility rules for AAIP entitlements.

Controls over data confidentiality and integrity

3.37 The ANAO’s analysis indicated that Industry had appropriate controls and practices in place to support the continued confidentiality and integrity of the information in TYIMS. However, the following minor issues were identified with regard to the management of users’ access:

• an instance where an employee had left the department, but still had access to TYIMS;
• an instance where an employee was given access to a particular user group incorrectly; and
• three instances of employees incorrectly having generic user accounts.

Analysis of financial assistance expenditure

3.38 The ANAO examined:

• whether selected financial assistance payments made between 1 July 2012 and 30 June 2014 were correct—ANAO’s testing focussed on financial assistance codes that had been affected by policy changes introduced during that period; and
• instances of irregular records to identify potential data deficiencies and erroneous transactions.

3.39 The ANAO’s analysis of nearly 920 000 financial assistance payment records worth approximately $1.2 billion indicated that the vast majority of payment calculations were correct. However, as outlined in Table 3.5, the ANAO identified some 960 incorrect payments with a value of almost $970 000.

Table 3.5: Incorrect financial assistance payments during 1 July 2012 and 30 June 2014 identified by ANAO analysis

Source: ANAO analysis.

Note A: At the time of this report, Education and Training was investigating a further 4 financial assistance payments with a value of $6 700 that were also potentially paid incorrectly.

3.40 Industry advised that the incorrect financial assistance payments shown in Table 3.5 were due to processing errors, rather than limitations in the functionality of TYIMS. Further, the ANAO was advised that:

• recovery action would be pursued in the case of the 700 overpayments—totalling $836 317; and
• the AACs will be advised to process corrections for the 259 underpayments—totalling $131 194.

3.41 The value and number of incorrect payments identified by the ANAO was relatively low compared to overall program expenditure during the period 1 July 2012 to 30 June 2014—representing less than 1 per cent. Nevertheless, as discussed in paragraphs 3.17 and 3.25, there would be value in Education and Training considering options for regularising data mining and analysis to help build assurance regarding program payments.

Other irregular records

3.42 The ANAO’s analysis identified a number of irregular records in TYIMS. Most significantly:

• 2 548 payments being made to employers with cancelled Australian Business Numbers (ABN)⁹⁰; and
• 27 655 apprentices’ records that potentially had been duplicated one or more times.

Payments to employers with cancelled ABNs

3.43 Section D4 of the AAIP Program Guidelines states that if an employer has not provided the department with an ABN the department shall withhold 46.5 per cent of the employer’s payment and remit it to the Australian Taxation Office. The requirement to withhold moneys from an employer’s payments should be triggered in cases where the employer’s ABN recorded by the department is invalid.

3.44 At the time of the audit, Industry advised—following a review of a sample of 200 of the exceptions identified by the ANAO—that it had not identified any incorrect payments. Industry further advised that its analysis indicated that the AACs have not been maintaining the accuracy of ABN data in TYIMS.

3.45 As a result, the department advised that:

• it would instruct the AACs on the importance of routinely checking the accuracy of standing or historical information recorded in TYIMS, such as the employers’ ABNs; and
• the Australian Apprenticeships Management System (AAMS)—the information and communications technology system replacing TYIMS⁹¹—will have an automatic link to the Australian Government’s ABN Register, which is intended to enable regular validation of employers’ ABNs.

3.46 There were positive results from Industry’s initial review of a sample of the exceptions identified by the ANAO. Nonetheless, there would be merit in Education and Training examining all of the exceptions in order to identify whether any employers have received financial assistance without having a valid ABN. In such cases, Education and Training should consider whether part of the moneys paid to the employers should have been withheld as required by the Program Guidelines.

Duplicate records

3.47 At the time of the audit, Industry advised that it had commenced work on identifying the most effective method of ‘cleaning’ the information in TYIMS, including merging duplicate records, prior to the migration of data to the new system—AAMS. In addition, Industry further advised that it intended incorporating enhanced data matching and search functionalities into the design of AAMS—which is aimed at helping prevent the creation of duplicate records and enabling timely assessments of the accuracy of duplicated fields.

3.48 In mid-March 2015, Education and Training advised that it had corrected over 7 800 of the 27 600 potentially duplicated apprenticeship records identified by the ANAO. The department further advised that the remaining 19 800 records would be analysed (and as necessary, corrected) as part of the data cleansing project being undertaken in readiness for the implementation of AAMS.

Conclusion

3.49 Contracts with the AACs are fit-for-purpose. Of note, the key service requirements are clear, and the five key performance indicators (KPIs) and the 15 associated measures are useful in assessing the AACs’ performance against those requirements. Most of the 17 benchmarks associated with two of the five KPIs (KPIs 2 and 3) are based on information dating from 31 December 2010. In this context, the time-difference between the date of the data used to set the benchmarks, and the dates of the actual assessments of the AACs’ performance, affects the usefulness and reliability of these measures.

3.50 A multi-faceted and well-targeted approach was adopted by Industry to monitor the performance of the AACs. In particular, Industry’s approach was designed to examine key areas of the AACs’ service delivery policies, practices and decision-making. Importantly, Industry’s performance monitoring was designed to measure the AACs’ compliance with contractual requirements. Among other things, this was achieved by: assessing the validity of a sample of financial assistance eligibility assessments made by the AACs; and examining the integrity of a sample of fee-for-service payments. The ANAO’s analysis indicated that, for the most part, Industry’s performance monitoring processes were operating as intended.

3.51 Industry’s processes for examining the AACs’ administration and Industry’s performance reporting guidance were well-designed. Monitoring undertaken by Industry indicated that the majority of AACs had achieved most of the 25 benchmarks associated with the 15 performance measures. In particular, the department’s monitoring indicated that there had been a general improvement in the quality of the AACs’ administration. The majority of AACs were meeting targets associated with the majority of measures relating to apprenticeship commencement and completion measures, but were not meeting the majority of measures relating to apprenticeship retention rates. The department advised that a number of external factors were likely to have affected performance in relation to retention rates, including a downturn in economic conditions and the associated tighter labour market.

3.52 ANAO analysis indicated that controls in place to protect the confidentiality and integrity of data in TYIMS were largely operating as intended; and that AAIP expenditure during the period 1 July 2012 to 30 June 2014 was materially correct. In this regard, however, the ANAO identified a small number of incorrect financial assistance payments—albeit with a relatively low value when compared to overall program expenditure.

3.53 The ANAO also identified a number of other inaccuracies among the data recorded in TYIMS. Most significantly, ANAO’s analysis found:

• more than 27 600 apprentice records that are potentially duplicated; and
• around 2 500 payment records containing invalid Australian Business Numbers (ABNs).

3.54 Before the program’s transfer to Education and Training, Industry advised that it was examining ways of ‘cleaning’ the data in TYIMS prior to migrating that data to the proposed replacement system—AAMS. Further, Industry advised that AAMS would incorporate improved data matching and validation functionality. Nevertheless, to further strengthen the department’s monitoring and compliance activities and help assure the continued accuracy and integrity of program expenditure (as well as help identify incorrect payments in a timely manner) there would be value in Education and Training examining options to regularise data analysis and data mining activities in relation to the AAIP. The design of a program of data analysis could focus on areas of higher risk and exposure to fraud.

Appendices

Please refer to the attached PDF for the Appendices:

• Appendix 1: Entity Responses
• Appendix 2: Estimated and Actual Expenditure in each Year of the Current Support Services Contract Round
• Appendix 3: Summary of Policy Changes to the AAIP since July 2012

Abbreviations

Glossary