Australia’s vocational education and training sector

1.1 The vocational education and training (VET) sector aims to deliver workplace specific skills and knowledge across a wide range of careers, industries and levels of personal development. VET service providers include technical and further education institutes, adult and community education providers, agricultural colleges, universities and schools, industry skill centres, commercial and enterprise training providers, as well as other private providers.

1.2 VET is an area of shared responsibility between the Australian Government, state and territory governments and industry. The Australian Government provides financial assistance to the states and territories to support the national training system, including the VET Student Loans (VSL) program. In 2017, of the estimated 4.2 million students participating in VET activities, 1.2 million were enrolled in the government-funded system.

The VET FEE-HELP scheme

1.3 Auditor-General Report No.31 of 2016–17 Administration of the VET FEE-HELP Scheme found that the VET FEE-HELP (VFH) scheme was not effectively designed or administered. Poor design and a lack of monitoring and control by the Department of Education and Training (the department) led to costs significantly increasing. The VFH scheme was weighted heavily towards supporting growth in the VET sector (increasing student access and supporting a sufficient number of providers), but appropriate quality and accountability frameworks addressing identified risks were not put in place. In effect, the department’s focus on increasing participation overrode integrity and accountability considerations that would have been expected given the inherent risks.

1.4 In addition, the administration of the VFH scheme did not safeguard its operation and did not support the achievement of scheme objectives relating to integrity, quality, value and sustainability. Similar to the scheme’s design and implementation failures, there were weaknesses identified in the department’s administrative processes for: approving VFH providers; developing and undertaking risk, fraud and compliance activities; controlling payments to providers; making information readily available to students about their rights and obligations under the VFH scheme; and managing and resolving student complaints.

1.5 In 2015 and 2016, the Australian Government introduced a suite of legislative changes to the VFH scheme to address many of the abovementioned deficiencies.¹ On 3 December 2015, in announcing measures to reform the VFH scheme designed to control growth and lift the performance of training providers ahead of a more fundamental re-design of the scheme, the Minister for Vocational Education and Skills stated that the reforms were intended to address ‘…unscrupulous marketing and enrolment practices by some training providers and restore the integrity of the VET FEE-HELP scheme’. At the same time, the Government announced its intention to introduce a new model to commence in 2017.²

The VET Student Loans program

1.6 On 5 October 2016 the Government announced that the VSL program would replace the VFH scheme. The program commenced on 1 January 2017 and is administered by the department. VET student loans are income-contingent loans available to eligible students. The core objective of the VSL program is to remove upfront financial barriers to VET training in order to make it more accessible to students who may not otherwise have access. The VSL program also aims to:

• ensure the delivery of quality training to students, in courses that have qualifications that are more likely to lead to employment outcomes; and
• ensure this delivery at costs that are affordable to both the student and the Government.³

1.7 In the Portfolio Additional Estimates Statements 2016–17, the Government committed $45 million over four years to implement the VSL program and to deliver additional compliance measures.⁴ An additional $36.2 million over four years was allocated in the 2018–19 Budget to fund the implementation of a new IT system to support VSL compliance and regulation.⁵ As at 30 June 2018:

• 42,220 students had accessed a VSL loan in 2017 and 43,294 from 1 January to 30 June 2018;
• $343.9 million in VET student loans were approved from 1 January 2017 to 30 June 2018, which is eight per cent of the $4.14 billion cap available for the two years commencing 1 January 2017 (that is, a $2.07 billion each calendar year);⁶
• 387 approved courses are on the eligible course list; and
• 180 registered training organisations had been approved to provide training under the VSL program.⁷

1.8 Figure 1.1 provides a comparison of student numbers and loan amounts under VFH and VSL from 2010 to 30 June 2018.

Figure 1.1: VFH and VSL student numbers and loan amounts, 2010 to 30 June 2018

Note: EFTSL is equivalent full time student load.

Source: ANAO analysis of Department of Education and Training data.

1.9 Figure 1.2 provides a comparison of student commencements and provider types under VFH and VSL from 2016 to 30 June 2018.

Figure 1.2: VFH / VSL commencements by provider type, 2016 to 30 June 2018

Source: ANAO analysis of Department of Education and Training data.

1.10 The key features of the VSL program include:

• the list of approved courses is based on an analysis of employer, state and territory government and Australian Government data. This list applies to all approved VSL providers, with additional courses approved for some specifically listed and not-for-profit providers;
• three loan caps set for courses depending on their estimated delivery cost (with some exceptions, such as aviation courses)⁸;
• a requirement for all VSL students to confirm engagement in their studies via an extension to the electronic Commonwealth assistance form system. This new requirement, effective from 1 July 2017, is intended to ensure students are active and represent legitimate enrolments;
• a new application process for providers wishing to participate in the VSL program. This involves the setting of a higher bar to entry, including consideration of the link the provider has with industry and other bodies relevant to the course, a range of financial criteria, student completion rates, employment outcomes, and past performance as an educational institution;
• strengthened legislative, compliance and payment conditions, including paying approved providers in arrears; the ability to cap provider loan amounts and student numbers; and enhanced powers to suspend poor performing providers from the program, cancel their payments and revoke their approval; and
• prohibition of approved providers from using ‘brokers’ or acting inappropriately in signing up students and limiting the subcontracting of training.

1.11 The VSL program and the loan process, which involves a number of stakeholders, is outlined in Figure 1.3.

Figure 1.3: Overview of the VET Student Loans process

Note a: From 2018-19, recipients are required to commence repaying their loans when they earn above $51,956.

Source: Adapted from Australian Government, VET Student Loans: Manual for Providers, Version 2.0, December 2017, p. 9.

1.12 A timeline of key events in the development and implementation of the program is included at Figure 1.4.

Figure 1.4: Timeline on design and implementation of the VET Student Loans program

Source: ANAO analysis of Department of Education and Training documentation.

Rationale for undertaking the audit

1.13 The VSL program was selected for audit because it is intended to address a number of significant deficiencies identified with the VFH scheme, as noted in:

• Auditor-General Report No. 31 of 2016–17 Administration of the VET FEE-HELP Scheme;
• Redesigning VET FEE-HELP: Discussion Paper (April 2016) and related submissions (April to June 2016); and
• The Senate Education and Employment References Committee report, Getting our money’s worth: the operation, regulation and funding of private vocational education and training (VET) providers in Australia (October 2015).

1.14 An assessment of the extent to which the department has incorporated lessons learnt from the former scheme into the new program allows design deficiencies to be addressed in a timely manner. The audit also provides assurance to Parliament about the robustness of the design and implementation of the VSL program.

Audit approach

Audit objective, criteria and scope

1.15 The objective of the audit was to assess the effectiveness of the department’s design and implementation of the VET Student Loans program.

1.16 To form a conclusion against the audit objective, the ANAO adopted the following high level criteria:

• Was an appropriate design process established to support the achievement of the Government’s policy objectives?
• Were sound arrangements established to support the implementation and management of the program?

1.17 The audit focused on:

• activities pursued by the department in designing the VSL program;
• the extent to which lessons learnt from the VFH scheme were incorporated in its design (including the learnings noted in Auditor-General Report No.31 of 2016–17 Administration of the VET FEE-HELP Scheme);
• stakeholder consultation on the design and implementation of the new loans program;
• the extent to which risks were considered during the design process and its implementation; and
• the adequacy of the department’s performance management approach.

Methodology

1.18 In undertaking the audit, the ANAO:

• reviewed and analysed policy documents, guidelines, procedures and relevant departmental files;
• interviewed: department managers involved in the design and implementation of the VSL program; selected VSL service providers; the peak VET bodies (TAFE Directors Australia, Australian Council for Private Education and Training and Community Colleges Australia); several state government agencies responsible for the VET sector; and representatives from the Australian Skills Quality Authority, Office of the Commonwealth Ombudsman, Department of Finance and the Australian Taxation Office; and
• received 32 submissions from a range of key stakeholders: 27 VSL-approved service providers (14 private and 13 public providers); three state entities responsible for TAFE sectors; and two general public contributions. The first two groups represented more than 900,000 VET students.

1.19 The audit was conducted in accordance with ANAO Auditing Standards at a cost to the ANAO of approximately $416,000.

1.20 The team members for this audit were Brendan Gaudry, Brendon Gittins, Barbara Das, Alana Tolman and Deborah Jackson.

Did the department have an appropriate process for designing the VSL program?

2.1 The department adopted a systematic approach to designing the VSL program which drew upon:

• Auditor-General Report No. 31 of 2016–17 and other VFH-related reviews;
• a range of task force, working group and committee deliberations of the VFH scheme and VSL program;
• learnings arising from legislative changes made to the VFH scheme in 2015 and 2016;
• the experiential learnings of personnel directly involved in implementing and administering the VFH scheme;
• stakeholder consultations; and
• the consideration of a range of options, particularly in early 2016 when developing the Redesigning VET FEE-HELP: Discussion Paper.

2.2 The comprehensive approach comprised the key elements of a policy design process, which included: identifying learnings and key issues within the VFH scheme; formulating a range of policy options, many of which emerged from the activities outlined above; consulting with stakeholders during the design; and establishing an evaluation strategy.⁹ Details of these key policy design inputs are noted in Table 2.1 and are discussed in more detail throughout Chapters 2 and 3 (paragraph numbers are indicated in the first column of the table).

Table 2.1: Key elements of the VSL design process

Source: ANAO analysis of Department of Education and Training documentation.

Were clear objectives developed for the program that align with the Government’s policy objectives?

2.3 When approving the VSL program, the Government’s policy intent was described in broad terms as introducing a scheme that is financially sustainable, meets student and industry needs, is outcomes focused and governed by effective legislation and compliance.

2.4 The objectives of the VSL program are described in key program documents, such as the Program Administration Plan, as:

• the program is fiscally sustainable and contributes to national economic growth;
• the program removes financial barriers to training and improves equity of access to higher level VET;
• the program promotes the delivery of quality and affordable training for students;
• the program balances industry needs, employment outcomes and student choice;
• the program is student centred through adequate protection for students (particularly from disadvantaged backgrounds) and access to information that enables informed decision making; and
• the program has program integrity, manages risk and promotes confidence in the regulated VET market.

2.5 These objectives were described as principles in the Program Implementation Plan and as objectives / principles in the Evaluation Strategy. The Program Implementation Plan expressed the core objective of the VSL program as: to make VET training more accessible to students who may not otherwise have access by removing upfront financial barriers. The plan added that the VSL program aims to:

• ensure the delivery of quality training to students, in courses that have qualifications that are more likely to lead to employment outcomes; and
• ensure this delivery at costs that are affordable to both the student and the Government.

2.6 As such, there is a lack of consistency about the program’s objectives across key program documentation.¹⁰ Nevertheless, the objectives broadly align with the government’s policy intent.

Were lessons learnt from the former VET FEE-HELP scheme reflected in the design of the program?

2.7 A range of reform options to the VFH scheme were considered at various times from late 2014, resulting in a series of changes to the scheme, effective from 1 April 2015, 1 July 2015 and 1 January 2016. The reform measures were designed to address the issues of rising fees and unscrupulous provider behaviour. The changes were informed by, among other things, the findings and recommendations of the Senate’s Education and Employment References Committee inquiry into the operation, regulation and funding of private VET providers in Australia¹¹, a 2014 taskforce and the VFH Reform Working Group. This, in turn, informed the department’s later thinking on what aspects of the VFH scheme needed to be reformed.

2.8 Subsequent to the Government’s announcement in December 2015 that a new model would be introduced in 2017, the department established the VET FEE-HELP Redesign Steering Committee in February 2016.

2.9 The Redesign Steering Committee was responsible for making decisions and providing advice on: project planning and risk management; policy development and costings; transitional strategies and implementation; stakeholder management and consultations; and any complementary and consequential measures that may be required.

2.10 The committee considered a number of issues, including:

• whether VET FEE-HELP is a possible mechanism for enhancing quality in VET;
• course eligibility and loan caps;
• student engagement requirements;
• building an informed market; and
• impacts on the VET market.

2.11 Papers addressing these areas succinctly canvassed the main issues and identified options, many of which were reflected in the final program design. This included, for example, considering the option of using state and territory Subsidised Training Lists, the Skills Shortage List or the Skilled Occupation List for the VSL eligible course listing methodology. Deliberations of the committee were used to brief the Minister for Vocational Education and Skills for a series of national consultations conducted during the first two weeks of April 2016.

Redesigning VET FEE-HELP: Discussion Paper

2.12 The Redesigning VET FEE-HELP: Discussion Paper, issued by the Australian Government in late April 2016, stated that the paper had been informed by feedback received from the Minister’s national consultations earlier that month. The discussion paper and the issues papers referred to above were also informed by VFH reforms.

2.13 The redesign discussion paper provided a detailed analysis of the take-up trends and history of the VFH scheme and the problems that beset it. It also posed a series of questions around options for change, focusing on the themes of: protecting students, regulating providers and managing the overall system. The department received 121 submissions by the due date of 30 June 2016.. These submissions covered: course list and caps; the use of third party providers and brokers; online learning; the role of tuition assurance; the proposal to establish a student support mechanism, either through an Ombudsman, the Australian Skills Quality Authority (ASQA) or a department complaints handling function; the use of completion rates to measure course quality; and VSL-provider application processes. The department considered the feedback in the design of the VSL program.

Lessons learnt from Auditor-General Report No. 31 of 2016–17

2.14 Auditor-General Report No. 31 of 2016–17 was tabled in December 2016 and the department adopted the emerging lessons learnt from this audit when designing and implementing the VSL program. Table 2.2 outlines the alignment between lessons learnt and key elements of the program’s final design. Further details are also noted in the remainder of this chapter and in Chapter 3.

Table 2.2: Integration of lessons learnt from the Auditor-General’s audit in the design and implementation of VSL

Note a: As at 31 March 2018, the VET Student Loans Ombudsman had received 48 complaints about the VSL program, compared to 5,193 complaints about the VFH scheme received since 1 July 2017. The most common issues identified through these VSL complaints were the loan amount, including loan caps, complainants’ applications for VET loan assistance, and course quality and progression.

Source: ANAO analysis.

Were the impacts on key stakeholders considered in the design of the program?

Impact on stakeholders

2.15 Building on reviews of the VFH scheme, the department assessed the impact of the new model on key stakeholders. For example, the Redesigning VET FEE-HELP: Discussion Paper included a detailed analysis of the impact on providers, students and in managing the overall VET system from an Australian Government perspective. The VET FEE-HELP Redesign Regulation Impact Statement, developed by the department in September 2016 as input to the Government’s consideration of the new VSL program, also drew on past analysis and considered the impact of the proposed model on students and service providers. In the Regulation Impact Statement, the department estimated that there would be no regulatory impact from the new program.

2.16 Some of the areas considered were:

• Student engagement: the department determined that this process would have a small regulatory cost, estimated at five minutes per fee period.¹² The department considered that this cost would be balanced by: increased awareness by students that their studies are not free and that the loans must be repaid; an increase in the enrolment of genuine students; and the encouragement of increased completion rates among engaged students. The Post-Implementation Review suggested that unit completion rates are at least 10 per cent higher under VSL than they were in the last four years of the VFH scheme. However, trends in enrolments and course completion rates, relative to the former VFH scheme, may not be known for some time.
• Loan caps: the department noted that the introduction of loan caps would result in some students incurring additional costs as a result of fee gaps as providers were expected to charge fees above the loan caps. However, the department considered that the introduction of loan caps and the competition for students should decrease fees, thereby minimising the impact on students of out-of-pocket fee costs. Figure 2.1 shows that average tuition fees have decreased, and that fees charged for the top 10 VSL courses, by commencements, are lower in 2017 than they were in 2016 under the VFH scheme.

Figure 2.1: Average tuition fees for top ten VSL courses, as at July 2018

Note: Data is for equivalent full time student load (EFTSL).

The composition and ranking of the top ten courses has changed over time. The top 10 courses as at July 2018 have been used to allow for a comparison of average tuition fees across the three years.

Source: ANAO analysis of Department of Education and Training data.

• Course mix and choices: the department suggested that students would have increased choice of training options under VSL. However, due to the lower number of VSL-eligible courses relative to VFH, student choice is more limited.
• Regulatory costs on providers: the department noted that the impact on providers would include new compliance and reporting arrangements and a requirement for all private providers who wish to participate in the new scheme to apply. These impacts were expected to be balanced by greater transparency and increased provider quality. In addition, it was anticipated that streamlining the provider application process would result in significant efficiencies. A June 2018 internal audit by the department found that there were opportunities to rationalise the information required in the application process.
• Closure of providers: the department noted that a number of existing VFH providers were likely to close as a result of the introduction of VSL (as had been occurring under VFH), which would increase tuition assurance events. The department advised that the likely number of closures and related tuition assurance activations under VSL was not modelled at the time due to time constraints and that it focused on developing stronger VSL tuition assurance requirements for providers and operators. The department undertook subsequent modelling when considering tuition assurance arrangements in 2018 and 2019. Under VSL, two operators were given provisional approval as tuition assurance scheme operators until 31 December 2017. On 10 November 2017, the department announced that it would administer the tuition assurance arrangements for 2018 while it considers future tuition protection.
• Consultation with regulators: the department met with regulators, including ASQA, to discuss the potential impacts of the new model.
• Student take-up of loans and fiscal sustainability: the VSL program was developed on the basis of the assumption that the take-up for VSL would be similar to 2016 VFH levels (115,000 EFTSL in 2016) — 135,000 EFTSL in 2017, increasing to 159,000 EFTSL in 2020 — and the value of loans under the program was capped at $2.07 billion each calendar year. The department advised the ANAO that 42,220 students accessed a VSL loan in 2017 and 43,294 students from 1 January to 30 June 2018. From 1 January 2017 to 30 June 2018, $343.9 million in VET student loans were approved. In August 2018 the department advised the ANAO that it does not have clear data on why take-up is lower than expected. As such, it has not analysed the reasons why program outcomes differ from forecast take-up levels and, therefore, whether it is achieving the Government’s policy intent of introducing a scheme that is, among other things, fiscally sustainable and meets student needs.

2.17 The department stated that its analysis of potential impacts of the program on key stakeholders was based on information that was available during the design process, but that it did not have sufficient data on provider and student behaviour to effectively model key elements of the program. However, data on the VFH scheme was available to the department that could have informed analysis of student and provider behaviour and Deloitte Access Economics, which was commissioned to review course eligibility and loans cap methodologies in early 2017, was able to access a range of relevant data.¹³ The department informed the ANAO that it has undertaken some modelling and analysis, for example, the total value of loans and change in the number of approved providers. It is also planning to examine: the interaction of the array of state and territory subsidy schemes and VSL; fee gaps for approved courses; provider coverage impacts on geographical access; and general VSL take-up trends, by equity groups.

2.18 As the program has now been operating for over a year and a half, more robust data will be available to the department to: monitor the progress of the program; assess the program’s impact on stakeholders; understand the reasons underlying the low take-up of VSL; and identify potential improvements to the program and its administration.

Impact on the department

2.21 The Government announced the VSL program on 5 October 2016, the legislation was assented to on 7 December 2016, and the program commenced on 1 January 2017.¹⁴ As a result, the time available to the department to plan for implementation and to develop necessary program documents and processes was limited, which was an implementation risk. The department advised the ANAO that it took a risk-based approach to implementing the new program and discussed the implementation risk with the Ministers. This approach involved the development of a series of Gantt charts, which in the design phase focused on the VFH grandfathering / opt-in process, transitional provider approvals, an IT second pass business case and milestones concerning the passage of VSL legislation. The department did not develop a risk-based implementation plan to prioritise tasks and activities in the period leading to commencement of the program.

2.22 Table 2.3 provides examples of the date at which some components of the program were decided or issued and the impact of the timeframe.

Table 2.3: The department’s readiness to implement the VSL program

Source: ANAO analysis of Department of Education and Training documentation.

2.23 While the department used various channels to communicate with providers and students, comprehensive manuals and relevant newsletters and factsheets were not available to stakeholders until after the program commenced. See Appendix 2 for stakeholder feedback, including further comments about this issue.

Were stakeholders appropriately consulted in the design and implementation phases of the program?

2.24 Figure 2.2 provides an overview of the stakeholder consultation processes used by the department in the design and implementation of the VSL program.

Figure 2.2: VSL stakeholder consultation

Source: ANAO analysis of Department of Education and Training documents.

Stakeholder consultation — design phase

2.25 As noted earlier, the department established the VET FEE-HELP Redesign Steering Committee in February 2016, with its remit including the provision of advice on stakeholder management and consultations. The Committee’s deliberations, including feedback received from the ministerial consultations held in early April 2016 and other departmental research inputs, culminated in the Redesigning VET FEE-HELP: Discussion Paper.

2.26 The discussion paper outlined how the Government intended to manage the consultation process. It provided a period of two months, to June 2016, for stakeholders to provide submissions and included a commitment to undertake a further detailed and wide consultation process regarding the options outlined in the paper. The department discussed the new program with state/territory representatives during June to August 2016. However, providers, industry and peak bodies, regulators and organisations that represent student groups were not formally consulted following the deadline for submissions to the discussion paper until after the announcement of the VSL program on 5 October 2016.¹⁵

2.27 The department outlined to the government two options to implement the new program. One option was a slower roll out of the new program, which would allow for more consultation; the other was a ‘hard and fast’ implementation, limiting further consultation. The government opted for the latter. The department advised the ANAO that the Minister for Education and Training determined in early August 2016 that sufficient consultation had been undertaken to progress with the design of a new program and sought a 1 January 2017 commencement date, which precluded further consultations. This meeting was not minuted and the decision was not documented at the time of the meeting. The department also advised that there was limited time to undertake further discussion. In its advice to the Minister for Education and Training in September 2016, it was noted that the same themes were continuing to emerge and that the design took into account the views of a broad range of stakeholders.

2.28 The department advised the Minister for Education and Training that ‘Undertaking a consultative approach provides the opportunity to engage stakeholders and gain their confidence in the new scheme. It provides adequate time to educate both stakeholders and students about the changes.’¹⁶ Stakeholders advised the ANAO that they felt marginalised by the department not consulting after the deadline for submissions to the discussion paper. Stakeholder engagement was important in achieving the objective of promoting confidence in the regulated VET market.

Stakeholder consultation — implementation phase

2.29 The department uses a number of fora to consult with stakeholders, including:

• the VET Student Loans Implementation Advisory Group (IAG) (see Box 1);
• Council of Australian Government’s Industry Skills Council and its associated Skills Senior Officials Network;
• an inter-departmental committee (IDC), comprising representatives from ASQA, Australian Competition and Consumer Commission, Australian Taxation Office and the Departments of Finance, Human Services, the Prime Minister and Cabinet and Treasury;
• an Interagency Working Group, which is chaired by the VSL Ombudsman and has broadly the same membership as the IDC; and
• annual provider information forums.

These fora provide an opportunity for the department to engage with stakeholders, communicate more broadly with the VET sector and share information about the program.

Source: ANAO analysis of Department of Education and Training documentation.

2.30 The main stakeholder consultation undertaken since January 2017 has been in relation to the post-implementation review and a review of eligible courses and caps methodologies. These are discussed from paragraph 3.55 to 3.60. Appendix 2 summarises stakeholder feedback received by the ANAO.

Was sound advice provided to the Government on program design and implementation?

2.31 The department provided a range of advice to the Ministers, covering for example:

• VFH Redesign submission feedback;
• pre-Government approval advice on the VSL proposal;
• submissions in response to initial VSL eligible courses list;
• provider application processes;
• analysis of changes in course fees from 2016 to 2017 by VSL providers;
• eligible courses and loan caps methodologies review; and
• results of the Post-Implementation Review.

2.32 The ANAO reviewed key advice provided by the department to the Government and found that it was generally clear and comprehensive. The advice included informative background information, supporting documentation and analysis as required. The ANAO noted several examples of detailed analysis being included in the advice. For example: advice to the Government about the Redesigning VET FEE-HELP: Discussion Paper included detailed analysis on the key issues raised in the 121 submissions received in response to the paper¹⁷; and options for the VSL eligible course list methodology was provided as input to commissioning the VET Student Loans – Eligible Course list and Loan Cap methodologies review.

2.33 As noted earlier, the department has stated that it did not have sufficient data on provider and student behaviour to effectively model key elements of the program, which impacted on the advice it provided to the Government about the new program. The Minister for Education and the Minister for Small and Family Business, Skills and Vocational Education also receive monthly reports that include statistics on the VSL program (discussed further in paragraphs 3.14 to 3.15). However, as at September 2018, the department has not provided detailed advice to the Government about the current outcomes of the program, including the reasons for the lower than anticipated take-up of VSL loans and whether it is achieving the Government’s policy intent.

Has an effective performance management system been established to measure and report outcomes from the program?

3.1 To support the implementation and administration of the VSL program, the department put in place the key components of an effective performance management system, including:

• an evaluation plan, including key performance indicators (KPIs);
• performance monitoring; and
• a range of internal and external reporting.

3.2 The department developed a detailed VET Student Loans Evaluation Strategy for the program, which was approved by the Minister for Education and Training in March 2017. The plan included evaluation questions that covered the program’s objectives, assessing the following areas:

• sustainability of the program and its affordability to students;
• quality of the training delivered;
• degree to which it achieved a balance between the needs of industry and the student;
• program integrity; and
• whether there are any unintended regulatory burdens.

3.3 The evaluation will be conducted in three stages from 2017 to 2019. The initial Post-Implementation Review (PIR), was finalised in December 2017. This is to be followed by a progress report scheduled for completion by 30 November 2018 and a final evaluation report scheduled for completion by 30 November 2019. The 2018 and 2019 reports are to focus on how VSL has addressed the policy intent of the program.

3.4 KPIs for the VSL program were outlined in the evaluation strategy. Six of the 19 KPIs focus on measuring the quality of VET training, with a further five linked to VSL’s integrity; the remainder are divided across the other objectives.¹⁸ The KPIs map to the program objectives, and they are relevant, reliable and complete, and baselined against VFH data where equivalent information is available (that is, for 16 of the 19 KPIs).

3.5 Data to measure performance against the KPIs is sourced from a variety of internal and external sources, such as internal IT systems, the Australian Government Actuary and the VET Student Loans Ombudsman. The VSL quarterly performance dashboard (discussed in paragraph 3.13) includes quantitative reporting against seven of the 19 KPIs. The department does not report against the qualitative KPIs in one document. Appendix 3 lists the KPIs, against the program objectives, and the ANAO’s analysis of their relevance, reliability and completeness. Appendix 4 shows the department’s performance against the KPIs as reported in the VSL quarterly performance dashboard.

3.6 The department has not developed a KPI to measure the impact of the VSL program in promoting confidence in the regulated VET market. The department advised the ANAO that the program’s design addresses the issues observed in the former VFH scheme, which will in turn help address the overarching objective of promoting confidence in the sector. The department also noted that the reputation of the VET sector is affected by additional influences beyond the VSL program; for example, its attractiveness to international and domestic students considering the choice between a VET and higher education qualification. Nevertheless, the department should develop a measure to aid ongoing monitoring of the program’s contribution to this objective.

3.11 During the initial implementation of the VSL program (January to June 2017), weekly executive reports were provided to members of the VET Student Loans Implementation Taskforce, as well as the Secretary and Deputy Secretary of the department. These reports were an effective mechanism for providing high-level status updates on the grandfathering process, provider approvals and closures, and the IT two-pass business case.

3.12 The ongoing performance of the VSL program is reported using the business-as-usual reports shown in Table 3.1.

Table 3.1: Internal and external reporting of VSL performance

Source: Department of Education and Training, VET Student Loans Program Administration Plan, January 2018.

3.13 The VSL quarterly performance dashboard is the key internal reporting mechanism, covering a range of the program’s KPIs. The report provides readers with an accessible overview of the program’s performance against the KPIs, with ’traffic light’ performance ratings based on comparisons with VFH performance. The dashboard report is considered at each quarterly meeting of the VET Student Loans Program Performance and Risk Committee (discussed in paragraph 3.22).

3.14 The Minister for Education and the Minister for Small and Family Business, Skills and Vocational Education receive monthly reports that include the following data:

• comparison of VFH/VSL providers by provider type;
• student commencements by month and cumulative by month;
• equivalent full time student load by month and cumulative by month;
• VET loan amount by month and cumulative by month;
• VET loan amount by reference month (month paid less two) and cumulative by month; and
• overview of all providers.

3.15 The monthly reports provided to the Ministers contain sufficient information to gain an understanding of student participation and loans approved. However, benchmarking against the former VFH scheme is limited to figures for VFH students grandfathered in 2017 and 2018.¹⁹ The inclusion of comparable pre-2017 VFH data, where available, to support comparison between VSL and VFH would allow greater oversight of the program’s overall performance.

3.16 The VSL six monthly and annual statistical reports are the primary form of public reporting on the performance of the VSL program. These reports meet the legislated reporting requirements set out in the VET Student Loans Act 2016 (VSL Act), and include information regarding:

• number of providers;
• value of loans;
• number of VSL-assisted students;
• course completions; and
• tuition fees charged.

3.17 In the department’s 2016–17 annual report, VSL performance information was aggregated as part of its reporting on the Higher Education Loans Program (HELP) program and the VET sector more broadly. Including key high-level VSL-specific information, such as the total number of students in receipt of VSL, in the annual report would align VSL reporting with the other HELP loans (including HECS-HELP and FEE-HELP) for which disaggregated student numbers are provided.

3.18 Overall, the VSL reports are designed to provide the Parliament, Government, the department’s Executive and the public with an overview of the program’s performance.

Are appropriate oversight arrangements in place to effectively manage the program?

Key players in VSL management

3.19 The current oversight arrangements for the program are shown in Figure 3.1.

Figure 3.1: VSL program governance

Note: The Minister for Education administers the Department of Education and Training, while the Minister for Small and Family Business, Skills and Vocational Education has portfolio responsibility for the VET sector and VSL program.

Source: ANAO adaption of Department of Education and Training documentation.

3.20 Management of the VSL program is largely the responsibility of the department’s Skills Programs Group, which includes the VET Student Loans and Skills Programs Compliance branches. These branches work closely together, meeting regularly to discuss the current status of issues and activities of each team. The Skills Market Group, which includes the Skills Outcomes and Financing branch, also has a role in administering the program.

3.21 A VET Student Loans Governance Committee was established by the department in January 2017 to discuss a range of issues related to the implementation of the program and to ensure strong links between the VET Student Loans Implementation Taskforce and the VET Student Loans branch.²⁰ The ANAO reviewed the minutes maintained by this Committee and notes that an issues register was created and reviewed weekly, and a record was kept of action items opened and closed. The Committee held its last meeting in September 2017, being superseded by the Performance and Risk Management Steering Committee.

3.22 The Performance and Risk Management Steering Committee, established in September 2017, was responsible for reviewing and improving program administration, including program direction, and delivering against the objectives of the VSL program and VFH scheme. In January 2018 it was renamed the VET Student Loans Program Performance and Risk Committee. This committee meets bi-monthly and comprises membership across the Skills Program and Skills Market Groups. As well as focusing on reviewing program performance (and the reporting thereof) and risk management, its remit includes a quarterly review of the internal Program Administration Plan developed in January 2018. Further details on the risk management deliberations of this committee are discussed in paragraphs 3.38.

Has appropriate guidance and training been provided to VSL program staff?

3.23 Key guidance materials available to departmental staff involved with the VSL program include the:

• VSL Program Implementation Plan and VSL Program Administration Plan 2018;
• VSL Manual for Providers;
• VET FEE-HELP Branch Business Plan 2016–17;
• Standard Operating Procedures developed for the VSL and Skills Programs Compliance branches; and
• an internal, detailed VSL overview document that outlines the key elements of the program, how it works, what features are supported by legislation or the VET Student Loans Rules 2016 (VSL Rules), key dates and a summary of 2015 and 2016 changes made to the VFH scheme that were retained under VSL.

Collectively, the documents provide useful guidance for departmental staff to administer the program.

3.24 The department has developed a comprehensive and well targeted VSL training program, which identified the officers to be trained and the timing of training. The program included: risk management and regulation, communications, provider application process, probity, project management, compliance and IT systems. The training program was not based on a formal training needs analysis; it was based on management’s prior experience with the VFH scheme and their understanding of the new VSL program.

3.25 In some cases the training was delivered when needed. For example, communications training was provided to call centre staff at the time of the Government’s announcement on 5 October 2016 to establish the new program. However, most training and related guidance was provided to relevant staff after the program had commenced on 1 January 2017, often to coincide with the release of operational plans and processes. The department has advised that it did not evaluate this training.

Data analytics capability

3.26 A key learning noted in Auditor-General Report No. 31 of 2016–17 was a need for the department to establish a strong data analytics capability and related management reporting processes to identify emerging threats and promote understanding and visibility of the outcomes of the scheme.

3.27 The department has recently expanded its VSL data analytics capacity. In August 2018 it established VSL Data Analytics and VSL Data and Reporting teams. Responsibilities of the teams include: data analytics for policy and compliance, IT system liaison and VFH/VSL data reporting. The department advised the ANAO that current systems do not support easy access to data and that its data analytics capability requires strengthening. These issues are expected to be addressed through the development of a new IT platform (scheduled to be fully operational from mid-2020) and related capability improvements.

Has an effective communications strategy been established that provides appropriate information to key stakeholders?

3.28 The department’s communications with key stakeholders are guided by a Communications Plan²¹ and a Stakeholder Engagement Plan. The department developed the Communications Plan in February 2017, with additional plans developed for specific campaigns; for example, related to compliance, provider fees and various aspects of the provider application process. The Stakeholder Engagement Plan, developed in April 2017, maps stakeholders and identifies key issues and mechanisms for targeting communications. The Stakeholder Engagement Plan provides a higher level summary of the engagement approach adopted by the department, with the Communications Plan providing greater detail of target audiences, key messages, communication channels and evaluation parameters.

3.29 To support the move from the VFH scheme to the VSL program and inform affected stakeholders about the changes and sources of support, the department undertook a range of communications activities as outlined in its plans. There were three phases to the campaign:

• Phase 1. October to December 2016 — VSL program announcement and closure of VFH;
• Phase 2. January to June 2017 — implementation of the VSL program; and
• Phase 3. July to 31 December 2017 — business-as-usual communications.

3.30 VSL program communications is multi-faceted, providing information to stakeholders, primarily students, and platforms through which stakeholders can contact the department. Communications channels include:

• VSL program and general student advisory websites and social media²²;
• a range of general program fact sheets targeted at students and providers;
• a series of targeted frequently asked questions pages on the department’s website on key topics, including the student opt-in process and changes in the VSL provider application process;
• VSL program inbox (VETStudentloans@education.gov.au) — mainly targeting providers;
• ad hoc provider newsletters and provider information forums;
• electronic mail-outs to students and providers;
• VSL student and provider information manuals;
• meetings with communications channelled through the various VET peak bodies;
• call centres, supported by the Department of Human Services and Australian Taxation Office; and
• providing information through the VSL Implementation Advisory Group.

3.31 The information was largely well targeted, clearly presented, accurate and fit-for-purpose, and was generally timely. The department’s approach to communications could have been improved in some areas. For example, stakeholders advised the ANAO that during the early stages of the program providers were not always informed of messages to students. Also, some of the communications to students was technical in nature. Combined with the lack of a comprehensive manual for providers, which was released in May 2017, this meant that it was difficult for providers to respond to student enquiries in a constructive and timely manner.

3.32 The department reviewed the impact of the communications and engagement activities in early 2017, concluding that the communications and stakeholder engagement measures appeared to work well. The review covered the period from the Minister for Education and Training’s announcement of the new program on 5 October 2016 through to March 2017.

3.33 Of the 32 submissions received by the ANAO from a range of stakeholders, 19 included some form of feedback on the communications activities undertaken by the department in implementing the VSL program. Appendix 2 summarises the key issues identified through the ANAO’s stakeholder consultation process.

Are appropriate risk management and compliance arrangements in place?

Risk

3.34 The department has established and maintains a comprehensive risk management framework for the VSL program, using the department’s existing risk management platform, RiskActive. The framework includes the VET FEE-HELP Scheme and VET Student Loans Program Risk Strategy, created in February 2017, which outlines processes such as governance, monitoring and risk escalations, and provides a high level overview of the VSL project risk plans (discussed below).

3.35 The VET Student Loans Program Risk Management Plan and the VET FEE HELP Scheme Risk Management Plan outline the risks relevant to the program. As at August 2018, these plans included eight VSL risks and six VFH risks, as outlined in Table 3.2.

Table 3.2: VSL and VFH risk ratings

Source: ANAO analysis of the VET Student Loans Program Risk Management Plan (updated 21 August 2018) and the VET FEE-HELP Scheme Risk Management Plan (updated 31 August 2018).

3.36 To complement the program risk management plan, project risk plans were also developed during the early stages of the VSL program (over the period November 2016 to February 2017). As at March 2017, eight project risk plans for the former VFH scheme and VSL program had been developed for: student grandfathering; digital media campaign for the VSL program; communication strategy; VFH and VSL payments transition; IT two-pass process; provider full approvals; provider closures; and the IT interim solution. As the program has matured, the department advised that five of the initial eight project risk plans have been retired and replaced with risk plans using Risk Active related to ongoing operational, compliance and monitoring business-as-usual activities.

3.37 The ANAO examined the VSL Risk Strategy and program and project risk management plans. The VSL Risk Strategy and its related plans have appropriately identified the key risks, personnel responsible for the management of program and project risks, roles and responsibilities, and risk escalation protocols that outline how issues are raised and resolved. In contrast, a robust risk management process was not in place under the former VFH scheme.²³

3.38 The VET Student Loans Program Governance Committee provided oversight of risk management quality assurance.²⁴ During the early implementation phase of the VSL program, this committee met weekly and considered risks linked to key implementation activities. Program risks were reviewed monthly and project risks analysed quarterly. Program reporting in the initial stages of implementation focused on the progress of key activities, rather than the management of key program risks.

3.39 Over time, the program’s risk management arrangements have matured, with the Performance and Risk Management Committee becoming more focused on the consideration of key program risks.

Compliance

3.40 The VSL compliance requirements are constituted through legislation and regulation.²⁵ Underpinning this legislation has been the development of a compliance strategy, related compliance management plans and a draft general compliance business plan.

3.41 The compliance strategy notes that risk is an integral component of compliance and that a risk-based approach allows the department to prioritise resources and pursue appropriate compliance actions. For example, the department informed the ANAO that its activities are based on elevated risk factors that identify suspected non-compliance or fraudulent practices. The strategy links to the risk management plans. The strategy does not detail how compliance activities are prioritised or the process by which the level of resources are allocated to these activities.

3.42 The key components of the department’s compliance activities are prevention, detection and response.

3.43 In terms of prevention, approved VSL providers are required to satisfy a range of eligibility and assessment criteria to qualify for, and maintain, approved course provider status under the VSL Act. As noted earlier, the VSL program also introduced restricted course eligibility, the capping of loan amounts payable for particular courses, and the setting of payment caps on approved course providers, which contribute to its prevention strategy.

3.44 The department has sound processes in place to monitor and enforce the ongoing compliance of approved providers against the program’s overall requirements and the specific conditions imposed on each provider as part of their approval. The department’s prevention activity is supported by the Program Assurance Plan, a systematic and data-rich plan that aims to address potential non-compliance issues before they develop into compliance issues. Education and communications also form part of this strategy.

3.45 In terms of detection, the department states that it operates through four main mechanisms:

• payments – analysis of self-verified data from providers for any potential anomalies or instances of non-compliance before making payment;
• a range of audit activities:
  • desktop audits — assesses provider and departmental data, provider websites, media reports and other information sources for compliance against information publication. These audits, undertaken on a rolling basis, do not generally involve contact with providers during the audit process. As at 14 August 2018, of the 60 providers audited in 2018, 35 providers were compliant with publication requirements, 24 were in contravention of a small number of minor publication requirements and a compliance notice had been issued to one provider;
  • provider audits (health checks) — assesses provider practices, such as student marketing, to address potential patterns of non-compliance and identify providers with data anomalies and issue. As at 14 August 2018, 40 provider health checks had been undertaken in 2018, with five providers identified as having data pattern concerns; and
  • site audits — assesses whether providers are complying with their requirements under the relevant law and whether students enrolled by providers are bona fide students. Since late 2017, the department has commissioned the Australian Skills Quality Authority (ASQA) to undertake 12 compliance audits — see Box 2 for further details.
• surveys and complaints – to ascertain students’ experiences and feedback on course marketing and enrolment practices. Since 1 July 2017, VSL complaints have been managed by the VET Student Loans Ombudsman (VSL Ombudsman).
• the gathering and analysis of relevant information and intelligence from a number of regulators, Australian and state government agencies to assist in detecting potential non-compliance, and by reviews of media and other open-source information – see next section for further details on collaboration between the department, key regulators and other government entities.

Source: ANAO analysis of Department of Education and Training documentation.

3.46 A VET loans provider compliance table has been developed and is maintained by the department, which includes comprehensive details of those providers the department, the Australian Competition and Consumer Commission (ACCC)²⁷ and ASQA have investigated and/or are taking action against. This provides a clear picture of which entity is undertaking which form of activity, which assists the three entities to manage their remit.

3.47 The third component of the department’s compliance strategy is response. This involves investigations to address issues of serious non-compliance, undertaking of a range of compliance actions, and imposing civil penalties, as prescribed under the VSL Act. As at 14 August 2018, there have been no investigations of VSL providers.

3.48 One of the key challenges in VSL program compliance has been the process by which operational matters related to lower levels of non-compliance become significant enough to be escalated as compliance issues. The department has drafted a General Compliance Business Plan 2018–19 with the aim to better coordinate and delineate the escalation process. While the plan is in draft, the department has advised that the activities outlined are now business as usual activities and that the plan will be finalised as part of the department’s business planning process.

Have appropriate mechanisms been established to operationalise regulatory roles and responsibilities?

3.49 The department collaborates with key regulators involved in the VET sector through: formal MoUs, which outline the roles and responsibilities of each party; regular meetings to discuss emerging issues and the recent activities of each entity; and informal exchanges and sharing of information and intelligence, as agreed within the respective MoUs. Table 3.3 outlines the key features of the arrangements.

Table 3.3: Arrangements between the department and key entities

Source: ANAO analysis.

3.50 The department also gathers information and intelligence to assist it to detect potential non-compliance and exploitative behaviours. This is sourced from a review of media and other open-source information and information sharing arrangements with:

• entities such as the Australian Taxation Office, Department of Human Services and the Department of the Prime Minister and Cabinet;
• regulators such as ASQA, TEQSA, ACCC and state and territory-based Offices of Fair Trading;
• tuition assurance scheme operators; and
• the VSL Ombudsman (in its role as administrator of the VSL external dispute resolution scheme).

3.51 In ANAO interviews with ASQA, the VSL Ombudsman, the Department of Finance and the Australian Taxation Office, officers noted that their respective roles and responsibilities are clearer and more focused as a result of legislative changes associated with the VSL program. They also stated that a more collaborative and coordinated approach has been adopted by the department for managing the VSL program.

Is an evaluation strategy in place to assess whether program objectives are being achieved?

3.52 The department has reviewed the progress of VSL, or aspects of the program, at several stages. The key reviews undertaken to date are discussed below.

Internal health check review

3.53 The department’s internal audit section undertook a health check review in early 2017. The objective was to provide assurance to the department’s Executive on the early implementation of the program. The review concluded that the department had made considerable progress in developing a governance framework for implementing the Program. It made four operational recommendations and three management improvement suggestions. The recommendations were aimed at improving risk assessments and reporting, performance management and program compliance.

3.54 The department monitors the implementation of internal audit recommendations through reports to the department’s Audit and Assurance Committee and in progress reports periodically requested by department senior executives. The committee papers recorded that the recommendations were closed by 1 March 2018.

Evaluation strategy

3.55 As discussed earlier, the VET Student Loans Evaluation Strategy provides a framework for the PIR, and includes key evaluation questions, review timetable and key performance indicators.

3.56 The first phase of the department’s three year evaluation was based on data analysis, a range of stakeholder interviews and student surveys. The resultant PIR report (December 2017) concluded that the VSL program is meeting its objective of fiscal sustainability, is addressing provider misconduct and early indicators suggest that training quality has improved. The review also concluded that some program features designed to increase integrity have had an adverse impact on other objectives, for example, the focus on fiscally sustainability and unscrupulous providers needs to be balanced with student access to the VSL program.

3.57 The report made 22 recommendations related to the objectives of the program. The department agreed to 10 recommendations, eight were agreed in principle, two were noted and two were not agreed. The department monitors PIR recommendation implementation using a spreadsheet that notes the proposed response, timeframe for implementation, and manager assigned the responsibility to action the recommendation. Following on from the PIR, the department has developed several papers, including papers on earned autonomy for providers and improving student access to the program, and is pursuing a number of issues to inform potential options.

3.58 One of the recommendations related to a key stakeholder concern – the two-state eligible course list process.²⁸ The review recommended that the department streamline the process. The department agreed in principle to this recommendation, but noted that the 2017 eligible courses and caps review had concluded that there was not enough evidence to warrant significant change. The spreadsheet notes that the department will consider how to operationalise this recommendation in the context of the review of courses and loan caps (see below) and the final PIR evaluation, due to be completed by the end of 2019.

3.59 The next phase of the evaluation is scheduled for completion by 30 November 2018, with a final evaluation report to be completed by 30 November 2019. The 2018 and 2019 reports are to focus on how VSL has addressed the policy intent of the program.

Review of courses and loan caps

3.60 The VET Student Loans – Course list and loan cap methodologies, Final Report concluded that the early phases of the VSL program appear to be broadly in line with its policy intent, and that there was little evidence on which to base immediate actions to change the eligible course list and loan cap settings.²⁹ The report notes that the conclusion should be understood in the context of the limited amount of data available and the short time period in which the program has been operating. The report also notes that marginal changes could be made in the near term, for example: considering approaches for rebasing loan caps, in collaboration with states and territories; and developing a clear monitoring and evaluation approach to overcome limitations in the current dataset.

Implementing changes

3.61 It is reasonable for the department to adopt an evidence-based approach to adjustments, including how the new program and its significant changes (relative to the VFH scheme) is progressing. Effectively monitoring the impact of the program, including changes in provider and student behaviour, and taking action as appropriate should mitigate the risk that the type of problems that beset the former VFH scheme will eventuate.

Appendix 1 Entity response

Appendix 2 Feedback from stakeholder consultation

1. The Australian National Audit Office (ANAO) engaged with a range of VSL-approved service providers (and in the case of the publicly-funded TAFE sector, the entities to which they report) seeking their views on the design and implementation of the VSL program in relation to:

• the consistency, fairness and transparency of the departments’ activities;
• the effectiveness of the departments’ engagement with your entity in relation to the design and early implementation of the new loans program; and
• any other matters related to the two high level audit criteria.

2. Thirty-two submissions were received, representing over 900,000 of VET students enrolled in the government-funded system, as follows:

• three State entities responsible for their TAFE sectors;
• 13 public VET providers;
• 14 private providers, including one not-for-profit; and
• two other public submissions.

3. The ANAO’s stakeholder consultation process also included face-to-face interviews and teleconferences with stakeholders across Australia and the establishment of an online portal through which submissions could be made to the audit. The information gained through interviews and submissions provided additional information for the ANAO to discuss with VSL program personnel within the department and were considered in developing audit findings.

4. Table A.1 represents the key issues identified through the ANAO’s stakeholder consultation processes as received by the ANAO and the action or position that the department has taken.

Table A.1: Stakeholder feedback on VSL program design and implementation

Source: 32 submissions received by the ANAO and interviews with stakeholders, including members of the VSL Implementation Advisory Group and state departments responsible for TAFEs.

Appendix 3 VSL key performance indicators

Table A.2: ANAO assessment of VSL key performance indicators

Note: Assessed against the Department of Finance’s Quick Reference Guide - RMG 131 Developing good performance information, April 2015, which describes the characteristics of good performance information as:

• Relevant — performance information should clearly state who benefits and how they benefit from the entity’s activities.
• Reliable — performance information should use information sources and methodologies that are fit-for-purpose and verifiable.
• Complete — performance information should help stakeholders judge whether the purposes of an entity are being achieved.

In the instances when a KPI has been rated as mostly relevant, the link between the KPI and the project’s outcomes and/or beneficiaries could have been more clearly stated.

Source: ANAO analysis of VSL key performance indicators.

Appendix 4 Performance against key performance indicators

Table A.3: VSL performance against quantitative indicators

Note: KPIs as reported in the VET Student Loans Quarterly Performance Report for Quarter 1 2018.

Source: ANAO analysis of VSL key performance indicators.