Introduction

1. State and territory governments currently fund over 90 per cent of all medical specialist training in Australia through specialist trainee¹ positions in public teaching hospitals. Over the past 15 years the Australian Government has also contributed to medical specialist training through a range of Commonwealth-funded programs administered by the Department of Health (Health).²

2. Australian Government programs were consolidated in 2009 as the medical Specialist Training Program (STP), in order to create a simpler and more flexible funding program. In March 2010 the then Government announced additional funding of $144.5 million over four years to increase the number of specialist training positions under the STP from around 360 to 900 by 2014.³ The funding increase—part of a wider National Health and Hospitals Network initiative—was intended to assist in addressing a forecast shortage of specialists in Australia by drawing on the private sector and other non-traditional avenues for training.⁴

Specialist Training Program

3. The Government intended that medical specialties with shortages were to be targeted through the expanded STP, including general surgery, pathology, radiology, dermatology, obstetrics and gynaecology. Priority was also given to providing training positions ‘where Australians need them, such as in rural and regional areas’.⁵

4. The objectives of the STP are to⁶:

• increase the capacity of the health care sector to provide high quality, appropriate training opportunities to facilitate the required educational experiences for specialists in training;
• supplement the available specialist workforce in outer metropolitan, rural and remote locations; and
• develop specialist training arrangements beyond traditional inner metropolitan teaching hospitals.

5. The STP is an executive grants program⁷ involving annual competitive funding rounds. The Australian Government provides financial assistance to hospitals and other medical facilities or health organisations (known as ‘settings’) to employ specialist trainees. Depending on the circumstances, STP grants to settings can consist of a number of components, as follows:

• the primary component is in the form of salary support to settings to assist with the cost of employing a specialist trainee in a specified training position. Salary support is set at $100 000⁸ per full-time equivalent (FTE) per year;
• training positions outside metropolitan areas are eligible for an additional rural loading of $20 000 per FTE per year; and
• training positions in a private sector setting are eligible for the Private Infrastructure and Clinical Supervision allowance, consisting of a supervision allowance of $30 000 per FTE per year⁹ and a training infrastructure allowance of $10 000 per FTE once every three years.

6. As a consequence, an STP grant will typically provide financial assistance of between $100 000 to $153 333 per FTE per year for a training position.

7. Under the STP, specialist trainees are placed in specified training positions within public or private hospitals, other medical facilities or health organisations that are accredited for the purpose of specialist training by the relevant specialist medical college (college). Specialist trainees will generally rotate through a number of training positions at different hospitals or facilities during their specialist training in order to gain a broad range of experience and skills. Completion of specialist training normally takes between three and six years depending on the speciality involved. Subject to meeting any other college requirements, specialist trainees are then eligible to apply for Fellowship¹⁰ of the relevant college and be recognised as a fully qualified specialist.

8. While Health has overall responsibility for STP administration, the department does not have a direct contractual relationship with the settings, which are the grant recipients. The administration of STP grant funds is managed through separate agreements between Health and the colleges.¹¹ Under this ‘college administration’ model, all grant funding for STP training positions for a particular specialty is provided by Health to the relevant college.

9. Overall, STP expenditure from 1 July 2010 to 31 December 2014 has been $379 million. While the STP is an ongoing initiative, current funding agreements with colleges expire at the end of 2015. As at January 2015, the Government has not made a decision on future STP funding.

Previous audit coverage

10. The ANAO has not previously examined the STP. ANAO Performance Audit Report No.34 2010–11 General Practice Education and Training examined the management of general practice vocational education and training programs by General Practice Education and Training Limited (GPET), then a Commonwealth company.

Audit objective and criteria

11. The audit objective was to assess the effectiveness of the Department of Health’s (Health) administration of the STP. The audit focused on key aspects of Health’s administration of the STP since the consolidation of funding programs in 2009, and the achievement of key program targets and objectives. To assess the department’s grants administration, the ANAO focused on the fourth annual grant funding round (the 2014 round) which was completed in December 2013 and funded training positions from the beginning of 2014.

Overall conclusion

12. Australian Government programs providing support for medical specialist training were consolidated in 2009 as the Specialist Training Program (STP), and in March 2010 the then Government announced additional funding of $144.5 million over four years to increase the number of specialist training positions funded under the STP from around 360 to 900 by 2014. The funding increase was intended to help address a forecast shortage of specialists in Australia by tapping into the private sector and other non-traditional training settings. The STP is a grants program, with four annual competitive funding rounds conducted since its expansion in 2010. While the Department of Health (Health) has overall responsibility for STP administration, the department receives advice from state health services¹² and specialist medical colleges as part of the grants assessment process, and disburses grants through the colleges.

13. Health has made substantial progress towards achieving the key STP targets and objectives, adopting a generally sound administrative approach which has improved over time. The STP training targets established in March 2010 have largely been met, with college reports indicating that some 93 per cent of training positions have been filled¹³, and some 89 per cent of funded training positions have been located in non-traditional settings. However, in the 2014 grant funding round Health adopted an internal review and rescoring procedure which was not documented, and the department did not strictly adhere to the published selection criteria; an approach which affected the transparency and to an extent the equity of the assessment process when viewed in terms of the application form and other explanatory material that informed applicants’ expectations about how grants would be selected.

14. The ANAO’s analysis of specialist medical college reports provided to Health indicates that, on a full-time equivalent (FTE) basis, around 833 training positions were filled as at 30 June 2014, representing some 93 per cent of the target of 900 training positions announced in March 2010. Further, college reporting indicates that the STP has been successful in utilising non-traditional settings to expand the number of specialist training opportunities. The most recent (July 2014) reporting indicates that 800 (89 per cent) of the training positions are in ‘expanded’ (non-traditional) settings, with 369 (41 per cent) in regional or rural areas.

15. The 2014 funding round, which was completed in December 2013 with a view to funding training positions from early 2014, increased the number of funded training positions from 750 to 900. The first stage of the assessment process for the 2014 round was soundly-based and benefited from third-party assessment of applications by state health services and colleges. However, Health decided not to fund some highly-rated applications recommended by the state health services and colleges as it sought to obtain a relatively even distribution of the new training positions against the population. In adopting this approach, which also featured in the previous (2013) funding round¹⁴, the department effectively applied a selection criterion that was not documented in the application form or other explanatory material made available to applicants. Although the program’s funding priorities, which underpinned the assessment criteria, were reviewed between rounds, Health did not take the opportunity to incorporate a reference to the approach adopted on population distribution in the 2014 round application form or explanatory material. Nevertheless, when considered in the context of the program’s intended outcomes (which include achieving a better geographical distribution of specialist services) the department’s approach in relation to this matter was not unreasonable.¹⁵

16. Further, the department advised the ANAO that it also adopted an internal review and rescoring procedure in the 2014 round¹⁶, to address differences in assessment scores from the state health services and colleges¹⁷; another departure from documented processes. Health’s approach was also inconsistent with information provided to applicants that the department would ‘collate’ assessment results received from state health services and the colleges, and advice to the Health Minister in March 2013 that the department’s selection of applicants was ‘essentially administrative’ as individual assessments were based on recommendations received from the state health services and colleges.

17. The ANAO has made one recommendation aimed at improving the transparency and equity of Health’s grants administration by: reviewing program guidelines and assessment criteria to incorporate lessons learned from funding rounds; and providing operational guidance to staff on moderation or other quality control processes to be applied to assessments by third-party advisers.

Key findings by chapter

Assessment and Selection of Applications (Chapter 2)

18. In the 2014 funding round, applications for STP grants were open to a broad range of organisations, consistent with the general program objective that specialist training occur beyond traditional teaching settings. The assessment process for the selection of grants was also outlined in explanatory material to the application form. Stakeholders informed the ANAO that the opening of the 2014 round was well publicised amongst potential applicants, and in the event, some 467 applications were received for the 150 available grants.

19. Health trialled the use of shared services for the 2014 funding round, to make the application process easier for applicants and to achieve administrative efficiencies. However, stakeholders advised the ANAO that while the electronic application process had improved overall administration compared to previous rounds, the trial suffered from insufficient testing prior to implementation. In particular, the trial encountered significant technical problems relating to the receipt and processing of applications, and the anticipated benefits and efficiencies were not fully realised.

20. The assessment criteria and selection processes outlined in the department’s assessment plan, application form and other publicly available explanatory material for the 2014 round reflected Australian Government priorities, which were informed by published research of Health Workforce Australia. Further, the involvement of specialist medical colleges and state health services in the assessment of applications strengthened the assessment process. In particular, the state health services and colleges provided third-party advice to Health on the educational merit of applications, the potential impact of applications on health services, and the extent to which applications met program funding priorities.

21. ANAO testing of Health’s final assessment of applications for the 2014 funding round indicated that the department adopted an internal review and rescoring process after receiving input from state health services and the colleges. Around 250 of the 467 applications received in the 2014 STP funding round were reviewed as part of this process, which was not documented. ANAO analysis indicates that the rescoring directly affected funding outcomes for 13 applications, representing some 2.8 per cent of all (467) 2014 round applications. For 12 of these applications, the final score assigned by Health was above that which was calculated by the ANAO based on input from the specialist medical colleges and state health services, indicating these applications likely benefited from the review and rescoring process to the extent that they were offered grants. As there were only 150 grants available through the 2014 round, the elevation of the 12 applications meant that some applicants that may have otherwise been offered a grant were not. One application was scored down by the department and as a consequence of this, was not offered a grant.

22. Further, the department decided not to fund some highly-rated applications received as part of the 2014 funding round. Specifically, 13 applications were placed on a ‘reserve’ list¹⁸, as the relevant setting had submitted applications for two or more training positions in the same specialty.¹⁹ In these cases, Health funded only one place in order to obtain ‘a relatively even distribution of the new training positions against the population data’. In adopting this approach, Health effectively applied a selection criterion that was not documented in the application form or other explanatory material made available to applicants for the 2014 funding round. Health’s approach was inconsistent with the instructions provided to applicants for completing the program application form, which indicated that the department would ‘collate’ assessment results received from state health services and the colleges. Further, the approach adopted was not consistent with advice provided to the Health Minister in March 2013 that the department’s selection of applicants to be funded under the STP was ‘essentially administrative’, and that individual assessments were based on recommendations received from the state health services and colleges.

23. The lack of appropriate record-keeping and quality control in the internal review and rescoring process, and the use of a selection criterion that was not contained in the 2014 round application form or other explanatory material available to applicants, affected the transparency and to an extent the equity of the assessment process when viewed in terms of the application form and other explanatory material that informed applicants’ expectations about how grants would be selected.²⁰ However, as previously noted, the department’s approach in using a selection criterion that incorporated population distribution considerations was not unreasonable in the context of the program’s intended outcomes.

Administration of Funding Agreements (Chapter 3)

24. A feature of the STP is that while Health has overall responsibility for its administration, the department does not have a direct contractual relationship with individual settings, which are the grant recipients. Rather, the administration of STP grant funds is managed through separate agreements between Health and the respective colleges. Under this ‘college administration’ model, developed by the department in 2010, all grant funding for STP training positions within a particular specialty is provided by Health to the relevant college.

25. The colleges and settings interviewed by the ANAO indicated that the ‘college administration’ model generally worked well. However, some potential risks were not fully assessed by Health when developing the model, including the risk that the Australian Government’s financial framework requirements might apply to the colleges if they handled public money.²¹ It is prudent for government entities to consider, at the design stage, the full implications of complex financial and administrative arrangements—such as those involving third-party administration of government programs—so as to avoid potential compliance and reputational risks.²²

26. The department relies on reports received from the colleges to inform its oversight of the college administration model.²³ Since 2012 Health has sought additional management and performance information from the colleges, particularly relating to financial issues. However, the presentation of financial information by colleges—particularly income and expenditure—has varied significantly, sometimes making it difficult for Health to assess how funds have been spent. Where variations in reporting have occurred, Health has undertaken follow-up communication with colleges to determine their actual financial position.

27. As at 31 December 2013, total surpluses of STP funds held by the colleges were $36.28 million.²⁴ By 30 June 2014, total surpluses had risen to $56.31 million. The surpluses can be partly attributed to timing issues, including delays in the submission of invoices by training settings to the colleges. During 2014, Health responded more actively where surpluses were identified, by withholding a proportion of scheduled progress payments. In consequence, $23.89 million that was due to be paid following receipt of the July 2014 college reports was withheld.

Program Performance and Evaluation (Chapter 4)

28. The STP has had key performance indicators (KPIs) in place since the consolidation of the program in 2009. However, explicit outcome-linked KPIs were only developed in 2013, and the colleges have reported against these KPIs since January 2014.

29. College reporting against the program’s KPIs indicates that the STP has been successful in utilising non-traditional settings to expand the number of training positions for specialist trainees, with 89 percent of STP-funded positions being located in non-traditional settings. In discussions with the ANAO, stakeholders also suggested that the expanded range of work environments has contributed to the overall quality of training.

30. ANAO analysis of college reporting indicates that on a FTE basis, around 833 training positions were filled as at 30 June 2014, representing 93 per cent of the target of 900 positions. Overall, the additional specialist trainee positions funded by the STP have boosted the availability of specialist services, including in regional and rural areas. However, it remains unclear to what extent the STP has, or will, contribute to an improved geographical distribution of specialist services to meet community need, over the longer term.

Summary of entity response

31. The Department of Health agrees with the audit recommendation. The findings of the audit will be of value in the future administration of the Medical Specialist Training Programme.

32. Health’s full response is provided at Appendix 1.

Recommendation

Training medical specialists in Australia

1.1 In Australia, medical graduates must undergo a 12 month period of additional training before becoming fully qualified as medical practitioners (doctors) and receiving a provider number to enable billing under Medicare.²⁵ Most commonly, this training involves an internship at a public hospital, followed by a further 12 months as a resident medical officer. After this, doctors may undertake further training, either in general practice or in a particular medical specialty.²⁶ In the latter case, a doctor will enter a training program under the auspices of the relevant specialist medical college (college).²⁷

1.2 Under these college programs, specialist trainees (also known as registrars) are placed at specified training positions²⁸ within public or private hospitals, other medical facilities or health organisations that are accredited for the purpose of the specialist training programs by the relevant college.²⁹ Specialist trainees will generally rotate through a number of training positions at different hospitals or facilities (which are called ‘settings’) during their specialist training in order to gain a broad range of experience and skills.

1.3 On completion of specialist training—which may take between three and six years³⁰—and subject to meeting any other requirements of the relevant college, specialist trainees are eligible to apply for Fellowship³¹ of the college and be recognised as a fully qualified specialist. Whilst undergoing training, specialist trainees may choose to focus on a particular sub-speciality or discipline.³² Alternatively, they may train across a broader spectrum: in some cases this may lead to specialist trainees qualifying within a ‘generalist’ specialist stream.³³ An illustrative medical education and training pathway for a specialist is shown at Figure 1.1.

Figure 1.1: Medical specialist education pathway in Australia

Source: Health Workforce 2025—Volume 3 Medical Specialties, p. 406.

1.4 There has been substantial growth in the overall number of fully qualified specialists in recent years. Excluding general practice specialists, the number of actively-practising college Fellows has risen from 26 946 in 2008 to 32 702 in 2012, an increase of 21.4 per cent.³⁴,³⁵ Excluding general practice, the number of specialist trainees undergoing specialist training increased from 10 649 in 2009 to 13 801 in 2013, an increase of 29.6 per cent.³⁶ In comparison, Australia’s population increased by 7.7 per cent between 2008 and 2013.³⁷

The Specialist Training Program

Background

1.5 State and territory governments currently fund over 90 per cent of all medical specialist training in Australia through specialist trainee positions in public teaching hospitals. Over the last 15 years the Australian Government has also contributed medical specialist training through a range of programs administered by the Department of Health (Health). These programs have funded a broad range of activities, including those with a specific geographic focus³⁸, directed at a particular specialty³⁹, or dealing with doctors trained outside Australia.⁴⁰ In 2009, programs funded by the Australian Government were consolidated into the medical Specialist Training Program (STP) in order to create a ‘simpler, more flexible funding program’.⁴¹

1.6 In March 2010, the then Australian Government announced a range of significant health policy and funding measures under the National Health and Hospitals Network initiative. A component of the initiative was additional funding of $144.5 million over four years to increase the number of specialist training positions under the STP from around 360 to 900 by 2014.⁴²

1.7 The 2010 Australian Government announcement noted that work by the Australian Medical Workforce Advisory Committee⁴³ and the colleges suggested there would be a shortage of 1280 specialists in Australia by 2020. The expansion of the STP aimed to deliver 680 additional specialists by 2020 by drawing on the private sector and other non-traditional avenues for training. The remainder of the projected shortfall was to be addressed by state health services⁴⁴ increasing the number of training positions in their public teaching hospitals. Specialties where shortages existed were to be targeted through the expanded STP, including general surgery, pathology, radiology, dermatology, obstetrics and gynaecology. Priority was also given to providing training positions ‘where Australians need them, such as in rural and regional areas’.⁴⁵

1.8 The objectives of the STP are to⁴⁶:

• increase the capacity of the health care sector to provide high quality, appropriate training opportunities to facilitate the required educational experiences for specialists in training;
• supplement the available specialist workforce in outer metropolitan, rural and remote locations; and
• develop specialist training arrangements beyond traditional inner metropolitan teaching hospitals.

1.9 The objectives outlined above were to be ‘achieved without an associated loss to the capacity of the public health care system to deliver services’.⁴⁷’⁴⁸

1.10 Associated with the STP objectives are nine ‘expected outcomes’⁴⁹:

• rotation of specialist trainees through an integrated range of settings beyond traditional inner metropolitan teaching hospitals, including a range of public settings (including regional, rural and ambulatory settings), the private sector (hospitals and rooms), community settings and non-clinical environments;
• increased number and better distribution of specialist services;
• increased capacity within the sector to train specialists;
• improved quality of specialist training with trainees gaining appropriate skills not otherwise available through traditional settings;
• development of system wide education and infrastructure support projects to enhance training opportunities for eligible trainees;
• improved access to appropriate training for overseas trained specialists seeking Fellowship with a college;
• increased flexibility within the specialist workforce;
• development of specialist training initiatives that complement those currently provided by state health services; and
• establishment of processes which enable effective and efficient administration of specialist training positions, with reduced complexity for both stakeholders and the administering department.

1.11 The STP is administered by the Australian Government through the Department of Health (Health).⁵⁰ Formal stakeholder input into the operation of the STP is through two main sources: the Medical Training Review Panel⁵¹ and the STP inter-college forum.⁵²

How the program is delivered

1.12 The STP is an executive grants program⁵³ involving annual competitive funding rounds.

Funding

1.13 The grants are provided to hospitals and other medical facilities or health organisations to employ specialist trainees. Depending on the circumstances, STP grants to settings can consist of a number of components, as follows:

• The primary element is in the form of salary support to settings to financially assist them to employ a specialist trainee at a specified training position. Salary support is set at $100 000 (ex GST)⁵⁴ per full-time equivalent (FTE) per year.⁵⁵
• Training positions outside metropolitan areas are eligible for an additional rural loading of $20 000 per FTE per year.⁵⁶ ANAO discussions with settings indicate this ‘rural loading’ is used to subsidise a range of expenses, including covering transport costs or, in some cases, providing specialist trainees with accommodation and/or a vehicle.
• Where the position is in a private sector setting, it is also eligible for the Private Infrastructure and Clinical Supervision allowance. This consists of a clinical supervision allowance of $30 000 per FTE per position, per year—which is intended to recognise the time and effort involved in a specialist supervising a specialist trainee whilst they are undertaking training—and a training infrastructure allowance of $10 000 per FTE once every three years.⁵⁷ Consistent with the intent of the STP to utilise non-traditional avenues for specialist training, the Private Infrastructure and Clinical Supervision provides an incentive for the private sector to participate in the program.

1.14 As a consequence, an STP grant will typically provide financial assistance of between $100 000 to $153 333 per FTE per year for a training position.

1.15 In order to avoid cost-shifting⁵⁸, the STP only funds ‘new’ training positions. Training positions that have been funded from any source for more than 12 months out of the last three years are ineligible for STP grants.^59,60 Applications must also be accompanied by a letter of support from the local hospital network as well as the relevant college.

Administration

1.16 A feature of the STP is that while Health has overall responsibility for its administration, the department does not have a direct contractual relationship with the settings, which are the grant recipients. Rather, the administration of STP grant funds is managed through separate agreements between Health and the colleges. Under this ‘college administration’ model, all grant funding for STP training positions within a particular specialty is provided by Health to the relevant college. Under their respective funding agreements with Health, the colleges have responsibility for:

• disbursing the STP grant funds to the relevant settings through periodic payments;
• oversight of the conduct of the funded training positions through contractual and liaison arrangements with the settings;
• the development and implementation of strategic training and education projects to support the network of STP training positions;
• promoting the integration of training provided through STP training positions with that provided by state health services; and
• providing progress and financial reporting to the department.

1.17 Colleges receive funding from Health to undertake the administrative functions referred to in paragraph 1.16.⁶¹ The amount of administrative funding provided by Health varies between specialist medical colleges when measured on a ‘per position’ basis. However, for the majority of specialist medical colleges the amount of administrative funding is approximately $5000 to $10 000 per position, per year.⁶²

Progress in implementing the Specialist Training Program

1.18 Since the expansion of the STP announced in 2010⁶³, four annual STP competitive funding rounds have been conducted by Health.⁶⁴ The 2014 round⁶⁵, which was completed in December 2013, increased the number of training positions funded under the program from 750 to 900. The progressive expansion in the number of STP training positions through the four funding rounds is shown in Table 1.1. As at December 2014, the Australian Government has not announced whether any further funding rounds will be undertaken, or whether supported training positions will have funding extended beyond the current contractual commitments that cease at the end of December 2015.

Table 1.1: Expansion of STP training positions 2011–2014

Source: Department of Health Annual Reports.

1.19 While the STP has achieved the target of 900 training positions, some funded settings have subsequently withdrawn from the program or have had periodic difficulties in attracting or retaining specialist trainees in the training positions. In such cases, colleges have utilised so-called ‘reserve lists’ of highly ranked unsuccessful applicants from the previous funding round to boost the number of occupied training positions. While calculating the exact number of occupied training positions is difficult due to some variations in the relevant college reports and/or late reporting by settings, ANAO analysis indicates that around 833 of the 900 training positions⁶⁶ (93 per cent) were occupied during the January to June 2014 reporting period. Several colleges with higher vacancy rates⁶⁷, or which experienced underspends of previously provided STP funds⁶⁸, indicated in their July 2014 progress reports that they will draw on reserve lists from the 2014 funding round to increase the number of occupied training positions in the second half of 2014 and in 2015.

1.20 ANAO analysis of Health’s financial records indicates that over the four years from 2010–11 to 2013–14, some $336.53 million has been expended on the STP, as shown in Table 1.2. In the first six months of 2014–15, expenditure has slowed over that of the previous year, with $42.46 million spent to 31 December 2014. This reflects the department’s decision to withhold a proportion of scheduled progress payments in response to the build–up of significant surpluses of STP funds held by the colleges.⁶⁹

Table 1.2: STP expenditure 2010–11 to 2014–15

Source: Department of Health data.

Notes: The above figures do not include the amounts for two other initiatives, Emergency Department Workforce (Doctors and Nurses) and Specialist Training in the Tasmanian public health system, which are largely delivered under STP agreements with the colleges. These two initiatives provide funds, including to support specialist training positions, of up to $123.83 million from 2010–11 to 2015–16. Further details are at paragraphs 1.23–1.28.

Distribution of Specialist Training Program training positions

1.21 The national distribution of all funded STP training positions is shown in Figure 1.2. The distribution broadly corresponds to population distribution, with the exception of the Northern Territory, which has 4.5 per cent of STP training positions but only one per cent of Australia’s population.⁷⁰

Figure 1.2: Funded STP training positions 2010–14 by State and Territory

Source: ANAO analysis of Department of Health data.

1.22 The distribution of STP training positions by medical speciality is shown in Figure 1.3. Some 40.9 per cent of training positions are in specialties that fall within the auspices of the Royal Australasian College of Physicians. These cover a broad range of specialties, including public health medicine, paediatrics, geriatric medicine, medical oncology and cardiology. The significant proportion of psychiatry training positions (18.2 per cent) is consistent with the increased emphasis on mental health in Australian health policy in recent years.

Figure 1.3: Funded STP training positions 2010–14 by medical specialty

Source: ANAO analysis of Department of Health data.

Note: Percentages do not equal 100 per cent due to rounding up and down of original data. Not shown are the two training positions funded through the STP that are administered by the Australasian College for Emergency Medicine. These represent 0.2 per cent of all STP training positions.

Delivery of other activities through the Specialist Training Program

Emergency Department Workforce (Doctors and Nurses)

1.23 In July 2010, the then Australian Government announced the $96 million Emergency Department Workforce (Doctors and Nurses) measure. This was intended to increase the capacity of the healthcare sector to train emergency department specialists, nurses and support staff, as well as training general practitioners in emergency medicine.

1.24 A significant component of this measure has been delivered through the STP. Grant applications for new training positions for specialist trainees specialising in emergency medicine have been considered through the annual STP funding rounds.⁷¹ Each year, 22 additional training positions have been funded through this process—these are additional to the 900 training positions targeted by the STP process. During the January to June 2014 reporting period, 87 training positions were filled⁷², an outcome very close to the target of 90 training positions specified in the relevant funding agreement.⁷³ The grants are administered through a funding agreement between Health and the Australasian College for Emergency Medicine (ACEM) using the STP model, although ACEM’s reporting obligations are slightly different to the other colleges.⁷⁴

Specialist Training in the Tasmanian public health system

1.25 In June 2012, the Australian Government announced a $325 million Tasmanian ‘health assistance’ package. One element of the package was the $39.6 million Training more Specialist Doctors in Tasmania initiative (the ‘Tasmanian project’), which provided targeted grants to specifically address the public health needs of Tasmania and the medical specialist workforce in Tasmania’s public hospitals.⁷⁵ Tasmania’s difficulty in recruiting and retaining its health workforce, including specialist trainees and specialists, was a key issue in the 2004 review of the Tasmanian health system by an expert advisory group.⁷⁶ The provision of specialist services also featured in the 2012 preliminary report of The Commission on Delivery of Health Services in Tasmania.⁷⁷

1.26 Following discussions between Health, the Tasmanian Department of Health and Human Services, Tasmanian regional health organisations and colleges, a detailed implementation plan for the Tasmanian project was approved by the Australian Health Minister in June 2013. The project involves Health providing grants to fund an initial 38 FTE specialist trainees and 10.7 FTE specialist supervisor positions in specified settings in the Tasmanian public sector health system in 2013–14, rising to 51 FTE and 14.5 FTE positions in 2015–16.⁷⁸ The Tasmanian project explicitly funds the full cost of the relevant positions.⁷⁹ On a FTE basis, this equates to around $120 000–180 000 per year for a specialist trainee, depending on their speciality and seniority, and an average of $365 000 for a specialist supervisor.

1.27 STP funding agreements were amended in 2013 to provide for delivery and administration of Tasmanian project funding through the agreements. As is the case for STP funds, Tasmanian project funds are administered by the relevant college, but there are different reporting requirements and payment schedules applying to the Tasmanian project funds. The colleges do not receive further administration funding for the Tasmanian project over and above the STP funding that they receive from Health.

1.28 As at June 2014, progress in filling funded Tasmanian positions has been relatively slow. In part, this was due to a previously unrecognised legal issue concerning medical indemnity, with resolution of this matter also delayed by a change of government in Tasmania in March 2014. During the January to June 2014 reporting period, 31.35 FTE specialist trainees and 7.46 FTE supervisors were in place, around 80 per cent of the target for 2013–14.

Grants administration framework

1.29 The STP is subject to the Australian Government’s framework for grants administration, which has operated since 2009. Commonwealth Grant Guidelines were introduced under the Financial Management and Accountability Act 1997 (FMA Act) and Financial Management and Accountability Regulations (FMA Regulations) in July 2009 and updated in June 2013. Since July 2014, the Commonwealth Grants Rules and Guidelines have operated pursuant to the Public Governance, Performance and Accountability Act 2013 (PGPA Act) and related Rules. Since 2009, the Commonwealth Grant Guidelines and subsequently the Commonwealth Grants Rules and Guidelines have outlined a generally consistent legislative, policy and reporting framework for administering grants, and seven key principles for better grants administration. For the purposes of this audit, clear reference is made to which set of guidelines and/or rules (2009, 2013 or 2014) are relevant to the discussion of Health’s administration of the STP at any one time.

Health grants policy

1.30 In 2011, following the completion of a review of administrative arrangements in the Health and Ageing portfolio, the Australian Government decided to implement a range of changes within the portfolio, including consolidating 159 programs (many of them grant programs) into 18 internal ‘flexible funds’.⁸⁰ The flexible funds concept was intended to ‘reduce red tape, provide increased flexibility to respond to emerging issues and deliver better value with public money’⁸¹ The STP falls within one of these flexible funds, the Health Workforce Fund. The primary objective of the Health Workforce Fund is to strengthen the capacity of the health workforce to deliver high quality care.⁸² Policy responsibility for the STP, as for other Health Workforce Fund matters, lies with the department’s Health Workforce Division. However, administration of the STP funding agreements, including the assessment of college reporting and the authorisation of progress payments, is undertaken by the department’s Grant Services Division.⁸³

Previous reviews of the Specialist Training Program and specialist training

1.31 The ANAO has not previously examined the STP. ANAO Performance Audit Report No.34 2010–11 General Practice Education and Training examined the management of general practice vocational education and training programs by General Practice Education and Training Limited (GPET), then a Commonwealth company.

1.32 A review of Health’s administration of the funding agreement with one of the colleges in receipt of STP grant funding was carried out by the department’s internal audit unit in 2012. The review indicated that ‘substantial improvement’ was required in key aspects of the department’s administration. Key findings related to unspent grant funds, a substantial shortfall in the number of training positions actually filled, and inconsistencies in financial reporting.

1.33 In 2012, the then Australian Government commissioned a review (the Mason Review) to assess the appropriateness, effectiveness and efficiency of the health workforce programs and activities, and their alignment with Australia’s workforce priorities.⁸⁴ The review’s April 2013 report observed that:

The STP has been highly successful in extending vocational training into new settings, particularly in the rural and private sectors. It has also demonstrated that specialist colleges can take a flexible approach to accrediting new positions and to supporting networked training arrangements involving multiple health care settings, sometimes in different regions.⁸⁵

1.34 However, the review also made some observations about career pathways for medical graduates wishing to specialise, and recommended a full review of the STP to inform its future direction and consider whether existing training positions were meeting the program’s objectives.

1.35 Observations about the overall coordination of training pathways for specialists were also made by Health Workforce Australia in its 2012 report Health Workforce 2025:

While better organised and targeted at a national level, the Commonwealth [specialist and GP training programs] lack the requisite level of coordination and alignment with state and territory approaches to best leverage the collective tax payer funded training investments of both levels of government. Rectifying these shortcomings is essential to achieving the long-term workforce outcomes required.⁸⁶

1.36 The coordination of medical training is being considered through the National Medical Training Advisory Network, formed in 2012 under Health Workforce Australia.⁸⁷ Health advised that the National Medical Training Advisory Network is developing a series of rolling medical training plans, including in relation to specialists, to inform the Australian Government, health and education sectors.⁸⁸ The department further advised that the first specialist training plan (for psychiatrists) is expected to be provided to Health Ministers for their consideration around February or March 2015.

Audit objective, criteria and methodology

1.37 The audit objective was to assess the effectiveness of the Department of Health’s (Health) administration of the Specialist Training Program (STP). The audit focused on key aspects of Health’s administration of the STP since the consolidation of funding programs in 2009, and the achievement of key program targets and objectives. To assess the department’s grants administration, the ANAO focused on the fourth annual grant funding round (the 2014 round) which was completed in December 2013 and funded training positions from the beginning of 2014.

1.38 The audit methodology involved:

• conducting over 30 interviews with primary stakeholders of the STP, including specialist medical colleges, state health services, the Australian Medical Association, and settings that received STP grants;
• undertaking a detailed review of key aspects of the application and assessment process for the 2014 funding round, including how Health handled the assessment results received from the colleges and jurisdictions;
• reviewing the process through which existing funding agreements were varied to take account of the outcomes of the 2014 funding round; and
• examining college reporting for 2013–14, Health’s analysis of that reporting and the making of progress payments.

1.39 The audit reviewed Health’s administration of the STP against the Australian Government’s resource management and grants frameworks and the ANAO’s grants administration better practice guide.⁸⁹

1.40 The audit was conducted in accordance with the ANAO Auditing Standards at a cost to the ANAO of approximately $450 140.

Report structure

1.41 The structure of the audit report is outlined in Table 1.3.

Table 1.3: Structure of the audit report

Introduction

2.1 The 2014 funding round was the fourth to be conducted following the Australian Government’s decision in 2010 to significantly expand the Specialist Training Program (STP). The 2014 round was intended to fund an additional 150 training positions, so as to bring the total number of STP training positions up to the target of 900.

2.2 The application, assessment and selection process for the 2014 funding round broadly followed the model established in previous rounds. The only significant change was that applications were to be completed and submitted electronically utilising a ‘smartform’, with storage and processing of the applications occurring through an existing Department of Social Services (Social Services) online funding management system, FOFMS. This process was intended to improve Health’s grants administration by making it easier to apply and achieving efficiencies in the department’s handling of applications during the assessment and selection stages.

Development of grant guidelines and application documentation

2.3 Under the Australian Government’s grants framework, all grant activities must be administered according to a set of approved guidelines. These guidelines must also be made publicly available.⁹⁰ The STP guidelines were originally developed by Health in 2009 and subsequently approved by the Expenditure Review Committee of Cabinet in December 2009, as then required by Australian Government policy. Consistent with changes in Australian Government policy, subsequent revisions of the STP guidelines (including in relation to the 2014 round) were subject to a risk-based assessment, rather than Expenditure Review Committee consideration, involving consultation with the Department of the Prime Minister and Cabinet and the then Department of Finance and Deregulation.

2.4 The STP grant guidelines were subsequently revised and renamed as the STP Operational Framework (Operational Framework) in 2011.⁹¹ The Operational Framework sets out: the program objectives and expected outcomes; governance arrangements; available funding amounts; eligibility requirements; and an overview of the application and assessment process. The Operational Framework is supplemented by the STP Priority Framework (Priority Framework), which sets out the funding priorities for each round.

2.5 Stakeholder input into the Operational Framework and Priority Framework for the 2014 round was received through the Medical Training Review Panel. In addition, work by Health Workforce Australia informed the development of the Priority Framework.⁹²

2.6 Priorities for the 2014 round were:

• private sector healthcare settings;
• settings in regional, rural and remote areas;
• non-hospital settings including aged care, community health and aboriginal medical services;
• specific areas of medicine (obstetrics and gynaecology, ophthalmology, anatomical pathology, diagnostic radiology, radiation oncology, medical oncology, geriatric medicine and psychiatry);
• generalist training⁹³; and
• dual training.⁹⁴

2.7 In addition, the Priority Framework specified that preference would be given in the assessment process to training positions which demonstrated:

• their capacity to be filled with Indigenous trainees;
• specialist trainee involvement with clinical academic research or teaching junior doctors and/or medical students; or
• capacity for an individual trainee to complete the majority of training requirements for Fellowship⁹⁵ in an on-going position in a rural, regional or remote setting.

2.8 The Priority Framework for the 2014 round was approved by the Minister for Health in early March 2013. As with the previous round, the Minister delegated authority to approve the 2014 round application documentation to the department’s relevant First Assistant Secretary. The Minister also delegated authority to the First Assistant Secretary to approve the ‘outcomes’ of the round—that is, the selection of applications to be funded. In seeking the Minister’s approval, the department advised that ‘the Priority Framework is the key document that determines the composition of the outcomes’ with the actual approval of outcomes being ‘essentially administrative, as the assessment is based on the recommendations from the colleges and [state] health services’.

2.9 Immediately after receiving the Minister’s approval, the First Assistant Secretary approved the application documentation. This documentation included both public material (the Operational Framework, the Priority Framework, the application form and the associated guidance material for applicants) as well as the internal assessment plan. The documentation was also reviewed by Health’s internal grants advisory area⁹⁶, although the relevant clearance indicates the review was done before the assessment plan was drafted. Other than a slightly different set of priorities, the only material change in the 2014 assessment plan over the 2013 round was the insertion of a section on ‘probity, accountability and ethics’.

Application process

2.10 As in previous rounds, a broad range of organisations were eligible under the Operational Framework to apply for an STP grant, including medical education providers, state health services (including local hospital networks and regional hospitals), community health organisations, private healthcare organisations and settings, and Aboriginal community–controlled health services.

2.11 The application process opened on 20 March 2013 and closed on 1 May 2013. Stakeholders advised the ANAO that the opening of the 2014 round was well publicised amongst potential applicants, including through the colleges advising their Fellows and other contacts. Stakeholders also considered that the six week period was generally sufficient to develop an application with all necessary supporting documentation⁹⁷ and the information required in the application was proportionate to the nature of the grants on offer and the objectives of the STP. Some 467 applications were received for the 150 training positions on offer, with another 44 applications for the 22 Emergency Department Workforce training positions.⁹⁸ The number of applications was similar to previous rounds, indicating on-going demand for STP funding.

Use of shared services to support the 2014 round

2.12 As noted by the 2014 National Commission of Audit⁹⁹, one way that agencies can achieve efficiencies and reduce costs is through the appropriate use of shared services. Shared services involve the single provision of particular functions to more than one agency. They commonly cover activities in the areas of human resources, information management, communication, technology, procurement and financial management.

2.13 Since 2005, the Department of Social Services¹⁰⁰ has administered the FaHCSIA Online Funding Management System (FOFMS). One of the main capabilities of FOFMS is the administration of grant activities. In 2012, under an agreement with Social Services, Health started to progressively roll out the use of FOFMS to manage a range of payments it administered.¹⁰¹

2.14 In late 2012, Health decided to use FOFMS to support the upcoming STP 2014 funding round. This involved the development of a tailored ‘smartform’ by Social Services¹⁰² that allowed applicants to complete and submit applications electronically, including letters of support from the relevant college and the local hospital network. Development of the smartform and integrated FOFMS process took place from October 2012 to March 2013. It was originally anticipated that the project would enable applications to open around early March, however the operational smartform was not delivered to Health until 18 March 2013.

2.15 Stakeholders advised the ANAO that they generally considered the electronic application process to be an improvement on previous rounds, although some problems were encountered when unexplained error messages were received when attempting to complete the form or where there were difficulties in attaching the required supporting documentation.

2.16 However, a number of more significant technical problems arose when receiving and processing applications, which reduced the overall efficiency and effectiveness of the application process. Notably, FOFMS experienced an outage on the day immediately before applications closed, and applications could not be submitted during this time. Some applicants instead emailed applications to the Health STP mailbox, and the volume of data caused the mailbox to crash. As a consequence, these applications had to be manually loaded into FOFMS by departmental staff. Due to the various technical issues encountered, a large number of duplicate applications were stored in and extracted from FOFMS, requiring significant work by departmental staff to identify which applications were duplicates.¹⁰³ Further, Health had anticipated that it could implement automated screening of ineligible applications by FOFMS—for example, where applications did not include the required letters of support—but the technical problems precluded this.

2.17 A post-implementation review of the project by Social Services, with the involvement of Health, resulted in 23 recommendations intended to inform any future activity of this sort. Further development of the smartform application concept to support a FOFMS-based grants funding round for the Rural and Regional Teaching Infrastructure Program was undertaken by Health in 2014. A specific risk management process, derived directly from the review of the STP pilot, and involving more extensive testing, training and technical support, was developed to inform the use of FOFMS for the Rural and Regional Teaching Infrastructure Program funding round. However, in October 2014, Health advised the ANAO that the Rural and Regional Teaching Infrastructure Program round, which was due to open for applications in September 2014, did not go ahead as the program was put ‘on hold’.

Assessment process

2.18 A key consideration in administering granting activities is to implement a selection process that identifies and recommends for funding those applications that will provide greatest value with public money in the context of government objectives for the granting activity, as set out in the grant guidelines.¹⁰⁴ Well-designed assessment processes can provide confidence that decision-makers have equitably and transparently selected applicants that best represent value for public money in the context of the program objectives and outcomes.¹⁰⁵

Assessment criteria

2.19 The instructions provided to applicants for completing the application form stated:

Each eligible application will be assessed by:

• the relevant specialist medical college for rating in terms of the position’s ability to meet the appropriate educational imperative; and
• the relevant state or territory government for rating in terms of jurisdictional areas of workforce need.

These assessments will be provided to the Department of Health and Ageing who will collate assessment results for the decision maker. The outcome of the 2014 STP Application Round will be based on the 2014 Priorities and further application weightings as detailed in the 2014 STP Priority Framework (Attachment B). These weightings are in no particular order. Additional information may be sought from the relevant college or jurisdiction to assist with this process.

2.20 Other sections of the lengthy instructions indicated that the colleges and state health services would also be involved in assessing applications against the Priority Framework, and would not be restricted to assessing educational and workforce need aspects.

2.21 In summary, applications would be assessed by third parties—the colleges and state health services—and the assessments collated by the department prior to consideration by a departmental decision-maker.

Assessment of applications by the colleges and state health services

2.22 Health’s internal assessment plan provided for an initial screening of applications by the department against mandatory requirements. However, the initial screening did not occur due to the problems with FOFMS, discussed above, and all applications were accepted as compliant and proceeded to assessment. ¹⁰⁶

2.23 The assessment process was conducted in two stages. The first stage was an assessment by the colleges and state health services. Health provided the colleges and state health services with a four–page template to assist in applying a consistent approach to assessments and a spreadsheet on which their assessment responses could be recorded. The template and spreadsheet provided for assessment against two key criteria:

• The extent to which the application met each of the Commonwealth priorities and preferences in the Priority Framework outlined in paragraph 2.6–2.7. Assessment responses were required to indicate ‘yes’ or ‘no’ against each priority and preference.
• An overall ‘global rating’ of the application using a four point sliding scale—strong support, moderate support, minimal support or no support. Health’s advice to the colleges and state health services was that, in addition to the application’s strength in addressing the Priority Framework, the global rating should reflect (in the case of the state health services assessments) ‘the benefit to health services’ and (for college assessments) the ‘educational merit’ of the application.¹⁰⁷

2.24 Applications were provided to the relevant colleges and state health services by Health on 17 May 2013.¹⁰⁸ Assessment responses were generally returned to the department by mid-June, although one was not received until late June.¹⁰⁹

2.25 The colleges and state health services advised the ANAO that they took a number of different approaches to how they assessed applications, including through established or ad hoc committees, providing the applications to local health networks or, in the case of the colleges, having senior officers or Fellows of the college advise on the applications. Similarly, the approaches adopted by the colleges and state health services in managing potential conflicts of interest varied. Most colleges and state health services told the ANAO that either formal or informal mechanisms were in place to manage the risk of real or perceived conflicts of interest.¹¹⁰ Nonetheless, in one instance, a senior manager of a public hospital was directly involved in advising the jurisdiction’s Chief Medical Officer on the assessment of applications from the public hospital at which that manager was employed. As such, the manager had a direct interest in the outcome of these applications.

2.26 The assessment documentation provided to colleges and state health services by Health did not address conflict of interest issues. Health advised the ANAO that:

The Department did not specifically request the colleges/jurisdictions to ensure that reasonable conflict of interest provisions were in place for the 2014 funding round. It should be noted that most colleges and jurisdictions had completed three assessment processes before with the operational framework and assessment guidelines distributed to the colleges/jurisdictions clearly explaining roles and included explanatory notes. The Invitation to Apply [documentation] also included a section on how the applications would be assessed and described the STP Complaints Handling Procedures.

No issues have been raised with the Department about possible conflict of interest issues arising during the 2014 funding round.

2.27 Since its introduction in 2009, the Australian Government’s grants administration framework has consistently highlighted the need to manage the risk of real or perceived conflicts of interest.¹¹¹ A recent ANAO performance audit of Commonwealth entities reviewed how conflicts of interests were managed for grants programs where applications were subject to assessment by third parties.¹¹² That audit observed that by its very nature, third-party assessment through peer review processes presents inherent conflict of interest issues. Appropriate management is therefore necessary to satisfy grants framework requirements that agencies ensure the impartiality of decision-making.¹¹³ While Health was the ultimate decision-maker regarding the selection of successful applicants, the college and state health services assessments informed Health’s decisions and hence funding outcomes. Should any future funding rounds be conducted and assessed in a similar way, there would be merit in the department providing appropriate guidance to colleges and state health services regarding its expectations around the management of potential conflicts of interest.

Calculation of scores by Health

2.28 The second stage of the assessment was undertaken by Health. The assessment plan provided that, subject to receiving at least a moderate global rating from both the college and state health services, applications would be assessed against the Priority Framework with the ‘final ranking based on the number of priorities the proposed training position addresses’.

2.29 In fact, all applications were assessed against the Priority Framework. As in the previous round, Health did this by taking the college and state health services assessment responses—relating to whether individual applications met the priorities and preferences set out in the Priority Framework—and converting them into a score. Table 2.1 shows the relative weighting of the various priorities and preferences used to calculate an applicant’s score. The weightings were not included in the application documents provided to applicants or the departmental assessment plan.

Table 2.1: Relative weighting of priorities and preferences

Source: 2014 round Health assessment report.

Notes: A priority setting means a setting in the private health sector, in a regional, rural or remote area, or an aged care, community health or Aboriginal medical service or facility.

2.30 By the end of the second stage of assessment, each application had received two scores out of 12 from Health: one based on the college assessment, and one on the relevant state health service assessment.

2.31 To assess the accuracy of Health’s calculation of scores, the ANAO replicated this scoring process based on the Health weightings reported in Table 2.1. The ANAO analysis was based on the college and state health service assessment responses for 449 out of 467 applications assessed in the 2014 round.¹¹⁴ The analysis identified 178 applications (39.6 per cent of the applications included in the ANAO analysis) in which the score recorded by Health on its assessment database varied from that calculated by the ANAO.

2.32 The ANAO raised the issue of variances with Health, as there was nothing in the assessment report or other records examined by the ANAO to explain the variances. Health advised that for some applications, the scores initially calculated by the departmental assessment team (based on the college assessment response) were different to those calculated from the state health service assessment response. For example, using the weightings in Table 2.1, a college assessment response might equate to a score of eight, but the relevant state health service assessment for the same application might only equate to a score of six. In effect, this meant that the relevant college and state health service had different views on the extent to which an application met the various priorities and preferences contained in the Priority Framework.

2.33 Health advised the ANAO that the differences in the calculated scores had prompted the department’s assessment team to review the ‘initial’ scores of those applications receiving either strong or moderate global ratings, totalling around 250 applications. Following this review, the Health team then assigned a ‘final’ score to the application that they considered most accurately reflected the information in the application. In relation to this ‘review and rescoring’ procedure, Health was able to provide the ANAO with working papers that recorded the initially assigned application scores. However, Health did not document any of the key decisions involved in this procedure–in particular no records were kept of the specific reasons for making changes to the scores of individual applications. The department acknowledged that in hindsight, these changes should have been recorded on the assessment database. The Health assessment team advised that they could not recall any discussions with the colleges or state health services about this subsequent departmental review and rescoring process¹¹⁵, nor could they recall any discussion with the relevant Assistant Secretary within Health.

2.34 The department also acknowledged that there was no further quality control undertaken during the review and rescoring process and that this may have led to some variations or input errors that affected the final scores assigned by Health.

Ranking of applications by Health

2.35 Under the 2014 round, funding was available for 150 new training positions. While the global rating provided by the colleges and state health services was the primary factor in ranking applications and determining funding outcomes, the scores assigned by Health also played a role in the ranking process, and in some cases the department exercised discretion to not fund applications.¹¹⁶

2.36 As Table 2.2 illustrates, 147 applications of the total of 467 applications received a strong global rating by both the relevant college and state health services (a ‘strong/strong’ rating). Of these, 121 (82.3 per cent of all strong/strong applications) were ranked by Health in the ‘top 150’—that is, selected as one of the 150 applications to be offered STP grant funding.¹¹⁷ A further 22 strong/strong applications (15.0 per cent of all strong/strong applications) were placed on the reserve list, and four (2.7 per cent) were unsuccessful. Of the remaining 29 applications that were ranked in the top 150, all received a strong rating from the relevant college and a moderate rating from the relevant state health services, or vice-versa (a ‘strong/moderate’ rating). These 29 successful applications represented 27.1 per cent of all strong/moderate applications. The remaining 78 strong/moderate applications were placed on the reserve list.

Table 2.2: Ranking of 2014 round applications

Source: 2014 round Health assessment report.

Note: The term ‘Other’ in the rating column means either minimal support or no support.

2.37 As noted in paragraph 2.35, in some cases the department exercised a discretion to not fund applications. Of the 22 strong/strong rated applications that were placed on the reserve list, 13 (representing 8.8 per cent of all strong/strong applications) involved situations where the hospital or medical organisation had submitted applications for two or more training positions in the same specialty.¹¹⁸ In such cases, Health funded only one place in order to obtain ‘a relatively even distribution of the new training positions against the population data’, an approach which also featured in the previous (2013) funding round.¹¹⁹ As a result, some relatively high–scoring, strongly–rated applications were not offered funding. The remainder of the unfunded strongly–rated applications were either relatively low–scoring (between two and four), or, in the case of the four unsuccessful applications, were from Tasmanian public hospitals and were not funded on the basis that specialist training in the Tasmanian public sector was being separately supported through the Tasmanian project.

2.38 The 29 ‘strong/moderate applications ranked by Health in the top 150 had scores ranging from a high of nine to a low of five, with three applications scoring five. However there were also 29 applications with the same ‘strong/moderate’ rating that scored either six or seven and which were placed on the reserve list. Health was unable to provide any specific evidence as to why the three applications scoring five were funded in preference to the 29 applications scoring six or seven.

Transparency of the scoring and ranking process

2.39 As noted in paragraph 2.8, Health had advised its Minister in March 2013 that the approval of assessment outcomes by the departmental delegate was ‘essentially administrative, as the assessment is based on the recommendations from the colleges and [state] health services’. While the college and state health service global ratings were the single most influential factor in deciding which applications received funding, the relevant score subsequently assigned by Health was also a factor. As such, the assessment team’s decision to change the initial scores across a substantial proportion of the applications through a review and rescoring procedure that was not documented was a feature of the assessment process.

2.40 Health’s approach was inconsistent with information provided to applicants—in the instructions for completing the program application form—that the department would collate assessment results received from the state health services and colleges.¹²⁰ The failure to document the additional review and rescoring procedure in the program guidelines and instructions; the lack of record–keeping regarding the reasons for changing initial assessment scores; and the failure to provide advice on the procedure in the assessment report¹²¹ provided to the decision-maker, meant that the procedure was not consistent with sound practice.¹²² As discussed, Health acknowledged also that that there were no specific quality control measures in place for the additional review and rescoring, further increasing the risks introduced by the procedure.

2.41 As a result of the variances observed by the ANAO, the department undertook a complete rescoring of applications for the 2014 round.¹²³ As at December 2014, this work remained incomplete.

2.42 Assessing the impact of the undocumented review and rescoring process was made difficult as there was not always a strict ‘cut off’ score for funding. However, there were 12 applications in the top 150 for which the scores calculated by the ANAO¹²⁴ were between two and three and a half points out of 12 but were ultimately assigned a score by Health of between four and seven. In these cases, the department’s rescoring increased the prospects of applications being funded. Conversely, there was one application on the reserve list (which received a strong/strong rating) where the ANAO calculated a score of four but Health assigned a score of two. Had that application been assigned a score of four, it may have been placed in the top 150 and offered funding.

2.43 The ANAO’s analysis of the rescoring of the 13 applications discussed above (which together represented 2.8 per cent of the 467 applications received in the 2014 round) is illustrated in Figure 2.1. The hatched horizontal line at the score of four represents the general minimum score for applications placed in the top 150. As the figure shows, there were 12 applications where the score calculated by the ANAO, based on the college and state health service assessment responses (the ANAO score), was under four, but which were given a final score by Health of four or more; with one application having an ANAO score of four but a Health score of two.

Figure 2.1: Effect of rescoring process on selected applications

Source: ANAO analysis of college and state health services assessment responses and Health data.

2.44 In addition to the review and rescoring process, the department also decided not to fund some applications that received a strong/strong global rating and a relatively high score.¹²⁵ As discussed in paragraph 2.37, 13 applications were placed on the reserve list where the setting had submitted applications for two or more training positions in the same specialty.¹²⁶ In such cases, Health funded only one place in order to obtain ‘a relatively even distribution of the new training positions against the population data’. In adopting this approach, Health effectively applied a selection criterion that was not documented in the 2014 round application form or other explanatory material available to applicants. In adopting this approach, which also featured in the previous (2013) funding round¹²⁷, the department effectively applied a selection criterion that was not documented in the application form or other explanatory material made available to applicants.¹²⁸ Although the program’s funding priorities, which underpinned the assessment criteria, were reviewed between rounds, Health did not take the opportunity to incorporate a reference to the approach adopted on population distribution in the 2014 round application form or explanatory material. Nevertheless, when considered in the context of the program’s intended outcomes (which include achieving a better geographical distribution of specialist services) the department’s approach in relation to this matter was not unreasonable.

Recommendation No.1

2.45 To improve transparency and equity in the administration of grants, the ANAO recommends that the Department of Health:

• review program guidelines and assessment criteria at the conclusion of grant funding rounds, to incorporate lessons learned; and
• provide operational guidance to staff on moderation or other quality control processes to be applied where applications have been assessed by third-party advisers.

Department of Health Response:

2.46 Agreed.

2014 round outcomes

2.47 A total of 150 STP training positions were funded nationally in the 2014 round. Figure 2.2 shows the proportion of training positions located in each state and territory jurisdiction. The proportions broadly correspond to each state and territory’s share of population.¹²⁹ However, the Northern Territory was funded for nine training positions; representing six per cent of all 150 training positions, in contrast to its population share of about one per cent.

Figure 2.2: Funded 2014 round STP training positions by State and Territory

Source: ANAO analysis of Department of Health data.

2.48 The distribution of STP training positions by medical speciality is shown in Figure 2.3. The proportion of specialties funded under the program generally aligned with the number of applications received for each specialty. For example, approximately half of all applications fell within the auspices of the Royal Australasian College of Physicians, and these specialties received 47.3 per cent of training positions.

Figure 2.3: Funded 2014 round STP training positions by medical specialty

Source: ANAO analysis of Department of Health data.

2.49 Approximately 45.3 per cent of successful applications were in medical specialties listed in the Priority Framework for the 2014 round with a similar number (47.3 per cent) incorporating generalist training, another specified priority.¹³⁰ Applications having a rural or regional component were well represented (53.3 per cent)) with applications for training positions wholly or partly in the private sector at around the same levels (55.3 per cent). Fortythree percent involved a component of training in non-hospital settings. Eight per cent of successful applications identified a specific capacity to be filled by an indigenous trainee.

Advising applicants and the provision of feedback

2.50 The selection of the top 150 applications was approved by the departmental delegate on 19 July 2013. Immediately thereafter, letters were sent to all applicants advising whether they were successful or not (in the latter case, applicants were also advised whether they were placed on the reserve list and thus may be potentially funded at a later stage). The letters to successful applicants stated that a formal offer of grant funding was dependent on a contract first being executed between the department and the relevant college.

2.51 The letters also provided the STP mailbox email address should applicants ‘wish to discuss this matter further’. Departmental records indicate that 25 unsuccessful applicants sought feedback. In response to an invitation by the ANAO, some applicants contacted the ANAO in relation to the usefulness of the verbal feedback they received from Health through this process. Views were mixed, although the small number of applicants responding to the ANAO made it difficult to draw any firm conclusions.

2.52 Lists of successful applicants were also provided by the department to colleges and state health services. The colleges and state health services informed the ANAO that they considered the outcomes of the round to be generally consistent with their assessments, although several commented they would have appreciated additional feedback on the reasons why applications within their (state or college) jurisdiction were not successful.

Briefing the Health Minister

2.53 The Minister received departmental advice on the outcomes of the 2014 round through a minute submitted immediately after the Health delegate approved the outcomes. The minute did not require any decision by the Minister—it was for ‘noting’ only.¹³¹ Health advised the ANAO that this was the department’s only communication with the Minister or Minister’s office about the grant outcomes before the applicants were advised by correspondence.

Approval of funding

2.54 Following the delegate’s approval of the outcomes of the assessment process and the applicants being advised, Health started negotiations with the colleges to vary the existing funding agreements. However, this process was not completed before the caretaker period for the September 2013 Australian Federal election took effect.

2.55 The 2014 round STP grants were subsequently considered by the incoming Australian Government, and their implementation was agreed initially by the Finance Minister and then the Health Minister in late 2013, as part of a wider consideration of proposed grants. Following the review process, the departmental delegate gave financial approval under FMA Regulation 9 before formal offers were made to vary the existing funding agreements.¹³² The department provided the delegate with advice on the 2014 funding round assessment process and the role of the colleges in administering the STP training positions selected through that process.¹³³

Conclusion

2.56 In the 2014 funding round, applications for STP grants were open to a broad range of organisations, consistent with the general program objective that specialist training occur beyond traditional teaching settings. The assessment process for the selection of grants was outlined in explanatory material to the application form. Applications were received from: state health services (including local hospital networks and regional hospitals), community health organisations, private healthcare organisations and settings, and Aboriginal community–controlled health services. Stakeholders informed the ANAO that the opening of the 2014 round was well publicised amongst potential applicants, and in the event, some 467 applications were received for the 150 available grants.

2.57 Health trialled the use of shared services for the 2014 funding round, to make the application process easier for applicants and to achieve administrative efficiencies. Applications could be completed and submitted electronically utilising a ‘smartform’, and were subsequently stored and processed through an existing Department of Social Services funding management system, FOFMS.¹³⁴ However, there were some shortcomings in the approach adopted by Health. Stakeholders advised the ANAO that while the electronic application process had improved overall administration compared to previous rounds, the trial suffered from insufficient testing prior to implementation. In particular, the trial encountered a number of significant technical problems relating to the receipt and processing of applications, and the anticipated benefits and efficiencies were not fully realised.

2.58 The assessment criteria and selection processes outlined in the department’s assessment plan for the 2014 round reflected Australian Government priorities, which were informed by published research of Health Workforce Australia. Further, the involvement of specialist medical colleges and state health services in the assessment of applications strengthened the assessment process. In particular, the state health services and colleges provided third-party advice to Health on the educational merit of applications, the potential impact of applications on health services, and the extent to which applications met program funding priorities.

2.59 ANAO testing of Health’s final assessment of applications for the 2014 funding round indicated that the department adopted an internal review and rescoring process after receiving input from state health services and the colleges. Around 250 of the 467 applications received in the 2014 STP funding round were reviewed as part of this process, which was not documented. ANAO analysis indicates that the rescoring directly affected funding outcomes for 13 applications, representing some 2.8 per cent of all (467) 2014 round applications. For 12 of these applications, the final score assigned by Health was above that which was calculated by the ANAO based on input from the specialist medical colleges and state health services, indicating these applications likely benefitted from the review and rescoring process to the extent that they were offered grants. As there were only 150 grants available through the 2014 round, the elevation of the 12 applications meant that some applicants that may have otherwise been offered a grant were not. One application was scored down by the department and as a consequence of this, was not offered a grant.

2.60 The department decided not to fund some highly-rated applications received as part of the 2014 funding round. Specifically, 13 applications were placed on a ‘reserve’ list¹³⁵, as the relevant setting had submitted applications for two or more training positions in the same specialty.¹³⁶ In these cases, Health funded only one place in order to obtain ‘a relatively even distribution of the new training positions against the population data’. In adopting this approach, Health effectively applied a selection criterion that was not documented in the application form or other explanatory material made available to applicants for the 2014 funding round.¹³⁷ Health’s approach was inconsistent with instructions provided to applicants for completing the application form, which indicated that the department would ‘collate’ assessment results received from state health services and the colleges. Further, the approach adopted was not consistent with advice provided to the Health Minister in March 2013 that the department’s selection of applicants to be funded under the STP was ‘essentially administrative’, as individual assessments were based on recommendations received from the state health services and colleges.

2.61 The lack of appropriate record-keeping and quality control in the internal review and rescoring process, and the use of a selection criterion that was not contained in the 2014 round application form or other explanatory material available to applicants, affected the transparency and to an extent the equity of the assessment process when viewed in terms of the application form and other explanatory material that informed applicants’ expectations about how grants would be selected.¹³⁸ However, the department’s approach in relation to this matter—of using a selection criterion that incorporated population distribution considerations—was not unreasonable in the context of the program’s intended outcomes.

Introduction

3.1 The Specialist Training Program (STP) funds¹³⁹ are disbursed under 13 individual funding agreements between Health and each college participating in the program.¹⁴⁰ These agreements give effect to a ‘college administration’ model as discussed in the following paragraphs.

The college administration model

3.2 Health’s administration of STP grants has undergone a considerable transition over the life of the program. Initially, some of the grants were administered by colleges¹⁴¹, while the majority were administered by Health, including through its state offices. However, from 2010 the department relinquished its direct role in dealing with the settings and the relevant colleges assumed responsibility in this regard.¹⁴² When seeking approval to move to the ‘college administration’ model in 2009, the department advised the Minister that:

… the current arrangement [through which the Department manages funding agreements primarily with individual practice settings] is fragmented and whilst it delivers a range of training posts, these are not integrated in a network arrangement or closely aligned with the skill needs of individual registrars. The new arrangement brings together the management of training posts function within organisations that are already responsible for the quality of training arrangements, responsible for the delivery of training curriculum and assessment of registrars for Fellowship. It is proposed that this arrangement provides a better platform for the delivery of networks of high quality training positions.

3.3 Under this model, the administration of STP grant funds is governed through separate funding agreements between Health and the respective colleges. For example, all grant funding for STP training positions providing training in surgery is provided to the Royal Australasian College of Surgeons (RACS). Under the relevant funding agreement, RACS is responsible for:

• disbursing funds to the individual settings¹⁴³ that operate STP training positions;
• oversighting the conduct of the STP training positions through contractual¹⁴⁴ and liaison arrangements with the individual settings;
• the development and implementation of strategic training and education projects to support the network of STP positions;
• promoting the integration of training provided through STP training positions with that provided by state health services; and
• providing periodic progress and financial reporting to Health.

3.4 The STP funding agreements each consist of a ‘heads of agreement’ based on Health’s standard–form grant contract, and a project schedule. The project schedule contains the detailed STP–specific obligations, mentioned above, lists the training positions administered by the college, the amounts of the various funding elements¹⁴⁵, the progress payment schedule, and details of the periodic and final reporting required from the colleges.¹⁴⁶

3.5 The project schedules accompanying funding agreements are amended as necessary through formal deeds of variation. Variations are typically made after every funding round (to reflect increases to the number of training positions being administered by the relevant college), when major changes have been made to reporting requirements and the like, or when additional specialist training grant funding is provided. The most recent variations were made between late 2013 and early 2014 to incorporate the results of the 2014 funding round.

3.6 Over 2010–2016¹⁴⁷, the colleges may receive up to $33.68 million¹⁴⁸ from the Australian Government for the administration of training positions and related tasks.

The application of the Financial Management and Accountability Act 1997 to the specialist training colleges

Section 12 of the FMA Act

3.7 The purpose of the Australian Government’s resource management framework is to promote the proper use of public resources.¹⁴⁹ Where the Commonwealth uses third parties to make payments on its behalf, it is necessary for Australian Government entities to assess the possible application of framework requirements.

3.8 Section 12 of the Financial Management and Accountability Act 1997 (FMA Act) established special requirements for Australian agencies that entered into agreements or arrangements for the receipt, custody or payment of public money by ‘outsiders’, including third parties involved in government administration such as the colleges. Agencies either had to ensure that outsiders complied with all the requirements of the Commonwealth financial framework, or alternatively could make special arrangements through a ‘Section 12 agreement’, which specified a set of requirements to be met by the outsider.¹⁵⁰

3.9 Health did not fully assess the potential implications of section 12 for the college administration model when it was first proposed in 2009, and the department did not seek legal advice on the issue at that time.¹⁵¹ The department sought legal advice from the Australian Government Solicitor during the course of the audit, after the matter was raised by the ANAO. Health’s advice indicated that while there were ‘arguments either way’ on the matter, ‘on balance’, college staff were not subject to the FMA Act in administering STP grant funds.

3.10 In July 2014, the FMA Act was replaced by the Public Governance, Performance and Accountability Act 2013 (PGPA Act). While the PGPA Act does not include a provision comparable to section 12 of the FMA Act, it places a responsibility on entities to assess risks before entering into financial arrangements.¹⁵² It is prudent for Australian Government entities to consider, at the design stage, the full implications of complex financial and administrative arrangements.¹⁵³

How Specialist Training Program administration funding is calculated

3.11 In 2010, Health decided that the administration funding provided to colleges would be set at a maximum of 10 per cent of the total salary support component of the training positions being administered by the colleges. This equated to an average upper limit of $10 000 per position, per year, although the actual amount payable to each college was to be negotiated on a case-by-case basis to ‘ensure a reasonable allocation’ for each college. The relevant record of that decision does not indicate as to why $10 000 was considered an appropriate upper limit.

3.12 With each successive funding round increasing the total number of training positions funded by the STP, individual colleges had a corresponding rise in the number of training positions they administered. As a consequence, the level of administrative funding provided by Health to the colleges also generally increased. However, for the majority of colleges, the annual funding measured on a ‘per position’ basis has generally declined over time, indicating efficiencies have been achieved by the relevant colleges over the course of the program. Figure 3.1 illustrates this trend through the last four funding rounds, with funding levels as at the 2014 round ranging from $5098 to $10 263 per position, per year. Two further colleges (not shown in Figure 3.1), with relatively small numbers of administered training positions, are funded at a somewhat higher level ($11 053 and $15 914), with the remaining college being funded at $32 800 per position, per year. This last college had only commenced participation in the STP from the 2013 funding round, and was also considered by Health to require additional funding for the administration of strategic support projects.

Figure 3.1: Trends in STP administration funding 2011–2014

Source: Department of Health data.

Notes: The Department of Health administered the Anaesthetic STP training positions in 2011. The Australian and New Zealand College of Anaesthetists administered all relevant STP training positions from 2012.

3.13 As part of negotiations for additional administrative funding flowing to colleges from the 2014 round, Health required colleges to provide estimates of forward budget requirements and reviewed 2013 college expenditure reports. While the level of detail in the expenditure reports and budget estimates varied significantly, email trails of the negotiation processes examined by the ANAO indicate that in a number of cases, the department negotiated amounts downwards from initial college budget estimates. In some cases no additional funding was provided where underspends of administrative funding provided under previous STP funding rounds were identified.

3.14 To maintain consistency of approach in its funding of colleges, there would be benefit in the department clearly documenting the basis for agreed administration funding at the conclusion of negotiation processes.

Salary support and rural loadings

3.15 Participating colleges receive two discrete elements of STP funding from Health which are then on-paid to the settings¹⁵⁴:

• salary support; and
• rural loading.

3.16 Salary support funding is provided through the colleges to settings to assist them with the cost of specialist trainee salaries. Each full-time equivalent (FTE) position receives $100 000 per year. Salary support is by far the largest funding element of the STP.¹⁵⁵ Collectively, the funding agreements provide that up to $387.55 million in salary support is payable to settings via the colleges between 2010 and 2016.¹⁵⁶

3.17 The salary support funding provided to each college is based on a simple calculation of the number of FTE training positions administered by the college. In relation to the additional salary support funding flowing from the 2014 round, ANAO analysis indicated that the amounts in the funding agreement variations were, with one minor exception¹⁵⁷, calculated in accordance with Health’s funding methodology.¹⁵⁸

3.18 Before making a salary support payment, colleges require STP settings to provide them with invoices or other documentation showing that the position was filled by a specialist trainee. Payments are generally made twice a year in arrears. The six-monthly college reporting provided to Health in July 2014 indicates that a small number of settings were late in providing relevant documentation, delaying payments and contributing to underspends by colleges against their projected annual expenditure.

3.19 The settings consulted by the ANAO indicated that there was an increasing gap between the full cost of employing specialist trainees (particularly more senior specialist trainees) and the amount of the STP grant.¹⁵⁹ Grant funding amounts for salary support and rural loading (at $100 000 and $20 000 respectively) were set by the then Australian Government in 2009 through the program guidelines and have not been revised. A review of the STP ‘funding model’ for 2011–2014 (including the individual components such as salary support and rural loading) was undertaken by Health in 2010. However, the review was silent on whether the funding levels of these components remained appropriate going forward.¹⁶⁰

3.20 Private sector settings advised the ANAO that where the specialist trainee was seconded to a private sector STP position from a public sector hospital, the specialist trainee generally remained on the public sector payroll. In such cases, the public hospital would require the private sector setting to reimburse the specialist trainee’s salary (and in most instances, on-costs such as superannuation) so that the public hospital could fund the ‘backfilling’ of the specialist trainee’s position.¹⁶¹ In effect, the STP grant funds were used by the private sector setting towards these reimbursement costs. A simplified illustration¹⁶² of this process is shown in Figure 3.2.

Figure 3.2: Simplified STP funding flow

Source: STP Operational Framework and ANAO discussions with stakeholders.

3.21 If the position is partly or wholly in a regional or rural area¹⁶³, settings are eligible to receive a rural loading allowance of up to $20 000 per FTE per year. Collectively, the funding agreements provide that up to $31.83 million¹⁶⁴ is payable for rural loading to eligible settings via the colleges for the period 2010–2016.¹⁶⁵

3.22 In relation to the rural loading funding flowing from the 2014 round, ANAO analysis indicated some relatively small variations between the funding amounts contained in the deeds of variation and the information contained in relevant appendices (which form the basis for calculating these amounts).

3.23 Discussions with stakeholders also indicated that colleges adopted different approaches in the way that rural loading is paid to settings for STP training positions. Some colleges indicated they will not pay the rural loading allowance unless settings provide appropriate receipts or other documentation, while other colleges allocate the rural loading allowance automatically. Where STP training involves rotations between metropolitan and rural or regional settings, one college commented that significant effort was required to confirm whether the rotations¹⁶⁶ had in fact taken place as originally planned, and checks were not done on a regular basis.

Execution of the 2014 round variations

3.24 The Commonwealth financial framework in operation until 1 July 2014 required that all proposals to commit public money receive formal approval under FMA Regulation 9 before Australian Government agencies entered into contracts or funding agreements.¹⁶⁷ Funding agreements were also required to be consistent with the Regulation 9 approval.¹⁶⁸ Financial approvals for the 2014 STP funding round were for sums ‘up to’ certain specified amounts. ANAO analysis indicates that the amounts contained in Health’s deeds of variation with the colleges did not exceed the maximum limit approved, and the number of new training positions to be funded was consistent with the relevant approvals.

Public grants reporting

3.25 Effective disclosure and reporting arrangements for grants administration is essential for reasons of transparency and public accountability. Reliable and timely information on grants awarded is a precondition for public and parliamentary confidence in the quality and integrity of grants administration.¹⁶⁹

3.26 Under the 2013 Commonwealth Grant Guidelines, which applied at the time the 2014 round funding agreements were entered into, an agency was required to publish, on its website, information on individual grants no later than 14 working days after the grant agreement took effect.¹⁷⁰ Where existing agreements were varied—as with the STP—this timeframe did not apply, but it was considered ‘better practice for agency staff to amend the website to reflect the variation as soon as practical’.¹⁷¹

3.27 The ANAO examined the department’s grants and tenders homepage as at 31 July 2014, and found that only a minority of the variations had been reported on Health’s grants reporting website. The department advised the ANAO that the exact publication dates had not been recorded for the variations prior to January 2014—only the financial year the data was published. Health advised the ANAO that variations data cannot be extracted from FOFMS, and the department is therefore reliant on manual reporting of variations. Health further advised the ANAO that a system for manual data capture was put in place in January 2014 by its Grant Services Division pending the introduction of an automated system, implemented from 9 January 2015. The new system is intended to include variations in grant reporting, to ensure that all new and varied grants are published within the reporting timeframes required by the Commonwealth Grants Rules and Guidelines. ¹⁷²

Administration of the reporting and payment regime

The reporting regime

3.28 The STP and the Tasmanian project funds are administered by the same colleges, through a single funding agreement between each of the relevant colleges and Health. The funding agreements outline the responsibilities of all parties, including: payment of funding; use of funding; audit and monitoring; reporting requirements; and termination of the grant. Under the funding agreements, the specialist medical colleges are required to provide separate sixmonthly progress reports for both STP and the Tasmanian project by the end of January and July each year.

3.29 The funding agreement terms for payment of STP and the Tasmanian project funding differ. Under the funding agreement the terms for the payment of STP funding to the colleges are dependent on Health’s acceptance of the report provided. Whereas the terms for the Tasmanian project funding payments are not explicitly linked to satisfactory reporting or any other defined deliverable from the specialist medical colleges—rather the payments are made on a set date in June and December each year.

3.30 Despite these differences in payment conditions, the funding agreement terms provide Health with the discretionary power to:

• defer, reduce or not make any payment under the funding agreement if the specialist medical college has either overspent or underspent funding; and
• request repayments of funds in situations where the department has formed a reasonable opinion that funds have been spent or committed other than in accordance with the terms of the funding agreement.^173,174

3.31 The reporting requirements for STP training positions have evolved since the first agreements were executed in 2010. Increased detail in financial reporting¹⁷⁵ was required from late 2012, and specific key performance indicator (KPI) reporting was developed in late 2013.

3.32 More detailed financial reporting was introduced following an internal Health review of its administration of one STP funding agreement. The review highlighted several areas of concern including:

• a significant level of unspent STP funds being held by the relevant college;
• STP funds being mixed with other college funds in banking and investment arrangements; and
• a substantial level of vacancies in STP training positions managed by the college.

3.33 Reporting against the Tasmanian project is relatively light touch, requiring colleges to advise on:

• current expenditure and funds ‘committed’;
• whether planned specialist trainees, supervisor and coordinator positions have been filled;
• the extent to which outcomes have been achieved;
• key issues and barriers to successful implementation of the project (including a ‘handling strategy’ in respect of these);
• commentary on uncommitted funds; and
• anticipated key actions for the next six months.

3.34 Annual college STP audited financial statements must also cover Tasmanian project revenue and expenditure.

3.35 The timing of the periodic reports allows colleges to collect the required information from the settings in order to provide the necessary information to Health on the preceding period—reports submitted to Health in July will cover the preceding January-June period. Colleges generally report that they are able to obtain most of the information required, although some reports examined by the ANAO stated that up to five percent of settings had not provided relevant information by the reporting date. ANAO analysis of the January and July 2014 college reports indicated that around 90 per cent of reports were received by Health within a day or two of the due date, although on a number of occasions the department has subsequently sought additional information.

3.36 The general view from the settings interviewed by the ANAO was that the reporting requirements were not excessive, and that relationships with the colleges on STP matters were good. The only negative issue raised by settings on a fairly regular basis was that, if a setting was hosting multiple STP training positions across different specialties, they had to provide different forms of reports to the relevant colleges, as each college had somewhat different reporting requirements.

3.37 On balance, the department’s reporting requirements have had regard to the proportionality principle of the Australian Government’s grants framework, which provides that grants processes should be commensurate with the scale, nature, complexity and risk involved in the granting activity.¹⁷⁶

Assessment of college reporting by the Department

3.38 A well-drafted funding agreement alone is not sufficient to ensure the objectives of the grant are met. Agency staff should ensure that funding agreements are supported by ongoing communication, active grants management and performance monitoring requirements, which are proportional to the risks involved.¹⁷⁷

3.39 Under the STP, assessment of college reports is undertaken by a departmental grants officer, who makes a written recommendation to a decision maker as to whether the report should be accepted. As noted above, the grants officer sometimes seeks further information from the college before making a recommendation. The assessments are based on a standardised template and have two separate components: assessment against the KPIs, and assessment of the financial information. Typically the assessments are completed and then considered by the decision maker within one to two weeks of all the required information being received by Health. Acceptance of a report by Health triggers the execution of a progress payment in FOFMS, although as noted in paragraph 3.43, Health may decide to withhold part (or the whole) of the scheduled payment due to underspends by the college.

ANAO analysis of reporting and assessments

3.40 A key element of the department’s STP monitoring approach is college reporting against outcome-oriented KPIs that were developed in consultation with the colleges during 2013.¹⁷⁸ While these were required to be reported on from January 2014, it was evident from the ANAO’s analysis of the July 2014 STP reports that for some KPIs, a minority of colleges were unable to collect the relevant information to allow reporting. The colleges raised this issue with Health at the February 2014 annual intercollege STP meeting. While at the time the department agreed to review the issue, there has been no further work by Health on this.¹⁷⁹

3.41 A fundamental KPI against which the colleges must report is the number of STP training positions filled and the corresponding vacancy rate. The department advised the ANAO that the colleges were required to report against this KPI on a FTE basis, but the actual reports did not always do so. There were also occasions where the spreadsheet showing the status of each position was inconsistent with the level of vacancies contained in the KPI reporting template, although the inconsistencies were relatively minor in terms of the number of training positions involved. In some cases, a small number of settings had not reported to colleges, and these reports noted that the actual overall vacancy rate might vary somewhat from the rate in the July 2014 report.

3.42 ANAO analysis indicates that around 833 of the 900 FTE training positions were filled as at 30 June 2014, equating to 93 per cent of training positions filled on an FTE basis. Health’s assessment of the July reports rated a number of colleges as ‘unsatisfactory’ on this issue, but recommended acceptance of the report on the basis there was evidence that the relevant college had committed to a specific strategy to fill significant numbers of additional STP training positions in the second half of 2014 and/or 2015.

3.43 As at 31 December 2013, collective surpluses of STP funds held by the colleges were $36.28 million.¹⁸⁰ By 30 June 2014, this had risen to $56.31 million, equivalent to 16.4 per cent of the total STP funds that had been paid to the colleges by Health under the relevant agreements to 30 June 2014.¹⁸¹ While the surpluses can be partly attributed to timing issues¹⁸², Health has responded by withholding a proportion of the scheduled progress payments. Some $23.89 million that was due to be paid following receipt of the July 2014 reports was withheld.¹⁸³

3.44 Expenditure of 2013–14 funds for the Tasmanian project was also relatively low.¹⁸⁴ Colleges received $10.81 million in funding in 2013–14 with around half received in June 2014, but progress reports indicate that only $1.59 million had been spent. However, the level of ‘committed’ funding through contracts with the relevant settings was considerably higher.

3.45 The presentation of financial information by colleges, particularly income and expenditure, varied significantly from college to college and sometimes made it difficult to assess how funds had been spent, particularly on administration. In one case, Health concluded in its assessment that the relevant college’s reported cash surplus was some $600 000 below that indicated in the college’s bank statements included in its financial report.¹⁸⁵

Progress payments to colleges

3.46 Progress payments are managed through FOFMS, in part to allow the department to automate payment processing. The department advised that some other payments are date based and automatically released to reduce the administrative burden on Health and the colleges.

3.47 The ANAO examined FOFMS payment records over the last two years. Only one payment was not explicitly approved by Health, and this was a result of the department deciding to withhold a portion of an earlier scheduled payment to a college (some $453 000) until the next progress report was submitted. The release of the withheld funds was to be considered as part of the next progress report assessment by Health. Through an oversight, there was no reference to the withheld amount (and hence no recommendation as to whether it should be released) in the subsequent assessment report approved by the Health, however the amount was released in FOFMS regardless. The department’s experience in this case is a reminder of the need to apply relevant financial controls.

3.48 As noted in paragraph 3.43 Health decided to withhold some $23.89 million in payments scheduled to be made on acceptance of the July 2014 college progress reports. The ANAO’s examination of FOFMS records indicates that the subsequent payments were reduced consistent with the amounts approved by the decision maker.

Conclusion

3.49 A feature of the STP is that while Health has overall responsibility for its administration, the department does not have a direct contractual relationship with individual settings, which are the grant recipients. Rather, the administration of STP grant funds is managed through separate agreements between Health and the respective colleges. Under this ‘college administration’ model, developed by the department in 2010, all grant funding for STP training positions within a particular specialty is provided by Health to the relevant college. The colleges may receive up to $33.68 million in 2010–2016 for the administration of training positions and related tasks. Most colleges receive between $5 098 and $10 263 per position for administrative purposes, although colleges with relatively small numbers of training positions to administer tend to receive somewhat higher funding.

3.50 The colleges and the settings interviewed by the ANAO indicated that the ‘college administration’ model generally worked well. However, some potential risks were not fully assessed by Health when developing the model, including the risk that the Australian Government’s financial framework requirements might apply to colleges if they handled public money. It is prudent for Australian Government entities to consider, at the design stage, the full implications of complex financial administrative arrangements—such as those involving third-party administration of government programs—so as to avoid potential compliance and reputational risks.

3.51 The department relies on reports received from the colleges to inform its oversight of the college administration model. Since 2012 Health has sought additional management and performance information from the colleges, particularly relating to financial issues. On balance, the department’s reporting requirements have had regard to the proportionality principle of the Australian Government’s grants framework, which provides that grants processes should be commensurate with the scale, nature, complexity and risk involved in the granting activity.

3.52 However, the presentation of financial information by colleges–particularly income and expenditure–has varied significantly, sometimes making it difficult for Health to assess how funds have been spent. Where variations in reporting have occurred, Health has undertaken follow-up communication with colleges to determine their actual financial position.

3.53 As at 31 December 2013, total surpluses of STP funds held by the colleges were $36.28 million. By 30 June 2014, total surpluses had risen to $56.31 million. The surpluses can be partly attributed to timing issues, including delays in the submission of invoices by training settings to the colleges. During 2014, Health has responded more actively where surpluses were identified by withholding a proportion of scheduled progress payments. In consequence, $23.89 million that was due to be paid following receipt of the July 2014 college reports was withheld.

Introduction

4.1 A key principle of grants administration is the achievement of government policy outcomes.¹⁸⁶ An effective performance framework facilitates assessment of the extent to which grants of financial assistance have contributed towards achieving the stated program outcomes and objectives.¹⁸⁷

4.2 The Parliament and the public’s consideration of a program’s performance, in relation to its impact and cost effectiveness, rely heavily on reliable and appropriate performance information.¹⁸⁸ Within the context of the Australian Government’s Outcomes and Programs Framework, key performance indicators (KPIs) are established to provide information on the effectiveness of programs in achieving objectives in support of outcomes.

Development of key performance indicators

4.3 When the then Australian Government was considering the proposed expansion of the STP in 2010, Health advised Ministers that the key performance measure for the program was the ‘take up rate and completion of vocational training placements’ which would be ‘assessed annually’. This simple performance measure was reflected in the first of several KPIs included in the program guidelines for the first funding round of the expanded program:

• training positions identified under each funding agreement being filled by an eligible specialist trainee, for the duration of the term specified.

4.4 Other KPIs adopted for the first funding round were:

• appropriate flow of funds identified as a contribution to salary, to the employer of the specialist trainee;
• improved quality of specialist training with specialist trainees gaining appropriate skills not otherwise available through traditional settings;
• the number of specialist international medical graduates supported and attaining College Fellowship;
• demonstration that the specialist training initiatives developed through resources supplied as part of the Agreement complement initiatives currently provided within the state health services; and
• processes established which enable effective and efficient administration of specialist training positions.

4.5 Apart from noting that Health would require periodic information from STP training positions, the program guidelines were silent on how the KPI data would be collected and assessed. This was addressed through a STP funding agreement variation executed in late 2012, which contained a commitment from colleges to work with Health to develop new KPIs that would form part of the college’s periodic reporting obligations. This general approach was consistent with a finding of the 2013 Mason Review of health workforce programs that more ‘sophisticated’ KPIs should be developed for the STP and included in funding agreements, with a ‘broader focus on sustainability, health workforce distribution outcomes and impact on trainee career decisions’.¹⁸⁹

4.6 The new KPIs, information on how they were linked with existing STP outcomes, and reporting requirements, were incorporated into the funding variations flowing from the 2014 funding round. The KPIs are shown in Table 4.1. Reporting against the KPIs started from January 2014, and form part of the colleges’ six-monthly reporting obligations.¹⁹⁰

Table 4.1: Current STP key performance indicators

Source: Department of Health.

Note: KPIs measuring the timely provision of complete periodic reports, and participation in meetings or teleconferences involving the department, have been excluded from the above table as they are not directly related to the achievement of STP outcomes.¹⁹¹

4.7 The new KPIs are clearly linked to STP outcomes. The majority are also quantifiable, allowing trends to be measured over time, although Health has not set any targets for the desired level of achievement. In most cases the KPIs constitute proxy measures, which are indirect measures of effectiveness. Health confirmed that it did not undertake selective verification activities regarding the accuracy of KPI reporting by the colleges¹⁹², nor is this identified as a risk in the department’s 2014 STP risk management plan.

Program performance and the future of the Specialist Training Program

Program performance

4.8 Colleges report against the KPIs through a template provided by Health. Based on the July 2014 reports received from colleges, the majority of KPIs were clearly reported against. For other KPIs, a number of colleges qualified their figures in some way or stated that they were unable to report because they did not collect relevant information.¹⁹³ The ANAO aggregated the July 2014 KPI reporting from the eleven colleges, with the results shown in Table 4.2.

Table 4.2: Aggregated college KPI results from July 2014 reporting

Source: ANAO analysis of July 2014 STP progress reports from colleges.

Note: The above percentages have been rounded to the nearest whole number.

4.9 A key program outcome relates to exposing specialist trainees to an increased range of work environments and patient needs, to improve the quality of their training. Overall, based on July 2014 college reports, ANAO analysis indicated that 89 per cent¹⁹⁴ of STP training took place in expanded settings—that is, outside traditional metropolitan teaching hospitals. This aspect of the STP was consistently rated as one of the program’s strengths by the settings in discussions with the ANAO, a point also noted in specialist trainee surveys conducted by the colleges.

4.10 One setting advised the ANAO that the focus of STP on expanded settings—which are generally smaller in size than metropolitan teaching hospitals—had been influential in encouraging the relevant college to adopt a more flexible approach in accrediting training positions in non-traditional settings for specialist training purposes. A similar point was made in the Mason Review findings, which cited examples of ‘hub and spoke’ STP training models. Under this model, STP funding has facilitated an increased range of training opportunities by tapping into training capacity in smaller training settings that would not otherwise be able to support a specialist training position on its own. STP–funded ‘strategic support’ projects have also contributed in this regard.¹⁹⁵ Collectively, these developments have contributed to the achievement of another key program outcome—increased capacity within the healthcare sector to train specialists.

4.11 In addition to exposing specialist trainees to a more diverse range of healthcare working environments, STP training positions in regional and rural areas are intended to contribute to a better distribution of specialist services, a program outcome of direct benefit to the community.¹⁹⁶ Overall, 41 per cent of STP training positions were in rural or regional settings, with the figures for individual colleges ranging from 17 per cent to 75 per cent. The settings interviewed by the ANAO noted that it could sometimes be challenging to attract specialist trainees to rural or regional locations, and that STP funding (including the availability of the $20 000 rural loading) was of substantial assistance in this respect.

4.12 ANAO analysis of July 2014 college reports indicated that the overall proportion of STP training positions in expanded settings that are integrated with training provided by the relevant state health service providers is high at 82 per cent.¹⁹⁷ While college reporting generally does not provide any substantial information on the nature of the ‘integration’, the figures suggest that the STP may be making a contribution towards improved coordination between the Commonwealth, states and territories in terms of specialist training pathways. This issue was highlighted as requiring attention by the Mason Review findings in 2013 and Health Workforce Australia in its 2012 Health Workforce 2025198 report. The broader coordination issue is now being examined as part of the work of the National Medical Training Advisory Network. ¹⁹⁹

Evaluation and the future of the Specialist Training Program

4.13 In March 2012, the Health Minister’s policy approval was sought to extend STP funding to 2015–16, and as part of that request the department advised that:

it is intended that specialist training activities…will be evaluated in 2013 followed by annual reviews of individual training posts that will commence in the 2014 academic year. This evaluation will examine the effectiveness of the STP…in achieving the Government target of delivering an additional 680 additional specialists into the health sector by 2020.²⁰⁰

4.14 A high-level evaluation of the STP was undertaken in 2012–13 as part of the wider Mason Review of health workforce programs. The Mason Review recommended that²⁰¹:

While STP has been a well received and apparently successful program, it is important that a full evaluation of the program should be carried out to verify that settings such as the mix of positions are optimal, and to inform the future development of the scheme.

In addition, existing STP posts should be reviewed by colleges (in discussion with the Department and other program stakeholders) to ensure they are meeting the objectives of the program. This may provide the opportunity to redirect funds to new training posts that may better meet emerging workforce priorities.

4.15 In publicly releasing the findings of the Mason Review in May 2013, the then government announced that it would take steps to implement a number of the Review’s recommendations. However, none of these directly related to the STP.²⁰² The issue of a more detailed evaluation of the STP was not subsequently raised in departmental advice to the Minister, although the possibility of such an evaluation was discussed with colleges in early 2014 at the STP inter-college forum.²⁰³ Health advised the ANAO that in view of the commencement of the ANAO’s performance audit the department did not progress a planned evaluation, particularly as it may have duplicated the collection of information and had a significant impact on the colleges.

Conclusion

4.16 The STP has had KPIs in place since the consolidation of the program in 2009. However, explicit outcome-linked KPIs were only developed in 2013, and the colleges having reported against these KPIs since January 2014.

4.17 College reporting against the program’s KPIs indicates that the STP has been successful in utilising non-traditional settings to expand the number of training places for specialist trainees, with 89 per cent of STP-funded positions being located in non-traditional settings. In discussions with the ANAO, stakeholders have also suggested that the expanded range of work environments has contributed to the overall quality of training.

4.18 ANAO analysis of college reporting indicates that on a full-time equivalent basis, around 833 positions were filled as at 30 June 2014, representing 93 per cent of the target of 900 positions. Overall, the additional specialist trainee positions funded by the STP have boosted the availability of specialist services, including in regional and rural areas. However, it remains unclear to what extent the STP has, or will, contribute to an improved geographical distribution of specialist services to meet community need, over the longer term.

4.19 A relatively high-level evaluation of the STP was undertaken in 2012–13 as part of the wider Mason Review²⁰⁴ of health workforce programs. While the review concluded that the STP has been a well received and apparently successful program, it recommended a full evaluation to verify that policy settings, such as the mix of positions, are optimal and to inform the scheme’s future development. The Review also recommended that existing STP posts be reviewed by colleges (in discussion with the department and other program stakeholders) to: ensure they were meeting the objectives of the program; and with a view to redirecting funds to new training posts that may better meet emerging workforce priorities. In publicly releasing the findings of the Mason Review in May 2013, the then government announced that it would take steps to implement a number of the Review’s recommendations. However, none of these directly related to the STP.²⁰⁵

4.20 As discussed, Health advised the ANAO that it did not progress a planned evaluation in view of the commencement of this performance audit.

Appendices

Please refer to the attached PDF for the Appendices:

• Appendix 1: Entity Response
• Appendix 2: Electoral Status of Selected 2014 round Grant Applications

Abbreviations