This audit

The objective of this audit was to form an opinion on the Australian Research Council's (ARC's) management of research grants. To achieve this, ANAO centred the audit around the following aspects of ARC's grants administration:

• governance and structure, particularly the roles and responsibilities of those parties involved in administering ARC's grants (Chapter 2);
• the processes for assessing and selecting ARC grants (Chapter 3);
• post-award management of grants under the Funding Agreements (Agreements) between ARC and those universities that receive and administer the ARC grants to researchers (Chapter 4); and
• ARC's monitoring of its grant programs for management, performance improvement and reporting (Chapter 5).

In its assessment, ANAO considered ARC's compliance with relevant sections of the Australian Research Council Act 2001 (ARC Act) and the Financial Management and Accountability Act 1997 (FMA Act). The assessment also took account of the ANAO's Better Practice Guides, particularly the Better Practice Guide—Administration of Grants.
The audit focused mainly on ARC's administration of Discovery Projects, the largest scheme in ARC's National Competitive Grants Program (NCGP).

About the Australian Research Council

ARC is a small agency of about 65 staff within the Education, Science and Training portfolio. It was established as an independent body under the ARC Act, and is also a statutory agency under the Public Service Act 1999, and a prescribed agency under the FMA Act.

The role of the ARC Board (Board) is to decide ARC's goals, priorities, policies and strategies, and ensure that ARC's functions are performed properly, efficiently and effectively. It also establishes committees to assist in ARC's work, provides advice to the Minister for Education, Science and Training (Minister), and may initiate inquiries.

Under its Act, ARC has two main functions; to advise the Australian Government on research matters, and to administer grants through the NCGP. At any point in time, the NCGP provides around 5 000 new and ongoing research grants in about 40 universities and 15 other eligible research institutions. Grant recipients are largely university-based researchers, with the Group of Eight universities receiving the largest number of grants and around 70 per cent of NCGP funding.

ARC's 2005–06 budget of $571 million accounts for about 10 per cent of the Australian Government's $5.3 billion annual science and innovation budget. Funding for the NCGP in 2004–05 was $481.4 million, with 54 per cent ($260 million) of this allocated to Discovery, one of two main elements of the NCGP. Discovery Projects grants range from $20 000 to $500 000 per annum, and from one to five years duration.

ARC holds one Discovery Projects grant round each year, for which it receives between 3 500 and 4 000 applications. ARC has a complex decision-making process. The round begins when ARC releases the Discovery Projects funding rules (usually in December). To assess and select grants, ARC uses a system of peer review, in which Australian and international assessors in relevant fields of research score and rank each application based on ARC's grant selection criteria. For final assessment and ranking, ARC sends each application to several assessors and then to one of six selection advisory panels constituted from ARC's College of Experts.

Selection advisory panels make recommendations for successful and unsuccessful applications to ARC's Board. Once endorsed, the Board forwards its recommendations to the Minister for approval. Around October to November the Minister announces the successful grants; the entire grants cycle taking up to eleven months to complete.

ARC's grant processes are underpinned by the ARC Act, funding rules and many procedures. Of crucial importance to the grants process is ARC's implementation and monitoring of its guidelines and processes for the assessment and selection of grants, as well as guidelines for managing conflict of interest.

In July 2005, the then Minister announced that he would change ARC's governance in light of the Government-commissioned review of the governance of statutory authorities and office holders (Uhrig Review), retiring the Board by early 2006. The Australian Government introduced legislation into Parliament to amend the ARC Act on 30 March 2006.

Key findings

Structure and governance (Chapter 2)

ARC's structure was consistent with the ARC Act, comprising a Board, a Chief Executive Officer (CEO), committees of the Board and staff appointed under the Public Service Act 1999. The Board was involved in key functions defined by the ARC Act, including strategic planning and recommending grants for Ministerial approval.

Central to ARC's structure are the Executive Directors, who have key responsibilities in overseeing the peer review and grant selection processes in ARC's six discipline areas. ANAO found that while ARC appointed the EDs for three-year terms, it needed to be careful to ensure ARC's business continuity given the likely loss of knowledge with the end of their terms of appointment. While ARC staggered the appointment of the EDs, the risks associated with their turn-over, such as loss of corporate knowledge, were not identified in ARC's risk management plan. Better documentation of administrative procedures would also assist ARC to maintain business continuity and a sound knowledge of grant processes among ARC staff.

ANAO found that while ARC is part of the Education, Science and Training Portfolio, there was little formal documentation describing the arrangements between the Department of Education, Science and Training (DEST) and ARC for key administrative processes such as clearing of key documents or exchanging information and data. ANAO noted that the then Minister intended to ‘retire' ARC's Board in 2006. The Secretary of the Department of Education, Science and Training (Secretary) would therefore no longer provide links between the two agencies through her Board membership. ANAO considers there would be benefit in ARC and DEST agreeing on a Memorandum of Understanding about their consultative and reporting activities, to ensure regular and ongoing information exchange on key policy and administrative matters. This step would be consistent with the Government's intent of a clearer role for departmental secretaries in such relationships following the Uhrig Review.

ARC operates in an environment where any perceived breaches of conflict of interest could easily undermine its reputation. ANAO found that ARC had developed conflict of interest guidelines and was making a concerted effort to implement these across its various committees. Notwithstanding, the audit identified areas that could be strengthened. To demonstrate sound practice and improve the visibility of ARC's grant selection process, ARC is advised to revise its guidelines, review conflict of interest declarations annually, and establish a register of interests for its College of Experts and other committees.

Assessment and selection (Chapter 3)

While ARC had internal timetables for each of its individual funding schemes, ARC's draft business plan did not consolidate timelines, clearly convey ARC's plan for the scheduling and delivery of the grant schemes on an annual basis, or clearly illustrate the links between the key administrative tasks in delivering the funding schemes, ARC's resources, or operational targets.
ANAO also found that ARC did not produce an external annual calendar. As preparing an ARC grant application can be lengthy and complex, ARC should provide stakeholders with timely information on the scheduling of its key activities and timelines. ANAO considers that publication of an annual calendar would strengthen ARC's business planning, and enhance the visibility of the various NCGP schemes, providing ARC staff and stakeholders with timely information on scheduling of key activities. This would assist stakeholders in planning their activities and resources to meet ARC timelines and requirements.

ANAO found that ARC had a structured process for developing and revising funding rules, and overall, the clarity and consistency of funding rules had improved over the 2004–06 period. While ARC was found to be implementing its funding rules, the timeliness in releasing the funding rules was inconsistent. ANAO surveys also found that ARC's clients rated this aspect of ARC's performance as poor. ANAO suggests that ARC improve its performance in this area by implementing consistent timeframes for release of funding rules and by setting targets to monitor its performance.

The process for assessment and selection of grants is also complex and lengthy. ARC produced a range of guidelines and procedures to assist staff and committees in carrying out their various roles, although these did not cover all aspects of the grant selection process. There were some processes where documentation should more clearly describe the roles and responsibilities of those involved in decisions, and the basis for making decisions, for example, the determination of eligibility, and requests from researchers for particular assessors to be excluded from assessing their applications (‘requests not to assess').

Selection criteria and their overall weightings were available to applicants and grant assessors. However, application forms did not clearly indicate whether applicants had to address all or some of the sub-points under each criteria, whether there was any weighting applied to these sub-points, and how applications were assessed compared with other applications if the applicant had not addressed all points. ARC should revise its instructions to applicants and assessors, to make sure that the intent and relevance of the sub-points to the selection process is fully explained.

ARC's Administration Handbook states that ‘workbooks are the official record of selection meetings'. While each project had a workbook record, ANAO found that in most instances these did not provide a complete record of the selection panel's key actions and decisions. For example, while reasons for ineligibility were recorded in the workbook, reasons for excluded (not funded) applications or key actions taken at the meeting were not routinely recorded in the workbooks. A better approach would be for ARC to document a more comprehensive summary of key actions and decisions from selection meetings, as a hard-copy record on each researcher's file.

ANAO also found that while the majority of approved grants received less funding than requested, ARC's decision making in this area was not transparent. For example, ARC maintained limited documentation of each panel's discussions or reasons for reducing funding. It also provided little or no information to grantees on this part of the grant process, so grant recipients were unaware of which areas of their projects were considered strengths or weaknesses by grant selection advisory panels. Notifying successful applicants of the reasons for significant budget reductions would allow greater understanding of the funding process, help applicants to develop future applications, and assist grantees in making informed decisions about the best use of ARC grant money.

Post-award grant management (Chapter 4)

ARC has a decentralised model of grants management whereby each administering organisation (mainly universities) coordinates and administers grants on behalf of ARC. ARC had established Funding Agreements (Agreements) with the universities to support the administration of grants. These reflected the legislative requirements of the ARC Act and FMA Act and followed ANAO's Better Practice Guide. Universities generally found the conditions on the award of grants clear, although they identified some areas that caused them difficulty in administering the grants.

Significant weaknesses were found in ARC's post-award management of grants, with few systematic processes in place to monitor progress and final reporting of grants under the conditions of the Agreements. In particular, ARC awarded full funding to few successful Discovery Projects applications. In 2003 and 2004, less than five per cent of Discovery Projects grants received 100 per cent of requested funding, about 30 per cent received 80 per cent or more, and 20 per cent received less than 50 per cent of their requested funding. Despite this, neither the Agreements nor ARC procedures clearly defined ARC's expectations for partially funded grants or the process for researchers to submit revised project plans.

ANAO also found that ARC frequently approved project variations through annual progress reports. This meant that ARC was approving variations retrospectively, often a year or more into the project, or when finished in the case of one-year projects. At this stage substantial funds could already have been spent. ARC had not analysed whether partial funding of grants affected the success of projects or the quality of research results.

ANAO sampling showed that ARC was making timely and accurate monthly grant payments to universities. However, ANAO also found that reporting requirements under the Agreements were often not fully met, with a substantial number of progress reports and final reports on projects either not submitted or submitted late. This diminished ARC's ability to assure the Government that all grants met their objectives, that funds were used as intended, and that ARC goals were being fully met.

ANAO noted that ARC was aware of weaknesses in this area of administration, recognising that its existing systems could not effectively handle key aspects of post-award management. ANAO considers that ARC should give further priority to improving post-award management, while maintaining its focus on selecting projects and paying grants. ARC advised that it was undertaking an Information Technology (IT) system redevelopment to address these weaknesses.

ARC had recently carried out audits in two universities, and advised ANAO that it would commence a rolling institutional review program in the second quarter of 2006. ANAO considers that implementing such a program would help ARC and universities to improve the compliance and accountability of projects under the Agreements.

Monitoring and performance (Chapter 5)

ARC complied with the ARC Act's requirement of preparing strategic plans and reporting progress against these in its annual report. However, ANAO detected areas where ARC could improve its performance management framework and be more consistent with the Department of Finance and Administration's (Finance) outcomes and outputs framework.

In particular, ANAO found there was no obvious link between ARC's effectiveness indicators and its 10 Key Performance Indicators, and reporting against the effectiveness indicators was minimal. The separation of indicators against administered and departmental items or output and outcomes in the Portfolio Budget Statements was also not well defined. ANAO suggests ARC use more targets and a wider range of quality and quantity measures to describe its performance, and its performance reporting to Finance's requirements.

ARC draws on several sources of data to generate information for internal performance management and external reporting. However, there were a significant number of final reports submitted late or not at all. This limited ARC's capacity to generate accurate aggregate data to support its performance management and monitoring, and inform policy.

While ARC periodically commissioned reviews and studies to measure performance, a more planned approach to reviews, to provide information at regular intervals, would assist ARC in determining the long-term results of ARC funded grants. During the audit, ARC advised that it was now staffing a Research Evaluation Team, which will assist in addressing these issues.

ANAO found that ARC had incorporated the national research priorities (NRPs) into the NCGP, as directed by the Minister under the Australian Government's Backing Australia's Ability (BAA) initiatives. Most researchers responding to ANAO surveys were aware of the NRPs. However, there was uncertainty about how these influenced the grant selection process, or whether it was to the applicant's advantage to designate an NRP in their grant application. ANAO considered that greater clarification was required to ensure that researchers were providing all necessary information.

ANAO also found that ARC was not capturing all relevant information from grant applicants, due to limitations of the application form. One way for the ARC to improve the validity and reliability of its data on NRPs is by implementing a more rigorous approach to data collection and monitoring. ARC advised ANAO that its IT system redevelopment project currently underway is designed to address this.

ANAO found that, for many years, ARC had not commissioned research to determine client needs. ANAO surveys showed that although universities were generally satisfied with ARC's level of service, there were specific services where ARC could better meet client needs, for example by: producing a calendar showing major ARC events and timelines to help researchers and universities in planning for ARC grant rounds; enhancing information to unsuccessful applicants; by providing clearer information to grantees who receive partially funded grants; and by informing clients in a clearer and more comprehensive way of its monitoring of its performance.

Overall audit conclusions

The audit identified that ARC was meeting the requirements of the ARC Act in administering grants for basic and applied research. However, shortcomings in ARC's administrative processes meant that ARC was not in a position to determine and inform the Government about whether all grants met their objectives, that funds were used as intended, and that ARC goals were being fully met. While ARC had a strong focus on selecting the best applications, it had few systematic processes to enable effective or timely post-award management of grants.

ARC had established governance and organisational structures that supported the National Competitive Grants Program (NCGP), ARC's vehicle for administering grants. Overall, the ARC had developed a workable decentralised model with universities for administering grants on ARC's behalf. Funding agreements were in place and ARC's monthly payments to universities were accurate and timely. However, ARC had few systematic processes to monitor progress and final reporting of projects. As a result, ARC could not fully account that grants were used as intended, or assess the extent that ARC's output and outcome were being met.

ARC has a substantial peer-review process in place, with a strong focus on research merit and national benefit. This enables ARC to select and fund high calibre research.

ANAO has made nine recommendations and a number of suggestions to strengthen ARC's management of grants, with particular emphasis on improving the effectiveness, transparency and accountability of ARC's grants administration. These recommendations are particularly important in light of changes to ARC's governance announced by the then Minister following the Uhrig Review.

Agency response

ARC provided the following short form response on the audit:

The audit found that the ARC was meeting the requirements of the ARC Act in administering grants for basic and applied research. It made a number of recommendations and suggestions to strengthen the process.

The ARC acknowledges that it has to date focussed its efforts on the selection process and that ideally additional work is required to strengthen its capability in the area of post-award management and monitoring. To assist with this process, the ARC is undertaking an extensive systems redevelopment project aimed at integrating all aspects of grants management (scheduled for completion in 2007). It also conducted a number of pilot post-award institutional reviews during 2005 as a precursor to an ongoing program of reviews which now will commence in June 2006.

The ARC agrees with the report's nine recommendations and is proceeding with their implementation.

ARC also provided a response to each of the recommendations. These appear immediately following each recommendation in the body of the report.