Introduction

1. The Commonwealth Scientific and Industrial Research Organisation¹ (CSIRO) is Australia's largest research and development organisation. CSIRO has an annual budget of over $1.3 billion and employs 6680 staff, located across 56 sites within Australia and overseas. CSIRO’s primary functions are to:

• carry out scientific research to assist Australian industry and to further the interests of the Australian community;
• contribute to the national and international objectives and responsibilities of the Australian Government; and
• encourage or facilitate the application and use of the results of its own or any other scientific research.

2. CSIRO established the Flagship Program to address major national challenges and opportunities through large-scale multi-disciplinary research partnerships. The Flagship Program comprises 10 individual National Research Flagships (Flagships). Each Flagship has an overarching goal which is framed around addressing the National Research Priorities which apply to all Australian Government science agencies such as CSIRO, and Australian Government competitive grant schemes for public sector research.

3. The genesis of the Flagship Program came from internal concerns surrounding the future of the organisation. A factor contributing to these concerns were reductions in funding to CSIRO, combined with a lack of new funding for the organisation under the Australian Government’s Backing Australia’s Ability initiative.^[2] CSIRO concluded that its competitive advantage lay in its capacity to assemble large multi-disciplinary research teams to undertake research associated with major national goals. Consequently, in 2002, CSIRO commenced developing the Flagship Program, with its launch by the then Prime Minister taking place in 2003. The Flagship Program now represents a significant proportion of the research activities undertaken by CSIRO.

4. Since the Program was established CSIRO has invested over $2 billion in research projects under the umbrella of the Flagship Program. This funding has come through a range of sources including direct Budget measures, an ongoing process of redirecting CSIRO research activities and associated funding to the Flagship Program, and external revenue derived through specific research projects.

5. The Flagship Program started with six Flagships and since 2003 the number of Flagships has increased to 10. The 10 Flagships and their goals are set out in Table S.1. The Flagships that were the focus of this audit (the ‘selected Flagships’) are highlighted in the table.

Table S.1: The current goals and launch dates of the Flagships

Note ^: These dates are the official launch dates. The planning, development and establishment of these Flagships occurred prior to these dates.

Source: CSIRO documentation.

6. CSIRO regards Flagships as vehicles for government interaction on key policy areas. In line with this interaction, research undertaken by Flagships has been influenced by a number of Commonwealth, state and territory government policy initiatives such as: the National Water Initiative (2004); the Department of Agriculture, Fisheries and Forestry’s Rural Research and Development Priorities (2007), the Council of Australian Government’s National Climate Change Adaptation Framework (2007), the Department of Climate Change and Energy Efficiency’s National Framework for Climate Change Science (2009) and the Department of Climate Change and Energy Efficiency’s 2010 Position Paper on Adapting to Climate Change in Australia. A number of these policy initiatives have provided research grants to CSIRO.

7. In addition to the Flagships, a key component of the Flagship Program is the Flagship Collaboration Fund, which was established in 2004. The Flagship Collaboration Fund is designed to further strengthen collaboration between the Flagships, universities and other publicly funded research institutions by building partnerships with these organisations in support of delivering Flagship goals. Since its inception, the Flagship Collaboration Fund has received Budget funding of $114.3 million.

Organisational change program

8. Through a succession of strategic plans since 2000, CSIRO has undertaken a significant organisational change program. This program has encompassed structural, planning, reporting and administrative processes across the organisation. Key structural changes have included:

• the introduction of a standardised planning and reporting framework across research activities, known as the Portfolio Performance Framework;^[3]
• the transition to a matrix-based management model to facilitate multi-disciplinary research by allowing Flagship projects to draw on the capabilities from across the various research divisions within CSIRO; and
• the development of an organisation-wide annual investment process, known as the Science Investment Process, to allocate resources across research activities.

9. To a large degree the development and implementation of the Flagship Program has acted as a driver for this change process as the Flagships rely on the effective operation of the matrix-based management model, adopted by CSIRO, to undertake multi-disciplinary research. The Science Investment Process uses the Portfolio Performance Framework as the basis for decision-making and to direct research activities across the matrix.

Audit objective, criteria and scope

10. The objective of the audit was to assess the effectiveness of CSIRO’s development and administration of selected National Research Flagships. In assessing CSIRO’s performance, the ANAO examined whether:

• mechanisms were in place to develop and implement the Flagships, within the context of the broader CSIRO change program;
• governance arrangements for Flagships incorporated sound oversight, planning and reporting arrangements; and
• periodic review activities were used to assess and improve the operation of the Flagships.

11. In addition to examining overarching governance arrangements, the ANAO focused on four selected Flagships where there was potential for interrelated issues (see Table S 1). These Flagships are undertaking research in the fields of energy, water, agriculture and climate adaptation. Within this context there is increasing acknowledgement in the scientific community that the areas of energy security, water security, food security, climate change mitigation and climate adaptation interact, and that there is a need to understand this interaction.^[4]

12. The audit does not provide an opinion on the scientific merits of Flagship research, with the examination of individual research projects underpinning Flagships only being used to gain an understanding of the operation of the program.

Overall conclusion

13. The Flagship Program was launched in 2003, and since that time has become a core component of the framework within which CSIRO undertakes research activities. In 2011–12, CSIRO will allocate some $566.18 million^5] (or 43 per cent of budgeted revenue) towards funding the 10 Flagships within the program.^[6] The Flagship Program provides a model for undertaking multi-disciplinary research to address the Australian Government’s National Research Priorities. Flagship research is directed at long-term goals and focuses on applied research in contemporary fields. The issues within these fields of research are often complex, to some extent interrelated, and are areas where advances in knowledge may have significant benefit for Australia.

14. The Flagship Program has provided the basis for a significant realignment of the research activities in CSIRO over several years. This realignment has been supported by a series of initiatives implemented through a large-scale organisational change program. Consequently, the Flagship Program has developed in an environment of ongoing changes to business processes while also acting as a key driver for the organisational change program. This approach contributed to the effective development of the Flagship Program within CSIRO.

15. CSIRO’s approach to administering the Flagship Program has evolved as the program has matured. As part of this, CSIRO has used the experiences from the progressive rollout of individual Flagships to inform the development and implementation of subsequent Flagships. CSIRO has also used the organisational change process as a base to support the administration of the program. In that regard, the transition to a matrix-based management model has been a particularly challenging exercise. CSIRO has adopted a continuous improvement approach to administering the Flagship Program and has actively refined and modified change initiatives related to the Flagship Program to enhance organisational outcomes. This approach has provided a sound structural framework for administering Flagship research.

16. Given the nature of the Flagship Program in terms of its long-term goals, partnership arrangements and research paths, which are subject to external drivers that change over time, the evolution of the program and the individual Flagships will necessarily be a continuous process. This process requires not only effective external engagement but also an ongoing focus on refining internal arrangements for managing complex multi-disciplinary research activities in a matrix management environment. Within this context there are opportunities to improve the administration of the program, particularly around performance management arrangements and reporting on budget performance. More broadly, there are opportunities to improve the overarching governance, direction-setting and internal coordination of the program.

17. A key intended outcome of establishing the Flagship Program was to undertake research focused on delivering economic, social and environmental impact. This focus on impact continues to be the case, as reflected in the Flagship goals. The evaluation of impact is a complex exercise confronting many research organisations, as the impact of research may be somewhat removed from the production of the research output.^[7] While there is evidence that the research of the selected Flagships has contributed to policy and investment decisions by government, CSIRO does not have a systematic process for measuring the impact of Flagship research. Instead, CSIRO has relied on external consultants to estimate the impact of research through a series of individual studies that have been based on a range of assumptions in areas of ongoing uncertainty. The 2010 Lapsing Program Review^[8] recommended that CSIRO improve its measurement of the social, environmental and economic value of its research. In response, CSIRO established a project in late 2010 (Impact 2020 Project), which is designed to improve the way impact is measured.

18. The change program has provided a defined structure for planning which incorporates performance management and reporting arrangements. CSIRO has developed an internal reporting framework at the research stream level based around Annual Performance Goals (APGs). The implementation of these arrangements for the Flagship Program is yet to provide a stable reporting regime which demonstrates the performance of individual Flagships over time. APGs are variable in format and content, making the comparison of performance across research streams within the Flagship Program difficult. Reporting processes for APGs are not metrics-based, making them a subjective indicator of long-term performance. This reflects the nature of the planning structure introduced by the change program which aggregates projects into research streams. As this aggregation occurs, the ability to develop objective measurable performance indicators decreases.

19. External reporting for the Flagship Program is based on Key Performance Indicators (KPIs) which have been modified as CSIRO has transitioned to reporting within an outcome/program framework. This ongoing modification of KPIs limits the capacity to analyse the performance of the Flagship Program over time. The current KPIs focus on indicating performance against a mixture of outcomes and outputs. In the absence of systematic mechanisms for measuring the impact of Flagship research, the statement against the outcome-focused KPI is subjective. For a number of the KPIs the data presented in support of KPI achievement is not specific to the Flagship Program, thereby providing limited insight into the performance of the program, relative to other research activities undertaken by CSIRO.

20. A comparison of Flagship financial data from key external documents including the Portfolio Budget Statements, CSIRO Operational Plans and Annual Reports, revealed that these documents contained different financial information for the same financial year. Further, there was also a high degree of variability in the way this data was presented. These factors diminish the transparency and accuracy of the Flagship financial performance information available to external stakeholders.

21. Achieving an appropriate balance in the level of executive oversight of the Flagship Program has presented some challenges to CSIRO. Initially, a Flagship Oversight Committee was the primary governance body with direct responsibility for managing the Flagship Program, including direction-setting and governance. However, the role of this committee was subsumed by another committee in 2008, and this resulted in a dilution in the direct oversight of the Flagship Program, particularly in the area of overarching governance of the program. In late 2010, the Flagship Oversight Committee was re-established to provide a stronger focus on the governance and coordination of the Flagship Program, consistent with its significance to CSIRO.

22. Since the Flagship Program commenced, there have been a number of internal and external reviews and audits of CSIRO activities. CSIRO has also sought external input through a range of advisory committees. These activities have been wide ranging and both directly and indirectly assessed the Flagship Program, individual Flagships and the broader CSIRO administrative arrangements encompassing the Flagship Program. It is evident that these activities have influenced the way CSIRO administers the Flagship Program and the direction of its research. As the Flagship Program relies on the effective management of research activities across CSIRO and the ongoing interaction and support of stakeholders, improving mechanisms for capturing and consolidating the findings of reviews, internal audits and stakeholder input would assist CSIRO in identifying areas where improvements can be made to organisational performance and responsiveness.

23. To help improve the effectiveness of CSIRO’s administration of the Flagship Program, the ANAO has made two recommendations aimed at enhancing both the financial reporting arrangements and the use of the insights captured through review activities.

Key findings

Organisational realignment (Chapter 2)

24. Since 2000, CSIRO has undertaken a significant program of organisational change. Structural change initiatives have included implementing a common planning framework (Portfolio Performance Framework) across CSIRO which is underpinned by a Science Investment Process, the consolidation of corporate functions, and the progressive implementation of an operating model based around a matrix-based management model. To a large degree, these initiatives have now transitioned to business-as-usual within CSIRO.

25. The Portfolio Performance Framework is a standardised planning framework that was piloted in two Flagships in 2002 before being applied across the organisation. In 2004, CSIRO undertook a review of the implementation of the Portfolio Performance Framework. The review of the framework made generally positive findings with respect to the implementation of the strategic elements of the framework for the Flagship Program.

26. Investment decisions surrounding Flagships made through the Science Investment Process were based on recommendations to CSIRO’s Science Sub Committee by the Flagship Oversight Committee up until 2008. Within CSIRO, the Flagship Oversight Committee was also regarded as a positive influence in terms of governance, although there was some tension surrounding the extent of oversight that it provided. In 2008, the Flagship Oversight Committee was dissolved through a change to the committee structure in CSIRO. This resulted in the newly established CSIRO Appraisal and Investment Committee assuming responsibility for input to investment decisions relating to the Flagship Program. In late 2010, the Flagship Oversight Committee was re-established.

27. The Flagship Program has been in place for over eight years and today represents an organisation-wide initiative. A key area where this is evident is the Flagships’ reliance on research capabilities from across CSIRO to undertake multi-disciplinary research. CSIRO has transitioned to a matrix-based management model to facilitate this multi-disciplinary research. Over several years, CSIRO has attempted to improve the operation of the matrix-based management model, although several recurrent issues have been encountered. These issues include uncertainty surrounding roles and responsibilities, and the complexity of undertaking and administering research in a matrix environment. CSIRO continues to work on optimising the operation of the matrix-based management model.

28. There are linkages between the research areas of the selected Flagships. Through the Science Investment Process there is some evidence of consideration of this relationship. Also, the nature of multi-disciplinary research in a matrix-based management model can provide some cross-fertilisation of ideas and knowledge across the Flagships. However, there was no apparent structure around this interaction within CSIRO at the time of audit fieldwork. Given recent statements by the Office of the Chief Scientist about the intersection of issues that are the focus of these Flagships, and the intent that Flagships undertake multi-disciplinary research, this is an area of the program that could be strengthened. The recently re-established Flagship Oversight Committee should provide a vehicle to influence and develop strategies to promote interaction where Flagship research interrelates.

Evolution of the selected Flagships (Chapter 3)

29. To assess the key characteristics underpinning the development and ongoing operation of the Flagship Program, the ANAO examined four Flagships (the ‘selected Flagships’). This included a comparison of the factors that have influenced the development of two of the original Flagships—the Water for a Healthy Country Flagship and the Energy Transformed Flagship; and two new Flagships—the Climate Adaptation Flagship and the Sustainable Agriculture Flagship.

30. In many areas, CSIRO had a history of undertaking research prior to the introduction of the Flagship Program. As such, while some Flagships received new funding at their outset, the research undertaken, to an extent, often represents a consolidation and redirection of existing research. Further, the introduction of the Flagships has occurred in an environment where Australia has been facing many challenges, such as water scarcity, and this is reflected in Flagship goals, themes, streams and research projects.

31. The drivers and demand for research vary between Flagships and/or can change over time. This can result in difficulties in defining goals and the focus of research. The Energy Transformed Flagship is an example of this, and the Flagship underwent a significant restructure in 2009 following CSIRO-commissioned review activities.

Funding and expenditure (Chapter 4)

32. Through the Flagship Program, CSIRO has received additional funding from government and progressively increased the proportion of the remainder of its budget that it directs to the Flagships. In 2009–10, CSIRO allocated $534.9 million to the Flagship Program. This comprised $363.2 million from government funding and $171.7 million of external revenue. External revenue is derived from a number of sources including: Australian, state, territory and local government agencies; the Australian private sector; and overseas entities.

33. The 2002 report on the implementation of the Portfolio Performance Framework commented that the framework must fit within the current organisation-wide planning, budgeting and reporting processes. The process was intended to provide a direct relationship between Portfolio Performance Framework and budgeting processes including the Portfolio Budget Statements (PBS), the operational plan and the annual report. A comparison of the budget information contained in the Quadrennium Funding Agreement^[9], PBS, operational plans and annual reports shows inconsistencies in the figures contained in the documents and instances where information has not been included. It is also apparent that the budget figures contained in external documents contain some inaccuracies, primarily due to weaknesses in the way CSIRO prepared budget data for inclusion in the PBS. This is an issue which CSIRO is seeking to address. The inaccuracies in the PBS combined with the lack of continuity in the financial data included in key external documents have reduced the transparency of the Flagship financial performance information available to external stakeholders.

34. The nature of the research partnership arrangements means that the capacity of a Flagship to attract and retain external revenue, from a diverse range of stakeholders, is an important factor in setting the research direction of a Flagship and determining the potential impact of Flagship research. CSIRO was unable to provide the ANAO with consistent client contract financial information for all of the selected Flagships, which limited the capacity to compare Flagship performance on this basis. However, based on the client contract financial information provided, it was apparent that the selected Flagships received a large proportion of their external revenue from Australian Government agencies that are responsible for developing and implementing policy initiatives, or for administering ongoing research programs such as Cooperative Research Centres and Research and Development Corporations.

35. While each Flagship has a number of research projects, many rely on a small number of high-value contracts. This highlights the importance of CSIRO’s ability to identify and establish partnerships for the Flagships. As research under these projects is completed, CSIRO is conscious that it will need to identify new research opportunities that are consistent with the external drivers for research in the respective fields.

Performance reporting (Chapter 5)

36. Through the Portfolio Performance Framework, CSIRO introduced a framework for internal performance reporting. At the pinnacle of this framework are the Flagship goals. For the selected Flagships, these goals are very broad in scope and their achievement will be subject to a range of external factors. As such, they are more aspirational than a baseline for the measurement of progress and performance.

37. Consistent with the intent of the Flagship Program, the Flagship goals are impact-focused. The evaluation of research impact is a complex exercise, particularly as users of research may take into account a range of other factors in making decisions and, potentially, there can be a significant time lag between when research output is produced and when the impact of that research is realised.

38. While there is evidence that the research of the selected Flagships has influenced policy and investment decisions by government, CSIRO does not have in place systematic processes for measuring the impact of Flagship research. Since the program commenced, economic consultants have been engaged on a number of occasions to assist in estimating the impact of research, although even in cases where Flagship research has been prepared specifically to inform decision making, a range of assumptions are necessarily required to estimate the economic impact of a specific research output. This includes the impact of factors beyond CSIRO’s control. Where research is less mature or the external environment less certain, an increasing range of assumptions need to be applied. Further, Flagship research may also have social and environmental impacts not directly measurable in an economic sense.

39. CSIRO has introduced Flagship roadmaps, which are intended to demonstrate progress towards achieving Flagship goals by indicating the progress of research themes within a Flagship. These roadmaps appear in both internal and external reports. While this is a positive initiative, the measurement of progress against these roadmaps is limited by the lack of a systematic process for monitoring and assessing the impact of research.

40. At the research stream level, CSIRO has implemented Annual Performance Goals (APGs) to report performance. APGs are not metrics or milestone-based and do not indicate the quality of underlying science. Over time, CSIRO has attempted to improve the consistency in defining and reporting against APGs. The APGs for the selected Flagships indicate variability in terms of timeframe, definition and measurability. These factors combine to limit the capacity to use APGs to compare performance across the Flagship Program and the performance of individual Flagships over time.

41. External reporting for the Flagship Program is based around a series of Key Performance Indicators (KPIs). Within the 2009–10 CSIRO Annual Report, the Flagship Program reported against five KPIs. This was the first year that these KPIs had been included in the Annual Report, meaning that a direct analysis of performance over time was not possible. The ANAO reviewed the KPIs and found that in some instances they were not a direct indicator of performance. In other cases the data presented in support of KPI achievement was either not Flagship-specific, or was presented in such a way that it risked being misleading.

42. Through the Lapsing Program Review, CSIRO has recognised that there is an opportunity to enhance its overall performance reporting arrangements. The review also identified a need to improve monitoring and measurement of economic, social and environmental impact. In late 2010, CSIRO commenced the Impact 2020 Project. The goal of this project is to increase the visibility of CSIRO’s future impact pipeline for the next 10‑20 years. The project charter outlines that to achieve this goal it is likely that modifications will be required to the Science Investment Process, project planning and management arrangements within CSIRO. Achieving such an outcome will be particularly challenging and until such arrangements are established, CSIRO should consider how to best present the impact of Flagship research through external reporting.

Internal and external reviews (Chapter 6)

43. Since establishing the Flagship Program, CSIRO has commissioned a number of reviews that have either been Flagship-specific or have touched on aspects of the program, individual Flagships or the broader CSIRO operational environment. Periodic review and acting on resulting recommendations can contribute to a continuous improvement approach to management. Consistent with this outcome, CSIRO has used reviews and stakeholder feedback to assist in defining the future research direction for individual Flagships. For example, a review of the Energy Group, which incorporated two Flagships, resulted in a significant change to the research focus of the Energy Transformed Flagship.

44. Reviews that have focused specifically on the Flagships have been generally positive in their findings. Flagship-specific and broader reviews have also identified a range of cultural, resourcing and structural issues surrounding the Flagships and the broader change program undertaken within CSIRO. It is important that the outcomes of reviews are not considered in isolation as to do so may limit the capacity to draw strategic insights, identify systemic issues and recognise points of leverage where organisational outcomes can be improved. This is particularly the case in situations where organisations undertake an extensive volume of reviews and seek input from a range of stakeholders, as has been in the case in CSIRO. Accordingly, CSIRO should investigate options to improve the mechanisms for capturing and consolidating the findings of reviews to improve overall organisational performance.

Summary of agency response

45. CSIRO welcomes the ANAO report and accepts its key findings and recommendations. CSIRO is committed to the further development of the administration, governance and performance management of the Flagship program and the findings of this review will assist their evolution.

46. In relation to the report's recommendations, both are accepted.

Footnotes

[1] CSIRO is constituted and operates under the provisions of the Science and Industry Research Act 1949. CSIRO is a statutory authority, so it is also subject to the Commonwealth Authorities and Companies Act 1997. The CSIRO Board is responsible to the Australian Government for the overall strategy, governance and performance of CSIRO. The CSIRO Chief Executive is responsible for conducting the affairs of CSIRO in accordance with the strategy, plans and policies approved by the CSIRO Board.

[2] Under the 2001 Backing Australia’s Ability initiative, the Australian Government provided $3 billion in new funding over five years to a range of research programs to address issues in Australia’s science and innovation system, including programs directed to: public sector and business research and development; adoption of technology; and commercialisation of research. None of the programs funded under this initiative specifically provided funding to CSIRO.

[3] The Portfolio Performance Framework structure comprises: · Themes — major areas of research contributing to a Flagship goal; · Streams — a collection of related projects that address a particular aspect of a theme; and · Projects — core units of research undertaken.

[4] Examples include Office of the Chief Scientist, Securing Australia’s future: PMSEIC reports on food security and energy-water-carbon intersections [Internet]; 2010, available from <http://www.chiefscientist.gov.au/2010/12/securing-australia%e2%80%99s-future-pmseic-releases-expert-reports-on-food-security-and-energy-water-carbon-intersections> [accessed 18 February 2011] and R Garnaut, Climate Change Update 2011 – Transforming Rural Land Use [Internet]; 2011, available from <http://www.garnautreview.org.au/update-2011/update-papers/up4-transforming-rural-land-use.html> [accessed 17 March 2011].

[5] Comprises $339.5 million from Government funding and $226.7 million from external revenue.

[6] Department of Innovation, Industry, Science and Research, Portfolio Budget Statements 2011–12, DIISR, 2011, pp. 211-215.

[7] In 2009 the Organisation for Economic Co-operation and Development (OECD) released a report titled Measuring Government Activity which stated with respect to measuring the outcomes of government programs: Commonly one cannot hold particular organisations – or even governments – fully responsible for outcomes in the same way that one can hold them responsible for outputs. On the other hand, they are not entirely without responsibility either: very frequently they make a contribution to the final outcome but cannot wholly determine it.

[8] Government guidelines require that a review of CSIRO be conducted at the end of each funding agreement, known as a Lapsing Program Review. The review is undertaken by an Inter-Departmental Committee and examines whether CSIRO is operating appropriately, effectively and efficiently and may recommend the continuation of funding.

[9] A four-year agreement between the Australian Government and CSIRO that sets out funding principles for CSIRO. The current Quadrennium Funding Agreement 2007–08 to 2010–11 is in its final year of operation. As part of the 2011–12 Budget, the Australian Government announced a continuation of funding for CSIRO. The funding will form the basis of a new Quadrennium Funding Agreement expected to operate over four years from 2011–12. At the conclusion of the audit, the Quadrennium Funding Agreement 2011–12 to 2014–15 was yet to be finalised.