Background

1. The Defence Science and Technology Group (DSTG or the Group) is part of the Department of Defence. Its purpose is to provide specialist science and technology services to the Australian Defence Organisation (Defence). The Group’s core roles are to support the Australian Defence Force (ADF) on operations, sustain and improve existing defence capabilities, assist Defence to be a smart buyer of defence equipment and undertake research aimed at future-proofing the ADF.

2. DSTG is led by the Chief Defence Scientist and employs some 2300 scientists, engineers, information technology specialists and technicians. It is a geographically dispersed organisation with its headquarters in Canberra and its largest facilities in Melbourne and Adelaide. DSTG’s budget for 2015–16 is $432 million, which is 1.4 per cent of the total Defence budget.

3. In April 2013, DSTG released a new strategic plan. While developing this plan the Group identified that it needed to: more effectively address the longer-term science and technology needs of Defence, be more agile in its response to the changing science and technology needs of Defence, and collaborate more effectively with external organisations. Following the release of its strategic plan, DSTG has been undertaking an active process of reform to address these issues.

Audit objective and criteria

4. The objective of this audit was to assess how effectively DSTG administers the science and technology work it undertakes for the Australian Defence Organisation (Defence).

5. To form a conclusion against this objective, the ANAO adopted the following high-level criteria:

• DSTG’s strategic plan supports the effective delivery of science and technology work for Defence. In particular: DSTG’s strategic plan addresses the Group’s key challenges; and DSTG has procedures in place to implement its strategic plan and monitor its progress in doing so.
• DSTG recognises Defence’s strategic priorities for science and technology work. DSTG has aligned its science and technology capabilities with these priorities and has sound processes for aligning its science and technology work with these priorities.
• DSTG has sound processes for defining, monitoring and delivering individual pieces of science and technology work. These processes include regularly reviewing and reporting on individual pieces of work and considering the need to continue with them.
• DSTG has a strategy for effectively collaborating and partnering with other science organisations and industry to deliver science and technology work to Defence.
• DSTG measures outputs and outcomes from its science and technology work and how satisfied its clients are.

Conclusion

6. Since 2013, the Defence Science and Technology Group has implemented a number of key initiatives to improve the effectiveness with which it administers the science and technology work it undertakes for the Australian Defence Organisation. In April 2013, the Group released a new strategic plan, with a focus on becoming more innovative and collaborative, and better aligning its work with wider Defence priorities. Following the release of this plan, DSTG implemented a significant organisational restructure to embed a more client-focused strategic direction.

7. There remains scope for DSTG to build on these reforms by improving the use of its Management Information System and strengthening the Group’s performance monitoring and reporting framework. In particular, DSTG’s Management Information System captures valuable business data regarding client requirements, tasks and deliverables, but DSTG does not effectively harness this data to inform senior management’s oversight and review of the Group’s science and technology work at a strategic level.

Supporting findings

DSTG’s strategic plan

8. DSTG consulted widely, between April 2012 and April 2013, to develop its new strategic plan. During this consultation process the Group identified that: not enough of its science and technology work was addressing the longer-term needs of Defence; it was difficult for the Group to quickly grow a new science and technology capability or divest itself of an existing capability; and its dealings with external organisations were more transactional than strategic.

9. To address these challenges DSTG’s strategy is to be a more valued, collaborative and innovative organisation. DSTG has also articulated 10 roles that it performs, four of them core roles:

• supporting Australian Defence Force operations;
• supporting the acquisition of new major Defence capability;
• supporting the sustainment of current Defence capability; and
• future proofing the Australian Defence Force.

10. These core roles provide a useful framework for DSTG to demonstrate how its work aligns with key Defence activities.

11. To implement its strategic plan DSTG developed 10 strategic initiatives and allocated $24 million to implement these initiatives over five years. DSTG’s senior managers review the progress of these initiatives every three months.

12. Following the release of its strategic plan DSTG restructured its organisation to better align its science and technology capabilities with the major areas of Defence that need science and technology support.

Administering DSTG’s science and technology work

13. DSTG records information about its science and technology work in its Management Information System using three key attributes: client requirements, tasks, and deliverables. The system provides a sound structure for DSTG to manage its science and technology work. However, DSTG mostly uses its system to record, organise, track and report on individual pieces of work and does not use it effectively to group individual pieces of work together and link them to a strategic goal. DSTG advised the ANAO that it recognises this shortcoming and is intending to address it through one of its strategic initiatives.

14. The Group’s deliverables vary widely in terms of the type of work undertaken and the level of management oversight needed to ensure delivery. DSTG does not differentiate between different types of deliverables and has not identified the appropriate level of management oversight required for each type of deliverable.

15. DSTG does not have a consistent approach to managing individual pieces of science and technology work, instead the Group relies on a variety of processes developed locally within its Major Science and Technology Capabilities. These processes do not provide DSTG, at a strategic level, with assurance that the progress of individual pieces of work is assessed against planned milestones or other metrics, or how issues arising from such assessments are addressed.

Collaborating and partnering with external organisations

16. DSTG has established a sound strategic framework for collaborating with external organisations. This framework includes strategic alliances with industry organisations, and partnerships with universities and other research organisations.

17. DSTG is trialling a set of metrics to measure the performance of its collaborations and partnerships with external organisations.

18. DSTG is developing a culture of collaborating with external organisations by:

• training an external engagement manager for each division to support staff in developing and managing relationships with external organisations;
• seeking to simplify the process by which small-to-medium enterprises can use DSTG’s intellectual property; and
• establishing an industry secondment program for DSTG staff.

19. DSTG has recently changed the way it manages risks associated with collaborating with external organisations. In 2013 DSTG disbanded its Probity Board, which was responsible for providing advice on the probity of collaborations and partnerships. At that time, some functions of the Probity Board were taken up by DSTG’s Advisory Board. In 2015, DSTG also abolished its Advisory Board. Following the abolition of these boards, DSTG has adopted Defence’s probity arrangements. There would be merit in DSTG updating its strategic plan, which contains references to the Probity Board, to reflect these new arrangements.

Demonstrating organisational performance

20. DSTG consults extensively within Defence to develop a prioritised list of Defence’s science and technology needs. However, DSTG has recognised that shortcomings with the data held in its Management Information System make it difficult for the Group to demonstrate quantitatively the extent to which its portfolio of science and technology work aligns with Defence’s strategic priorities.

21. In 2014 DSTG introduced a structured program of benchmarking its Major Science and Technology Capabilities (MSTCs). This program consists of annual internal reviews of all MSTCs, and a program of rolling external reviews for a quarter of the Group’s MSTCs each year.

22. The Group also conducted client satisfaction surveys in 2013 and 2014. The surveys indicated that DSTG’s clients were generally satisfied with DSTG but had specific concerns relating to: DSTG’s approach for reporting the progress of work; the length of time DSTG took to complete deliverables; and DSTG’s approach to project managing deliverables.

23. DSTG has traditionally described the value of its science and technology work in terms of high-level outcomes such as: lives saved, risks reduced, money saved and capability enhanced. In line with this approach, DSTG recently engaged ACIL Allen to undertake an economic assessment of 10 selected DSTG projects. This assessment calculated the economic benefit to Australia of these projects to be $5.1 billion.

24. DSTG could strengthen its performance management framework by also reporting internally on its efficiency and administrative effectiveness in managing science and technology work, having regard to time and cost expectations.

Recommendations

Summary of entity response

The Department of Defence’s response to the proposed report is provided below.

Defence welcomes and acknowledges the findings contained in the audit report on Managing science and technology work for Defence – Defence Science and Technology Group and agrees with the one recommendation.

Defence has made significant progress on improving management and oversight of science and technology [S&T] work since the introduction of DST Group’s Strategic Plan. Work has been progressing to improve transparency and traceability of how DST Group manages S&T capabilities, program planning and the execution of the program.

Defence has assigned resources to deliver enhanced client engagement, program planning, consistent project management and governance, and changes to the supporting management information system.

Introduction

1.1 The Defence Science and Technology Group (DSTG or the Group) is part of the Department of Defence.¹ Its purpose is to provide scientific advice and technological solutions to support the Australian Defence Organisation (Defence). The Group’s core roles are to: support the Australian Defence Force (ADF) on operations; sustain and improve existing defence capabilities; assist Defence to be a smart buyer of defence equipment; and undertake research aimed at future-proofing the ADF.

1.2 DSTG is led by the Chief Defence Scientist and employs some 2300 scientists, engineers, information technology specialists and technicians.² It is a geographically dispersed organisation with its headquarters in Canberra and its two largest facilities in Melbourne and Edinburgh (near Adelaide). It also has facilities in Brisbane, Sydney, Rockingham (near Perth), Scottsdale (in Tasmania) and Innisfail (in northern Queensland). DSTG’s 2015–16 budget is $432 million, which is 1.4 per cent of the total Defence budget.³

1.3 Figure 1.1 shows DSTG’s organisational structure. Reporting to the Chief Defence Scientist are three Chiefs in corporate roles and seven Chiefs of Division. DSTG’s science and technology work is undertaken in its seven divisions. Each division is divided into Major Science and Technology Capabilities (MSTC) and, in turn, these MSTCs are made up of Science and Technology Capabilities (STC). For example, DSTG’s Land Division has four MSTCs and 13 STCs.

Figure 1.1: DSTG high-level organisational structure

Source: DSTG.

1.4 The Chief Defence Scientist reports to the Secretary of the Department of Defence.

Recent reviews of DSTG

1.5 In June 2013, the then newly appointed Chief Defence Scientist requested that an external panel⁴ conduct a review of how well DSTG manages: its collaborations with external organisations; the intellectual property it creates; and the transfer of technology to other parties.

1.6 The panel concluded that:

• DSTG’s high-level strategy was disconnected from the day-to-day practices and culture further down in the organisation;
• collaborating with external organisations and commercialising technology are secondary functions for DSTG and a move to increase these activities will need to be accompanied by greater recognition of their value and importance across DSTG;
• to improve how DSTG collaborates with industry and other external organisations DSTG should set up a consultancy unit, a customer feedback survey and an external engagement advisory committee; and
• DSTG should measure the outputs, outcomes and impacts of its activities.

1.7 In October 2013, the Australian Government established a National Commission of Audit to review and report on the performance, functions and roles of the Commonwealth government. The Commission recommended that the Government consider outsourcing DSTG.⁵

The Commission supports the Government’s commitment to undertake a first-principles review of Defence’s departmental structure and major processes. Opportunities for further rationalisation of 25 Defence non-principal bodies should be considered in this context.

The Defence Science and Technology Organisation should be assessed for its outsourcing potential.

First Principles Review of Defence

1.8 The report of the First Principles Review of Defence, released in April 2015, stated that DSTG ‘is valued by Defence and national security agencies for its contribution to saving lives, reducing risk, saving money and enhancing capabilities.’⁶ The report considered that DSTG ‘provides a range of specialised and technical services across both Defence and the national security community and has developed state-of-the-art technology that is being used across the world today.’⁷ The report also stated that, on the basis of information reviewed by the review team, ‘there is no clear case for outsourcing the Defence Science and Technology Organisation and, in fact, this approach may be detrimental to the support it offers to Defence.’⁸

1.9 That said, the report concluded that DSTG ‘struggled to articulate clearly to the review team the value that it contributes to Defence outcomes’.⁹ As a consequence the report recommended that DSTG ‘clearly articulate its value proposition. This would include examples and actual amounts of value created’.¹⁰ The report also recommended that DSTG:

• become part of the proposed Capability Acquisition and Sustainment Group¹¹;
• rationalise its senior leadership;
• establish strong partnerships with key academic and research institutions to leverage the knowledge of scientists and create pathways into and out of academia and industry; and
• disband its Advisory Board.

1.10 More broadly, the report recommended that Defence review its developmental research priorities, how these priorities align with future force requirements, and how Defence could use allied partners to promote innovation and make the most valuable contribution to future Defence capability.

1.11 The Government agreed with all but one of the recommendations in the report. It did not accept the recommendation to move DSTG into Defence’s Capability Acquisition and Sustainment Group, and DSTG remains a separate Group within Defence, albeit with a name change.

Audit approach

1.12 The objective of this audit was to assess how effectively DSTG administers the science and technology work it undertakes for the Australian Defence Organisation (Defence).

1.13 To form a conclusion against this objective, the ANAO adopted the following high-level criteria:

• DSTG’s strategic plan supports the effective delivery of science and technology work for Defence. In particular: DSTG’s strategic plan addresses the Group’s key challenges; and DSTG has procedures in place to implement its strategic plan and monitor its progress in doing so.
• DSTG recognises Defence’s strategic priorities for science and technology work. DSTG has aligned its science and technology capabilities to address these priorities and has sound processes for aligning its science and technology work with these priorities.
• DSTG has sound processes for defining, monitoring and delivering individual pieces of science and technology work. These processes include regularly reviewing and reporting on individual pieces of work and considering the need to continue with them.
• DSTG has a strategy for effectively collaborating and partnering with other science organisations and industry to deliver science and technology work to Defence.
• DSTG measures outputs and outcomes from its science and technology work and how satisfied its clients are.

1.14 DSTG has been undertaking an active process of reform, following the release of its new strategic plan in April 2013. The audit focused on the progress DSTG has made in reforming the way it: administers its science and technology work; collaborates with external organisations; and demonstrates its organisational performance.

1.15 The audit focused on the interface between DSTG and Defence and did not consider the interface between DSTG and other national security organisations. The audit did not examine DSTG’s Strategic Research Investment program¹², instead focusing on how DSTG addresses the more immediate science and technology needs of Defence.

Audit method

1.16 The audit team reviewed Defence records and visited DSTG’s head office in Canberra and its two largest facilities at Fisherman’s Bend in Melbourne and Edinburgh, near Adelaide. During these visits, the audit team conducted interviews with DSTG managers and scientists.

1.17 The audit was conducted in accordance with the ANAO auditing standards at a cost to the ANAO of approximately $461 127.

Does DSTG’s strategic plan address the Group’s key challenges?

2.1 DSTG’s Science Strategy and Policy Branch, with the assistance of a major consulting firm and an internal steering committee, designed a thorough process for developing DSTG’s new strategic plan. This process included gathering information using four ‘radars’:

• a tactical radar to understand the Defence and whole-of-government environment in which DSTG would operate over the next five years;
• a strategic radar to gain insights from external organisations;
• a people radar to gauge staff attitudes; and
• a technical radar to determine the broad direction DSTG’s science and technology capabilities would take over the next five years.

2.2 DSTG developed its strategic plan over a year and assigned more than 50 people from across the Group to gather the information needed.¹³ During this process DSTG identified the following key challenges that needed to be addressed by the plan:

• DSTG’s overall strategy was not well understood within the Group and different areas of DSTG did not easily share knowledge with each other;
• DSTG’s science and technology work was mostly focused on addressing the short-term needs of Defence, and not enough of its work was focused on the longer-term needs of Defence;
• DSTG found it difficult to respond quickly to marked changes in Defence’s needs by either quickly growing a new science and technology capability or divesting itself of an existing capability;
• DSTG’s business processes varied significantly across the Group, and these processes were not well documented, communicated or followed by DSTG personnel; and
• DSTG’s dealings with external organisations (such as universities, defence industry organisations, and other research organisations) were driven by short-term needs rather than longer-term goals.

2.3 DSTG’s strategic plan also describes important external challenges facing the Group. The plan states that:

Key challenges facing DSTO include the increased blurring of state and non-state threats, military modernisation in the Asia-Pacific region, global access to commercial off-the-shelf technology and the rapid progression of cyber capabilities and other disruptive technologies. These external challenges coincide with a tightening resource environment for Defence.¹⁴

2.4 DSTG released its draft strategic plan in December 2012. The Group then consulted further with staff and stakeholders, and amended the plan based on this feedback. The Secretary of the Department of Defence and the Chief of the Defence Force endorsed DSTG’s strategic plan on 13 March 2013 and the Minister for Defence Science and Personnel endorsed it on 12 April 2013.

DSTG’s vision and strategy

2.5 To address the challenges facing the Group, DSTG’s strategic plan sets out its vision and strategy. DSTG’s vision is:

… to be a world leader in defence science and technology – indispensable in supporting and transforming Australia’s defence and national security.¹⁵

2.6 DSTG’s strategy is to be more valued, collaborative and innovative. In particular, DSTG’s goal is to be a valued advisor to government, support current Australian Defence Force operations, and play a vital role in ensuring Australia’s defence and national security capabilities remain at the leading edge by:

• collaborating more with industry; and
• more effectively integrating a number of innovations to address a Defence science and technology problem.

2.7 DSTG’s strategy also includes becoming a more efficient and effective organisation that supports and develops its workforce. To underpin this vision and strategy, the strategic plan articulates 10 roles for DSTG (see Table 2.1).

Table 2.1: DSTG’s roles

Source: DSTG Strategic Plan 2013–18.

2.8 The core roles set out in DSTG’s strategic plan provide a useful framework for DSTG to demonstrate how its portfolio of science and technology work aligns with key Defence activities.

2.9 DSTG’s strategic plan lists DSTG’s attributes which provide value to its stakeholders across these 10 roles. The attributes most frequently identified are the Group’s: defence domain expertise, science and technology skills, facilities, and partnerships and linkages with industry and academia. The plan also mentions DSTG’s ‘rigorous independence’ as an important attribute.

Does DSTG have sound processes for implementing its strategic plan and monitoring its progress in doing so?

2.10 DSTG’s strategic initiatives are arranged into four themes (see Table 2.2).

Table 2.2: DSTG’s strategic initiatives

Source: DSTG.

2.11 DSTG has developed project plans for its 10 strategic initiatives and assigned project teams to implement these plans. As of January 2015, DSTG had allocated $24 million to implement its strategic initiatives over the five years covered by the strategic plan, 2013–18.

2.12 Every three months since the launch of the plan, DSTG’s senior managers have reviewed the status of every strategic initiative. During these reviews, management has heard presentations from the project teams, assessed the current and prospective status of the initiatives, considered likely issues and risks, and agreed any actions or refocusing of effort. Records of these reviews indicate that senior management has been active and willing to candidly discuss areas of concern and make adjustments. For example, in respect to strategic initiative D2, strategic engagement with a client focus, in December 2014, the review process indicated that:

The initiative appears to be heading towards becoming too process-driven. A more strategic approach is required that involves greater engagement from Divisions, particularly the Senior Leadership Team. Membership of the working group needs broadening in order to seek input and support from Divisions, particularly Research Leaders.

2.13 To address these concerns DSTG restructured strategic initiative D2 and developed a new project plan to implement it. The DSTG Leadership team approved this new project plan in July 2015.¹⁶

2.14 In addition, the DSTG Leadership Team often discusses the status of the strategic initiatives at its monthly meetings, and updates of how the initiatives are progressing are regularly provided to all staff.

2.15 To measure progress in implementing its 10 strategic initiatives, DSTG has developed qualitative success measures for each initiative. Examples of these measures include:

• ‘measurable improvement in research quality and client outcomes’ for strategic initiative D1;
• ‘processes for planning, managing and reporting of the science and technology program are consistent and efficient’ for strategic initiative D2; and
• ‘recognition of DSTO as an innovative organisation, both within Defence and externally’ for strategic initiative T1.

2.16 Including quantitative as well as qualitative measures would make it easier for DSTG to monitor and report on its progress in implementing the 10 strategic initiatives. Between August 2013 and April 2015, DSTG’s senior managers assessed the status of each strategic initiative as being on track, or broadly on track.

Has DSTG aligned its science and technology capabilities with Defence’s strategic priorities?

2.17 DSTG’s strategic plan states that the Group needs to build distinctive and sustainable capabilities which demonstrably create impact. Strategic initiative D2 strategic engagement with a client focus aims to improve the way DSTG takes into account Defence’s priorities at a strategic level.¹⁸

2.18 Following the release of its strategic plan, DSTG underwent two organisational restructures. In July 2013, DSTG reduced the number of its divisions from 11 to seven. The intention of this restructure was to better align the Group’s divisions with the major areas of Defence that need science and technology support.¹⁹ Between March and June 2014, DSTG introduced a standard approach for organising science and technology capabilities within the new divisions.

2.19 A Major Science and Technology Capability (MSTC) is now the fundamental unit of organisation within DSTG. DSTG named and defined each MSTC in defence science terms: that is, the science the MSTC undertakes plus the defence domain in which the science is applied. With the introduction of MSTCs, it was also DSTG’s intention to avoid any duplication of science and technology capability across more than one MSTC.

3.1 DSTG undertakes a wide range of science and technology work, in a range of scientific fields, across a geographically dispersed organisation. During audit fieldwork, the ANAO consulted with scientists from 11 of DSTG’s Major Science and Technology Capabilities (MSTC) and four of its Science and Technology Capabilities (see Figure 3.1 – the MSTCs visited by ANAO are coloured light blue). The diversity of science undertaken within these MSTCs includes researching: threats posed by weapons; aircraft structures; mobile communications; electromagnetic and environmental signatures made by ADF platforms; high frequency radar; patterns in data; and human/technology interactions. A summary of the science and technology work undertaken in each of the 11 MSTCs visited by the ANAO is in Appendix 2.

3.2 To support its staff in administering their science and technology work, DSTG developed a Management Information System (MIS) in 1987. This system is used to define work, store information, generate progress reports, and track spending. In 2007, DSTG upgraded its MIS to a web-based tool, accessed from DSTG’s intranet.

Figure 3.1: DSTG’s seven Divisions and 39 Major Science and Technology Capabilities

Source: DSTG.

3.3 DSTG’s strategic plan recognises the importance of improving the Group’s processes for planning, managing and reporting its science and technology work. One of the success measures for strategic initiative D2, strategic engagement with a client focus, is to make these processes more efficient, and ensure they are used consistently across DSTG.²⁰

3.4 In the first year of strategic initiative D2, between April and July 2014, DSTG engaged an external contractor, Ernst and Young, to assess the current state of its business processes. Ernst and Young found that:

• DSTG’s business processes varied significantly across the Group, and these processes were not well documented, communicated or followed;
• to report on the progress of their work, DSTG personnel often needed to manually manipulate data from various spreadsheets and reconcile data between various databases of DSTG’s MIS; and
• Secret and Top Secret information cannot be stored on DSTG’s MIS resulting in the details of work which contained classified information being stored elsewhere.²¹

Does DSTG effectively use its Management Information System?

Management Information System

3.5 Within its MIS, DSTG uses client requirements to record Defence’s science and technology needs. DSTG then assigns these client requirements to either a new or existing task. Tasks are the basic unit DSTG uses to manage its program of work; they are managed by a Task Leader and usually involve coordinating the work of scientists from different MSTCs and divisions. More than one client requirement can be attached to a task. The task’s activities describe the work that needs to be done to produce the deliverables, and the deliverables are DSTG’s products. Deliverables address a specific client requirement and are intended to be unique and verifiable (see Figure 3.2).

Figure 3.2: DSTG’s client requirement, task and deliverable structure

Source: ANAO analysis of DSTG data.

3.6 The MIS provides a sound structure for recording, organising, tracking and reporting on DSTG’s science and technology work. As at March 2015, DSTG had 1033 approved client requirements, 133 tasks and 1211 deliverables in its MIS.

DSTG’s use of its Management Information System

3.7 To assess DSTG’s use of its MIS, the ANAO analysed data from the MIS and reviewed, in more depth, a sample of 60 deliverables and associated client requirements.²²

DSTG is not effectively using the hierarchical structure in its Management Information System

3.8 DSTG mostly uses its MIS to record, organise, track and report on individual pieces of science and technology work. As a result of this focus on individual items of work:

• as at March 2015, there were a similar number of client requirements and deliverables in the MIS;
• between February 2013 and February 2015, 47 per cent of client requirements in the MIS had just one deliverable associated with them; and
• in the sample of 60 deliverables and client requirements reviewed by the ANAO, 15 per cent of the client requirements were defined so specifically they were indistinguishable from their associated deliverable.

3.9 DSTG does not effectively use its MIS to group individual pieces of work and link each group to a strategic goal. This makes it difficult for DSTG’s senior managers to use information in the MIS to oversee the strategic direction of the Group’s science and technology program and to understand how DSTG’s resources are being allocated and used to achieve its strategic goals.

3.10 DSTG recognises that it needs a more tiered approach to structuring its work. As part of strategic initiative D2, DSTG is proposing to establish a process for managing the investment of resources across the Group. DSTG intends that MSTCs would have between one and five investment goals, which convey the strategic intent of each MSTC, rather than the large number of client requirements each MSTC currently manages.²³ DSTG also recognises that it needs to regularly review these investment goals and determine whether each goal should continue, be changed, completed or terminated.

DSTG does not distinguish between different types of deliverables

3.11 DSTG’s deliverables vary widely in terms of type of work undertaken and the length of time allocated to complete them. The ANAO identified five main types of deliverables in DSTG’s MIS (see Table 3.1). These five deliberable types would benefit from different management approaches to ensure successful delivery. However, DSTG has not defined, at a strategic level, the management oversight appropriate to each type, and the five deliverable types are not differentiated within the MIS. Of the deliverables captured in DSTG’s MIS in March 2015, the length of time DSTG had allocated to complete them varied from less than a year to over nine years. In total, 87 per cent of deliverables were expected to take more than a year to complete.

Table 3.1: Five types of deliverables

Note a: A project has a defined beginning and end and is undertaken to accomplish a specific goal.

Source: ANAO analysis of DSTG data.

Poorly defined deliverables

3.12 The ANAO also assessed whether the deliverable descriptions used in the MIS clearly defined the output that DSTG intended to deliver to Defence and its purpose. Some 60 per cent of DSTG’s deliverables in the sample examined by the ANAO were either poorly or only partially defined. Examples of poorly defined, partially defined, and well defined deliverables are shown in Table 3.2. Including unclear or inconsistent deliverable descriptions in the Management Information System reduces the system’s value as a source of key management information.

Table 3.2: Examples of poorly defined, partially defined and well defined deliverables

Source: ANAO analysis of DSTG data.

Does DSTG have sound processes for monitoring, managing and delivering its science and technology work?

3.13 As discussed, DSTG has not determined, at a strategic level, appropriate management oversight arrangements for its science and technology work. In practice, DSTG relies on processes developed locally in its MSTCs for this important management function.

3.14 During 2014–15, each of the 11 MSTCs that ANAO visited was responsible for 60 deliverables on average. Of those 60 deliverables, on average, 11 were new to each MSTC and eight were completed by each MSTC in that year. The ANAO discussed with these 11 MSTCs the processes they use to monitor and review deliverables, and report progress to clients.²⁴

3.15 Appropriately, all the MSTCs placed an emphasis on communicating the progress of work to clients. This communication took place through various means, including:

• annual task reviews and client science and technology conferences;
• other formal meetings including steering groups, working groups, project teams and committees; and
• frequent interactions between desk officer and DSTG scientists by phone, email or in person.

3.16 The 11 MSTCs also described to the ANAO the internal processes they use for monitoring and reviewing work (see Table 3.3). These processes mostly involve face-to-face meetings of the MSTC leader and the leaders of the Science and Technology Capabilities (STC) within the MSTC, or meetings of all staff in each STC. The outcomes of these meetings are mostly documented in notebooks, meeting minutes, briefings or spreadsheets.

Table 3.3: Examples of process used by MSTCs to monitor and review science and technology work.

Source: ANAO analysis of DSTG information.

3.17 Individual scientists consulted by the ANAO were able to speak knowledgeably about the progress of their science and technology work. However, the differences in local processes limits the assurance that DSTG can obtain, at a strategic level, that the progress of individual pieces of science and technology work is being consistently assessed against planned milestones or other metrics, or how issues arising from such assessments are addressed. Action to address issues may involve, for example, changing a project’s scope or timeline, or delaying, transitioning or cancelling a project.

3.18 To supplement these local processes, DSTG personnel are required to generate formal reports on the progress of their work using information in the MIS three times per year. These reports, known as ‘desk officer’, ‘one-star’ and ‘two-star’ reports, are reviewed by senior DSTG managers and are provided to appropriate one-star and two-star managers within each Defence client.²⁵

3.19 The desk officer reports include information on the status of deliverables. There is a status field in DSTG’s MIS for deliverables that includes the options: on-track, behind schedule, cancelled and completed. For deliverables reported in DSTG’s MIS as at March 2015:

• 76 per cent were recorded as on-track or ahead of schedule;
• 11 per cent were recorded as late;
• 8 per cent were recorded as having been completed; and
• 5 per cent were recorded as having been transferred or cancelled.

3.20 For the sample of 60 deliverables reviewed, the ANAO observed that:

• For most of the deliverables in the sample reported as being either ‘on-track’ or ‘ahead of schedule’, the value of this status report was diminished because:
  • the description of the deliverable itself was poorly defined and it was not clear what was to be delivered;
  • the time-frame given for the deliverable to be completed was several years and no intermediate milestones were identified;
  • the due date for completing the deliverable had passed; or
  • the type of work being undertaken (for example, on-going quality assurance testing, activity based work or responding to requests from clients for science and technology advice as required) did not fit with a status report of ‘on-track’.
• For many of the deliverables in the sample reported as being behind schedule, the status had been agreed to by the client to allow DSTG to undertake higher priority work for them. However, there was no clear distinction made between deliverables where lateness was a concern for the client (and therefore needed closer monitoring by DSTG management) and those deliverables where the client had agreed to a longer schedule.

3.21 DSTG has been aware of problems with the data collected in its MIS since at least 2009. At that time, DSTG commissioned a Rapid Prototyping Development and Evaluation²⁶ Quicklook to review whether DSTG’s MIS was comparable to industry best practice. The review found that the quality of data in DSTG’s MIS varied greatly and that this variety reduced the quality and accuracy of the reporting that came out of the MIS. The review also found that DSTG staff often use local processes and systems to manage their work and viewed the need to enter data into the MIS as additional work. The review concluded that in similar industries, information systems are the primary source of information for reporting to management, but this was not the case with DSTG’s MIS. Furthermore, the review found that there was not corporate acceptance of the value of the MIS across DSTG. It recommended that DSTG undertake additional work to improve its MIS, focusing on:

• the metrics needed at different levels of management within DSTG and how the MIS can be used to measure these metrics; and
• the reliability and accessibility of data held in the MIS.

3.22 The ANAO’s findings in this audit indicate that MIS data quality remains variable and this variability limits the value of the system in supporting strategic oversight of DSTG’s science and technology work.

4.1 One of the key goals of DSTG’s strategic plan is for DSTG to become more collaborative. The plan states that DSTG will leverage world-class capabilities, and gain access to a wide variety of skills through strategic alliances and partnerships with defence industry organisations, academia and other research organisations, and that these partnerships will assist DSTG in developing and commercialising its inventions.

4.2 In April 2013²⁷, the Chief Defence Scientist appointed an external panel²⁸ to conduct a review of DSTG’s external collaborations and provide guidance on how the Group could improve them. In July 2013, the panel made 52 recommendations. In summary, the panel recommended that DSTG:

• reform its Business and Commercialisation Office;
• improve its external engagement culture;
• make its systems and processes for engaging with external organisations more active and efficient;
• use performance measures to monitor outputs, outcomes and impacts of engagement activities; and

retain some commercial revenue from its inventions.

4.3 DSTG accepted 49 of the 52 recommendations, with some modifications to 13 of them. DSTG developed 10 engagement initiatives²⁹ to address these recommendations and, in December 2013, began a three year plan to implement these initiatives. To date, DSTG has revived its strategic frameworks for collaborating with industry, developed processes to oversee its participation in collaborations with external organisations, and replaced its Business and Commercialisation Office with its Technology Partnerships Office. This new Office is intended to be a hub of engagement expertise within DSTG and, in particular, it is to:

• professionalise and simplify collaboration practices;
• establish a strategic framework for collaborating with external organisations;³⁰ and
• improve DSTG’s processes for collaborating with external organisations.

Is DSTG applying a more strategic approach to the way it collaborates with industry and other science organisations?

Strategic alliances with industry

4.4 Since August 2013, DSTG has signed nine new strategic alliances with strategically important industry organisations. The purpose of these alliances is to formalise long-term relationships between DSTG and these organisations. Each alliance identifies particular areas of cooperation between the two parties³¹ which have the potential to form the basis of joint projects. As at October 2015, DSTG had nine such projects underway (see Table 4.1) and has approved a further 14 projects, which have not yet started.

Table 4.1: DSTG’s strategic alliances with industry

Source: DSTG.

4.5 In September 2015, DSTG’s Technology Partnership Office advised the ANAO that during the first 12 months of the alliances, the following key outcomes were achieved: a greater understanding by industry organisations about DSTG’s future research priorities, and the relationship between these priorities and Defence projects; and a greater understanding by DSTG of industry’s capabilities and its research and business priorities. DSTG’s Technology Partnerships Office stated that:

[Strategic] alliances are long term partnership relationships, which take time to develop. This is particularly [the case] when alliance projects do not involve the transfer of [money] between the organisations, but rely [instead] on in-kind and collaborative engagement. For alliance relationships to be successful, the key foundations … must be established. As the relationship builds, business and research interests intersect and joint projects commence.

4.6 Overall, while the fundamentals of the DSTG’s strategic alliances are sound, the alliances are still in the very early stages of producing science and technology work for Defence. DSTG also acknowledges that there are challenges in attracting industry partners to collaborate on projects.

Measuring the performance of DSTG’s strategic alliances

4.7 DSTG runs a graduate program in science leadership and in 2014, a group of graduates on this program developed, as a project, a comprehensive set of metrics to measure the performance of DSTG’s collaborations with external organisations. Following on from this work, DSTG is currently trialling a set of 61 metrics to measure the performance of each project undertaken within a strategic alliance and the overall health of each alliance. These metrics are based on five tenets of collaborative success: communication; trust; commitment to the collaboration; leadership and governance; and outputs and outcomes. Performance of these tenets is assessed at three levels—line worker, middle management and senior management—using a five point scale.

4.8 As at October 2015, DSTG had discussed with two of its strategic alliance partners, BAE and ASC, trialling these metrics before the end of 2015. DSTG intends to roll out the metrics to other alliances during 2016. Using these metrics will provide DSTG with useful data for managing and reviewing the performance of its strategic alliances.

Defence Science Partnerships with universities

4.9 In August 2013, the DSTG Leadership Team noted the importance of the Group developing closer ties with the university sector. Consequently DSTG developed a Defence Science Partnerships Program which outlines a system of collaborating with universities to produce mutually beneficial research. DSTG expects:

that these long term partnerships will allow academics to better understand Defence needs and priorities, will result in research outcomes with greater impact and relevance, and will achieve more efficient use of national resources by aligning with other research funding sources.

4.10 Defence Science Partnerships are bilateral agreements between DSTG and a particular research institution. Each Defence Science Partnership is managed by an executive committee, composed of equal numbers of members from each party, and is to meet at least twice a year. The committee provides strategic and policy direction and will evaluate the partnerships against agreed performance measures.³²

4.11 The parties to each Defence Science Partnership negotiate activity agreements which lead to joint projects. These activity agreements generally use one of 12 standard templates. If there is no relevant template, the parties negotiate a speciality agreement. As at September 2015, DSTG was conducting work, through 159 activity agreements, with 28 universities.³³ These agreements are in addition to the 222 agreements between DSTG and universities that were in place before the Defence Science Partnerships were introduced.

4.12 In May 2015, at DSTG’s Partnerships Week³⁴, DSTG’s Director of University Engagement³⁵ noted that DSTG has experienced some difficulty in implementing its new Defence Science Partnerships in a timely fashion due to: personnel in both DSTG and universities still familiarising themselves with processes; some universities undertaking extensive legal reviews of the partnership agreements; and delays in securing all signatures. In spite of these difficulties, processing times for new agreements have fallen from an average of 86 days in 2013–14 to 38 days in 2014–15. DSTG advised the ANAO that it started a review of its Defence Science Partnerships in July 2015.

Strategic relationship agreements with publicly funded research entities

4.13 DSTG has established strategic relationship agreements with several publicly funded research entities—the Commonwealth Scientific and Industrial Research Organisation (CSIRO) in June 2013, the Australian Nuclear Science and Technology Organisation (ANSTO) in 2006, and the Bureau of Meteorology (BOM) in August 2013. These strategic relationship agreements involve sharing resources, personnel and facilities to perform collaborative projects and are overseen by a senior steering committee. To date, DSTG has undertaken a number of joint research projects with CSIRO and ANSTO³⁶, exchanged staff with CSIRO through a secondment program, developed training programs with ANSTO, and conducted forums with BOM.

4.14 DSTG’s Technology Partnerships Office advised the ANAO that it intends to introduce standard agreements for DSTG’s publicly funded research entity collaborations, similar to the Defence Science Partnerships.

International collaborations

4.15 At the time of the audit, DSTG’s mechanisms for collaborating with international organisations mainly took the form of Memoranda of Understanding, treaties, or similar instruments to participate in bilateral and multilateral research projects with other countries.³⁷ Several senior DSTG scientists informed the ANAO that, for relatively little investment, DSTG is able to access significant benefits through these agreements, including access to new technologies, research and facilities. In 2012–13 DSTG spent $24.1 million on its international collaborations, increasing to $27.2 million in 2013–14.

4.16 In October 2014, the DSTG Leadership Team identified that while the international engagement arrangements:

serve the purpose for which they were designed, what is currently missing from all of DSTO’s international activities is a holistic system for managing all of DSTO’s international relationships, aligned to strategic direction, to meet desired outcomes.

4.17 To this end, DSTG is developing an international engagement framework to:

• align DSTG’s international engagement to DSTG’s strategic direction;
• support Defence’s international policy initiatives, and its Capability Plan and the Approved Major Capability Investment Program;
• support formal international agreements; and
• build relationships that will support new capabilities, and will strengthen regional ties and security in Australia’s region.

Is DSTG actively developing a culture of collaborating with external organisations?

4.18 To help promote a culture of external collaboration, DSTG has trained one person in each division to become an external engagement manager. These managers are responsible for supporting staff to develop and manage external relationships. In particular, they:

• coordinate the divisions’ external agreements with industry, academia and government bodies;
• run staff training sessions on collaborating with external organisations;
• assist in implementing strategic initiative T1, fostering innovation; and
• update and maintain the divisional agreements in DSTG’s web-based customer relationship management system.

4.19 DSTG has had difficulty collaborating with small-to-medium enterprises (SME) to commercialise its inventions. Traditionally, DSTG used licenses that were up to 30 pages long, to enable industry to use its research to support Defence capability and the Australian economy. DSTG considers that SMEs often found these licenses to be a prohibitive barrier to collaboration.

4.20 To enable SMEs to more easily commercialise some of DSTG’s inventions, the Technology Partnerships Office has refreshed its suite of collaboration and commercialisation agreement templates. One of these templates simplifies access to DSTG’s intellectual property whilst complying with the Commonwealth and Defence policies, in a document of three pages.³⁸ Another innovation is a collaborative R&D template which DSTG has recently rewritten using plain language with the intention of making it easier for SMEs to use.³⁹

4.21 Furthermore, in May 2015, DSTG replaced its numerous divisional contracting panels with a single point of contact for SMEs wanting to collaborate with DSTG. The Research, Scientific, Engineering and Other Technical Services (ReSET) panel consists of a number of SMEs that provide standing offers of services to DSTG in 22 skill categories. The intention of this panel is to make it easier for DSTG to use the capabilities of these SMEs. As at October 2015, DSTG had arranged 133 contracts valued at $17.6 million with SMEs using the ReSET panel.

4.22 In January 2014, DSTG started to offer its staff opportunities for secondment to industry for up to 36 months. The aim of these secondments is to build stronger links with industry, give industry access to DSTG’s research capability, and increase the skills of DSTG staff. During the secondment the salary of the DSTG employee is paid for by the host organisation, and any intellectual property generated by the DSTG employee remains the property of the host organisation. As at September 2015, DSTG had seven staff on industry placements.

Does DSTG have sound arrangements to manage risk when collaborating with external organisations?

Defence internal audit of procurement practices

4.23 In March 2015, Defence Internal Audit prepared an audit of aspects of DSTG’s interactions with external parties. The audit focussed on working-level practices, rather than DSTG’s strategic framework for collaborating with external organisations. The report concluded that:

DSTO’s achievements at a strategic level … have been undermined by poor procurement practices, which have exposed Defence to unnecessary risk, cost and non-compliance with the [Commonwealth Procurement Rules].

4.24 The audit made seven recommendations and in June 2015, DSTG advised ANAO that it had completed implementing five of the seven recommendations, and parts of another one. While DSTG has made good progress in addressing the recommendations of the internal audit, the Group needs to maintain its focus on improving procurement practices.

4.25 In addressing these recommendations, DSTG promoted its Defence Science Partnerships program as a way in which DSTG is applying more rigour to its interactions with external parties. DSTG highlighted the good practice of the program’s standardised templates, common costing model and intellectual property arrangements. DSTG also noted that it has developed procurement and evaluation plans. These plans walk staff through the steps required for carrying out simple and complex procurements, and evaluating tender responses. DSTG has also committed to improving its process for approving prepayments of equipment or services and signing and amending contracts.

DSTG’s probity arrangements

4.26 In 2011, DSTG established the DSTO Probity Board to produce a probity framework for its interactions with industry, and to provide risk management advice to the Chief Defence Scientist on specific industry engagements. The Board produced a guide which provided: advice on when to engage with industry; a set of guiding probity principles and examples of where conflicts of interest could arise; and examples of the types of industry interactions in which probity concerns might arise.

4.27 Once the guide was delivered DSTG abolished the Board, at the request of the then Assistant Minister for Defence, as a savings measure.⁴⁰ The Chief Defence Scientist proposed that its functions be replaced by:

• introducing an agenda item for consideration of probity issues at DSTO Advisory Board Meetings⁴¹;
• having the chair of the Probity Board join the DSTO Advisory Board;
• seeking advice on probity issues from former members of the Probity Board; and
• having the DSTO Advisory Board consider advice received on probity issues.

4.28 The Assistant Minister for Defence agreed to the Advisory Board considering probity issues, but did not agree to the chair of the Probity Board joining the advisory board. In April 2015, the report of the First Principles Review of Defence recommended that DSTG disband its Advisory Board and Defence agreed with this recommendation.

4.29 In October 2015, DSTG advised the ANAO that it is now using Defence’s probity arrangements. These arrangements comprise guidance from the Department of Finance and specific Defence policies which address areas such as managing conflicts of interest and accepting gifts and hospitality. At the time of the audit, DSTG also had specific probity measures in place for some of its engagement mechanisms. For example, strategic alliance management committee meetings consider probity issues as a standing agenda item and DSTG advised that the Chief Defence Scientist can request probity advice on specific issues from independent experts.⁴² There would be merit in DSTG updating its strategic plan to align with these revised arrangements, as the plan currently identifies an ‘innovative framework for engagement that actively uses advice from the DSTO Probity Board’ as a success measure for implementing its partnership role.⁴³

Does DSTG have sound processes for demonstrating how its science and technology work aligns with Defence’s strategic priorities?

5.1 One of DSTG’s key performance indicators in Defence’s Portfolio Budget Statement 2014–15 addressed the alignment of DSTG’s science and technology work with Defence’s strategic priorities. The key performance indicator was that:⁴⁴

[DSTG’s] applied research program is strategically balanced and aligned with the needs of Defence in support of operations, the current force, capability development and acquisition, and the needs of national security agencies.

5.2 Defence reported in its Annual Report 2014–15 that DSTG met this key performance indicator.⁴⁵

5.3 DSTG undertakes an annual process to develop its program of science and technology work for the coming financial year. To improve how well its science and technology work aligns with the strategic priorities of Defence, in July 2013, DSTG organised its clients⁴⁶ into six domains and appointed Domain Program Managers, who are at a Chief of Division level, to oversee the science and technology program for each domain.⁴⁷ The role of the Domain Program Managers during this annual process is to consult with two-star level managers within Defence to develop a prioritised list of science and technology needs for each domain.

5.4 As discussed in Chapter 3, DSTG records its science and technology work in its Management Information System (MIS) as a cascading series of records: client requirements, tasks and deliverables. As part of DSTG’s annual planning process, DSTG’s clients review their existing client requirements and determine whether these requirements are still relevant and prioritised correctly, or whether they need to be amended or cancelled.

5.5 The list of updated client requirements is then reviewed by DSTG’s Chief of Program Office and his staff who consider the impact of whole-of-government and Defence strategic priorities on each client requirement. In its MIS, DSTG then:

• links each client requirement to one of the ten DSTG roles listed in its strategic plan⁴⁸;
• assigns the client’s priority to each client requirement (priority one, two or three)⁴⁹; and
• describes the Defence and whole-of-government priority of each client requirement.

5.6 While this is a sound process, DSTG has a large number of client requirements and deliverables in its MIS and it does not effectively aggregate information about these client requirements and deliverables to quantitatively demonstrate the extent to which its science and technology program, as a whole, aligns with the strategic priorities of Defence. DSTG recognises this shortcoming and, as discussed in Chapter 3, the Group intends to introduce a more tiered approach to structuring its work. DSTG advised the ANAO that this approach would involve each MSTC identifying a small number of investment goals, between one and five, which would convey the strategic intent of each MSTC more clearly than the large number of client requirements that each MSTC currently manages.

Does DSTG have sound processes for assessing its science and technology capabilities and client satisfaction?

5.7 One of DSTG’s key performance indicators in Defence’s Portfolio Budget Statement 2014–15 addressed DSTG’s processes for assessing its science and technology capabilities and its client satisfaction. The key performance indicator⁵⁰ was that:

[DSTG’s] science and technology capability is contributed to by a workforce with world-class expertise and facilities, measured through benchmarking and client feedback.

5.8 Defence reported in its Annual Report 2014–15 that DSTG had met this key performance indicator.⁵¹

Benchmarking program of MSTCs

5.9 A key action of DSTG’s strategic initiative D1, science and technology excellence is to implement an annual rolling program to benchmark DSTG’s Major Science and Technology Capabilities (MSTCs). DSTG started its benchmarking program in 2014. The program consists of annual internal reviews of all MSTCs, and a program of rolling external reviews for a quarter of MSTCs each year. DSTG intends this benchmarking program to inform its MSTC investment decisions, assist MSTC leaders to manage their capability, and to inform the Group’s Leadership Team about the overall health of each MSTC.

5.10 The internal review involved 22 questions addressing the following six dimensions, rated from 1 – 5:

• strategy, planning and leadership including business and governance processes;
• delivery and impact to Defence, National Security and other areas outside the MSTC;
• quality and technical review of research including its intrinsic merit and peer recognition;
• engagement and partnering with internal and external stakeholders;
• research infrastructure and capabilities to undertake current and future work; and
• innovation and future focus including the balance between fundamental scientific research and research applied to client work.

5.11 The MSTC leader, in consultation with their staff, conducts the internal review of their own MSTC. Table 5.1 shows the combined average results for all MSTCs of internal benchmarking against each of the six dimensions assessed.

Table 5.1: Internal benchmarking results against the six dimensions

Source: DSTG.

5.12 Of the 22 questions, the highest scoring question was the one dealing with client satisfaction, with an average score of 3.9. The lowest scoring question dealt with training, with an average score of 1.7.

5.13 The external assessments are undertaken by a panel of reviewers, comprising external experts and a number of MSTC leaders from other MSTCs. The panels also address the six dimensions, using the same 22 sub-questions as the internal reviews. In 2015, nine MSTCs were reviewed by an external panel.

5.14 The panels examine background materials prepared by the MSTC leader, which appropriately do not include the results of the MSTC’s internal review. In 2015, each external assessment was conducted over a week, and included a pre-review meeting between the Chief Defence Scientist and the panel, discussions with and presentations from the MSTC’s staff, and a further meeting with the Chief Defence Scientist on the last day of the assessment.

5.15 Figure 5.1 shows the combined average results for the nine MSTCs externally reviewed against each of the six dimensions assessed. The external review results are broadly consistent with the internal benchmarking results.

Figure 5.1: External benchmarking results

Source: ANAO analysis of DSTG data.

Measuring client satisfaction

5.16 DSTG regularly consults with, and seeks feedback from, its clients. To formalise this feedback, one action of DSTG’s strategic initiative, D2 strategic engagement with a client focus, is to ‘implement an improved client feedback loop on the quality and timeliness of DSTG support by the end of 2013–14’.

DSTG’s 2013 client satisfaction survey

5.17 DSTG conducted its first client survey of Defence desk officers in November 2013. Desk officers are the first point of contact for DSTG Task Leaders when discussing the work they are undertaking.⁵²

5.18 DSTG surveyed 265 desk officers, and 30 per cent responded. DSTG attributed this low response rate, in part, to conducting the survey at the end of the year. The survey addressed:

• how well DSTG communicates and engages with its clients;
• how well DSTG is organised and managed;
• the quality and timeliness of DSTG’s outputs; and
• the quality of DSTG’s outcomes.

5.19 The survey responses indicated that DSTG’s clients were satisfied with DSTG. Desk officers were very satisfied with their working-level relationships with DSTG personnel and in particular, they rated having DSTG liaison personnel posted within Defence as extremely useful. Desk officers commented on difficulties in electronically sharing Top Secret information with DSTG personnel and noted this has become more of an issue with DSTG’s reduced travel budgets.

5.20 The desk officers were concerned with the length of time DSTG took to complete deliverables. Related to this concern, desk officers were also critical of DSTG’s project management of deliverables, citing scope creep as a particular problem. In response to this feedback, DSTG refreshed its Task Leader training and delivered it to Task Leaders in the first quarter of 2015. DSTG intends to deliver this training annually to Task Leaders.

DSTG’s 2014 client satisfaction survey

5.21 DSTG next surveyed Defence’s desk officers a year later and the response rate increased to 40 per cent. The survey addressed similar themes as canvassed in the previous year.

5.22 Again, the survey responses indicated that desk officers were very satisfied with their working-level relationship with DSTG personnel. Desk officers were concerned about DSTG’s approach to reporting the progress of work, the length of time DSTG took to complete deliverables, and DSTG’s approach to project managing deliverables.

Canvassing the perspectives of senior management within Defence

5.23 Between May and August 2014, DSTG’s Domain Program Managers held discussions with two-star level Defence leaders to understand how satisfied they were with DSTG. These discussions were less useful than expected as senior defence personnel were not always available and did not fully engage with the process.

5.24 In December 2014, the Group’s Leadership Team decided that discussions with senior defence managers would, in future, form part of the annual planning discussions between senior defence managers and the DSTG Domain Program Managers (see paragraph 5.4).

Does DSTG have sound processes for demonstrating the outputs and outcomes of its science and technology work?

5.25 Three of DSTG’s key performance indicators in the Defence Portfolio Budget Statement 2014–15 addressed the outputs and outcomes of DSTG’s science and technology work. These key performance indicators⁵³ were that:

• [DSTG’s] advice to Defence and the Government on science and technology matters is valued through its contribution to improved Defence and national security outcomes;
• [DSTG’s] research program outputs enable enhanced Defence and national security capability, treat risks and resources; and
• [DSTG’s] program outcomes are delivered on time, in scope and within agreed resources.

5.26 The Defence Annual Report 2014–15 reported that DSTG met the first two of these key performance indicators during that year. Defence also reported that DSTG had only substantially met the third of these key performance indicators, observing that ‘budget management and rebalancing within the department led to Defence clients agreeing to some medium and lower priority tasks being cancelled or deferred.’⁵⁴ DSTG advised the ANAO that it arrived at this assessment of its performance by considering formal and informal client feedback and the outcomes of its:

• annual planning process;
• ongoing arrangements for renegotiating use of the Group’s resources and capabilities with its clients; and
• MSTC benchmarking program.

5.27 In March 2015, DSTG engaged ACIL Allen Consulting to undertake an economic study of the value DSTG delivers to Defence in terms of cost savings, capability enhancements and Australian export sales. DSTG had previously undertaken a similar economic study in 2003. That study calculated that the economic benefit resulting from six exemplar DSTO projects over the previous 10 years was $6.5 billion.

5.28 In 2015, ACIL Allen assessed the economic benefits of 10 selected projects DSTG had undertaken since 2003. ACIL Allen calculated the economic benefits of these projects as $5.1 billion. ACIL Allen acknowledged several limitations in its estimates of the economic benefits and costs of the projects. In particular, the limitations included that the consultants had:

• valued the benefits and costs in 2015 dollars, because it was difficult to establish the actual year the cost occurred or benefit was realised;
• used only unclassified data, and so could not make a complete assessment of the benefits of the projects;
• undertaken limited work to determine the value of DSTG’s options portfolio (that is, the value arising out of DSTG’s work when it generates new options to support Defence capability and decision making); and
• found costs of projects more difficult to estimate than benefits due to the poor quality of DSTG’s records.

Opportunities to strengthen reporting arrangements

5.29 Calculating the economic benefit of DSTG’s science and technology work in terms of cost savings, capability enhancements and Australian export sales is a measure of the value DSTG produces, but these high-level outcomes are difficult to measure. In part, this is because these outcomes are realised when DSTG’s science and technology work is being used by Defence. As such, they can only be measured some time after DSTG has delivered work to Defence.

5.30 To mitigate the difficulty of making timely assessments of high-level outcomes, as an intermediate activity, DSTG could also report on its efficiency and administrative effectiveness in managing science and technology work. For example, there is scope to report on: the number of products DSTG delivers to Defence in a given period of time; and the time taken and resources used to deliver these products. Such information should be available in DSTG’s Management Information System. This reporting activity would usefully complement DSTG’s periodic economic assessment of the high-level outcomes of its work, the benchmarking of its science and technology capabilities, and client satisfaction surveys. Improvements in the performance monitoring and reporting framework would also assist DSTG to address issues raised by its clients in relation to the Group’s approach to project management. By implementing Recommendation 1 in Chapter 3 of this audit report, DSTG would have a sound basis for this improved reporting.

Please refer to the attached PDF of the report for the Appendices:

• Appendix 1: Defence’s response
• Appendix 2: Major Science and Technology Capabilities visited by ANAO during audit fieldwork