Background

The terrorist attacks of September 11, 2001 in the United States of America introduced a new and confronting dimension to the international security environment. This was reinforced by subsequent terrorist attacks in Bali, Madrid, Jakarta, London and, most recently, again in Bali. Since the attacks in 2001, Australia's national counter-terrorism alert level has remained assessed at the ‘medium' threat level, meaning a terrorist attack within Australia could occur.

The Australian Government's response has been to strengthen and upgrade national security and the national counter-terrorism arrangements, involving the commitment of some $5.6 billion in additional budget funding since 2001. State and Territory governments have also strengthened and upgraded their arrangements.

Under the Constitution of Australia, the State or Territory in which a terrorist incident occurs holds the primary operational responsibility for the management of that incident. The first responders will be the respective State/Territory police and emergency services. In planning to respond to a terrorist incident the Australian Government and States and Territories' roles interconnect under the coordination mandate of the National Counter-Terrorism Committee (NCTC).

Two Australian Government agencies represented on the NCTC also provide administrative and secretariat support to the ongoing operations of the other various counter-terrorism committees. The Department of the Prime Minister and Cabinet (PM&C) has responsibility for the coordination of counter-terrorism policy, and the Attorney-General's Department (AGD) has responsibility for operational coordination of counter-terrorism, including the management of the NCTC's National Capability Development Exercise Programme ¹ (the Exercise Programme). Both agencies also have a significant role in arranging for the ongoing process of evaluation of Australia's national counter-terrorism arrangements.

Evaluations, reviews or assessments of activities such as the counter-terrorism coordination arrangements, play an important role in aiding judgments about the performance of the activity and its appropriateness, efficiency and effectiveness. The outcomes of evaluations can inform decision-making, form the foundation for continuing improvements and provide greater accountability. Given the significant funds committed, and efforts extended by all governments towards delivering the basic, viable nation-wide counter-terrorism capability, regular and structured evaluation should play a vital role in the improvement of, and accountability for, Australia's domestic counter-terrorism arrangements.

Thus, the focus of the audit was to review the main evaluation methods employed to assess the various aspects of the national counter-terrorism coordination arrangements. The key evaluation methods considered were:

• Australian Government commissioned evaluations or reviews;
• NCTC commissioned evaluations, reviews or assessments; and
• exercises conducted under the Exercise Programme.

The objectives of the audit were to:

• assess the effectiveness of the key evaluation methods used to review the efficacy of the Australian Government's national counter-terrorism coordination arrangements; and
• examine the effectiveness of the links between the key evaluation methods, and how the key evaluation methods contribute to the process of continuous improvement.

The audit criteria assessed:

• the strategic alignment between the key evaluation methods employed and the counter-terrorism coordination frameworks;
• aspects of the coverage of the key evaluation methods employed;
• the adequacy of the performance measures used for the evaluations;
• the available mechanisms for reporting the outcomes of the evaluations; and
• how the effectiveness of the national counter-terrorism arrangements is being increased through a formal process of continuous improvement.

The audit did not examine Australia's international counter-terrorism arrangements or the broader elements of national security. Nor did it attempt to assess the appropriateness of the current framework governing the coordination arrangements or the capability of participants to respond to a domestic terrorist incident.

Key findings

The framework for the national coordination of Australia's counter-terrorism arrangements (Chapter 2)

There are well defined frameworks in place for the coordination of both the Australian Government response and the national response for countering terrorism. Since September 2001 an increasing number of agencies at all levels of government, as well as the private sector, have been drawn into these frameworks.

The Australian National Audit Office (ANAO) concluded that the policy and operational coordination roles of PM&C and AGD respectively, for both the Australian Government and national frameworks, had been defined with respect to how the agencies would coordinate a response to any terrorist incident. Their roles in the provision of administrative and secretariat support to the ongoing operations of the various counter-terrorism committees established by the Australian Government or the NCTC had also been defined. With respect to the level of secretariat support provided by PM&C or AGD, the majority of agencies were highly positive, commenting that the agencies provided professional and proactive support. The ANAO noted the work underway to inform newly involved agencies with respect to the policy and operational coordination roles of PM&C and AGD as well as the national arrangements set out in the NCT Plan and Handbook . ²

Given the extensive agendas of the various counter-terrorism committees, the ANAO suggests that there would be benefit in the greater allocation of time for the committees to consider the strategic issues relating to the national counter-terrorism arrangements. The ANAO notes that, since audit fieldwork, specific meetings to consider strategic issues have recently commenced and that the review of the NCTC sub-committees offers further solutions that would afford the committees greater opportunity to focus on strategic issues.

The ANAO considers that there are opportunities for PM&C and AGD to further support the committees' greater focus on strategic directions through the provision of more coordinated reporting and analysis of the outcomes from the range of evaluations conducted. How this could be provided is addressed in detail in the report.

Nature and extent of the key evaluation methods used (Chapter 3)

The ANAO concluded that the national counter-terrorism arrangements are subject to frequent, ongoing and multi-level evaluations, ranging from reviews of the implementation of policy initiatives through to operational reviews. The evaluations include a mixture of commissioned reviews into particular functional aspects, major reviews of the arrangements and the practical, operational testing under the Exercise Programme. The ANAO further concluded that there were sufficient avenues for evaluation of the national counter-terrorism arrangements.

Strategic alignment between the key evaluation methods used and the national counter-terrorism coordination frameworks (Chapter 4)

The ANAO found that the evaluation strategies were aligned with the Australian Government and national counter-terrorism frameworks that they were intended to review, and that the evaluations covered matters of capability, coordination and the effectiveness of the frameworks.

However, there was limited effort taken to correlate strategies or objectives across the key evaluation methods used. While recognising that a fully integrated, whole-of-government evaluation strategy would be difficult and time consuming to develop and implement, the ANAO considers that there are interim measures that could be applied to provide a greater whole-of-government perspective to the existing evaluation methods employed.

The ANAO concluded that AGD and PM&C could better coordinate and amalgamate the outcomes arising from the various reviews and exercises and should also provide greater strategic analysis of these outcomes and recommendations. Such analyses could be used to effectively underpin greater efficiency in the whole-of-government efforts to improve the counter-terrorism arrangements.

In addition, the ANAO concluded that there would be benefit in establishing a more strategic approach to the planning of the Exercise Programme to better ensure that priority areas are tested and that each of the exercises within the programme better contributes to the continuous improvement of the national arrangements. The ANAO further concluded that, for recall and accountability purposes, the decisions made in the setting of the programme should be documented.

Evaluation coverage (Chapter 5)

Generally, the major reviews undertaken to evaluate aspects of the counter-terrorism arrangements, whether commissioned by the NCTC or by the Australian Government, took into account, and consulted with, relevant agencies during the conduct of the evaluations.

Opportunities to participate in the Exercise Programme had initially not kept pace with the rapid expansion in the number of agencies involved in counter-terrorism and the broadened scope of the exercises, but the ANAO concluded that PM&C and AGD had identified this issue and were taking steps to encourage the newly involved agencies. The ANAO also noted the comments by AGD and PM&C that agencies do not always take up the opportunity to participate, but further concluded that a more structured approach to the planning and setting of the annual Exercise Programme, with advance specification of the scenarios and exercise objectives, would assist agencies to better plan their participation.

Although the Exercise Programme has a focus on coordination as well as capability, the mechanisms for effectively evaluating coordination and for assigning responsibility for subsequent corrective action have not been formalised to the same extent as those for the NCTC designated capabilities. ³

The ANAO concluded that the national counter-terrorism arrangements would benefit from a greater focus on inter-agency coordination and the creation of more transparent and systematic mechanisms to facilitate taking coordination issues through a continuous improvement cycle model. The ANAO considers that the NCTC Exercise Lessons Learned Database being developed by the AGD would eventually be an important component in this regard.

The ANAO considers that, in light of the increased complexity and magnitude of the Exercise Programme, it could be difficult to sustain high levels of agency participation, and hence the degree of exercise coverage achieved to date. The ANAO concluded that there could be benefits in the use of more contemporary and varied training aids, which may allow for efficiencies and flexibility in the delivery of the programme and thus may contribute to an expanded coverage, and sustainability, of the programme. Such aids may include the use of computer-based simulations or videos. The AGD, in conjunction with the NCTC exercise management capability advisers, should explore options for the incorporation of such aids into the Exercise Programme.

Performance measures used for evaluations (Chapter 6)

The ANAO observed that the exercises fulfil a multiplicity of valuable functions that include evaluating, testing, practising and training, the building of intra-agency and inter-agency relationships, providing a valuable opportunity to share information and increasing awareness of the roles and functions of the various other agencies. All of these are important factors that contribute to the ability of agencies to respond collectively in a crisis.

However, the ANAO considers that the work in progress to clarify the definition of the ‘basic, viable nation-wide capability' ⁴, and the development of operational plans to better specify how capability is to be achieved, would together provide an overarching ‘common goal' for the counter-terrorism agencies to work towards in a whole-of-government context. The common goal would provide a framework against which agencies can measure and report in a whole-of-government context.

Greater alignment between the objectives and performance indicators of the participating agencies with the broad exercise objectives and indicators and those of other relevant participating agencies would:

• reduce the risk of issues of inter-agency performance being overlooked; and
• allow for a more tightly focussed assessment of agencies' ability to effectively deliver the aspects of the NCT Handbook being tested.

The ANAO concluded that the training and testing components within the exercises conducted under the Exercise Programme should be more clearly identified and differentiated. Training tends to target the performance of the staff and people involved and to be accompanied by a coaching approach, and testing and evaluation focuses on the effectiveness of the procedures and the capacity of the trained staff to deliver them. The identification and differentiation of the training and testing components would allow the exercises to be used more effectively as a means of evaluating the capability and coordination required to deliver the agreed, predetermined level of basic, viable nation-wide capability.

Mechanisms for reporting the outcomes of evaluations (Chapter 7)

The ANAO found that the reporting mechanisms for the reviews and exercises cater for the majority of government stakeholders involved in counter-terrorism. Similarly the ANAO noted that additional links have been built, or are being built, to other coordination frameworks, such as the Australian Health Disaster Management Policy Committee and the Australian Emergency Management Committee.

The ANAO also found that formal reporting mechanisms were in place that generally allowed for the timely reporting of the costs and other outcomes of major reviews and exercises. However, for recall and accountability purposes, the ANAO considers that when a review is delayed, the reasons for these delays should be better documented.

The ANAO noted that the work being progressed by the NCTC combined with the more stringent setting, measuring, and hence reporting of performance against objectives and performance indicators, and greater data analysis, would allow for more comparable evaluations in the future.

Effective use of counter-terrorism evaluations as a basis for continuous improvement (Chapter 8)

The ANAO considers that, in accordance with the IGA, those involved in the delivery of counter-terrorism capability had, since 2001, exerted significant effort across a broad range of fronts to strengthen Australia's capability. New agencies have been brought into the arrangements. Numerous reviews and exercises have been conducted, all focussed on identifying areas for improvement, and aimed at continually strengthening the arrangements and the way the agencies work together.

The ANAO found that a process for continuous improvement was largely in place for the counter-terrorism arrangements, but that all elements of the process could benefit from further strengthening, better coordination or greater transparency.

The ANAO considers that strategic analysis of the recommendations arising from the reviews and exercises would facilitate a more streamlined approach for the agencies implementing the corrective actions, so that the current arrangements to respond to terrorist incidents reflect the accumulation of the best knowledge and experience available.

The ANAO further found that little was done to collectively track the implementation actions, but notes that the NCTC Exercise Lessons Learned Database being developed by the AGD will be of assistance in this regard. The ANAO concluded that the database should be expanded to allow for the integration of implementation actions arising from the exercises with those arising from the major reviews.

In addition, the ANAO considers that the database should be reviewed to ensure that its capacity is commensurate with the functions it will be required to perform. The ANAO further considers that it is important that adequate priority and resources are assigned to the Evaluation Section within AGD and the development and management of the Lessons Learned Database.

Overall audit conclusion

Overall, the ANAO concluded that a range of strategies and processes were in place to evaluate the national counter-terrorism coordination arrangements and to ensure that the arrangements were subject to regular evaluation and continuous improvement. Since September 2001, the arrangements have undergone continuous and rapid evolution to embrace a widening circle of relevant agencies and organisations as well as continuous re-positioning to meet emerging risks and threats.

The policy and operational coordination functions undertaken by PM&C and the AGD are important contributions to the national coordination arrangements. Between them, the two agencies provide a wide range of services and support and, where necessary, guidance, to the counter-terrorism committees and the range of agencies across governments and jurisdictions.

The ANAO found that, generally, strategies and processes were in place for evaluation of the arrangements and the management of continuous improvement. The ANAO has identified some aspects of the evaluation strategy and the continuous improvement cycle that require strengthening, better coordination and greater transparency. To achieve this, PM&C and AGD will need to take a greater role in the management of evaluations and the strategic analyses required to use the outcomes to best effect.

The ANAO acknowledges that the effective measurement and evaluation of performance across multiple functions, jurisdictions and agencies represents a significant challenge. The ANAO has made nine recommendations arising out of this audit aimed at improving the efficiency and effectiveness of the key evaluation methods used to assess the national counter-terrorism coordination arrangements and their ongoing improvement.

Agency response to the audit

Attorney-General's Department

The Department welcomes the report. The Department notes the ANAO's conclusion that since 2001, in accordance with the IGA, those involved in the delivery of counter-terrorism capability had exerted significant efforts across a broad range of fronts to strengthen Australia's capability. Effective measurement and evaluation of performance in this area is a significant challenge, and the Department welcomes the ANAO's finding that strategies and processes to evaluate the National Capability Development Exercise Program and manage continuous improvement are generally effective.

The Department has worked hard internally and with stakeholders to put practices and procedures in place that address these challenges. Initiatives that will build upon existing processes include the lessons learned database, establishment of a dedicated evaluation unit, and development of a rolling four-year capability development exercise program. The recommendations made by the ANAO will be a valuable tool for the Department as it continues its efforts to improve performance in this area.

Department of the Prime Minister and Cabinet

The Department of the Prime Minister and Cabinet (PM&C) thanks the Australian National Audit Office (ANAO) for its report. PM&C notes that the report concludes that there are well-defined frameworks in place for the coordination of national response to terrorism and that the national counter-terrorism arrangements are subject to frequent, ongoing and multi-level evaluations. PM&C also notes that since September 2001, the national arrangements have undergone continuous and rapid evolution, and that strategies and processes were generally in place for evaluation of the arrangements and the management of continuous improvement. PM&C accepts the report's recommendations, noting that steps are being taken by both PM&C and the Attorney-General's Department (AGD) to address the issues identified by the report, including in relation to the national counter-terrorism exercise programme.

Footnotes

1 The Exercise Programme seeks to test, maintain and strengthen counter-terrorism and consequence management capabilities, command and control and interoperability. The programme is delivered through four types of exercise – Tactical, Investigation and Consequence Management, Discussion and Multi-Jurisdictional. These exercise types are outlined in Chapter Three.

2 The Handbook supports the National Counter-Terrorism Plan. The purpose of the Handbook is to outline the coordination arrangements, procedures and protocols that assist the implementation and integration of counter-terrorism and emergency management arrangements in Australia.

3 The NCTC has 11 designated capabilities that are developed and maintained that contribute to the basic, viable nation-wide capability.

4 The ‘basic, viable nation-wide counter-terrorism capability' is the NCTC term used to describe the base level of Australia's capability/preparedness to respond to terrorist incidents.