Australian Transport Safety Bureau

1.1 The Australian Transport Safety Bureau (ATSB) was established by the Transport Safety Investigation Act 2003 (TSI Act) as Australia’s national transport safety investigation agency. It seeks to improve safety and public confidence in the aviation, marine and rail modes of transport through:

• independent investigation of transport accidents and other safety occurrences;
• safety data recording, analysis and research; and
• fostering safety awareness, knowledge and action.

1.2 Under the TSI Act, the ATSB focusses on the prevention of future accidents and the improvement of safety. It is not a function of the ATSB to apportion blame or provide a means for determining liability. The ATSB does not investigate for the purpose of taking administrative, regulatory or criminal action.

1.3 In 2017–18, the ATSB received 15,766 aviation notifications³, 518 rail notifications and 238 marine notifications in the form of telephone calls, emails and website contact. This resulted in 6,350 being assessed as a safety occurrence and so came under consideration for investigation. Under the TSI Act (section 21), the ATSB may investigate any transport safety matter, and must investigate a transport safety matter if directed in writing by the Minister to do so. Figure 1.1 illustrates the types of investigation categories that the ATSB currently undertake. While not all of the reported occurrences are investigated, the details of each occurrence are retained within the ATSB’s occurrence database, which is analysed to identify emerging trends and issues.

Figure 1.1: Investigation categories and expected timeframes across aviation, marine and rail.

Source: ATSB data.

1.4 There has not been a major catastrophic event within Australia, as categorised by the ATSB that has required a major investigation to be undertaken under the TSI Act. The ATSB reports on complex investigations, which is a combination of investigations categorised as complex and defined.⁴ The ATSB also reports on short investigations which have been undertaken since 2010. In late-2017, the ATSB introduced and reported against occurrence briefs. These briefs are concise reports that detail the facts surrounding an occurrence, as provided in the initial notification and from any follow-up information. For the purposes of this audit, the ANAO has focussed on complex, defined and short investigations.

Operational context

1.5 The ATSB is a non-corporate Commonwealth entity under the Public Governance, Performance and Accountability Act 2013 (PGPA Act).⁵ In 2017–18, the ATSB had a full time Chief Commissioner; three part-time Commissioners and 109 staff across its offices within Australia. The staffing profile included 53 aviation, marine and rail safety investigators.

1.6 There has been growth across all three of the transport modes the ATSB is responsible for investigating. The May 2017 Budget included additional funding of $11.9 million for the ATSB over five years from 2016–17. The increase in funding was provided to the ATSB to replenish its workforce, and re-profile its capital investment strategies to meet its projected needs in technical equipment, data warehousing and core enterprise systems.

Audit rationale and approach

1.7 This topic was selected for audit as part of a series of performance audits focussing on the efficiency of entities. The audit was undertaken in the early stages of a significant organisational change program within the ATSB aimed at enhancing operational efficiency and effectiveness.

Audit objective, criteria and scope

1.8 The objective of this audit was to examine the efficiency of the ATSB’s investigation of transport accidents and safety occurrences.

1.9 To form a conclusion against the audit objective, the following high-level criteria were adopted:

• Has the ATSB put in place efficient processes for the investigation of transport accidents and safety occurrences?
• How well does the ATSB’s investigation efficiency compare to its own previous performance, as well as relevant international comparator organisations?

1.10 The scope of this audit covered the ATSB’s activities and processes for the conduct of investigations. The ATSB’s operations support functions, systems and processes such as financial and workflow management were also included given the influence they have on operational efficiency.

Audit methodology

1.11 This audit applied the ANAO’s methodology for auditing efficiency, ANAO Special Considerations for Efficiency Auditing Methodology and Guidance, which is based on a general model for assessing public sector performance. Efficiency is defined as ‘the performance principle relating to the minimisation of inputs employed to deliver the intended outputs in terms of quality, quantity and timing’.⁶

1.12 The methodology recognises that an examination of efficiency needs to be ‘fit-for-purpose’ for each entity or subject matter being audited. In most cases, this is likely to include:

• identifying if the audited entity has its own efficiency measures in place;
• identifying the relevant inputs and outputs, as well as the policy outcome(s) being sought;
• determining appropriate performance measures, drawing on data for inputs and outputs;
• identifying suitable comparators to benchmark against, to identify relative efficiency; identifying the key operational processes that are used to transform inputs into outputs (or outcomes) and the linkages between these elements; and
• undertaking appropriate audit procedures to understand and account for any material differences in the comparison of measured efficiency.

1.13 Specific audit procedures undertaken include:

• analysis of entity records;
• interviews of management and key stakeholders;
• reviewing and assessing the case prioritisation model used to determine which notifications are investigated⁷;
• analysis of available data; and
• benchmarking with relevant international comparators and the ATSB’s past performance drawing on data obtained by the ATSB during the audit.⁸

1.14 The audit was conducted in accordance with the ANAO Auditing Standards at a cost to the ANAO of approximately $301,000.

1.15 The team members for this audit were Michelle Mant, Amanda Ronald, Alexander Wilkinson and Brian Boyd.

Does the ATSB have appropriate performance measures to enable an assessment of its investigation efficiency?

2.1 The ATSB has defined deliverables, strategies and key performance indicators (KPIs). These have been published in the Portfolio Budget Statements (PBS) and in its Corporate Plan.

2.2 Externally, the ATSB has reported against its KPIs in its Annual Report. More detailed performance reporting has been provided internally against the established KPIs to the ATSB Executive, Commission and Audit Committee. Quarterly reports have also been provided to the Minister.

Time efficiency

2.3 Since 2009, the ATSB has consistently had a KPI that has focussed on the completion and/or publication of complex investigations within a 12 month target timeframe. Since 2010, the ATSB has also had a KPI addressing the timeliness of its short investigation reports against a target timeframe. Although there is no information provided by the ATSB in its public performance reporting of the age of ongoing active investigations, the ATSB does report this information internally.

2.4 Between 2014–15 and 2016–17, the ATSB measured and reported against three KPIs focussed on the timeliness of investigation activities (see Table 2.1). Although the numbers of reports to be completed per year has been amended, the ATSB’s focus on timeliness and publishing complex investigation reports within 12 months has remained a consistent KPI. The ATSB has provided a ‘performance-at-a-glance’ table which provides three years of trend data relating to the completion rates of investigation reports against targets.

Table 2.1: ATSB key deliverables and key performance indicators

Note: In 2016–17, the ATSB amended the timeframe for the completion of short investigation reports. The ATSB advised the ANAO that the two month time frame was an aspirational target, which was set when the short investigation report was shorter and less independent data collection was conducted by team members. In 2016 it was decided that short investigation reports should be expanded to include findings along with a short safety analysis section. This process was developed to ensure independent data collection by the ATSB was robust enough to support an expanded scope investigation if potential safety issues were identified.

Source: ANAO analysis of the ATSB’s data.

2.5 As illustrated by Table 2.2, at no stage over the past eight years has the ATSB achieved its timeframe targets for either complex or short investigations. As an overall average, the ATSB identified that it has generally taken around 19 months to complete a complex report and five and a half months to complete a short investigation report.

Table 2.2: ATSB performance against timeframe targets

Source: ATSB advice to Commission as at 30 November 2018.

2.6 In 2016–17, the ATSB amended the timeframe for the completion of short investigation reports, doubling it from 90 per cent within two months to 90 per cent within four months. The ATSB achieved a completion rate of 88 per cent against the 90 per cent target. The ATSB advised the ANAO that timeframe was amended to reflect the change in the format of the short investigation reports which were expanded to include findings and a short safety analysis. The Commission agreed to a four month target as the evidence provided suggested that they would be able to reach close to 90 per cent of reports to be completed within that timeframe. There was also a dedicated team that worked solely on the short investigations. In June 2017, this team was disbanded⁹ with only 41 per cent of short investigation reports completed within the four month timeframe for 2017–18.¹⁰

2.7 The ATSB advised in January 2019:

It is unclear exactly what caused the drop in short investigation performance and the drop was likely due to multiple factors. Re-establishing a short investigations team is one way this issue could be addressed.

2.10 In October 2018, the ATSB advised the ANAO that the target of completing 90 per cent of complex investigations within 12 months was driven by the ATSB’s perspective as to what may be considered acceptable to industry and directly involved parties, including the next of kin. The ATSB further advised the ANAO that the 12 month target also took into account the International Civil Aviation Organization’s (ICAO) International Standards and Recommended Practices manual ‘Annex 13 to the Convention on International Civil Aviation, Aircraft Accident and Incident Investigation, Eleventh Edition, July 2016’ (Annex 13). Specifically, section 6.5 of Annex 13 states:

In the interest of accident prevention, the State conducting the investigation of an accident or incident shall make the Final Report publicly available as soon as possible, and if possible, within twelve months.

2.11 On 30 November 2018, advice was provided to the Commission on proposed amendments to the ATSB’s timeliness KPIs. This advice drew on analysis of performance over time as well as the results of an initial benchmarking exercise undertaken as part of this ANAO performance audit on target timeframes (if any) and actual performance by a number of comparable international transport investigation agencies. Table 2.3 illustrates the changes being considered by the ATSB at the time ANAO audit fieldwork was completed.

Table 2.3: Proposed changes to ATSB’s timeliness KPIs

Note: The ATSB advised the ANAO in January 2019 that it intends to gradually adjust the numbers and timeframes over the coming financial years to demonstrate that it is tracking towards the revised targets from 2020–21 onwards. To achieve these amended targets, the ATSB will need to carefully manage a reduction in the number of active complex investigations that it has in progress over the next two reporting periods.

Source: ATSB data.

Resource efficiency

2.12 The ATSB has been meeting its target output for investigations completed (50 to 60 complex investigations and 100 short investigations per annum). These targets have been achieved in an environment where the ATSB has had a large number of ongoing active investigations at the end of each month (see Figure 2.1).

Figure 2.1: Ongoing active investigations and the number commenced since January 2014

Source: ANAO analysis of the ATSB’s data.

2.13 As at 30 June 2018, the ATSB had 95 ongoing active complex investigations and 38 ongoing active short investigations, totalling 133 investigations. As at 30 November 2018, the ATSB had 85 ongoing active complex investigations and 37 ongoing active short investigations (122 in total). The ATSB has attempted to reduce the number of ongoing active investigations and to take on less investigations.

2.14 In terms of resources consumed to produce investigation outputs, the ATSB reports internally on calendar and investigator (effort) days. Externally the KPI is based on calendar days only. Reporting solely on calendar days does not give a full picture as the effort logged for an individual investigation that has been underway for a considerable period of time can be minimal (for example, as a result of waiting for external parties to undertake their own investigations or external parties responding to requests from the ATSB).

2.15 Investigator (effort) days are collected by investigators logging a percentage of each work day working on individual investigations. The percentage is converted to hours, based on a 40 hour week. Calendar days are based on recorded investigation dates and workflow stages of the investigation. Table 2.4 outlines the investigation categories and expected calendar and investigator (effort) days (previous and current). Consideration is being given to establishing workload targets for investigators.

Table 2.4: Investigation categories—expected calendar and investigator (effort) days

Note: Occurrence briefs were not introduced as ATSB products until mid-2017. The occurrence briefs have expected calendar days of 30, expected investigator days of 1 day.

Source: ATSB data.

2.16 The amendments in 2017 reflect analysis and work completed in 2015, based on investigations that were considered to be high quality, within scope and completed by competent investigators. The analysis highlighted that the expected calendar and investigator (effort) days did not represent the full effort for an investigation.

2.17 Figure 2.2 shows the ongoing active complex investigations as at November 2018 in terms of how they are tracking against expected calendar days and investigator (effort) days. It illustrates that, for many investigations, it continues to take the ATSB longer and require more investigator effort than is targeted.

Figure 2.2: Ongoing active complex investigations as at November 2018

Source: ANAO analysis of the ATSB’s data.

Does the ATSB collect relevant and reliable information on its investigation efficiency?

2.20 The ATSB collects a broad range of data relating to its accident investigations, occurrence data for the three modes and information that could assist in an assessment of its investigation efficiency. The information systems used by the ATSB to support this are the:

• Safety Investigation Information Management System (SIIMS¹¹); and
• Pacific Timesheet.

2.21 The focus of what is reported internally and publicly has mainly involved an examination of the time taken/measured to complete investigation reports (directly linked to its KPIs). Since 2012, this has been complemented by the collection of investigator effort logging.

2.22 ATSB collects data for many aspects of its investigation activities, including:

• effort given to each investigation;
• time taken to complete each investigation phase;
• workflow stages of an investigation;
• trend effort/day data; and
• investigators assigned to individual investigations.

2.23 Information collected and recorded in the SIIMS database allows the amount of time taken to complete individual investigations to be calculated. Executive Directors, Directors and team managers automatically receive a closure report for all investigations closed in the previous week, for their team, branch or division. These reports detail the time and effort taken for each investigation closed.

2.24 Similarly, the timeliness of active investigations can be calculated using data collected in SIIMS.

2.25 Although the ATSB collects data on a range of investigation activities and workflow stages, limited analysis has been completed in this area. The ATSB advised the ANAO in October 2018 and January 2019 that the key reasons for this relate to:

• staff capacity (due to the focus on completing investigations with a finite number of investigators compared to number of investigations) and the reliance on a small number of the ATSB staff with the skills to undertake this analysis¹²;
• unreliability of data as workflows are suspended and reset; and
• the focus has been directed towards KPI targets and reducing the backlog of older investigations.

2.26 Analysis of workflow stages and key milestones reported in SIIMS would allow the ATSB to identify periods within an investigation whereby the ATSB did not have control over the process. For example, the ATSB relies on the cooperation of various external international entities, such as aircraft manufacturers, when undertaking aviation investigations.

Data limitations

2.27 Due to system limitations, the ATSB has limited visibility across all current investigations. This has increased the difficulty for the ATSB in determining whether there is capacity to initiate new investigations without impacting the timeframes of existing investigations, or having a complete picture of investigator workloads.

2.28 Another limitation is that finance and human resources data is completely separate from the investigation and occurrence data, making visibility of holistic data impossible without considerable manual effort (for example complete costs associated with individual investigations). Manual workarounds increases the risk of errors in performance information produced and reported and currently prevents the ATSB from undertaking a wider range of analysis.

2.29 The ATSB has developed a data strategy that plans to address the limitations raised above and increase the reporting capacity and visibility of all data held within ATSB databases.

Does the ATSB’s processes for prioritising the oversighting and conduct of investigations support the efficient use of resources?

2.30 Each year, the ATSB receives more than 15,000 notifications of transport incidents and accidents across the aviation, rail and marine modes. As illustrated by Figure 2.3, the trend has been for an increasing number of notifications over the last nine years.

Figure 2.3: Number of notifications received per year since 2009–10

Source: ANAO analysis of the ATSB’s data.

2.31 A notification must be assessed to be a transport safety matter in accordance with section 23 of the TSI Act. The assessed notification must be classified by the anticipated level of resources, or complexity and time required to complete the investigation.

2.32 The Safety Reporting Team Leader runs a daily notifications received report and reads through all notifications received with the past 24 hours. When assessing an aviation notification, Safety Reporting staff:

• perform an initial assessment on the information relating to the notification;
• obtain (if necessary) further information from other parties;
• assess further (when there is sufficient information about the incident); and
• classify the occurrence reports as accidents, serious incidents, incidents (collectively known as occurrences) or events.¹³

2.33 Currently the notifications process is a largely manual undertaking for six full time employees (FTEs) who are required to enter and quality assure more than 15,000 aviation notifications per year. The proposed data strategy aims to automate the notification process, thereby reducing processing time and human effort to process notifications and increase the safety analysis capability through more up to date aviation dataset.

2.34 At 1.30pm each work day, there is a meeting at which the Transport Safety Directors, Manager Safety Reporting and key staff work through the details of occurrences provided by the Safety Reporting team. An assessment is made and, if required, further information is sought to clarify the facts. It is at this meeting that the Directors will often decide to seek more information to further inform their decision making process, to either investigate an occurrence or not. In seeking further information, the ATSB is also able to register more full or accurate details within its occurrence database. Of the 50 to 70 notifications received each day, about five per cent (around three) are accidents and serious incidents, which are prioritised for processing into occurrences, whether they are investigated or not.

2.35 The 1.30pm meetings are an efficient and effective process for prioritising and oversighting the conduct and decision to seek further information on the facts of an occurrence, prior to making a decision whether to investigate. As part of the broader 2017–18 organisational change initiatives, and to address the ongoing issues that the ATSB has faced with a backlog of investigations yet to be completed, starting in early 2018 the ATSB has been attempting to better match the number of new investigations to the available resources.

Resource allocation

2.36 The ATSB has limited visibility over the resource pool of investigators. This has proven problematic in having an understanding of whether there is capacity to initiate an investigation, whether there are sufficient resources for the investigation and what impact, if any, a further investigation will have on existing investigations. This situation has contributed to investigation delays.

2.37 The ATSB has had an expectation that each investigator in charge (IIC) will manage two investigations simultaneously. Investigator resources are usually decided by investigation level, but also the type of investigation. These general practices follow for occurrence investigations, for example:

• short investigations — single investigator;
• defined investigations — one to two investigators; and
• complex investigations — team of investigators.

2.38 Resources are allocated on a case by case basis. For research and safety issue investigations, often a single investigator is primarily responsible for the bulk of an investigation at all levels. Resources will often be used for part of an investigation only, such as for recorders downloading and analysis, research tasks, and other specialty tasks. The resourcing approach is also affected by the workload of the IIC, availability of other resources, and the profile of the investigation.

2.39 Figure 2.4 illustrates the workload for the investigators as at November 2018, who are balancing investigations, training, mentoring and supervision of the new investigators. On average, an investigator is in charge of three investigations (a minimum of one and as many as six) as well as a team member on a further seven investigations (a minimum of no investigations solely as a team member and a maximum of 28 investigations as a team member, in addition to being in charge of six investigations). The workload allocated to some investigators has also contributed to a backlog with incomplete investigations (see paragraph 2.46). This allocation of investigations is likely to be contributing to the decline in the number of investigations completed per investigator (see Figure 3.5 on page 41).

Figure 2.4: Investigations per investigator as at November 2018

Note: Analysis of the SIIMS database identifies 54 IIC’s, rather than 53, due to one investigator being entered twice.

Source: ANAO analysis of the ATSB’s data.

2.40 As part the work being developed and implemented to promote a project management approach to investigations, the ATSB has undertaken to gain a better understanding of the current workload for all operational resources, both investigative and non-investigative. As part of this work, the ATSB is developing a tool to provide a holistic view of resource demand. This will be tracked through Microsoft Project and linked to a master resource pool. It is anticipated that this approach will produce a more interconnected reporting view to caseload and capacity at an individual, team and organisational level and will support decision making around the allocation and reallocation of resources to meet demand.

2.41 This approach will enable the ATSB to better track this information in the future, with investigations being managed similar to a project so they can be tracked at a task level. Managers are intended to have a clearer view of resource availability; accountability; level of effort; and duration metrics. It is anticipated that, once fully implemented, and once the model matures, there will be a greater capacity to track overall investigation efficiency, and have greater visibility of the existing commitments and capacity of the pool of investigator resources.

Investigation costs

2.42 The data and process for the budget for investigations is separate to that of the SIIMS investigation database, and therefore not visible to investigators. The ATSB advised the ANAO in January 2019 that it is the responsibility of Managers and Directors to consider costs when planning investigations, undertaking analysis or obtaining data (such as the cost of recovering voice and data recorders). There is no individual budget per investigation, but supplier and employee costs should be able to be tracked against individual investigations using a project code allocated to the investigation.¹⁴

2.43 For the 75 closed investigations in the 2017–18 financial year (across three investigation categories), Table 2.5 illustrates the overall investigation costs across the three investigation categories, showing the range, average and median. The data must be treated with some caution as the project code was not being consistently applied to each investigation, with IICs for only 30 of the investigations providing a cost code so that costs could be attributed to each investigation.

Table 2.5: Investigation costs

Note 1: Figures exclude the reopening in 2016 of the 2009 investigation into AO–2009–072—Ditching—IAI Westwind 1124A, VH–NGA, Norfolk Island, at a cost of $1.741million. Including this investigation increases the average cost to $240,593.

Source: ANAO analysis of the ATSB’s data.

Are opportunities to improve investigatory processes and practices identified and acted upon?

2.44 To address investigation efficiency and to attempt to meet performance targets the ATSB has introduced short investigation reports (2010) and defined investigation reports (2013).¹⁵ These two types of investigations were established as they are shorter therefore in principle require less effort, less resources, and have restricted scopes and overall shorter reports than the traditional complex investigations.

2.45 In 2017, the ATSB commenced an organisational change program, the ‘Evolution Program’. This program is intended to enable better resource allocation and utilisation and to maximise the efficiency and effectiveness of the ATSB operations by addressing the organisational structure, culture, efficiency, learning and development, talent management, succession planning and leadership of the ATSB.

2.46 In May 2017, the ATSB commenced the ‘Back on Track Program’. This was to address the ongoing backlog of open investigations that had exceeded their scheduled duration. The ATSB prioritised the completion of up to 56 investigations. This program of work has required a significant diversion of resources from the ATSB’s business as usual operations.

2.47 ATSB records examined by the ANAO evidence that, between September 2011 and November 2018 there have also been a range of opportunities to improve investigatory processes and practices identified. The opportunities were in areas such as:

• system enhancements including additional functionality to help investigators plan and manage investigations;
• making timeliness targets more realistic to be achieved;
• the development, introduction and administration of a public database schema for the automated submission and receipt of transport safety occurrence reports;
• the development of an enhanced ATSB data warehousing, analysis and business intelligence solution along with development of relevant policies and procedures;
• the investigation scoping process; and
• the extent of quality assurance activities during the investigation phase (rather than relying on high levels of quality control during the report review phase).

2.48 Until 2017, it was common for identified improvement opportunities to not be acted upon. For example, one key paper that was presented to the Executive in September 2015, demonstrated analysis of the ATSB’s own performance data highlighted that the then current effort and calendar days were not achievable, nor were they reflective of the full effort of an investigation. The analysis was based on investigations that were considered to have been exemplary in terms of being undertaken by competent investigators and completed within the expected scope for the investigation level. The analysis highlighted that these investigations were still exceeding the expected calendar and effort days.¹⁶ These issues were not addressed until the Evolution Program in 2017.

2.49 Table 2.6 sets out key recent initiatives in place or underway. As a number of these initiatives are underway or yet to be implemented, efficiency gains cannot be assessed at this stage.

Table 2.6: Key initiatives to improve investigatory process and practices

Note a: RACI is an abbreviation for: R=Responsible; A=Accountable; C=Consulted; and I= Informed.

Note b: A project to automate the notification process to address resource constraints and a growing backlog was presented for approval to the ATSB Executive in 2016. This project ran over time and budget. The automation of aviation notifications was not included as part of the initial project, but it did include updating the notifications module.

Source: ANAO analysis of the ATSB’s data.

What controls does the ATSB have in place to assure itself that improvements in efficiency do not compromise the quality of its investigations?

2.50 The ATSB was criticised for its investigation report into the 2009 ditching of an aircraft off Norfolk Island¹⁷, due to its delay (nearly three years after the accident) plus the lack of detailed analysis and useful recommendations for avoiding future incidents and accidents. Although the Aviation Safety Regulation Review 2014 stated this was an ‘aberration’ and not typical of the high standard that the ATSB usually attains, this had a negative impact on the ATSB’s reputation. Similarly, the December 2014 peer review report of the ATSB’s investigation methodologies and processes¹⁸ conducted by the Transportation Safety Board of Canada concluded that, when the ATSB methodology is adhered to, and the component tools and processes to challenge and strengthen analysis are applied, the investigation result is more defensible.

2.51 The ATSB has identified that its workflow processes have not been conducive to the timely completion and review of investigations and associated safety activities. The workflow processes incorporated high levels of quality control during the draft report review phase, but contained limited quality assurance measures within the earlier investigation phases.¹⁹ In May 2017, in an Executive Paper, the ATSB acknowledged that its own investigation workflow processes and practices had directly contributed to its output timeliness issues, principally due to extensive review and rework requirements which were necessary to ensure that a quality report was released.

2.52 Since the broader organisational change initiatives in 2017, and in particular through the SIRF project, there have been major changes to the workflow processes for ATSB investigations. This has seen the increase in controls within the review process where there are multiple layers of review to gain the required level of quality assurance to release an investigation report. The improved workflow process for stages of an investigation currently include:

• planning meetings to ensure appropriate scope and resources;
• required analysis processes;
• safety factor reviews with managers and directors;
• safety factor executive briefs;
• technical review of reports;
• administrative reviews;
• director reviews;
• Commission reviews; and
• external reviews.

2.53 The ATSB advised the ANAO in October 2018 that since these processes have been introduced, and although in the early stages of implementation and application to current investigations, there has been notable improvements in quality and a reduction in the amount of rework required through the various review stages.

2.54 The ATSB further advised the ANAO in October 2018 that it is committed to looking for improvements across the range of investigatory processes and practices to improve efficiency, transparency and to better enable management decisions. As part of the work being undertaken to introduce a program managed approach to investigations, there is also a planned focus on strengthening the assurance processes that support the investigation process. For example, the ATSB is planning to establish an Investigation Review Board to more closely manage quality, risk mitigation and delivery issues.

Has the ATSB analysed changes in its investigation efficiency over time?

3.1 As discussed from paragraph 2.20 to paragraph 2.26, the ATSB collects data on a range of investigation activities and workflow stages.

3.2 Limited analysis of investigation efficiency has been undertaken by the ATSB. Notable examples of analysis that has been undertaken are:

• In 2014, the ATSB undertook analysis on draft/final report workflow data. This analysis helped to identify that the scheduled timeframes for manager review were unrealistic and has informed the introduction of changes in workflow process and introduction of earlier review points in the investigation process.
• In 2015, analysis was undertaken of expected investigator (effort) days and expected calendar days. The analysis led to a recommendation to revise timeframe expectations. In 2017, as part of the broader organisational change initiatives, the ATSB updated the expected investigator (effort) days based on the data previously provided in 2015. Current expected average effort for ATSB investigations is now:
  • complex — 44 to 80 days;
  • defined — 26 days; and
  • short — 13 days.

3.3 The new complex target reflected the older level three target (44 days) and level one target (80 days). Defined was double that of the previous short target, and increased from the old expected 18 days. The former 18 day target was not being achieved. The expected effort for short investigations remained the same at 13 days, as the data illustrated that 13 days was approximately average for short investigations in 2016–17. At this time, there was a dedicated team that was solely responsible for completing the short investigation reports.²⁰

3.4 In 2014, the Transportation Safety Board of Canada made a recommendation as part of a peer review that the ATSB review its schedules for the completion of various levels of investigation to ensure that realistic timelines are communicated to stakeholders. Although targets have been amended for investigations classified as complex or defined, more recent analysis has shown that they are still not reflective of actual effort and actual calendar days. In 2018, the ATSB analysed data over the past five years, which indicated that the average investigator (effort) days were tracking at:

• complex — 150 days;
• defined — 82 days; and
• short — 13 days.

Has the ATSB become more efficient over time in conducting investigations?

3.7 The ANAO examined the trends over time in the time taken by the ATSB to complete investigations in the complex, defined and short categories. As illustrated by Figure 3.1, the time taken to complete investigations in the:

• short category, remained relatively stable until 2018, averaging 131 days. Following the disbanding of the team focussed on those investigations the time taken in the first six months of 2018 to complete investigations increased to an average of 236 days (an 80 per cent increase on the average over the previous nine periods examined);
• defined category increased significantly between 2013 and 2016 from an average of 236 days in the six months to 31 December 2013 to an average of 697 days in the six months to 31 December 2016. Since then, the average has reduced over time and was 558 days for the first six months of 2018. This is still significantly above performance in the earlier part of the period examined by the ANAO (for example, more than double the time that was being taken in the last six months of 2013); and
• complex category, has increased significantly with those investigations completed in the first six months of 2018 taking, on average, nearly three years to be completed. The only period when the average time to complete reduced was in the last six months of 2016.

Figure 3.1: Average calendar days to complete investigations

Source: ATSB data.

3.8 The ANAO also analysed the inputs (in terms of effort being logged by investigators) being required to complete ATSB investigations. As illustrated by Figure 3.1 the ATSB has regularly exceeded the number of calendar days in comparison to the expected number of calendar days across complex, defined and short investigations. Further, as illustrated by Figure 3.2:

• for short investigations in 2018, they have remained relatively stable in regards to expected effort (13 days, with an average of 8.8 investigator (effort) days) to complete a short investigation; and
• for complex and defined categories of investigation, the trend is an ongoing increase, well above the expected investigator (effort) days sitting at an average of 150 effort days for complex (expected 80 investigator (effort) days) and an average of 82 (expected 26 investigator (effort) days) for defined.

Figure 3.2: Average investigator (effort) days

Source: ATSB data.

Has the ATSB compared its investigation efficiency to other relevant organisations?

3.9 The ATSB had not, prior to 2018, undertaken any benchmarking of its efficiency performance in comparison to other relevant transport safety organisations.

3.10 In October 2018, in connection with the ANAO audit and as part of the ATSB’s representation at the International Transportation Safety Association (ITSA) meeting, the ATSB proposed a benchmarking study to its international counterparts. The ATSB is to present a comprehensive paper to the ITSA 2019 annual meeting in Canada proposing a formal international benchmarking study to be undertaken.

3.11 As part of the audit process, in order to assist the ANAO to address the audit criteria and in order to learn lessons that may help to inform the future direction of the ATSB’s key performance indicators, the ATSB requested assistance from four international multi-modal transport safety investigation entities. The entities identified as suitable comparators are from the Asia Pacific region and North American region. Although not all multi-modal investigators, there would be benefits in consideration being given in future benchmarking exercises to also seeking to include comparators from Europe.²¹

3.12 One of the entities advised the ATSB that it did not have timeliness targets for investigations.²² The entity advised that it would attempt to assist the ATSB with its request, by retro-fitting its data. At the time audit fieldwork was concluded, information from this entity had not been received and therefore is not included in any of the analysis from paragraph 3.24 to paragraph 3.26.

3.13 Three years of data was requested, focussing on investigations into accidents and serious incidents (excluding safety studies/research) across all modes of transport each entity had in its jurisdiction. The request included information relating to the:

• number of investigations completed;
• percentage of investigations published within 12 months²³;
• median age at publishing;
• number of investigation staff participating in complex occurrence investigations; and
• number of active investigations.

How does the ATSB’s investigation efficiency compare to other relevant organisations?

3.14 As discussed in paragraph 3.13, three financial years of data was obtained from three countries as outlined in Table 3.1.

Table 3.1: Benchmarking data

Note: Staffing numbers for Country A and Country C are based on their average investigator staffing levels. Country B staffing numbers are from its Annual Reports.

Source: ATSB data.

3.15 Figure 3.3 (below) illustrates the ATSB’s performance against these three countries for the number of investigations completed; the median age for reports at the time of publishing; and staffing numbers.

Figure 3.3: ATSB’s performance against international comparators

Source: ATSB data.

3.16 Figure 3.4 (below) illustrates the ATSB’s performance against the three comparator countries for the percentage of reports published within 12 months.

Figure 3.4: Percentage of reports published within 12 months against international comparators

Source: ATSB data.

3.17 Based on the data received, on an annualised basis, the ATSB’s performance against the three comparator countries trend analysis shows:

• the ATSB has a downward trend for the number of investigations completed;
• similar to Country A, the ATSB’s median age for publishing remains steady, however Country B and Country C are showing an improvement in the median age it takes to publish their investigations; and
• comparatively, the ATSB is on a downward trend for publishing reports within 12 months, whereas, Country B and Country C are showing an upward trend.
• staffing numbers show little change.

3.18 The ATSB has adopted a more challenging timeliness performance target than the two comparators that have a target in place (Country A and Country B). Compared to the ATSB target of completing 90 per cent of complex investigations within 12 months, the relevant comparators had targets of 75 per cent of investigations completed within 450 days and 70 per cent of investigations completed within 440 work days. There were few instances reported where the comparators were able to complete investigations within 12 months (see Table 3.2). Across the three years, the ATSB performed better than two of the three countries for which data was obtained.

Table 3.2: Percentage of investigation reports published within 12 months

Note: Although Country C does not specify, they generally work to a two year timeframe.

Source: ATSB data.

3.19 As illustrated in Figure 3.5, the ATSB’s productivity is on a downward trend whereas the data indicates increasing productivity for Country’s A, B and C over the last three financial years. On a per investigator basis, in 2015–16 the ATSB was producing more reports than the comparators. By 2017–18, ATSB was only producing more reports per investigator than Country A, and produced significantly fewer reports than Country B. The over-allocation of investigator resources by the ATSB (as illustrated in Figure 2.4 on page 27) has likely contributed to reduced investigation closure rates for the ATSB.

Figure 3.5: Number of investigations completed as a ratio of the number of investigators

Source: ANAO analysis of ATSB data.

3.20 In January 2019 the ATSB advised the ANAO that yearly data can be highly variable and can be affected by things beyond an entity’s ability to control, such as several large accidents occurring in one year. The ATSB suggested that an average of the three years of data provided was a preferable approach. An average across the data provided for three financial years is shown below at Table 3.3. The analysis shows that, over the three year period analysed, the:

• ATSB produced significantly more investigation reports per investigator than two of the three comparators, but was less productive than one of the comparators;
• ATSB produced reports within 12 months more often than two of the three comparators (including the other comparator with who produced a similar number of total reports to the ATSB — the other two entities produced significantly fewer reports over the three years); and
• median age of the ATSB’s reports at publishing was greater than each of the three comparators.

Table 3.3: Three year average of benchmarking data

Source: ATSB data.

Additional benchmarking

3.24 The ANAO also undertook additional analysis of publicly available information of two countries to compare the ATSB’s performance against the numbers of commenced, completed and ongoing active investigations; and the number of investigator staff. Publicly available information on these metrics was not available for two of the comparator countries²⁴ and so the ANAO did not include either country in this analysis. Figure 3.6 illustrates the results of the ANAO’s analysis.²⁵ Figure 3.7 illustrates the ATSB’s performance against the two comparator countries for the average number of days taken to complete an investigation.

Figure 3.6: ATSB’s overall performance against international comparators

Source: ANAO analysis of the ATSB, Country A and Country B benchmarking data.

Figure 3.7: Average number of days to complete an investigation

Source: ANAO analysis of the ATSB, Country A and Country B benchmarking data.

3.25 As is illustrated above in Figure 3.6 and Figure 3.7, the ATSB has:

• commenced more investigations than the comparator countries for 2015–16 and 2017–18, but less than Country A in 2016–17;
• more ongoing investigations than all comparators across all the financial years; and
• took on average at least 100 days more than the comparators based on the reported average days to complete investigations.

3.26 Compared to Country A and Country B, the ATSB has commenced more investigations than it has completed. This is illustrated in Table 3.4. As set out in Table 2.6 on page 30, the ATSB introduced the Back on Track initiative to address a growing backlog of old investigations.

Table 3.4: Ratio of commenced and completed investigations over three financial years

Source: ANAO analysis.

3.27 When comparing the number of ongoing active investigations to the number of investigators, the ATSB is expecting greater productivity from its investigators than Country A and/or Country B, as shown below in Table 3.5.

Table 3.5: Ratio of ongoing active investigations over the number of investigators over three financial years

Source: ANAO analysis.

Appendix 1 Australian Transport Safety Bureau response