Overview

1.1 The Civil Aviation Safety Authority (CASA) was established on 6 July 1995 under the Civil Aviation Act 1988 (Civil Aviation Act) as Australia’s aviation safety regulator with a range of functions including: civil air operations in Australian territory; the operation of Australian aircraft outside Australian territory; classifying Australian administered airspace; and determining the services and facilities provided by approved air navigation service providers.

1.2 CASA is a corporate Commonwealth entity under the Public Governance, Performance and Accountability Act 2013 (PGPA Act) and is part of the Infrastructure, Transport, Regional Development and Communications portfolio. The CASA Board, which comprises a chair and the Director of Aviation Safety¹ and up to five other members, is the accountable authority of CASA under the PGPA Act.

1.3 CASA undertakes its functions in accordance with the Civil Aviation Act, the Airspace Act 2007, Australia’s obligations under the Convention on International Civil Aviation (Chicago Convention) and other international agreements. Under section 9A of the Civil Aviation Act, CASA is required to regard the safety of air navigation as the most important consideration when carrying out its functions.

1.4 Under the Civil Aviation Act, the Minister may give the board written directions about the performance of its functions or the exercise of its powers including the appropriate strategic direction for CASA and the manner in which CASA should perform its functions. The Minister’s most recent direction, Statement of Expectations for the Board of the Civil Aviation Safety Authority for the Period 31 January 2022 to 30 June 2023, was issued to the board on 13 January 2022 and sets out the Minister’s expectations for CASA in regard to its operations and performance.

1.5 CASA has a single outcome: ‘Maximise aviation safety through a regulatory regime, detailed technical material on safety standards, comprehensive aviation industry oversight, risk analysis, industry consultation, education and training’. In 2021–22, CASA’s expenditure is estimated to be $212 million with an average staffing level of 832.

Surveillance

1.6 CASA regulates the aviation industry using a suite of orders and regulations under the authority of the Civil Aviation Act. CASA issues the relevant authorisations required by persons and organisations to participate in aviation activities — in the form of licences, certificates, authorisations, approvals and permissions — if they meet the necessary standards and requirements.²

1.7 CASA is responsible for conducting surveillance on authorisation holders (AHs) within Australia and for Australian CASA AHs overseas. The main AH types include:

• Air Operator’s Certificate (AOC);
• recreational, private and commercial pilot flying training (Part 141 of the Civil Aviation Safety Regulations 1998 (CASR));
• integrated and multi-crew flying training (Part 142 of CASR);
• approved maintenance organisations (AMO) (Part 145 of CASR and regulation 30 of the Civil Aviation Regulations 1988 – CAR 30); and
• delegates and authorised persons (airworthiness).

1.8 In conducting surveillance activities, CASA’s primary objective is to determine whether an AH is fulfilling its obligations under the Civil Aviation Act, including managing safety risk and complying with legislation and regulations.

1.9 The National Oversight Plan is CASA’s overarching ‘approach’ to safety oversight with three of its four pillars underpinning surveillance including: the National Surveillance Selection Process (NSSP) (planned surveillance); response surveillance (unplanned in response to incidents, accidents or emerging risks); and campaign surveillance (coordinated surveillance activities of multiple AHs within a sector on a specific issue). The remaining pillar, regulatory services (which includes cost recovery and payment of fees), is not considered surveillance, but is considered when planning surveillance.

1.10 Surveillance activities are conducted by CASA staff in office locations across Australia (see Figure 1.1).

Figure 1.1: CASA office locations, June 2021

 

 

Note: CASA’s satellite office on Horn Island is not permanently staffed and is used to provide training and facilities for staff conducting surveillance activities and inspections.

Source: CASA annual report 2020–21, p. 6.

Organisational and surveillance activity changes

1.11 Since 2017, there have been changes to the CASA board, the Director of Aviation Safety and several changes to CASA’s approach to surveillance activities including the implementation of the:

• National Surveillance Selection Process, which was approved in January 2017 and came into effect on 1 July 2018. The NSSP involved the development of an annual planned surveillance schedule, based on a standardised approach to prioritising, classifying and scheduling surveillance activities;
• National Oversight Plan (NOP) on 1 July 2018 with the separation of planned, response and campaign surveillance as a result of the NSSP; and
• National Operating Model for regulatory oversight services (including the functions of guidance, entry control and surveillance) under the Regulatory Services and Surveillance Transformation program (RSST). The RSST (which commenced in July 2018 and was finalised in July 2021) involved a move to a national approach to managing surveillance activities based largely on dedicated surveillance teams.

1.12 The timing of these changes is summarised in Figure 1.2.

Figure 1.2: Timeline of CASA board changes and surveillance-related activities, January 2017 to January 2022

 

 

Source: CASA annual reports 2017–18 to 2020–21; CASA documentation.

Rationale for undertaking the audit

1.13 CASA’s surveillance activities assess the safety performance of aviation industry participants and whether they continue to meet their obligations under the Civil Aviation Act and associated regulations. The audit provides independent assurance to the Parliament on the appropriateness of CASA’s surveillance approach and its planning and conduct of surveillance activities.

1.14 CASA and its operations have not been the focus of an ANAO performance audit since 2010. Changes and events affecting CASA since 2010 have included:

• amendments to section 9A of the Civil Aviation Act on 7 November 2019 which required CASA to take into account the different risks associated with each aviation industry sector and to consider the economic and cost impact in setting safety standards; and
• the undertaking of the Aviation Safety Regulation Review over the period November 2013 to June 2014 and the implementation of surveillance-related recommendations.

Audit approach

Audit objective, criteria and scope

1.15 The objective of the audit was to assess whether CASA has implemented effective arrangements for the planning and conduct of surveillance activities.

1.16 To form a conclusion against this objective, the following high-level criteria were applied.

• Has CASA established an appropriate surveillance approach?
• Is CASA effectively monitoring compliance and regularly reviewing the effectiveness of its planning and conduct of surveillance activities?
• Is there regular reporting to the Board and the government?

1.17 The scope of the audit covered the period from 1 July 2018 until 31 December 2021.

Audit methodology

1.18 The audit methodology involved:

• review and analysis of relevant agency records;
• review and analysis of relevant inspection and audit data in Sky Sentinel (CASA’s IT surveillance management tool); and
• meetings with key staff and site visits to CASA’s national and regional offices.

1.19 The audit was conducted in accordance with ANAO Auditing Standards at a cost to the ANAO of approximately $642,406.

1.20 The team members for this audit were Sandra Dandie, Kai Swoboda and Brian Boyd.

2.1 An appropriate surveillance approach underpins confidence in the aviation sector. The ANAO examined:

• legislative and other requirements because they are the key elements of the regulatory framework which underpins CASA’s surveillance approach;
• policies, procedures and guidance for staff and stakeholders as they enable a consistent regulatory approach and clarifies expectations about how surveillance activities are conducted;
• risk management which supports decision making on the allocation of resources to surveillance activities; and
• whether CASA is meeting international standards directly related to surveillance activities because Australia is required to be compliant with the Chicago Convention.

Is CASA’s surveillance approach compliant with the relevant legislative frameworks?

Legislative and regulatory framework

2.2 As noted in paragraphs 1.1 to 1.4, CASA is subject to a range of legislation, regulation, and requirements in carrying out its functions. This includes the Public Governance, Performance and Accountability Rule 2014, the Regulator Performance Guide and consideration of the Commonwealth Risk Management Policy 2014.

2.3 The Civil Aviation Act 1988 (the Civil Aviation Act) does not contain detailed rules governing aviation safety, with these contained within the relevant regulations and other legislative instruments that are made under the Civil Aviation Act (see Appendix 3). The aviation safety regulations incorporate a mix of a prescriptive, compliance-based approach and an outcome-based (or performance-based) framework.³

2.4 One of CASA’s functions under the Act is to conduct ‘comprehensive aviation industry surveillance, including assessment of safety-related decisions taken by industry management at all levels for their impact on aviation safety’.

CASA’s surveillance framework and approach

2.5 CASA’s surveillance activities⁴ are directed towards assessing whether an authorisation holder (AH)⁵ complies with the requirements of the Civil Aviation Act and its associated regulations. CASA’s surveillance framework incorporates the processes, policies and guidance, IT systems, training and staffing arrangements that underpin the design and implementation of surveillance activities.

2.6 CASA’s surveillance approach, the National Oversight Plan (NOP), provides the basis for the annual planning and allocation of resources for CASA’s surveillance activities. As noted at paragraph 1.9, the NOP consists of four ‘pillars’ comprising: planned surveillance⁶; response surveillance; campaign surveillance; and regulatory services (the assessment and/or review of a regulatory approval, variation, exemption or instrument).

Planned surveillance

2.7 The National Surveillance Selection Process (NSSP) forms the basis of CASA’s planned surveillance activities, with the establishment of a prioritised annual schedule of surveillance activities from 2018–19 across most types of AHs.⁷ The NSSP manual notes that:

The NSSP is a systematic approach to planned surveillance selection, prioritising authorisation holders and creating a national planned surveillance schedule that aims to optimise resource usage for planned surveillance activities across CASA.

2.8 CASA notes that the nationally coordinated annual development of the NSSP is informed by updated AH data and a categorisation of AHs into one of three groups (A, B or C) that determines the prioritisation of surveillance activities based on an assessment of the risk consequence associated with the activities of each AH (see paragraph 2.50).

Response surveillance

2.9 Response surveillance events are those activities that are conducted on AHs in direct response to incidents, accidents, intelligence, or emerging safety risks. They are not NSSP, campaign surveillance or regulatory services events.

2.10 Decisions about undertaking response surveillance activities in response to safety occurrence information are made by surveillance managers and can include:

• desktop or site reviews;
• a review for the national manager such as a safety assurance review or a review for enforcement; and
• no further action.

Campaign surveillance

2.11 CASA defines its campaign surveillance as ‘coordinated surveillance activity focusing on multiple authorisation holders within an identified sector of the industry over a defined period. These are conducted to focus on a particular issue’. CASA advised the ANAO in August 2021 that there is no specific documented process for developing campaign surveillance activity. CASA noted:

It follows the basic process of the sector for the campaign being identified by the Executive, Terms of Refence identified and the National Manager Surveillance assigns responsibility for coordinating and managing the campaign to a suitably qualified Surveillance Manager (and then the surveillance is undertaken as outlined in the CASA Surveillance Manual).

National Oversight Plan

2.12 While there has been reference to the National Oversight Plan in CASA’s portfolio budget statements since 2020–21 as well as CASA’s corporate plans since the 2018–19 corporate plan, the NOP has not been documented to ensure a comprehensive surveillance approach, despite being CASA’s overarching model in conducting surveillance activities. Implementation of the NOP has also not been supported by a project plan, a monitoring and evaluation approach with defined review milestones and policy, procedures, and guidance to support campaign and response surveillance. A draft report of an internal review undertaken by CASA in 2020 came to similar conclusions.⁸

Governance arrangements

2.15 CASA’s governance framework for surveillance is largely incorporated into its general accountability and reporting framework (see Appendix 4). Key decision-making committees involved in surveillance oversight include:

• the Aviation Safety Committee (ASC) — an advisory and monitoring committee on matters related to regulatory and safety compliance, safety strategies, safety performance, international safety developments and emerging aviation safety risks; and
• the Oversight Executive Group (OEG) — an executive committee chaired by the executive manager, regulatory oversight that manages and administers decision-making on regulatory oversight including approving the NSSP schedule, determination of key focus areas for surveillance (see paragraph 2.65 for further information) and coordinating surveillance-related activities.

2.16 As of December 2021, the surveillance manual incorporated governance arrangements for surveillance activities for CASA’s regional offices and sectoral surveillance activities. This included monthly meetings of the ‘controlling office surveillance priority review group’ (SPRG) covering each regional office and weekly meetings of the ‘regional oversight committee’ (ROC) in each regional office. The purpose of these was to manage resource availability, detailed planning and identify necessary deviations for OEG consideration.

2.17 In August 2021, CASA advised the ANAO that this governance framework terminology was being phased out and was being replaced with weekly surveillance planning meetings and that, once this was finalised, manuals and policies would be updated to reflect the change.

2.18 CASA published an update to the surveillance manual in January 2022 with revised governance arrangements to reflect the national surveillance model that was implemented on an interim basis in December 2020 and finalised in July 2021.

Engagement with government agencies and stakeholders

2.19 The Australian Transport Safety Bureau (ATSB) is a key partner agency for CASA in its surveillance approach. The ATSB is responsible for the independent investigation of aviation accidents and other safety occurrences.

2.20 CASA has a formal Memorandum of Understanding (MOU) with the ATSB to guide the separation of responsibilities, communication/notification arrangements and cooperation in relation to commencing and conduction investigations. CASA has regular interactions with the ATSB about the status of ATSB safety investigations safety data.⁹ CASA also participates in biannual operational meetings with the ATSB as required under the MOU.

2.21 The Aviation Safety Advisory Panel (ASAP) is the primary advisory body through which CASA directs its engagement with industry and seeks input on current and future regulatory and associated policy approaches.¹⁰

2.22 CASA has regularly updated the ASAP on the progress of its organisational changes and surveillance approach since September 2019 and has also considered surveillance activities on COVID-19 pandemic affected AHs and the development of sector risk profiles.

Are there appropriate policies, procedures and guidance in place to support CASA’s surveillance approach?

Policies and procedures

2.23 The primary objective of CASA’s surveillance activities is to assess whether an AH is fulfilling its obligations under the Civil Aviation Act and associated regulations. CASA has a range of policies and procedures to support its inspectorate staff to plan and undertake surveillance activities to make such an assessment, with some of these publicly available (see Appendix 5). The provision of technical guidance, tools and the provision of safety critical information is a critical element identified by the International Civil Aviation Organization (ICAO) as part of complying with international obligations (see paragraph 2.79).

2.24 CASA’s surveillance manual and the NSSP manual are the key guiding documents for staff in preparing for and undertaking surveillance activities, including guidance on:

• the types of surveillance events conducted;
• governance and accountability arrangements for the review and approval of surveillance events and event reporting;
• processes required when conducting surveillance events including planning documents and arranging and documenting meetings;
• processes required to reschedule or not proceed with NSSP events;
• processes for the referral of AHs for enforcement action; and
• management of information within the IT systems that support surveillance activities.

Guidance

2.25 Guidance material needs to provide CASA staff with clear processes and requirements to be followed in undertaking surveillance activities. Where guidance material is unclear, there is a risk that CASA’s surveillance approach will be applied incorrectly.

2.26 CASA’s central guidance to inspectorate staff on key elements of the surveillance process are publicly available on the CASA website and information provided to AHs as part of the audit process provides links to this material.

2.27 The ANAO’s examination of CASA guidance material revealed instances where guidance was confusing and incomplete. CASA records include staff noting a lack of guidance or uncertainty in relation to some surveillance tasks. The ANAO also identified some areas where the surveillance manual could be improved. For example:

• Neither the CASA Surveillance Manual nor the NSSP Manual identify the interval between level 1 surveillance events. This is despite ICAO noting in section 1.6 of its Manual of Procedures for Operations Inspection, Certification and Continued Surveillance that: ‘The areas to be covered in the surveillance activities over a period should be similar to those examined during the original certification process’.
• CASA has issued guidance related to certain surveillance-related activities using Temporary Management Instructions (TMIs).¹¹ While elements of some TMIs have been incorporated into the surveillance manual at the termination of a TMI, there have been instances where there has been a significant gap between the expiry of a TMI and the issuing of a revised TMI.

2.28 The surveillance manual was updated nine times between the start of 2017 and December 2021. An update to the manual was published in January 2022 to reflect the changed CASA organisational structure that was implemented on an interim basis eleven months earlier (in December 2020 before being finalised in July 2021) and to remove material that was no longer relevant to existing surveillance practices. The NSSP manual, published in April 2019 some 10 months after the implementation of the NSSP in July 2018, has not yet been updated to reflect CASA’s changed organisational and surveillance governance arrangements and the implementation of the AOC subsequent issue review process from 1 July 2019.

Stakeholder communication

2.29 CASA’s stakeholder engagement and communication on surveillance activities includes direct contact with AHs, consultation through the Aviation Safety Advisory Panel (see paragraph 2.21) and broad industry communications via its website and regular industry updates.

2.30 CASA’s broad communication on surveillance activities to industry included information about the Regulatory Services and Surveillance Transformation (RSST) project implementation (see paragraph 1.11) over the period October 2019 to June 2021. CASA’s adoption of key focus areas for surveillance activities (see paragraph 2.65 for further detail) for the periods November 2020 to June 2021 and for 2021–22 was communicated to stakeholders using CASA’s industry communications channels and individual messages to AHs for the period November 2020 to June 2021.

Delegate oversight

2.31 The Civil Aviation Act and associated regulations provide that some of CASA’s powers and functions may be delegated to a ‘delegate’ or ‘authorised person’.¹² Industry delegates are managed across different areas of CASA and information about delegates is derived from several IT systems.

2.32 CASA has undertaken a range of activities since early 2017 to improve the systems used to store and report information about delegates. This included approving a workplan to update the industry delegates management manual in September 2018.

2.33 A CASA internal review in late 2019 that produced an updated inventory of industry delegates noted:

It cannot be confirmed that this is a complete list of delegates within CASA. In collecting data, the team discovered inconsistencies in the information provided and information able to be collected through other means, for example there are several delegates managed through the regional offices and these appear to be managed individually. There is no complete list of these (regional Office issued instruments) available to the research team and collating one was not feasible in the time allocated to this enquiry.

2.34 An update provided to the CASA board in April 2021 noted the ‘approximate’ numbers of delegates in different categories and that data on half of the delegate categories was held in the European Aviation Processing (EAP) system. Of the 12 delegate categories, five categories, covering approximately 1,652 delegates, were not included in the NSSP framework for scheduled surveillance.

2.35 As of December 2021, the latest version of the Industry Delegate Management manual was published more than 10 years earlier (in May 2011). CASA advised the ANAO in January 2022 that it is ‘currently reviewing the [manual] and intend to incorporate the [manual] into the existing Airworthiness Protocol Framework over time as part of continuous improvement’. CASA has not committed to a timeframe for the completion of this work. CASA is also considering incorporating additional classes of delegates within the NSSP in future years.

Does CASA apply a risk-based approach to planning and conducting surveillance activities?

Identification of relevant risks

2.36 Under section 16 of the Public Governance, Performance and Accountability Act 2013 (PGPA Act), CASA is required to establish and maintain an appropriate system of risk oversight and management. CASA has an enterprise risk management framework which incorporates CASA’s: Board policy on risk management; Board strategic risks and risk appetite statement; and risk management manual.

2.37 CASA’s risk management manual establishes risk management requirements and guidance to staff in identifying and managing risks in relation to operational/project risks and strategic risks.¹³ This includes a ‘CASA risk matrix’ to guide staff in rating the likelihood and consequence of risks as well as determining the effectiveness of risk controls. The risk management manual notes that:

Aviation safety risk is considered and managed in accordance with the CASA Aviation Safety System Manual [CASS]. The process for managing aviation safety risk is consistent with the CASA Risk Management Policy and aligns, where feasible, to the CASA Risk Management Manual.

Aviation safety risk is monitored by the Aviation Safety Committee.

2.38 The aviation safety system manual includes a risk assessment matrix that also incorporates a consequence and likelihood criteria rating. The consequence rating is between insignificant (no safety implications) and critical (aircraft accident with greater than nine fatalities) and the likelihood criteria includes consideration of quantitative assessment of the likelihood (where data is available).

2.39 CASA has advised that the:

CASS manual requires updating to accommodate the relatively recent publication of the CASA Risk Management Manual.

2.40 The ANAO considers that there is a lack of clarity in CASA risk management created by risk management guidance that is out of date.

Operational risks

2.41 Surveillance operational risks involve those risks that CASA face while conducting surveillance activities, such as workplace health and safety (WHS), surveillance conduct and IT system management. CASA identifies and records operational risk in a centralised risk register.¹⁴ Operational risks identified in business planning assessed as medium or high are reported to the Board Audit and Risk Committee (BARC) on a quarterly basis.

2.42 A key operational risk for CASA’s surveillance activities is the timely implementation of the surveillance related functionality of the European Aviation Processing (EAP) IT system. This is intended to replace CASA’s Sky Sentinel IT system (see Appendix 6).

2.43 CASA’s November 2021 risk management manual notes that operational risks are ‘captured in divisional business plans and presented on the business risk profile (reviewed quarterly)’. In February 2022, CASA advised the ANAO that risk ceased to be recorded in divisional risk registers following the release of the centralised risk register ‘so there is no need for them to appear there’. With the ANAO noting that this advice was inconsistent with the risk management manual, CASA advised the ANAO in March 2022 that:

CASA acknowledges that this excerpt would benefit from revision as it implies risks can still be recorded in divisional business plans and risk registers. In reality, risks are identified and assessed through business planning (which is conducted during preparation of divisional business plans) but the risks are now being recorded in the Risk Register not individual risk registers or business plans.

2.44 The ANAO considers that there would be merit in CASA promptly clarifying in the relevant manuals the process for the recording and assessment of risks and how these are to be reported.

Aviation safety risks

2.45 CASA holds and receives a wide range of aviation safety information and data to identify aviation safety risk at a sectoral and operator level including:

• surveillance/regulatory activities;
• sectoral analysis conducted by CASA;
• industry reporting such as mandatory defect reporting, public reporting of low flying/aviation safety concerns and occurrence reporting received from the ATSB; and
• accident reporting by the ATSB and other aviation regulators including individual accident reports and trend reporting.

2.46 CASA uses a range of separate approaches to store this aviation safety information such as the centralised risk register, documented Sector Safety Risk Profiles (SSRPs), surveillance activity information from Sky Sentinel and regulatory information held in the EAP system. Aviation safety risks were to be identified on the CASA aviation safety risk register prior to the adoption of a centralised register in December 2021.

2.47 Some of these approaches are yet to be fully implemented or are undergoing change so not all aviation safety risks are identified or are current.

• CASA changed its approach to sector risk profiling in August 2021 away from developing a published SSRP report to forming sectoral working groups that would convene for each sector every 18 months and publish a ‘sector risk focus document’. CASA’s work on SSRPs commenced in 2014 and only 7 of 14 SSRPs were finalised.
• CASA advised the ANAO in November 2021 that it is ‘currently populating the aviation safety risk register based on the outcomes of internal risk analysis workshops. To date Large Aeroplanes and Small Aeroplanes have been completed. The risk assessments are developed to support CASA’s decision-making by demonstrating how CASA controls (regulations) mitigate risk. In the future it is hoped that outputs of surveillance can be used in conjunction with these risk assessments to better understand the impact of surveillance trends on risks for different aviation sectors’.

2.48 The ANAO considers that CASA needs to finalise its work on identifying aviation safety risks and update these on a regular basis.

Surveillance planning

Consideration of risk consequence and risk likelihood

2.49 The scheduling of planned surveillance using the NSSP was implemented on 1 July 2018. Since the implementation of the NSSP, around 1,000 surveillance events have been scheduled annually (see Appendix 7).

2.50 According to the NSSP manual, the first step in prioritising authorisation holders (AHs) for the purpose of developing the annual surveillance schedule has been to allocate AHs into three groups (A, B and C) based solely on risk consequence with surveillance oversight based on their grouping (see Table 2.1).

• Group A authorisation holders are associated with a higher risk consequence, such as regular public transport carrying more than 9 passengers, aerodromes with high to very high levels of regular public transport or general aviation traffic and associated service and product providers.
• Group B authorisation holders are those which do not more appropriately belong in groups A and C.
• Group C authorisation holders are associated with a lower risk consequence, such as those involved in small operations involving single pilot and no passenger activity, aerodromes with occasional general aviation traffic and certain low volume product manufacturers. Charter operators are not included in this group.

Table 2.1: National Surveillance Selection Process AH groups and number of identified AHs, 2021–22

Note a: The AHPI (Authorisation Holder Performance Indicator) tool is an assessment completed by CASA staff using a word picture questionnaire. The questionnaire covers factors associated with the AHs management, organisation, operations and regulatory history.

Source: ANAO analysis of CASA documentation.

2.51 CASA’s risk management manual requires consideration of both risk consequence and risk likelihood (see paragraphs 2.37 to 2.39). CASA’s categorisation of authorisation holders solely based on risk consequence is inconsistent with these requirements. In February 2022, CASA advised the ANAO that:

The Minister’s Statement of Expectations makes clear the expectation that our focus has to be on the safety of the travelling public. To comply with this, the consequences of an accident must be a key factor, not just the likelihood. I also note that our approach is consistent with the Commonwealth risk management better practice guidance as well as ISO 31000:2018 (Risk Management).

2.52 The ANAO notes that the Commonwealth risk management guidance¹⁵ and ISO 31000:2018 both include a risk management approach that incorporates risk consequence and risk likelihood as part of the risk management process.

2.53 The ANAO suggests that CASA should incorporate risk likelihood into its risk assessment approach to group authorisation holders as part of its surveillance approach. If this approach is not adopted, the ANAO suggests CASA should clearly establish the basis for not considering risk likelihood in its allocation of authorisation holders into the three surveillance groups.

Consideration of sectoral and operator risks

2.54 As noted in Table 2.1, a measure of operator risk is incorporated into the NSSP scheduling for group B only. There is no consideration within the group allocation criteria for the prioritisation of surveillance according to identified sector risk, such as that identified in sector-based risk profiles (see paragraph 2.47).

2.55 Risk indicators used for surveillance planning by aviation regulators in New Zealand and Canada include several broader indicators of risk including national safety concerns and initiatives, sector trends and the presence of a safety management system. Aviation regulators in these two countries also use risk likelihood as a basis for prioritising surveillance.

2.56 CASA considered an alternative method in January 2020 to assess risk to prioritise surveillance resources to achieve the NSSP target in the second half of 2020–21. This approach proposed the use of an operator risk profile for AOCs and Approved Maintenance Organisations (AMOs) based on a formula that included the Authorisation Holder Performance Indicator (AHPI) score, location, incidents and number of aircraft operated/maintained and the number of regulatory services tasks completed, the latter of which was assumed to decrease risks as a result of AH interactions with CASA. CASA advised the ANAO in January 2022 that this approach was one option considered in reviewing alternative methods to achieve the NSSP target. The ANAO notes that the approach was not implemented and that CASA has advised the ANAO that the current approach includes the use of desktop surveillance events and ‘remote surveillance’ using a multimedia approach to validate AH compliance without physically attending on-site.

COVID-19 pandemic response and consideration of sectoral and operator risks

2.57 CASA paused surveillance for two weeks from 20 March 2020 to ‘enable a considered review of planned surveillance for the remainder of the NSSP year’. In May 2020, CASA established principles to adapt the NSSP schedule to consider emerging and changing risks for each sector and an assessment of all AHs against identified risks.

2.58 An interim methodology to give a more detailed consideration of risks at the sectoral and operator level was established in June 2020 to guide decision making on undertaking surveillance to October 2020. This was to consider those operators who were not able to be audited as part of the 2019–20 scheduled program, those operators included in the first three months of the 2020–21 program and those response activities on hand or that may arise over the period to October 2020.

2.59 Additional risk criteria considered in prioritising surveillance for these operators over this period included:

• sectors where there was an imminent increase in sector risk identified; and
• operators where organisational risks were rated higher using changes in the level of activity, the AHPI score, whether there had been a loss of key personnel and the share of staff who were no longer operational.

2.60 CASA advised the ANAO in December 2021 that this surveillance methodology was not completely implemented. CASA noted that the remaining NSSP events for 2019–20 were assessed to determine highest priority and the NSSP for 2020–21 was split into two tranches with the first including a three-month schedule that included organisations assessed to be a higher priority.

Sectoral approaches to surveillance prioritisation

2.61 There are several sectors that are not included in the NSSP prioritisation process or have a sector-based prioritisation process.

• Industry delegates such as ‘Designated Aviation Medical Examiners’ and ‘English language standard assessors’ are subject to alternate surveillance activities and are not included in the NSSP.
• The aerodrome and remotely piloted aircraft systems sector are prioritised separately and then included in the NSSP schedule. They are not rated consistently with the other sectors.

2.62 The ANAO considers that the segmented approach to surveillance prioritisation for industry delegates that are not included in the NSSP and the approach used for some sectors within the NSSP does not provide for a clear and consistent risk management approach across CASA. As noted in paragraph 2.35, CASA is considering incorporating additional classes of delegates within the NSSP in future years.

Surveillance conduct

Surveillance scope

2.65 Determining the scope of surveillance events incorporates elements of judgement by CASA staff in assessing risk and is informed by a range of information including previous surveillance events, and other safety data related to an AH. As part of the NOP, CASA intended to annually mandate from 1 July 2019 key focus areas (KFAs) for each sector that would be used by surveillance staff in their surveillance activities, with the intention that 80 per cent of effort would be directed towards the specified high risk areas.

2.66 The initial development of KFAs in early 2019 was informed by factors such as issues that arose from routine monitoring, high risk occurrence types and sector risks identified through the sector risk profiles. However, the development of the KFAs was delayed and none were published for the 2019–20 period.

2.67 KFAs for the period to 2020–21 were not published until November 2020, with CASA’s approach to developing the KFAs noting that:

COVID-19 significantly changed the aviation risk environment. This means that reliance on historical data and sector profiles, which is often a good predictor of future conditions, did not provide CASA with an appropriate focus this year. Instead RSSD [Regulatory Services and Surveillance Division] developed an alternative approach to utilise past data and sector risk profiles, in combination with COVID-19 sector-based survey information, inspector expertise and international benchmarking.

2.68 CASA announced the KFAs for 2021–22 in September 2021. CASA’s approach to KFAs for this period was largely based on a qualitative review by CASA’s surveillance staff, rather than a consideration of sector-based data on safety risks.

2.69 Delays to the identification and use of KFAs have limited the ability of CASA to target identified areas of risk within sectors. Further development of KFAs should incorporate quantitative data on risk in addition to qualitative reviews by CASA staff and be communicated to industry in advance of each year’s NSSP schedule commencing.

Managing conflicts of interest

2.70 There is a potential ongoing risk to CASA’s reputation and the effective conduct of surveillance activities that inspectorate staff may be influenced or appear to be influenced in their decision making by any professional background in the aviation industry, personal interests or the interests of family members.

2.71 CASA staff are required under an instruction issued by the CASA Director of Aviation Safety¹⁶ (see paragraph 1.2) to complete a conflict of interest (COI) declaration using CASA’s online COI management system within seven days of commencement with CASA.¹⁷ Staff are also required to review and confirm that their COI declaration remains correct during their annual performance cycle discussions and to immediately update their declaration when they become aware of a change in circumstances which might give rise to an actual, potential or perceived conflict of interest between their private or pecuniary interests and their duties as a CASA officer.¹⁸

2.72 A May 2021 internal audit report on CASA’s fraud control and integrity review found that more than 30 per cent of staff who had declared a conflict of interest when they had applied for a job with CASA had not made a declaration within seven days of their commencement and that CASA did not have a system in place to manage the requirement to annually review and confirm that COI declarations remained current.

2.73 Examination by the ANAO of a targeted sample of COI declarations made by selected senior CASA surveillance staff and an inspectorate team involved in regulatory activities revealed a range of issues associated with the implementation of CASA’s COI approach including:

• the non-declaration of previous employment within the aviation sector;
• senior surveillance staff not completing a COI declaration or making an initial declaration several years after their commencement or as required in November 2017 at the time of the implementation of the COI declaration management system;
• inconsistent practices observed for staff in making new declarations including where a conflict of interest had been noted by managers; and
• the holding of airline shares by several staff did not always result in action being taken to manage conflicts of interest.

2.74 CASA advised the ANAO in February 2022 that it had undertaken activities in response to the internal audit on fraud control and integrity review including:

• revising and re-releasing the conflict of interest training module in late 2021, requiring completion by staff of COI training every two years; and
• a planned amendment to the performance management system at the beginning of the 2022–23 cycle to prompt staff to discuss their COI declarations with their supervisors and update as necessary.

Does CASA’s surveillance approach comply with Australia’s obligations under international agreements and commitments?

International agreements and commitments

2.77 The Civil Aviation Act requires CASA’s surveillance activities to be performed in a manner consistent with Australia’s obligations under the Chicago Convention¹⁹ and any other agreements between Australia and other countries relating to the safety of air navigation.

2.78 CASA has entered into bilateral agreements with eight countries relating to the reciprocal recognition of technical approvals for airworthiness and memoranda of understanding for the promotion of civil aviation safety. CASA also has agreements with the European Aviation Safety Agency (EASA) regarding cooperation on the oversight of the Airbus A380 fleet and for the inspection of foreign aircraft under the EASA Safety Assessment of Foreign Aircraft (SAFA) program.

2.79 Australia is required to meet eight ‘critical elements’ under the Chicago Convention, five of which directly relate to surveillance activities (Table 2.2).

Table 2.2: Chicago Convention Annex 19 critical elements related to surveillance

Source: International Civil Aviation Organization, Annex 19 to the Convention on International Civil Aviation, Safety management, Second edition July 2016.

2.80 The International Civil Aviation Organization (ICAO), the international organisation established by the Chicago Convention, undertakes a range of oversight activities to ensure states are fulfilling their responsibilities under the convention. This has included two audits of Australia’s arrangements to implement the critical elements which scored Australia’s effective implementation as 83.38 per cent in 2008 and 94.98 per cent in 2017. In 2008, the effective implementation for critical element 3 (State civil aviation systems and safety oversight functions) was 90.93 per cent and for critical element 4 (technical personnel qualifications and training) it was 51.14 per cent. In 2017, the respective effective implementation rates for these critical elements were assessed to be 98.77 per cent and 92.31 per cent respectively.

Complying with international agreements and commitments

2.81 Annual calendar year targets for the inspection of foreign aircraft are agreed with EASA for the SAFA program each year. CASA has largely met its agreement with EASA, having completed 494 EASA inspections of foreign aircraft over the period July 2017 to December 2021. While CASA exceeded the target in 2019, CASA advised that it did not meet the 2020 target due to the inability to undertake inspections from March to October 2020 as a result of the COVID-19 pandemic restrictions.

2.82 In assessing compliance with the Chicago Convention requirements, the ANAO examined CASA’s performance against selected parts of the surveillance-related critical elements.

Inspectorate resourcing

2.83 ICAO critical element 3 requires that signatories to the Chicago Convention ensure that aviation regulators are established with sufficient and qualified personnel and are adequately resourced for the management of safety.

2.84 As of 30 June 2021, CASA employed 857 staff, with around 115 staff allocated to surveillance activities as of February 2022. Of the total staff at 30 June 2021, 338 staff worked in technical classifications (Table 2.3). Not all these technical staff undertake surveillance activities, with the separation of guidance and regulatory functions from surveillance across some areas. The number of technical staff has declined over almost all classifications, with an increase in staff employed in the ‘Aviation Safety Regulator – Other’ classification.

Table 2.3: CASA technical staff, 2017 to 2021

Note a: The ‘Aviation Safety Regulator Other’ category includes 84 different types of position classifications including cabin safety inspectors, dangerous goods inspectors, manufacturing inspectors and remotely piloted aircraft systems inspectors.

Source: CASA annual reports (2017 to 2020) and CASA documentation (June 2021).

2.85 Proposals by CASA staff for additional surveillance resources put to CASA management over recent years have included modelling of anticipated workloads across parts of CASA’s inspectorate, outlined key priorities that need to be met and noted the need for succession planning in some areas to cover staff absences considering the number of staff at, or approaching, retirement age.

2.86 Internal CASA communication on achieving the 2020–21 NSSP targets included advice in September 2020 to adjust the type of audit required to be conducted from a more detailed level 1 audit to a level 2 audit if a risk assessment supported this approach. Another proposal in January 2021 to achieve the remaining events in the last six months of the period was to reconsider, following a separate risk assessment process, whether the scheduled events should be carried out as a desktop audit, a virtual audit or an on-site audit.

2.87 In reviewing the resources required in May 2021 to fully deliver the NSSP program for the Regulatory Oversight Division in 2021–22, CASA staff estimated in a paper considered by the OEG that it required an additional 43 technical staff, including 26 airworthiness inspectors, nine flight operations inspectors and eight safety systems inspectors. CASA advised the ANAO that this paper was considered in the context of finalising staffing levels/allocation for the national operating model structure that was implemented on 12 July 2021.

2.88 CASA’s People Committee identified in May 2021 that CASA did not have a formal succession planning strategy for critical roles and that the lack of a formal approach to succession planning may impact on the ability of CASA to deliver on critical functions in the future. CASA advised the ANAO in February 2022 that there were several workforce planning projects underway to address these issues including a project to review critical roles and a review of the job family framework.²⁰

2.89 CASA’s compliance with critical element 3 is reduced by the lack of a resourcing model for surveillance activities and the absence of formalised succession planning. CASA should develop a surveillance resourcing framework to match anticipated levels of activity with the technical and other staff needed to meet these levels, including continuing with succession planning activities recently commenced. The framework should also consider comparisons with equivalent overseas regulators on surveillance staffing and the level of aviation activity in different sectors.

Technical personnel skills, training and competency

2.90 A training directive issued by the CASA’s Director of Aviation Safety (see paragraph 1.2) requires that all regulatory staff must be provided with the technical and administrative training to effectively fulfil safety oversight responsibilities.²¹ A stated purpose of the directive is to ensure that staff training complies with the ICAO critical element 4 requirements.

2.91 CASA has established a formal training program for its inspectorate staff that incorporates mandatory induction training covering general organisational policies as well as technical training which includes on the job elements and ongoing recurrency requirements. CASA aligns its training with ICAO requirements.

2.92 In late 2018, mandatory training completion rates for foundation training for CASA’s flying operations; airworthiness; and safety systems inspectors across all CASA regions were at or close to 100 per cent. Completion rates for on-the-job training and recurrent training varied across regions and the evidence showed that training had not been completed in full for these inspector types. While training data from September 2021 show that there has been an overall improvement in completion rates for these inspector types, with foundation and advanced training completion rates at or approaching 100 per cent, advanced training completion rates across 30 ‘other’ inspector types covering 135 inspectors was 71 per cent.

2.93 CASA delivered recurrency training for its inspectorate staff in 2017, 2019 and 2021. The attendance rate for the 2017 training was less than 60 per cent and the attendance rate for the 2019 training, delivered between June 2019 and February 2020, was 98.7 per cent. CASA advised the ANAO that the attendance rate for the 2021 training, delivered in October 2021, was 93.2 per cent, with seven staff yet to begin the training.

2.94 Staff in specified positions must hold and maintain flight crew licence qualification and to undertake flying currency and training (FCAT) to ensure they can adequately and safely conduct certain duties relevant to their position. Approvals to undertake FCAT are made on an ad hoc basis as individual inspectors look ahead at their own allocated jobs and identify the required currency for the tasks. Since 2017–18, annual expenditure on FCAT for CASA staff has declined by 70 per cent, from $2.2 million in 2017–18 to $0.7 million in 2020–21 (Figure 2.1).

Figure 2.1: Flying currency and training expenditure, 2017–18 to 2020–21

Source: ANAO analysis of CASA documentation.

2.95 Internal audits conducted during 2019 and early 2020 on CASA’s training management and FCAT included findings relating to monitoring and follow-up of progress against mandatory training requirements, assessment of formal competency prior to the allocation of tasks and assessment of training programs against currency requirements for job roles. Implementation of these recommendations is monitored by CASA’s Board Audit and Risk Committee. While CASA had largely implemented the recommendations from these audits by December 2021, the implementation of some recommendations had been delayed:

• evaluation of training programs to determine training relevance to roles (expected completion March 2022) and criteria to conduct and sign off on the on-the-job training component of training programs (expected completion 24 December 2021);
• strengthening controls to limit allocation of tasks to staff with confirmed competency (expected to be closed in April 2022)²²; and
• improved FCAT planning (expected completion March 2022).²³

2.96 CASA advised the ANAO in January 2022 that:

Subject to [Board Audit and Risk Committee] approval, the Training Internal Audit recommendations will meet the expected completion dates. However, should there be a need for a digital solution to be built and implemented, that will not be achievable by the current completion date.

[Regulatory Oversight Division] expects that the review/planning portion of the FCAT Internal Audit recommendations will met by 31 March 2022 … However, implementation may not be fully completed by March 2022 as any changes to processes/systems will require consultation as well as training of staff. It may also require a digital solution to be built and implemented.

2.97 While CASA’s surveillance staff have largely completed mandatory training requirements and undertaken recurrency training on a regular basis, without the finalisation of the internal audit recommendations noted in paragraph 2.95 CASA’s compliance with ICAO critical element four is reduced.

2.98 The ANAO suggests that the internal audit recommendations on CASA’s training management and FCAT, noted in paragraph 2.95 be fully implemented.

Other surveillance-related critical elements

2.99 ANAO analysis shows that CASA can improve on its compliance with other surveillance-related ICAO critical elements including:

• surveillance obligations (critical element 7) — formalising and documenting the NOP (paragraph 2.12);
• technical guidance, tools and provision of safety-critical information (critical element 5) — improvements to CASA’s guidance (paragraphs 2.23 to 2.28); and
• resolution of safety concerns (critical element 8) — improvements to CASA’s enforcement approach (paragraphs 3.37 to 3.40).

3.1 Surveillance is CASA’s method of monitoring aviation industry compliance with the regulatory framework. The ANAO examined whether CASA:

• is monitoring compliance because non-compliance with regulation by the aviation industry increases the risk of incidents and accidents; and
• has developed a process for its review and update of its surveillance approach as required by the Public Governance, Performance and Accountability (PGPA) Act 2013.

Does CASA effectively monitor industry compliance and act upon identified non-compliance?

3.2 CASA is responsible for conducting surveillance events on all civil aviation authorisation holders (AHs) within Australia and for Australian AHs overseas. CASA undertakes different levels of surveillance on AHs that CASA indicates will involve a broadly scoped systemic review or be more specific, narrowly scoped with a focus on a particular aspect of the authorisation. The CASA Surveillance Manual notes that the type of audit to be undertaken will be determined by the surveillance team (previously referred to as the authorisation management team)²⁴, that is, there is flexibility in determining the type of surveillance event to be undertaken.

3.3 A level 1 audit²⁵ is a: ‘…structured, forward-planned, larger-type surveillance event’. Level 1 audits include:

• Systems Audits — generally involves a multi-disciplinary team of inspectors undertaking a more comprehensive on-site assessment of activities that an operator is authorised to undertake;
• Health Checks — a more narrowly scoped systems audit requiring less resources with a focus on specific areas of the operator’s authorised activities; and
• Post Authorisation Review — a surveillance event that CASA mandates to be conducted within six to 15 months after an initial authorisation is issued ‘…to ensure standards are being maintained’.

3.4 A level 2 audit is less formal, shorter in duration with the scope of the audit more narrowly defined ‘…and may be in the form of checklist-based compliance and product checks of a specific section of its systems’.

3.5 Figure 3.1 shows an overall decline in the total number of all surveillance events (level 1 and 2) on AHs holding all certificate types from 2017–18 to 2020–21. There was a substantial rise in surveillance event activity in 2018–19, largely driven by a greater proportion of response events (not planned) compared to planned events conducted with declines in 2019–20 and 2020–21 over the period impacted by the COVID-19 pandemic.

Figure 3.1: Total number of surveillance events: 2017–18 to 2020–21

Notes: CASA data used for this figure cannot be disaggregated showing level 1 and 2 surveillance events.
CASA noted changes to regulations and the surveillance approach over this period.

Source: ANAO analysis of CASA data.

Assessment of an authorisation holder

3.6 The CASA Surveillance Manual notes that the first stage in CASA’s surveillance process is to assess the apparent risk to safety presented by an AH. Surveillance Teams (Authorisation Management Teams prior to January 2022) are expected to assess a variety of information including outstanding safety findings and safety finding history; system risk profile/history; time since the last level 1 and level 2 surveillance event; any available additional intelligence and results from the last authorisation holder performance indicator (AHPI) survey (discussed at paragraph 3.9).

3.7 Prior to January 2022, Authorisation Management Teams were also expected to formally discuss AH performance profiles within the team either one, six or 12 monthly, depending on the risk rating of the AH. There were no CASA meeting records to indicate there were formal discussions by the relevant oversight groups responsible for the review of AH performance profiles (pre and post-implementation of the National Oversight Plan), as required according to the CASA Surveillance Manual (CSM) prior to January 2022. CASA advised the ANAO in November 2021 that:

This level of monitoring suited the previous regional model, but not the national model so there is an element of ‘catching up’ to do in relation to the CSM. The previous model was conducive to considering authorisation holders by regional office as it was a smaller group to review. The new national model creates a much larger group to conduct these reviews on which is less efficient and the level of effort this requires may have been misunderstood. The national model is shifting to a more ‘data profile’ approach which will also include a monitoring portion but that is still developing.

3.8 Since the January 2022 update of the CSM, Surveillance Teams are not required as a team to formally discuss AH performance profiles.

Authorisation holder performance indicator (AHPI) survey

3.9 One of CASA’s key tools to inform its assessment of an AH and the prioritisation of surveillance scheduling for group B authorisation holders (AHs) is the AHPI. An AH is assessed by CASA and scored according to the AH’s response to the AHPI ‘word picture survey’. The survey asks questions that will be associated with the AH’s compliance covering relevant documents and procedures, decision making, assurance, training, and communication that, if deficient, are deemed by CASA to have the potential to trigger or contribute to adverse safety outcomes.

3.10 The highest 10 per cent of AHPI scores will have a category one oversight posture (prior to January 2022, formal discussion was expected to be held with the AH within one month of completing the AHPI, from January 2022 this requirement changed to 6 monthly), and the remainder category two (prior to January 2022, formal discussion was expected to be held with the AH within 6 months of completing the AHPI, from January 2022 this requirement changed to 12 monthly).

3.11 The CASA Surveillance Manual indicates that AHPI assessments are undertaken every 6 to 12 months, depending on the risk to safety posed by AHs. CASA was unable to provide time series data on the progress in undertaking AHPIs, including at a regional level, from 2018–19 to 2020–21 as ‘…Sky Sentinel is unable to produce reports at this level of detail’.

3.12 Figure 3.2 illustrates the ANAO’s point in time analysis of AHPIs overdue as of 30 June 2021.

Figure 3.2: Number of AHPIs overdue as of 30 June 2021

Source: ANAO analysis of CASA data.

3.13 As at 30 June 2021, there were nearly 500 AHs which CASA identified as having overdue AHPIs, or 21 per cent of all AHs. The proportion of overdue AHPIs considered by CASA to be high priority was 5 per cent.

3.14 CASA currently reports weekly on AHPIs due in the next month and AHPIs overdue to its National Surveillance Managers’ meeting. CASA in its reporting to the Aviation Safety Committee (ASC) in October 2021, indicated that ‘…a backlog of AHPIs exists due to previous prioritisation away from AHPIs in some regional offices. Current surveillance managers are now conducting a focussed campaign to complete AHPIs. Further, the forthcoming Surveillance Workshop will be tasked with reviewing the AHPI process to determine if there is a more effective way of achieving the safety outcome desired’ (further detail on the ASC is at paragraphs 2.15 and 3.52 to 3.56).

3.15 On 2 November 2021, there were 322 AHPIs overdue. As of 16 March 2022, there were 420 AHPIs overdue.

Planned Surveillance

3.16 The annual development of the NSSP schedule generally commences in April each year with a draft schedule developed by reviewing data on AHs in Sky Sentinel and the electronic document management system files; automatically including all Group A AHs; and prioritising Group B AHs (see paragraph 2.50 for further detail on groups). The NSSP annual schedule is then sent to managers and staff in regional offices for further refinement based on local knowledge with the final schedule approved by the Oversight Executive Group (OEG).

3.17 An AH can hold multiple related certificate types. Where an AH holds multiple related certificates, CASA noted in its NSSP 2020–21 draft schedule development guidance that subordinate or lower priority permissions are not included in the NSSP, although CASA also notes that all related permissions must be considered when preparing to conduct planned surveillance for relevant AHs.

3.18 Figure 3.3 shows NSSP surveillance events reported by CASA as either commenced or completed as a proportion of AHs from 2018–19 to 2020–21. It illustrates a significant decline over time in the proportion of AHs that are being subject to surveillance. CASA advised the ANAO in November 2021 that it is not possible to provide data on planned events prior to the introduction of the NSSP.

Figure 3.3: NSSP surveillance events as a proportion of all authorisation holders

Note: NSSP surveillance includes commenced or completed events in the relevant year.

Source: ANAO analysis of CASA data.

Response and campaign surveillance

3.19 Table 3.1 shows the number of response and campaign surveillance events undertaken by CASA between 2018–19 and 2020–21 (further detail on response and campaign events can be found at paragraphs 2.9 and 2.11). There was a 50 per cent decline in response events each year. There were 6 and 7 campaign surveillance events undertaken in 2018–19 and 2020–21 respectively, with no campaign surveillance events undertaken in 2019–20.

Table 3.1: Response and campaign events

Source: CASA data.

3.20 CASA calculates the number of response events by determining those events that occurred that were not: identified in Sky Sentinel as campaign events; or on the NSSP schedule. Prior to the introduction of the NSSP, CASA has indicated that ‘…there was no construct that differentiated between planned, response and campaign events’, and therefore the number of these events undertaken each year was not separately monitored.

3.21 The ANAO analysed CASA data for the period from 2018–19 and 2019–20 and found that there were substantial differences from the period when the response event was scheduled (CASA has advised the ANAO that this is a proposed date ‘which may get refined as planning evolves’) and when the response event started. There were 94 non-NSSP events where the period between ‘surveillance event scheduled start date’ and ‘surveillance event on-site start date’ was over 31 days. There were 41 non-NSSP events where the period between ‘surveillance event scheduled start date’ and ‘surveillance event on-site start date’ was over 62 days. There were 17 events that had a period of between 120 days and 369 days.

3.22 CASA advised the ANAO in December 2021 that:

There are a number of reasons why a surveillance event was created and kept open for a significant length of time, such as:

• The issue turned out to be a non-event and the person who raised the event didn’t go back into Sky Sentinel and cancel the event
• The event was created and then we couldn’t get to the location because of COVID travel restrictions
• The event was created as a follow up based on the results of a scheduled event and was used as a form of placeholder (this practice is no longer used)
• The event was created, and priorities changed and the event was put back to a more suitable date.

There is a lot more control now that we have a dedicated monitoring and response surveillance team and our surveillance reporting dashboard helps surveillance managers to monitor all of their scheduled and unscheduled surveillance events on a daily basis.

Monitoring an Air Operator’s Certificate (AOC) and an Approved Maintenance Organisation (AMO) certificate

3.23 The Air Operator’s Certificate (AOC) and the Approved Maintenance Organisation (AMO) certificate are the main certificate types that CASA oversights and, in 2020–21, made up 34 per cent and 29 per cent of all AHs respectively. The AOC is initially issued for 12 months and subsequent issue for either: three years for air transport (includes what is defined in CASA’s Air Operator’s Handbook as regular public transport and charter) and complex aerial work operations; or seven years for non-complex aerial work, and aerial application operations.²⁶

3.24 Figure 3.4 shows the number of surveillance events conducted on AOC and AMO AHs over the period from 2015–16 to 2020–21, and those events as a proportion of the total number of AOC and AMO AHs. The data shows a downward trend in the surveillance of AOCs and AMOs, albeit with increase in surveillance activity in 2018–19 (see paragraph 3.5). The ANAO’s analysis on the calendar years prior to the impact of the COVID-19 pandemic in 2020 also shows a slight downward trend in the surveillance of AOCs with AMOs.

Figure 3.4: Surveillance of AOCs and AMOs

Source: ANAO analysis of CASA data and CASA Annual Reports.

3.25 In planning surveillance, CASA categorise its AOCs and AMOs into groups A, B or C (see paragraph 2.50 for further detail on groups). As of 8 July 2021, CASA data showed an overall decline in AOC and AMO Authorisation Holders in Groups A and C, with an increase in Group B for AMO Authorisation Holders, seen in Table 3.2.

Table 3.2: AOC and AMO authorisation holders categorised by group

Source: ANAO analysis of CASA data.

3.26 The CASA NSSP Manual notes that Group A AHs should be subject to continuous surveillance and be subject to a surveillance event at least every year. CASA has not mandated the regularity of Level 1 audits, that is, the maximum period of time allowed between audits.

3.27 Figure 3.5 shows level 1 AOC and AMO surveillance events as a percentage of the number of AHs in that year for group A, B and C. While the data shows an increasing proportion of Group A AOC AHs being subject to level 1 audits, this is driven by a decline in AHs categorised into Group A and an increase in events from 2018–19 to 2020–21. For AMOs, the pattern continues for Groups A and B. While there was an increase in the ratio of level 1 surveillance events for Group C, this was driven by a substantial decline in AHs and only 1 surveillance event for Group C was conducted in 2018–19 and 2020–21.

Figure 3.5: Ratio of level 1 AOC and AMO surveillance events by group

Source: ANAO analysis of CASA data.

3.28 The ANAO examined CASA analysis on the conduct of AOC and AMO surveillance events undertaken in mid-2020. Findings suggested that there was an inconsistent approach to the surveillance of Group A large passenger carrying airlines (AHs that are to be subject to continuous surveillance) with significant variation in the extent of surveillance between airlines. For example, one major airline had not had a level 1 surveillance event for its AOC and AMO certificate for a period of four and three years respectively.

3.29 CASA issued a Temporary Management Instruction in November 2020 providing guidance on when an on-site surveillance event and a desktop surveillance event should be conducted prior to the renewal of an AOC (within 6 to 12 months of the AOC expiring for Group A and B AHs and within 3 months of the AOC expiring for all AHs respectively). Notwithstanding this, CASA does not monitor whether mandated surveillance events required for renewal of AOCs are commenced and completed within the required timeframes after being planned.

Aerodromes

3.30 Aerodrome surveillance is undertaken in the Regulatory Oversight Division where most surveillance is managed (further details in Appendix 4). Figure 3.6 shows the total number of aerodrome surveillance events and the number of those that were level 1 events from 2017–18 to 2020–21. There was an overall decline in level 1 surveillance events with the period prior to the potential impact of the COVID-19 pandemic showing the greatest decline.

Figure 3.6: Aerodrome surveillance

Source: ANAO analysis of CASA data.

Surveillance effort

3.31 In the NSSP schedule, the resources scheduled for the surveillance event (the number of hours or inspectors allocated) is dependent on whether the surveillance event is classified as a level 1 or level 2. Figure 3.7 shows the number of hours and inspectors (proxy for effort) taken to undertake both level 1 and level 2 surveillance events from 2017–18 to 2020–21. There has been a trend decline in the number of hours dedicated to level 1 and 2 surveillance events from 2017–18 to 2020–21. While there was an increase in the number of inspectors in 2018–19 dedicated to level 1 and 2 surveillance events, there was an overall trend decline from 2017–18 to 2020–21.

Figure 3.7: Surveillance effort — hours and inspectors

Source: ANAO analysis of CASA data.

COVID-19 pandemic

3.32 CASA expected the NOP to be impacted by the COVID-19 pandemic, with surveillance affected by restrictions which, to varying degrees, prevented on-site surveillance events. To address the impact of the COVID-19 pandemic, CASA planned to replace some on-site surveillance events with desktop and remote surveillance events where CASA deemed it appropriate. CASA identified that 230 out of a total of 306 NSSP surveillance events that weren’t conducted in 2019–20 (75 per cent) were due to the impact of the COVID-19 pandemic. CASA’s own observation has been that the number of desktop events did not increase substantially to compensate for on-site surveillance not being undertaken.

Metadata

3.33 Metadata²⁷ is necessary for developers and users of CASA data to accurately analyse and report on information collected in surveillance. In October 2019, the ANAO found, in assessing compliance against the government’s Digital Continuity 2020 policy (DC2020)²⁸ that CASA had partially implemented Australian Government Recordkeeping minimum metadata standards. CASA agreed to the ANAO recommendation that CASA should:

Complete the assessment of existing business systems and processes to ensure that information created, captured, stored used to deliver services, or inform decision making meets minimum metadata standards and functional requirements for the management, transferral, and disposal of information.

3.34 CASA inputs information into the Sky Sentinel system during surveillance activity on AHs and their interactions with CASA. Data from the Sky Sentinel system is relied on by CASA for reporting purposes. Sky Sentinel does not interact with other systems within CASA.

3.35 While CASA has an information quality control process in place, the ANAO’s examination of record keeping related to CASA surveillance events revealed that there were wide ranging omissions and errors.

3.36 Metadata for Sky Sentinel was last updated in November 2017 and does not account for the changes arising from the introduction of the NSSP and the NOP.

Acting on non-compliance

3.37 Critical element eight identified in ICAO’s international standards and practices on safety management (see Table 2.2) is on the resolution of safety concerns²⁹ and relates to processes to remedy identified issues including enforcement action. CASA reports that it enhances aviation safety by ‘taking appropriate enforcement actions when necessary’.

3.38 CASA identifies safety findings in its surveillance reports with AHs having 21 days to respond and address the findings. AHs that don’t address issues identified in safety findings can be referred to enforcement. The process involves the relevant regional Surveillance Manager recommending enforcement to the Executive Manager of the Regulatory and Oversight Division who, after reviewing the case, may refer the AH to CASA’s enforcement team.

3.39 As shown in Figure 3.8, while the number of safety findings for level 1 surveillance events has declined over the period from 2017–18 to 2020–21, this was combined with an initial increase in total surveillance events in 2018–19 followed by a decline thereafter so that the proportion of safety findings decreased before sharply increasing. For level 2 surveillance events, there was an increase for the period from 2017–18 to 2018–19 before declining for the period 2019–20 to 2020–21. As a proportion of total surveillance events, there was a slight decline.

Figure 3.8: Number and proportion of safety findings for surveillance events

Source: ANAO analysis of CASA data.

3.40 Since 2018–19, CASA has reported once, in its 2020–21 Annual Report, on outstanding safety findings (defined by CASA as not closed out). CASA advised the ANAO in October 2021 that it is unable to provide data on the number/type of surveillance referrals to enforcement and has indicated that it is working towards being able to develop a system to be able to report on this information.

Is CASA regularly reviewing, and where appropriate, updating its surveillance approach?

Review of the National Oversight Plan (NOP) and the National Surveillance Selection Process (NSSP)

3.43 While CASA’s NOP was implemented on 1 July 2018 with the introduction of the NSSP and defined and separately monitored campaign and response surveillance (see paragraph 1.11), the NOP is not supported by a review process including a proposed monitoring and evaluation approach (see paragraph 2.12).

3.44 The NSSP was expected to be reviewed and updated over time. An internal audit of governance arrangements supporting the NSSP in 2018, reported that:

Internal audit supports plans for further NSSP assurance activity as part of the 2018–19 Strategic Internal Audit Program.

3.45 The internal audit recommended:

that the NSSP project team pauses to conduct a review to consider the lessons identified, take action to remove ambiguity of the project throughout CASA, address the Project Management Plan review comments and finalise planning project artefacts, enforce project management disciplines, formalise milestones and track progress.

3.46 CASA management’s response was:

A review of the NSSP project was undertaken by EM RSSD and the PMP was finalised. The outcome of the review and commitment by CASA personnel resulted in the project delivering Phase 2 ‘on time’ and ‘within budget’.

3.47 However, the review was not documented, and there has been no comprehensive review of the NSSP approach since it was implemented including how risk is incorporated into surveillance planning.

3.48 CASA advised the ANAO in February 2022 that a review of the NSSP is expected to be completed by June 2022.

Safety Assurance Reviews

3.49 CASA’s process for review of aviation incidents and accidents, in addition to regulatory reviews related to the Australian Transport Safety Bureau and Corporate Integrated Reporting and Risk Information System (CIRRIS) incidents³⁰, includes an internal Safety Assurance Review (SAR).³¹ The purpose of the SAR is to determine whether any immediate safety related action is needed to be undertaken by CASA to prevent an incident or accident reoccurring.

3.50 The SAR considers the relevant AH profile; surveillance history; details of the incident; immediate action necessary for aviation safety; and learnings for CASA. While the CASA Executive Manager of the Regulatory Oversight Division (EM ROD) is responsible for commissioning the internal review, oversight responsibility was:

• not identified prior to June 2020;
• after June 2020 provided by the OEG, consistent with amendments to the CASA Surveillance Manual; and
• provided by both the OEG and the ASC from June 2021 until November 2021.

3.51 While CASA recommends that SARs are completed within 30 days of commissioning, this timeframe can be varied by the EM ROD. While CASA was able to provide eight SARs to the ANAO with incidents covering a three year period from 2019 to 2021, two of the reports were not dated. From CASA records, the ANAO identified a further three SARs commissioned, with one not finalised, dating back to 2019. In addition, ANAO analysis shows that there have been substantial delays between the commissioning of SARs and their finalisation (up to 484 days).

Aviation Safety Committee

3.52 CASA’s ASC first convened in April 2017. Membership of the ASC includes senior management across CASA who are scheduled to meet monthly. The ASC has evolved over time with its standing items for discussion being reviewed and updated since its inception.

3.53 Reporting on surveillance to the ASC has been primarily on:

• surveillance conducted compared to the schedule of planned surveillance;
• the outcome of campaign surveillance; and
• updates on the Regulatory Services and Surveillance Transformation (RSST).

3.54 CASA first developed a scorecard in July 2021 to provide the ASC with data on ‘available’ safety metrics and associated performance targets to ‘…support the ASC to monitor safety performance across all relevant elements’.

3.55 In June 2019, CASA provided a report to the ASC on its analysis comparison of international accident rates (accidents per million departures) from 2008 to 2018, shown in Table 3.3.

Table 3.3: Average accident rate from 2008 to 2018^a

Note a: CASA indicated that ‘Data has been extracted from the ICAO iStars API Data Service. The accident rate is based on scheduled commercial operations involving fixed-wing aircraft with a maximum take-off weight (MTOW) above 5700kg. Departures data is comprised of scheduled commercial operations that involve the transportation of passengers, cargo and mail for remuneration and is collated by the ICAO Air Transport Bureau. Estimates are made where data has not been provided by States’.

Source: CASA analysis.

3.56 In refining this work, the ASC commissioned a comparative International Benchmarking of Aviation Safety Performance within Air Transport study in mid-2019, comparing accident rates internationally ‘…to identify significant variations and where possible to understand why these variations exist to review’ and ‘…depending on the reasons for such variations, Australia may be able to learn from the experience of other States’.

3.57 Reporting on initial results from the study in May 2020 compared Australia to the United States of America on the average accident and fatal accident rate per million hours flown from 2006 to 2018. The analysis found that there was minimal difference between countries for fatal accidents. For non-fatal accident rates, results showed that Australia had almost three times the number of accidents per million flight hours than the USA for non-scheduled flights in the air transport segment. CASA advised the ANAO in December 2021 that ‘…there has not been further reporting to the ASC on international benchmarking. An updated version of the report will be completed in 2022’.

Third-party audits

3.58 The CASA Surveillance Manual indicates that third party audits are expected to be reviewed by CASA in preparation for a surveillance event. Third party audits refer to audits undertaken by non-CASA personnel to fulfil requirements of other authoritative bodies.³²

3.59 CASA commissioned an internal audit as a health check on third party industry audits as part of its 2019–20 Audit Program. The report was finalised in June 2020. The internal audit found that:

…existing policy and procedural guidance materials and training are insufficient to guide surveillance teams on how third-party audits should be considered and documented in a meaningful and consistent way. Interviews with nominated personnel and review of a sample of surveillance events identified challenges when considering third-party audit reports to inform the approach and scope of future surveillance events. Internal Audit was advised that given these challenges there was a reluctance by some members of CASA inspectorate to use third-party audit reports when determining the scope and timing of CASA scheduled surveillance.

3.60 CASA responded to the report recommendations by updating its manual and providing training.

3.61 The ANAO’s analysis of CASA surveillance reporting on third party audits suggests a lack of alignment between CASA’s application of its regulation and the scope and elements assessed in third party audits. In addition, CASA is not monitoring its use of third-party audits, with CASA noting:

Due to limitations within Sky Sentinel, there is no capacity to link receipt of a third-party audit. The [surveillance technical officers] provide a reference on Form 1189 (scoping for surveillance events) if there has been a third-party audit report sent to CASA, however a manual search would need to be done on all surveillance folders to identify if third party audits had been provided. Please note that it’s not mandatory for an Authorisation Holder to provide CASA with third party audit reports so there have been minimal received.

Quality assurance of surveillance

3.62 CASA’s Surveillance Framework Quality Assurance program was established in 2013–14. The program reviewed a select number of surveillance events on their consistency with the CASA Surveillance Manual. The program was undertaken outside of the division responsible for surveillance, providing a level of separation between surveillance and the review of surveillance. CASA does not have a current surveillance framework quality assurance program. CASA advised the ANAO in November 2021 that:

The area that conducted the - quality assurance reviews like your example is no longer part of CASA’s structure. We used to have a dedicated Continuous Improvement team that was disbanded a couple of years ago as a result of a structural change and the people from that team went onto different parts of the organisation. I was advised that this team was responsible for conducting the quality assurance review, but any findings were the responsibility of the relevant area for action or information.

3.63 CASA assessed its quality assurance practices in an internal audit, reporting in October 2019 that there was a:

Lack of internal management quality control and review requirements over surveillance activities, including consistent or regular (regime and intervals) oversight and monitoring of CASA’s compliance with the CSM.

3.64 CASA accepted the recommendation to mandate the evidencing of internal management quality control and review processes, noting the implementation date as December 2021.

Internal review of CASA’s approach to surveillance

3.65 CASA commissioned an internal review of its surveillance approach in March 2020. The objective was:

To conduct a review that will comprehensively examine CASA’s surveillance philosophy, capabilities and practises and prepare recommendations for any reform. The review will consider how CASA uses its current resources and powers to deliver its statutory objectives and assess CASA’s ability to perform as a capable and transparent regulator. The review may examine and provide options on how CASA could evolve its surveillance capabilities to support its strategic objectives and business needs.

3.66 The final draft of the review was reported to the CASA executive in June 2020 and the Executive indicated in September 2021 that the report was ‘…a great piece of work’ that would be ‘taken forward’. In February 2022, CASA advised the ANAO that the draft report had not been ‘validated or endorsed by management’ although CASA’s records did not evidence why an internal review that it had commissioned had not been subject to appropriate management consideration and, as appropriate, action on the recommendations.

3.67 The report was not filed in CASA’s electronic record management system or circulated more widely in CASA and was not considered by the OEG, despite the OEG having responsibility to ‘provide strategic direction for the management of regulatory oversight’ and to manage and administer decision making on regulatory oversight activities.

4.1 The Civil Aviation Act 1988 provides that the Civil Aviation Safety Authority (CASA) Board is responsible for deciding the objectives, strategies, and policies to be followed; ensuring CASA performs its functions in a proper, efficient and effective manner; and that CASA complies with directions given by the Minister under the Act.

4.2 To assess whether CASA is regularly reporting on its surveillance performance, the ANAO examined whether there was timely and accurate reporting on surveillance activity to the Board and Minister.

Has there been regular reporting to the Board on surveillance activities?

4.3 Under the Public Governance, Performance and Accountability Act 2013 (PGPA Act), the Board is the accountable authority in CASA (as a Corporate Commonwealth Entity) and has responsibilities to:

• properly govern the entity;
• establish and maintain appropriate systems relating to risk management and oversight and internal controls;
• encourage officials to cooperate with others to achieve common objectives;
• take into account the effects of imposing requirements on others; and
• keep the minister and the Finance Minister informed.³³

4.4 The CASA Board members are appointed on a part-time basis for up to three years by the Minister for Infrastructure, Transport and Regional Development. The Director of Aviation Safety (DAS) (who is also an ex-officio member of the Board) (see paragraph 1.2) manages CASA subject to Board directions and in accordance with policies determined by the Board.³⁴

4.5 The CASA executive regularly reports to its Board against a standard agenda at bi-monthly Board meetings (generally six meetings annually). In addition to keeping the CASA Board informed on the management of issues relating to service delivery, aviation safety management and regulation, staffing capacity and capability and stakeholder engagement, several reserved matters are approved by the Board including:

a suite of Board policies³⁵, strategic plans and reports which define and support their responsibility as the accountable authority, to meet their broader performance and direction setting obligations including strategic risk and financial management and corporate planning.

or endorsed, including on:

material changes in organisational direction pertaining to DAS/CEO/Executive Directives, strategic plans and reports which are deemed to have a significant impact on CASA’s stakeholders and /or reputation. Where there is no material change, the Board may be provided with progress updates.

4.6 To support the Board to meet performance and direction setting obligations, the CASA executive reports on performance in delivering one portfolio outcome to:

Maximise aviation safety through a regulatory regime, detailed technical material on safety standards, comprehensive aviation industry oversight, risk analysis, industry consultation, education, and training.³⁶

and the delivery of one corporate goal specific to surveillance, to:

Maintain and enhance a fair, effective and efficient aviation safety regulation system.³⁷

4.7 The CASA executive reports to the Board on three key performance indicators (KPIs) against CASA’s portfolio outcome and one KPI against CASA’s corporate outcome specific to surveillance, outlined in Table 4.1.

Table 4.1: Key performance indicators

Source: CASA 2021–22 Corporate Plan.

4.8 The ANAO assessed whether CASA’s performance reporting to the Board was appropriate and found that performance indicators for measuring against CASA’s portfolio outcome were only partly reliable or complete.³⁸

4.9 CASA does not reliably report against its KPIs relevant to surveillance. CASA financial year reporting³⁹ against KPIs on incident and accident data requires improvement because the key performance indicator relies on the availability of data which is published by the Bureau of Infrastructure, Transport and Regional Economics (BITRE) on a calendar year basis. In April 2021, the CASA Board were provided with a briefing from the CASA executive indicating that BITRE data used to calculate the accident and incident rates was deficient ‘…as “hours flown” data [is] often estimated leading to a weakness in the veracity of any calculated statistics’.

4.10 CASA annual reporting is focussed on trends rather than its identified KPIs on the number of incidents/accidents per hours flown. The DAS, in reporting to the Board in June 2021, acknowledged the need for a lead indicator and improvement in analysis and reporting with a CASA action item to:

Improve Accident Trend Data analysis of sector specific and/or aircraft specific matters to bring to the Board’s attention. A lead indicator is required that highlights emerging issues.

4.11 The CASA executive does not report to the Board against performance indicators to measure the effectiveness of its surveillance approach for its planned, targeted or response surveillance activities. Rather, it reports on the percentage of planned surveillance achieved in a year. While CASA’s method of calculating this outcome has changed over time, CASA does not provide this information in its reporting which impedes robust analysis of the data. There is limited information provided on the methodology it adopts in its calculation of surveillance event outcomes by CASA in its reporting.

4.12 There is minimal detail reported to the Board on the NSSP. The NSSP report does not provide information to the Board on AHPIs overdue, despite the AHPI being a key indicator used to assess risk (see paragraph 3.6 for further detail). The Board requested in April 2021 that NSSP reporting should be more granular. As at January 2022, surveillance reporting to the Board is not more granular and there is no reporting on overdue AHPI’s.

4.13 The primary CASA oversight body for aviation safety, the Aviation Safety Committee (ASC) commenced reporting to the CASA Board in June 2021 (see paragraphs 2.15 and 3.52 to 3.56 for further detail on the ASC) and CASA has indicated that ‘…the Aviation Safety Scorecard is a recent initiative that includes reporting on the AHPI assessments which now also goes to the Board’.

4.14 CASA’s internal audit of its performance framework in October 2021 found ‘CASA’s approach to developing, analysing and reporting CASA’s performance through its performance measures could be improved to ensure full compliance with the PGPA Act and PGPA Rule’. Specifically:

• The performance reporting terminology used in CASA’s Corporate Plan is not consistent with PGPA Rule and RMG requirements
• The methodology used in, and the structure of, CASA’s Corporate Plan makes it difficult to identify what performance measures CASA uses to determine whether its Purpose is being achieved, and what targets are applicable to these measures
• Aspects of CASA’s presentation of performance information in the Annual Performance Statement could be clarified to support readers’ understanding of CASA’s performance
• Although CASA’s performance measures broadly align with best practice, CASA’s performance measures could be more effective in demonstrating whether CASA’s is achieving its Purpose
• CASA’s performance measures could be documented more comprehensively and completely
• Internal guidance for performance reporting material should be updated to more accurately reflect CASA’s legislative obligations.

4.15 CASA agreed to all three recommendations to address these issues.

4.16 The ANAO found that reporting to the Board is not always as transparent as would be expected and does not reflect some of the performance issues and delays CASA has identified with its surveillance activity.⁴⁰

Use of non-government email services for official Australian Government business

4.19 Email communication is a widely used and accepted form of communication within the Australian Government. It provides evidence of the conduct of government business and is an important reservoir of information assets.

4.20 Retention of records outside of government designated email systems reduces transparency and contravenes various government policies related to protective security (Protective Security Policy Framework and the Australian Government Information Security Manual). As a Corporate Commonwealth entity, CASA is not obliged to comply with these policies, but it is considered good practice.

4.21 The ANAO examined the use of non-government email services for official Australian Government business and found that CASA routinely communicates, for the purpose of reporting, with all Board members through their private email addresses.

4.22 The ANAO suggests that all CASA official email communication be through government designated email systems.

Has CASA appropriately reported to the Minister on surveillance activities?

4.23 CASA is required to report to the Minister, consistent with the Regulator Performance Guide July 202141 and the Minister’s Statement of Expectations requires reporting on a quarterly basis.⁴²

4.24 The CASA Board Chair writes to the Minister generally at the beginning of each quarter providing a progress dashboard for the previous quarter on CASA’s delivery against the Minister’s Statement of Expectation and the CASA Corporate Plan.

4.25 The ANAO reviewed the regularity of the CASA quarterly performance reports over the period from 2017–18 to 2020–21. CASA provided all mandated reports to the Minister over this period, although the Minister requested more timely reporting in feedback to CASA for one of the reports.

4.26 CASA’s quarterly reporting on surveillance to the Minister is aligned with the reports provided to the CASA Board. There has been a marked reduction in the level of detail in surveillance reporting over time with recent reporting not accurately reflecting some of the performance issues and delays CASA has identified that increases risk to surveillance activity, including:

• while CASA has implemented its Learning and Management System, it has not established an effective monitoring system for key skills training (for instance, flying currency and training);
• the delay in establishing the EAP system; and
• decommissioning Sky Sentinel (which has been identified as being deficient in providing appropriate data for monitoring and reporting purposes).

Appendix 1 Entity response

 

 

 

 

 

 

Appendix 2 Improvements observed by the ANAO

1. The existence of independent external audit, and the accompanying potential for scrutiny improves performance. Improvements in administrative and management practices usually occur: in anticipation of ANAO audit activity; during an audit engagement; as interim findings are made; and/or after the audit has been completed and formal findings are communicated.

2. The Joint Committee of Public Accounts and Audit (JCPAA) has encouraged the ANAO to consider ways in which the ANAO could capture and describe some of these impacts. The ANAO’s 2021–22 Corporate Plan states that the ANAO’s annual performance statements will provide a narrative that will consider, amongst other matters, analysis of key improvements made by entities during a performance audit process based on information included in tabled performance audit reports.

3. Performance audits involve close engagement between the ANAO and the audited entity as well as other stakeholders involved in the program or activity being audited. Throughout the audit engagement, the ANAO outlines to the entity the preliminary audit findings, conclusions and potential audit recommendations. This ensures that final recommendations are appropriately targeted and encourages entities to take early remedial action on any identified matters during the course of an audit. Remedial actions entities may take during the audit include:

• strengthening governance arrangements;
• introducing or revising policies, strategies, guidelines or administrative processes; and
• initiating reviews or investigations.

4. In this context, the below actions were observed by the ANAO during the course of the audit. It is not clear whether these actions and/or the timing of these actions were planned in response to proposed or actual audit activity. The ANAO has not sought to obtain assurance over the source of these actions or whether they have been appropriately implemented.

• CASA issued an update to the surveillance manual in January 2022 (see paragraph 2.18).
• CASA’s Board agreed to update its conflict of interest policy to explicitly prohibit the trading of aviation shares (see paragraph 2.74).
• CASA has recently improved its conflict of interest declaration arrangements by revising annual training for staff and has proposed to amend documentation for staff performance discussions about updating conflict of interest declarations (see paragraph 2.74).
• CASA’s 2020–21 annual report (page 187) included a correction of an error relating to staffing (see paragraph 2.84).
• Recent work by CASA to address issues related to succession planning include a project to identify ‘critical roles’ (which commenced in October 2021) and to revise CASA’s ‘job family framework’ (which commenced in November 2021) (see paragraph 2.88). The latter project was identified in CASA’s 2020–2023 workforce plan as an activity to be undertaken in 2020–21.

Appendix 3 Australia’s aviation safety regulatory framework

Note: On 2 December 2021, new flight operations rules came into effect. These changes included a structural rearrangement of the regulatory framework by incorporating selected Manual of Standards into the Civil Aviation Safety Regulations 1998 and amendments to certain Civil Aviation Orders and the Civil Aviation Regulations 1988.

Source: Civil Aviation Act 1988 available at https://www.legislation.gov.au/Details/C2021C00060 [accessed 28 April 2022]; Airspace Act 2007 available at https://www.legislation.gov.au/Details/C2016C00178 [accessed 28 April 2022]; Civil Aviation Regulations 1988 available at https://www.legislation.gov.au/Details/F2021C01179 [accessed 28 April 2022]; Civil Aviation Safety Regulations 1998 available at https://www.legislation.gov.au/Details/F2022C00427 [accessed 28 April 2022]; Airspace Regulations 2007 available at https://www.legislation.gov.au/Details/F2019C00278 [accessed 28 April 2022]; Air Navigation Regulations 2016 available at https://www.legislation.gov.au/Details/F2018C00201 [accessed 28 April 2022]; The Civil Aviation Orders are available at https://www.casa.gov.au/rules-and-regulations/current-rules [accessed 28 April 2022]; The Manuals of standards are available at https://www.casa.gov.au/rules-and-regulations/current-rules/manual-standards [accessed 28 April 2022]; Airworthiness Directives are available at https://www.casa.gov.au/aircraft/airworthiness/airworthiness-directives [accessed 28 April 2022]; Australian Technical Standard Orders are in schedules to the Part 21 Manual of Standards Instrument 2016 available at https://www.legislation.gov.au/Details/F2017C01160 [accessed 28 April 2022].

Appendix 4 Key elements in the CASA surveillance governance framework

 

 

Source: ANAO analysis of CASA annual reports; CASA internal documentation.

Appendix 5 CASA’s surveillance policies and procedures for surveillance activities

Note a: The manual also sets out the processes to be followed when conducting surveillance on persons or organisations who are not authorisation holders, namely: Non-Air Operator’s Certificate holders for dangerous goods and approved self-administering aviation organisations.

Source: The surveillance manual and its 19 annexes available at https://www.casa.gov.au/search-centre/manuals-and-handbooks/surveillance-manual [accessed 28 April 2022]; The enforcement manual is available at https://www.casa.gov.au/sites/default/files/2021-09/enforcement-manual.pdf [accessed 28 April 2022]; CASA internal documents.

Appendix 6 European Aviation Processing system implementation

1. CASA’s project management framework includes project governance arrangements that require the development of project management plans, the maintenance of risk registers and regular reporting governance committees including the Major Programs Board.

2. Sky Sentinel is CASA’s key IT system to manage its surveillance activities. It is a bespoke IT system, implemented in 2012, that is used by surveillance staff to plan, conduct, and report on surveillance activities and to update AH information through the authorisation holder performance indicator (AHPI) tool (see paragraph 3.9). Not all information required by CASA surveillance staff to undertake proposed surveillance activities is held in Sky Sentinel, with regulatory activity information held on the EAP system and other IT systems such as the Electronic Document Records Management System holding information relevant to surveillance.

3. The need to use multiple systems to develop, manage and record surveillance activity that are not integrated reduces the efficiency of surveillance activities and limits the broader use of surveillance and regulatory data to inform surveillance activities, impeding effective surveillance of AHs.

4. CASA is progressively implementing functionality within the EAP system to guide and record surveillance activities which, when complete, will see the decommissioning of Sky Sentinel and other systems. The implementation of surveillance functionality has been delayed, with the original anticipated delivery in 2019–20 not achieved and now not anticipated to be implemented across all surveillance activities until 2022–23 (See Figure A.1 for a timeline on key events in the development of surveillance capacity in the EAP system).

5. A management-initiated review (MIR) of CASA’s management of the EAP project in mid-2020 noted that:

To date, the EAP project has significantly under-delivered. There has been partial realisation of anticipated business benefits associated with the implementation of EAP. There has been a long history of erratic cycles of approaches to the roll-out of components of EAP and the gap between what was required of EAP for the business to be fully effective and what has been delivered is significant.

…The MIR identified several factors contributing to the project not meeting the scope and timeframes within the 2018 [project management plan]. These factors are well known to management and include:

• the EAP project is more complex than originally scoped
• costing assumptions were not well understood or defined
• poor articulation in the 2018 project plan (a business case was not developed) of the costs associated with the project
• inefficient alignment of project and business requirements
• the significant additional work identified and the inability of the project to accommodate lessons from the initial project of work across the first two years of the project.

6. While CASA has identified key risks to the EAP project in the most recent business case and approved project plan, there is no consideration of risks associated with further delays in the project. These include consideration of the risks associated with continuing to operate legacy IT systems, the decommissioning of Sky Sentinel and the impact of delays to using a tool that provides for the allocation of staff to surveillance events based on their assessed competency. Figure A.1 shows the key events in the development of surveillance capacity in the European Aviation Processing system.

Figure A.1: Key events in the development of surveillance capacity in the European Aviation Processing system

 

 

Source: ANAO analysis of CASA documents.

Appendix 7 National Surveillance Selection Process activities, by group and event type

Source: ANAO analysis of CASA documentation.