Background

1. The aim of passenger screening is to prevent prohibited items such as weapons and explosives from being carried onto aircraft. Passenger screening is required at 62 security controlled airports across Australia. It involves the use of specialised equipment and screening personnel to detect and control prohibited items.

2. The Department of Infrastructure and Regional Development (the Department) regulates passenger screening through its administration of the Aviation Transport Security Act 2004 (the Act), and the Aviation Transport Security Regulations 2005 (the Regulations). The Act and Regulations establish a framework for aviation security, mandate minimum security standards for passenger screening and provide for the Department to undertake compliance activities to ensure legislated requirements are met.

3. Airport operators and screening authorities are responsible for delivering security on a day-to-day basis and must meet the minimum legislated security requirements outlined in the Act and the Regulations. Additionally: airport operators are required to operate in accordance with a Transport Security Program¹ that has been approved by the Department; and screening authorities must be authorised by the Department to carry out screening and operate in accordance with relevant Aviation Screening Notices. An airport operator may also be a screening authority. Throughout this report, the term industry participant is used when referring to both airport operators and screening authorities, unless a distinction is necessary.

4. The Department is responsible for ensuring that passenger screening is effective in detecting and controlling prohibited items, and for monitoring industry participants’ compliance with security requirements. The audit objective was to assess the effectiveness of the Department’s regulation of passenger security screening at Australian domestic airports.

Conclusion

5. The Department has implemented a regulatory framework that establishes minimum standards for passenger screening and a program of compliance activities at security controlled airports. However, the Department is unable to provide assurance that passenger screening is effective, or to what extent screening authorities comply with the Regulations, due to poor data and inadequate records. The Department does not have meaningful passenger screening performance targets or enforcement strategies and does not direct resources to areas with a higher risk of non-compliance.

Supporting findings

Regulating domestic passenger security screening

6. A sound regulatory framework has been established through the Act, Regulations and Aviation Screening Notices. The airport categorisation model identifies high risk airport types and applies specific regulatory requirements to those airports. The appointment of a screening authority and the approval of a Transport Security Program by the Department form the basis of the aviation security management arrangements between the Department and the aviation industry. Screening authorities are responsible for managing passenger screening in accordance with the requirements of the regulatory framework. The Department is responsible for administering the framework and monitoring compliance with regulatory requirements.

7. The Department’s approach to managing regulatory risk is based on the airport categorisation model that identifies airports with specific risk and operating profiles. The Department engages with industry participants to promote understanding of security obligations and ensure appropriate risk mitigation measures are in place. Engagement occurs through the regulatory approval process, industry forums and compliance activities.

8. The Department has not addressed a number of systemic issues that hamper its ability to implement a risk-based regulatory regime and provide assurance as to the effectiveness of passenger screening. The need to develop performance measures, analyse compliance data, implement an enforcement policy and provide adequate training have been identified in successive reviews but solutions are yet to be delivered.

9. A number of specific activities have been initiated to improve regulatory capability but progress has been delayed, which may reduce the effectiveness of passenger screening regulation.

Monitoring compliance

10. The Department conducts a planned program of compliance activities including audits, inspections and tests, which aim to assess the effectiveness of aviation security, including passenger screening. The compliance program is based on the airport categorisation model, which incorporates risk factors such as the maximum take-off weight of aircraft operating from the airport and the number of passengers departing the airport each year.

11. The compliance program does not incorporate the performance levels of individual airport operators or screening authorities, and non-compliance trends are not taken into account when prioritising compliance activities. The result is that non-compliance risk is not incorporated into the compliance program, and staff resources are not effectively deployed to areas that may require additional support or monitoring.

12. The Department does not have an enforcement policy for managing non-compliance with regulated requirements. In practice, non-compliance is managed through the corrective action plan process, which involves the development, approval and monitoring of a plan to rectify non-compliance. However, there is no clear escalation process for serious or systemic non-compliance, and no guidance on the application of the various sanctions that are available to the Department. Clear guidance on available enforcement options that are proportionate to the risks, and guidance on their application, would assist in the management of non-compliance.

Measuring performance

13. The Department has not established effective performance measures for passenger screening. The relevant key performance indicator in Programme 2.1 of the Department’s Portfolio Budget Statements relates to the number of compliance activities conducted by the Department and does not address the results of those activities. It does not include a quality measure that indicates whether aviation security meets a prescribed level of effectiveness. Similarly, the key performance indicators to be reported in 2015–16 as part of the Government’s Regulatory Performance Framework do not measure the effectiveness of aviation security generally, or passenger screening specifically.

14. The Department does not manage compliance data effectively. A new information management system, implemented in April 2014, does not meet the Department’s business requirements. The data is unreliable, there is inadequate reporting functionality, and training does not meet the needs of users. The Department does not have a robust system to collect, store and retrieve information about industry participants and their compliance with legislated security requirements. The project established to deliver the new information management system was closed in January 2016, and the Department reported that outstanding capability is to be addressed through other activities.

15. The Department has conducted limited analysis of compliance data. Poor data quality and lack of reporting capability in the current and previous systems have made analysis of data difficult. The reports produced from analysis activities reflect an amalgamation of available data, rather than a comprehensive analysis of compliance trends or systematic evaluation of compliance performance.

16. The Department does not report passenger screening compliance results to the Secretary, the Minister or to industry participants. Compliance activity results have been reported to the Office of Transport Security executive on a monthly or quarterly basis. These reports provided compliance results for the relevant reporting period, but did not support comparison of the data over time or identify trends.

Recommendations

Summary of entity response

17. The Department provided formal comments on the proposed audit report, which are included at Appendix 1. Its summary response is set out below.

The Department’s summary response

The Department thanks the ANAO for the audit and agrees with all the recommendations.

The Department notes that following significant review work in 2015 it is investing in the broad reform of its transport security regulatory operations. This is to ensure that the Office of Transport Security’s regulatory activities are well positioned to respond to changing threats and risks, future industry growth and diversification, and that its approvals and compliance operations are efficient. This reform program comprises three elements: redesign of the transport security operating model, the establishment of a competency based learning and development framework and improvements to the regulatory management system (RMS). The reform program will drive key changes to the way the Department regulates domestic passenger screening.

The Department notes the ANAO’s conclusion that it is unable to provide assurance that passenger screening is effective, or the extent screening authorities comply with Regulations due to poor data and inadequate records. A number of initiatives are underway to improve the quality of our data and record keeping with dedicated resources to remediate and improve the current data in RMS, with this activity due to be completed in early 2017. The Department has developed guidance to assist staff on how to use RMS which has improved the quality of new data being collected.

A Working Group has been established to develop a framework to measure the effectiveness and extent that screening authorities are complying with passenger screening regulations. The regular inspections and audits that are undertaken to monitor an airport’s compliance with passenger security screening requirements will be a key component of the framework. This includes testing the effectiveness of their ability to detect and control the entry of prohibited items and weapons into the sterile area.

The Department notes that it has already put in place mechanisms to improve its capacity to use compliance data and is gradually building this capacity. It is also currently revising its compliance approach to better incorporate non-compliance risk into its planning. A schedule for the revoking of grandfathering provisions for security screening has already been developed by the Department and industry stakeholders were advised of the changed requirements in early July.

Aviation security screening

1.1 Security screening in the aviation context is defined as:

The application of human, technical and/or other means to identify and/or detect weapons, explosives or other dangerous devices, articles, substances, or other prohibited items or behaviours which may be used to commit, or indicate an intention to commit, an act of unlawful interference against aviation.²

1.2 An unlawful interference is an act or attempted act that jeopardises the safety of civil aviation including the safety of aircraft, airports or people. The expected outcomes of screening are to mitigate against:

• unlawful seizure of an aircraft;
• hostage taking on board an aircraft;
• intrusion on board an aircraft of a weapon or other material capable of threatening the integrity of an airframe; and
• the use of the aircraft as a weapon.

1.3 The screening of passengers and their carry-on baggage is the most publicly visible part of the aviation security regime. It utilises a combination of specialist equipment and the judgement of screening personnel to detect and control prohibited items and identify certain behaviours. This security function needs to be performed effectively, to meet legislated requirements; and efficiently, in order to facilitate air travel. The ultimate aim of Australia’s screening regime is to ensure that no prohibited items, prescribed weapons, or explosives are carried onto an aircraft.

1.4 Passengers travelling on Australian domestic air services departing from security controlled airports³ are subject to three forms of screening prior to boarding:

• walk-through metal detection for all passengers;
• x-ray of all carry-on baggage; and
• explosive trace detection for randomly selected passengers and their carry-on baggage.

1.5 Passenger security screening, which is required at all security controlled airports from which a screened air service operates, is conducted by a screening authority that has been assessed as meeting specified requirements and authorised by the Department to provide screening services. In most instances, an airport or airline will act as the authorised screening authority, and may sub-contract day-to-day operations to a specialist screening provider.

Aviation security regulatory framework

1.6 Globally, aviation security has been strengthened considerably following the hijacking of four passenger planes used to attack targets in the United States of America on 11 September 2001. In response to the attacks, Australia implemented the Aviation Transport Security Act 2004 (the Act), and the Aviation Transport Security Regulations 2005 (the Regulations). At the time the legislation was introduced, the Government supported explicit regulation, rather than self-regulation or co-regulation, to ensure universal compliance with mandated security standards.

1.7 Australia’s aviation security framework has its origins in the Convention on International Civil Aviation, which is administered by the International Civil Aviation Organization. As a signatory to the Convention, Australia is obliged to regulate its aviation industry to safeguard against acts of unlawful interference with aviation.

1.8 The Department regulates aviation security through its administration of the Act and the Regulations. The legislation establishes a framework for preventive aviation security by mandating minimum security standards that industry participants are expected to meet. It also provides for the Department to undertake compliance activities to ensure industry participants comply with legislated requirements.

1.9 Airports are commercial in nature and the cost of operations, the smooth movement of passengers, as well as safety and security, are priorities for industry participants. The Department is responsible for ensuring industry participants meet legislated requirements and aviation security is maintained in a way that is cost effective to the Australian Government, industry and the travelling public.

1.10 On 4 December 2014, the Senate referred the matter of airport and aviation security to the Rural and Regional Affairs and Transport References Committee for inquiry and reporting. Submissions to the inquiry were received in January 2015, and the Committee was due to report on 19 May 2016. ⁴ However, the inquiry lapsed on 9 May 2016 with the dissolution of the Senate and the House of Representatives and reporting was not completed.

Roles and responsibilities

1.11 The Department advises the Government on the policy and regulatory framework for Australian airports and the aviation industry. Within the Department, the Office of Transport Security is responsible for administering the Act and the Regulations and for regulating the aviation industry. The Office of Transport Security’s organisational structure is shown at Table 1.1.

Table 1.1: The Office of Transport Security structure and responsibilities

Note: Branches relevant to aviation security are shaded.

Source: ANAO analysis of the Department’s documents.

1.12 As the regulator of security controlled airports⁵, the Department is responsible for ensuring that passenger screening is effective in detecting prohibited items and meeting expected outcomes (see paragraph 1.2). The Department administers aviation security legislation by:

• maintaining a regulatory framework to safeguard against unlawful interference with civil aviation;
• establishing minimum security requirements for civil aviation;
• regulating industry participants; and
• meeting Australia’s obligations under the Convention on International Civil Aviation.

1.13 Airport operators and screening authorities are responsible for delivering security on a day-to-day basis and must meet the minimum legislated security requirements outlined in the Act and the Regulations. Airport operators are required to develop and operate in accordance with a Transport Security Program that has been approved by the Department. Screening authorities must be authorised by the Department and operate in accordance with relevant Airport Screening Notices.

Audit approach

1.14 The audit objective was to assess the effectiveness of the Department’s regulation of passenger security screening at Australian domestic airports. To form a conclusion against the audit objective, the ANAO adopted the following high level criteria:

• an appropriate framework for assessing and mitigating risks and an effective plan for monitoring compliance have been established;
• an effective risk-based compliance program to communicate regulatory requirements and to monitor compliance has been implemented; and
• sound arrangements have been established to manage non-compliance with agreed security screening requirements.

1.15 The audit team examined departmental records, observed screening activities and consulted with departmental staff and a range of key stakeholders.

1.16 The audit was conducted in accordance with ANAO auditing standards at a cost to the ANAO of approximately $521 345.

Is a sound regulatory framework in place?

Regulatory approach

2.1 The Department’s regulatory approach is characterised by a strong relationship between industry and government, where minimum security requirements are supported by legislation and government has a role in monitoring and enforcing compliance. This arrangement recognises that industry participants have specialist capability and expertise to manage key infrastructure, and allows some discretion as to how they implement the Regulations.

Aviation security legislation

2.2 The Act and Regulations provide the overarching guidance for aviation security. The main purpose of the legislation is to establish a regulatory framework to safeguard against unlawful interference with aviation and specify minimum security requirements for civil aviation related activities. An airport categorisation model⁶ identifies security controlled airport types on the basis of inherent risk.⁷ It is designed to differentiate between different types of airports for the purpose of applying security measures, including measures related to passenger screening, and subjects them to regulation by the Department.

Aviation Screening Notices

2.3 Aviation Screening Notices (ASNs) are issued by the Department under Regulation 4.17, and specify the methods, techniques and equipment to be used for screening. They give operational effect to the Act and Regulations and prescribe how aviation security screening is to be undertaken. These notices acknowledge the operational differences between the airport categories. For example, ASN 2012 contains procedures for screening liquid, aerosol and gel products at international screening points that are not included in the ASN for category 4 and 5 airports as these airports do not operate international flights.

Transport Security Program

2.4 Operators of security controlled airports must submit, hold and maintain an approved Transport Security Program (TSP) that sets out the measures and procedures that the operator will implement to reduce the risk of unlawful interference with aviation. A TSP sets out the physical places within an airport that are subject to regulation, and how the airport operator will manage security for its operations, including for passenger screening. The TSP is developed by the airport operator, assessed and approved by the Department, and adherence is monitored through compliance activities.

Screening authorities

2.5 A screening authority is a body corporate—usually an airport operator or aircraft operator—that is authorised to conduct screening by a delegate of the Department. Applications to become a screening authority are assessed by the Department according to specific criteria, including knowledge and experience in managing security risks, screening equipment and security incidents. Screening authorities may contract the provision of screening services to other security related entities, known as screening service providers, which may, in turn, sub-contract to other entities. The screening authority remains legally responsible for demonstrating compliance with regulatory requirements, regardless of whether a third party is contracted to provide screening services.

Does the Department adopt a risk-based approach to regulating passenger screening?

Airport categorisation

2.6 The airport categorisation model identifies airports with specific risk and operating profiles for the purposes of applying a range of security measures, including those related to passenger screening. The Regulations specify criteria to consider when identifying security controlled airports and categorises them according to: the maximum take-off weight of aircraft operating from the airport; the number of passengers departing the airport each year; and international air service or regular open charter operations. All airports that operate aircraft with a maximum take-off weight of 5 700 kilograms and over are included in the categorisation model and are referred to as security controlled airports. Airports that operate with larger aircraft and higher passenger numbers represent a higher risk. As at 16 June 2016, there were 174 security controlled airports classified in categories 1 to 6 (see Table 2.1). The Department monitors compliance of category 1 to 5 airports.

Table 2.1: Security controlled airport categories and risk ratings

Source: ANAO analysis of the Department’s documents.

Regulatory approvals

2.7 The Department is responsible for assessing and approving requests to specify screening authorities and approve TSPs. The regulatory approval process, depicted as a simplified process in Figure 2.1, involves regular communication between industry participants and the Department. This helps to ensure that industry participants understand their security obligations and reduces the risk of non-compliance.

Figure 2.1: The regulatory approval process

Note: The shaded boxes are undertaken by the Department, and unshaded boxes by the industry participant.

Source: ANAO analysis of the Department’s documents.

2.8 The ANAO reviewed the regulatory approval process by assessing guidance material and a sample of regulatory approvals and found that the:

• guidance is clear and aligns with the legislation; and
• engagement process is effective in assisting industry participants to understand their security obligations.

Aviation security forums

2.9 The Department facilitates two stakeholder forums—the Aviation Security Advisory Forum (ASAF)⁸ and the Regional Industry Consultative Meeting (RICM). Two separate forums recognise the different security challenges and regulatory cost sensitivities faced by regional airport operators and those operating larger, international airports. The forums provide an opportunity for the Department to work constructively with key stakeholders to help manage risk. Meeting minutes from these forums indicate attendance by a broad range of stakeholders (see Table 2.2).

Table 2.2: Aviation security industry forums facilitated by the Department

Source: ANAO analysis of the Department’s documents.

2.10 The forums have an information sharing and education focus. For example, in November and December 2014 two workshops were held as part of the ASAF and RICM after the National Terrorism Alert Level was raised in September 2014. The purpose was to explore the potential implications of a heightened threat specific to the Australian aviation sector. Outcomes included sharing information on the alert level being raised, discussion around plausible risk scenarios that could arise in a heightened threat environment, and agreement on additional security measures that could be applied to prevent or reduce the likelihood of these risk scenarios from occurring.

2.11 Industry participants informed the ANAO that they value the opportunity to obtain and provide feedback on industry issues related to security in a constructive setting, and consider it an important tool for networking within the industry. The RICM terms of reference specify that meetings should be held four times per year. During 2014 and 2015 only three were conducted. The Department consulted with industry participants at the April 2016 RICM to determine the appropriate frequency of these forums and agreed to reduce the number of annual meetings to two, with options for extraordinary meetings and teleconferences. The terms of reference are to be updated accordingly.

Does the Department address gaps in regulatory capability?

2.12 The Department has commissioned several reviews relating to various aspects of aviation security in recent years.⁹ These reviews made over 100 recommendations aimed at improving specific policies and processes. The reviews highlighted a number of gaps in regulatory capability relevant to passenger screening, where the need for improvement has been repeatedly identified but solutions are yet to be achieved, including:

• implementation of an enforcement policy (see paragraphs 3.16 to 3.18);
• development of performance measures (see paragraphs 4.1 to 4.5);
• training and guidance for staff (see paragraphs 4.11 to 4.13); and
• analysis of compliance trends based on accurate, reliable data (see paragraphs 4.26 to 4.29).

2.13 A number of specific activities that are relevant to effective passenger screening regulation have been initiated by the Department but not completed. The following case studies highlight three such activities, where completion has been delayed and may reduce the effectiveness of passenger screening regulation.

2.14 The Department has recently established the Transport Security Operations Reform branch, which is responsible for improving the Department’s regulatory capability. Even so, not all improvements are part of the Transport Security Operations branch’s responsibility and a more integrated improvement approach is necessary.

Note a: The documents related to the application of security mitigation categories in the compliance program.

Note b: Those that referred to seven security mitigation categories did not include the category ‘screening and clearing’.

Note c: Inadequate guidance and training was identified as one factor in the flawed migration of data to the new information management system in 2014.

Does the Department monitor compliance with regulated security requirements?

Compliance activities

3.1 The Department has established a plan for monitoring compliance with regulated requirements through the National Quality Control Programme (NQCP). The NQCP includes regular audits, inspections, tests and capacity building activities, which are conducted by transport security inspectors from the Department’s regional offices.

Table 3.1: Types of compliance activities

Note a: The passenger screening system comprises the people, processes and procedures, and equipment used to detect and control prohibited items at the screening point.

Note b: In some cases, this may be a simulation of a prohibited item.

Source: ANAO analysis of the Department’s documents.

3.2 In 2014–15 the Department reported that it conducted 1216 compliance activities across a range of transport sectors. Of these, 905 activities (75 per cent) were conducted in the aviation sector. The majority of compliance activity was conducted at category 1 and 2 airports. Detailed analysis of these activities was not possible due to data integrity issues, which are discussed in Chapter 4.

3.3 Compliance activity findings are categorised as compliant, non-compliant or observation. Non-compliance represents a breach of transport security regulations and results in a non-compliance notice that is subject to a formal follow-up and acquittal process. Where non-compliance is identified, the extent of the non-compliance should be assessed and a corrective action plan developed, approved and monitored to ensure it is addressed. Observations represent potential security vulnerabilities and are provided to the industry participant as an observation notice that is not subject to formal follow-up arrangements.

3.4 Detailed guidance for managing corrective action plans is available, and discussions with transport security inspectors indicate sound knowledge of the process. However, as with most of the Department’s compliance activities, data on the number and status of non-compliances is difficult to extract from the information management system and the data is unreliable (see Chapter 4).

Does the Department apply a risk-based approach to compliance activity?

3.5 The NQCP team was established in 2014–15 and is responsible for developing a nationally consistent program of compliance activity in consultation with each regional office, and for reporting on compliance activities. The NQCP ‘Compliance Touch’ document details the minimum number and type of compliance activities required to be conducted at airports in each category annually.¹⁰

3.6 Additional NQCP activities are prioritised based on the airport categorisation model and the risk rating for each category (see Table 2.1). The process for determining the risk rating in the 2015–16 NQCP involves:

• prioritising those airport categories with more security mitigation measures in place, such as those operating international flights, to determine a risk mitigation score;
• applying ten adjustment score questions to each category to determine a quality control priority score; and
• applying the risk mitigation score and the quality control priority scores to the airport categories to calculate a revised risk rating.

3.7 The compliance history of airport categories and individual industry participants is also relevant in determining compliance risk. Past performance information can be used to identify systemic issues in: individual operations; regional or seasonal factors; or specific security functions. This information may indicate areas where additional monitoring or support is required to improve performance.

3.8 The Department has stated that it uses compliance activity results to identify trends and areas of industry participant operations that may require additional compliance activity.¹¹ However, the Department has limited information about the compliance history of individual industry participants. Likewise, the Department’s ability to identify systemic issues, understand seasonal variations or compare performance across industry is restricted by the absence of reliable data and analysis.

3.9 Experienced transport security inspectors, who have developed considerable knowledge of industry capability over time, can provide information about airport compliance based on their own learnings and by manually checking records of past compliance activity. However, without effective data collection, storage and retrieval to enable analysis of individual industry participant performance or the identification of systemic non-compliance, the Department is unable to assess the risk of non-compliance and apply resources where that risk is higher. Further, there is no feedback loop in the compliance cycle to indicate whether compliance is improving, or where additional monitoring may be required.

Does the Department manage non-compliance with regulated security requirements?

3.13 The Department has a range of responses available to manage non-compliance including:

• education;
• counselling;
• cautions;
• infringements;
• enforceable undertakings;
• powers to direct actions;
• control directions;
• injunctions;
• cancellation of a TSP; and
• prosecution.

3.14 The majority of responses are not applied and the Department encourages voluntary compliance through education, engagement and compliance activities.

3.15 When non-compliance is identified, a corrective action plan, developed by the industry participant to address the non-compliance, is approved and monitored by the Department. This approach is appropriate for most instances of non-compliance, where the industry participant is willing and able to implement appropriate actions to address the non-compliance in a timely manner. Where this is not the case, the Transport Security Compliance Manual advises that the matter may be escalated as a serious non-compliance¹² and should be referred to the National Investigations Unit for further action. However, the National Investigations Unit is no longer in operation and an alternate arrangement has not been documented.¹³

3.16 The Department has recognised the need to develop a clearly articulated enforcement policy, and provided an undated paper to the ANAO that details options for enforcement approaches, and states:

[The Office of Transport Security’s] preferred approach to enforcement is to facilitate compliance through education and awareness … there is a need to strengthen its enforcement capability to ensure [the Office of Transport Security] can identify non-compliance, apply sanction and deter future non-compliance.¹⁴

3.17 In response to a review in May 2013, the Department undertook to establish a compliance and enforcement policy. In November 2015 the Department informed the ANAO that the compliance and enforcement policy had been put on hold pending the development of a Regulatory Strategy. Work on the Regulatory Strategy has also been put on hold pending implementation of a new Transport Security Operations operating model.¹⁵ The operating model is scheduled for implementation in early 2017. This suggests that an enforcement policy will not be considered before 2017 at the earliest, four years after the need for a policy was identified.

3.18 The Department should develop a clear enforcement strategy, independent of other operational projects such as the Transport Security Operations operating model, which is communicated to relevant stakeholders, including industry participants. An appropriate range of enforcement options that are proportionate to the risks would assist in the effective management of non-compliance.

Does the Department measure the effectiveness of passenger screening?

Key performance indicators

4.1 The Department’s obligations for managing transport security are reflected in Programme 2.1–Transport Security in its Portfolio Budget Statements. The Programme 2.1 objective states that the aim is to ensure flexible and proportionate regulation that delivers measurable benefits. Programme 2.1 includes one deliverable, as shown in Table 4.1.

Table 4.1: Portfolio Budget Statements key performance indicator

Note a: Prior to 2015–16, to meet the key performance indicator, the compliance activity was defined as an audit, inspection or system test. From 2015–16, the compliance activity must be a full audit.

Source: Infrastructure and Regional Development Portfolio, Portfolio Budget Statement 2015–16, p. 41.

4.2 The Department reports that it has consistently met this performance indicator, with over 95 per cent of high risk industry participants being subject to a compliance activity every year. However, the performance indicator is not meaningful and is set below the Department’s usual level of activity. The indicator lacks sufficient detail to allow for an informed assessment of industry participant’s regulatory performance or demonstrate changes over time. The deliverable is a quantitative measure, and in isolation does not provide an effective means for Parliament to assess the quality of regulatory outcomes, only that a compliance activity has been undertaken.

4.3 The Department has established performance indicators, measures and evidence under the Government’s Regulatory Performance Framework. The framework includes five performance indicators (outlined in Appendix 3) against which the Department will report for the first time in 2016. Like the Portfolio Budget Statement’s performance indicator, the Regulatory Performance Framework indicators do not include a measure which, when reported against, would provide insight into the effectiveness of passenger screening.

4.4 In 2003, the ANAO recommended that the Department establish ‘… specific, practical, achievable and measurable performance requirements for aviation security …’.¹⁶ The Department agreed to consider the introduction of performance targets, noting that ‘development of a positive security culture within the aviation industry requires encouragement of a continuous improvement process through effective and comprehensive education and regulation’.¹⁷

4.5 The process of passenger screening and its effectiveness at detecting and controlling prohibited items and weapons is measurable through the system testing regime currently operated by the Department. Industry participants have indicated their support for the introduction of benchmarks for passenger screening to assist them to understand their performance in comparison to other operators, identify weaknesses and promote continuous improvement. Without performance measures, it is difficult to determine which screening authorities are performing passenger screening effectively and require less monitoring, and which are performing poorly and may require additional support to determine the cause and implement corrective actions.

Does the Department manage compliance data effectively?

4.8 In 2009, the Review of Aviation Security Screening Report described the Department’s security screening compliance data as ‘insufficient either to provide assurance about effectiveness or to direct targeted support …’.¹⁸ At that time the Department’s management of passenger screening information was characterised by a range of disparate systems, with difficulties in maintaining data fidelity and extracting meaningful reports. In 2011, the Regulatory Capacity Building Program was established to address functional gaps, replace unsuitable systems and introduce new tools to support staff through the introduction of a regulatory management solution.

4.9 Priority was given to replacing existing information systems with a new information management system, to be known as the Regulatory Management System (RMS). In April 2013, funding of $3.6 million was approved to contract the development of RMS and implementation was scheduled for July 2014. The proposed solution was expected to provide a single source of truth, allow for accurate, regular and repeatable reporting across a range of activities in the Office of Transport Security, and enable analysis of multiple data sets and their relationships.

4.10 The RMS system was first released in April 2014, when user acceptance testing revealed a number of unexpected system problems that delayed training for regional office staff. Since then, the Department has identified ongoing issues with training, data quality, and lack of reporting functionality. These issues persist today and impact the Department’s ability to provide assurance that passenger screening is effective.

Regulatory Management System deficiencies

Training for Regulatory Management System users

4.11 RMS user training and guidance was to be delivered in 2013–14 as part of the project, and funds were allocated for this purpose.¹⁹ However, specific details about which staff members would receive what training were not recorded in project documentation. The Department has not delivered instructor-led training for RMS users, and informed the ANAO that, since the implementation of RMS in April 2014, all training has been delivered on an ad-hoc basis by internal staff.

4.12 Several times between March and September 2015 the Department acknowledged, in internal reports, that training was not meeting user needs and that an instructor-led training course for all RMS users was to be delivered. In January 2016, the Department reported that training would be provided ‘after the [Office of Transport Security] reforms are completed.’

4.13 The use of consistent practices by trained system users is required to facilitate data quality and reporting. Inadequate training has resulted in a lack of user confidence and has contributed to data quality issues.

Data quality

4.17 The data in RMS is unreliable and a range of anomalies have been identified in all aspects of data held within the system. Data quality was identified by the Department as a risk in March 2014, and was recognised as an issue in October 2014. The problems apply to completeness and correctness of all aspects of RMS data, and stem from incomplete data migrated from legacy systems and inconsistent processes adopted by users who were not trained in the use of the system.

4.18 The Department commenced planning a data remediation exercise in March 2015 and has developed a detailed schedule of activities to be completed, including estimated durations and required resources. Completion is scheduled for October 2016. Schedule data provided by the Department indicates that the duration of the first remediation activity was eight weeks. However, at the current rate of progress, this task will take considerably longer than estimated.

4.19 The Department informed the ANAO that many of the resources identified to conduct the remediation work are transport security inspectors. This will have a negative impact on the compliance program, reducing the number of compliance activities completed. The Department has not calculated the extent of the impact. Given the current rate of progress, it is likely to be significant.

Reporting capability

4.20 In March 2015, one year after RMS was implemented, the reporting functionality was found to be ‘not working to specification’.²⁰ Specialist contractors have been engaged since June 2015 to analyse and develop detailed reporting requirements. The Department informed the ANAO that the contractor’s role is to: deliver a set of eight reports; propose and implement a final reporting solution; and support the data remediation project.

4.21 The eight reports were originally due for completion in September 2015. Each month since then, the Department has estimated that the reports will be completed in the following month. In April 2016 the Department informed the ANAO that four of the eight reports, described as complete in March, require ‘further adjustment and review’ and that one report is ‘not at all useful’. The Department has not developed a detailed schedule for the work required to complete the eight reports, but informed the ANAO that three contractors are expected to be retained until at least June, August and November 2016 respectively. As at 6 May 2016, the estimated cost for this work is around $977 000.

4.22 Two years after its implementation, RMS remains unable to produce standard reports such as industry participant contact details, regulatory approval status or compliance activity data.

Closure of the Regulatory Management System project

4.23 The Department informed the ANAO that the contractor responsible for developing and implementing RMS has met its contracted obligations, but the RMS project’s business requirements were not appropriately articulated in the contract specification. Consequently, the project has not delivered the expected benefits of providing: a single source of truth; accurate, regular and repeatable reporting; and analysis of multiple data sets and their relationships.

4.24 The RMS project was officially closed on 14 January 2016, with the budget of $3.6 million fully expended. However, the project has not delivered against its objectives and significant capability such as reliable data, basic reporting and user training remain outstanding. The RMS End Project Report notes that reporting capability, data remediation and training are to be delivered outside the RMS project.

4.25 The lack of data quality and reporting functionality indicates that there is a large body of work to be completed before RMS can deliver any tangible benefits to the Department. As a regulator, the Department should be able to report accurate information about the entities it regulates. With the closure of the RMS project, the Department should ensure appropriate resources are allocated to deliver the required capability.

Does the Department analyse compliance data?

4.26 The Department has conducted some analysis of compliance data. For example, limited analysis of system test results has been conducted and the results included in internal reports on a monthly or quarterly basis. The analysis shows the number of system tests conducted and the failure rate for each airport category for the period of the report. However, it was not possible from the data provided to determine whether system test failures were increasing or decreasing or whether certain screening authorities were performing better than others.²¹

4.27 The Department also produced six Regulatory Analysis Products (RAPs) in the period July 2014 to August 2015 that were intended to reflect analysis of aviation operations and identify trends. The reports were developed for internal use, with broad themes from the analysis discussed at industry forums. Two of the six reports provided a breakdown of compliance activity conducted at category 3 airports for a specified period and identified themes such as the rate of non-compliance in comparison with all other airport categories and the areas in which the non-compliance was found. The two reports were similar in presentation allowing comparison of information, although statistics such as the percentage of non-compliance and failed system tests were contained in the text and difficult to locate. The remaining four RAPs related to category 1 airports. The content and format of these reports varied markedly, precluding any comparison of the information contained in them. No further RAPs have been produced.

4.28 In September 2015, the Department conducted a pilot study to assess historical compliance data, captured in the legacy information management systems, for use in developing security mitigation categories. It utilised compliance results from the 2013–14 financial year and categorised them according to the security mitigation categories by state.²² The analysis identified that the most common non-compliances during the period were in the categories of security management, screening and clearing, and access control. The Department reported a number of issues identified during this activity including poor data quality and inconsistent recording of non-compliances.

4.29 The Department’s ability to analyse compliance data has been hampered by unreliable data and system limitations. Collection of data relies largely on manual processes to extract and verify information. Verified data is then manually consolidated in spreadsheets and filtered to produce reports. Analysis conducted by the Department does not enable comparison of compliance trends over time and it is not possible to determine if industry compliance is at an acceptable level, or whether it is improving or declining.

Does the Department report on the effectiveness of passenger screening?

Compliance activity reporting to the Office of Transport Security executive

4.30 Reports on compliance activity to the Office of Transport Security executive included system test results and failure rates for passenger screening tests conducted during the relevant reporting period. The reports did not include historical data and the way information was categorised in different reports made it difficult to identify trends over time.

4.31 Monthly reports between July 2015 and November 2015 attributed each non-compliance and system test failure to a specific airport. Over time, this information would enable trend analysis by airport, airport category and region. However, a new format introduced in December 2015 groups all non-compliances and system test failures into security mitigation categories and airport categories. While this is a positive step in terms of identifying the type of non-compliance, the absence of airport identification compromises the Department’s ability to effectively identify trends or systemic issues.

Compliance activity reporting to the Secretary and the Minister

4.32 Reporting outside of the Office of Transport Security consists of: a Weekly Issues Report to the Secretary that includes planned compliance activity; and annual reporting to the Minister against the Portfolio Budget Statement key performance indicator. Information provided to the Secretary and the Minister is insufficient to provide assurance about the effectiveness of passenger screening.

Compliance activity reporting to industry participants

4.33 The Department reported system test results for category 1 airports during 2013–14 and 2014–15. The reports compared the results for individual category 1 airports with overall category 1 results. The resulting report clearly indicated individual system test failures compared to national results, expressed as a percentage of the total tests conducted. This reporting ceased in March 2015 due to data quality issues that are yet to be resolved. Industry participants have indicated support for this kind of comparative analysis, which provides clear outcomes focused results.

Reporting requirements

4.34 Different stakeholders have different information requirements. Internally, the Department requires information about areas of non-compliance that would benefit from additional monitoring and support. This would allow the Department to direct departmental resources to areas of high risk, and reduce the burden of compliance activity for industry participants with a high rate of compliance.

4.35 Reporting to the Parliament enables transparency and accountability. It can provide information about the effectiveness of aviation security generally and passenger screening specifically. It can also enable assessment of security policy to ensure the level of regulation being applied is appropriate to provide effective security outcomes.

4.36 Industry requires practical analysis of non-compliance data to determine the cause and identify solutions to improve passenger screening performance. Better reporting of compliance results would help screening authorities manage passenger screening more effectively and encourage ongoing improvement.

4.37 However, an assessment of stakeholder information requirements and resolution of data quality issues is required before meaningful, accurate reporting can be delivered.

Appendix 1: Entity response

Appendix 2: Aviation Security Reviews

Note a: These reviews were conducted internally or by other Government entities at no cost to the Department.

Note b: The review report described these findings as ‘focus areas’.

Note c: The review report referred to ‘improvement opportunities’ and listed 4 high priority, 4 medium priority, and 10 low priority improvements.

Source: Analysis of documentation provided by the Department.

Appendix 3: Regulatory Performance Framework Key Performance Indicators, Measures and Evidence

Source: Department of Infrastructure and Regional Development, Office of Transport Security Key Performance Indicators, Measures and Evidence. Internet DIRD, available from <https://infrastructure.gov.au/department/deregulation/files/Office_of_Transport_Security_RPF_Metrics.pdf>.and accessed 10 May 2016.