Emerging technologies including artificial intelligence (AI) are increasingly a part of public services, with 56 public sector entities advising in the Australian National Audit Office (ANAO)’s 2023–24 financial statements audits that they have adopted AI in their operations. AI can offer the promise of better services, enhanced productivity and efficiency — and also has the potential for increased risk and unintended consequences.

AI is an area of public interest for the Australian Parliament, with two inquiries underway during the time of undertaking this audit. The Select Committee on Adopting AI reported in November 2024.¹ At the time of presenting this audit report to the Parliament, the Joint Committee of Public Accounts and Audit is conducting an inquiry into the use and governance of AI systems by public sector entities.² The Australian Government has policies and frameworks for agencies on the adoption and use of AI that are referred to in this audit.

The growing use of AI also brings new challenges and opportunities in auditing. As a first step in addressing these, the ANAO has identified providing assurance on the governance of the use of new technology as a way of bringing transparency and accountability to the Parliament in this area of emerging public administration. The Australian Taxation Office (ATO), as an agency that uses technology extensively in its administration of the tax and superannuation systems, was chosen as the first agency in this new line of audit work. I acknowledge the ATO’s work on governance to support rapidly emerging technologies and cooperation in the undertaking of this audit. I also acknowledge the assistance of the Digital Transformation Agency through consultation on this audit.

The ANAO will continue to focus on governance of AI while it develops the capability to undertake more technical auditing of the AI tools and processes used in the public sector. Building this capability will require investment in knowledge, methodology and skills to enable the ANAO to test more deeply how AI tools operate in practice.

Like audit offices around the world, the ANAO will seek to examine how AI can improve the audit process itself, in a profession where human judgement and scepticism are foundations in auditing standards. This work will progress through our relationships within the international public sector audit community over coming years.

Dr Caralee McLiesh PSM
Auditor-General

Introduction

1.1 Technologies considered to be artificial intelligence (AI) have been in existence for many years. AI has the potential to support public sector entities to drive efficiencies and productivity growth, to improve service delivery and to deliver on their purposes more effectively. On the other hand, poor implementation of AI in public service delivery could risk eroding trust in the public service or cause harm.⁹

1.2 Good governance and assurance arrangements support the delivery of ethical and lawful AI.¹⁰ Given the breadth of AI, the Organisation for Economic Co-operation and Development (OECD) has highlighted that there is no one-size-fits-all approach to AI governance:

different AI systems bring different benefits and risks. In comparing virtual assistants, self-driving vehicles and video recommendations for children, it is easy to see that the benefits and risks of each are very different. Their specificities will require different approaches to policy making and governance.¹¹

1.3 The Australian Taxation Office (ATO) employs AI in a variety of contexts to analyse large datasets for the provision of analysis and assessments.¹² The ATO’s primary use of AI involves AI models that it has built in house, and publicly available generative AI tools that it has assessed as low-risk and approved for use. As of 14 May 2024, the ATO had 43 AI models in production and, as of 18 June 2024, eight publicly available generative AI tools approved for use. Table 2.1 provides further information about the ATO’s use of AI.

Artificial intelligence

1.4 There is no single commonly agreed upon definition of AI. The OECD has identified that this may create challenges when developing legislation and regulation for AI.¹³ As technology has advanced and evolved, definitions have changed and may continue to change.¹⁴

1.5 The Australian Government (the government) has adopted the OECD’s definition of an AI system and suggests that entities should keep up to date on changes to this definition (Box 1).¹⁵

Note a: OECD, Explanatory memorandum on the updated OECD definition of an AI system, OECD, March 2024, p. 4, available from https://www.oecd-ilibrary.org/science-and-technology/explanatory-memorandum-on-the-updated-oecd-definition-of-an-ai-system_623da898-en [accessed 5 April 2024].

1.6 AI systems can be broadly categorised as:

• narrow or weak AI — an AI system that is ‘trained’ to deliver outputs for specific tasks to address specific problems, such as search engines or facial recognition; or
• general purpose or strong AI — an AI system that can be used for a broad range of tasks, both intended and unintended by developers, such as text or image generation.¹⁶

Artificial intelligence in the Australian Government sector

1.7 The government has stated that AI has the potential to enhance Australia’s wellbeing, quality of life and economic growth.¹⁷ In its Data and Digital Government Strategy published in December 2023, the government committed to adopting AI to ‘improve user experience, support evidence-based decisions and gain efficiencies in agency operations’ and to equip entities to safely engage with emerging technologies, including AI.¹⁸

OECD AI Principles and Australia’s AI Ethics Framework

1.8 The ethical use of AI involves promoting the responsible design, development and implementation of AI that is ‘innovative and trustworthy and that respects human rights and democratic values’.¹⁹ Ethical AI practices assist organisations to implement AI technologies that are safer, more reliable and fairer, and reduce the risks of negative outcomes.²⁰

1.9 The OECD has developed five AI principles around: inclusive growth, sustainable development and wellbeing; human rights and democratic values, including fairness and privacy; transparency and explainability; robustness, security and safety; and accountability.²¹

1.10 The government committed to meeting the OECD principles through Australia’s AI Ethics Framework which was published in November 2019.²² This framework ‘guides businesses and governments to responsibly design, develop and implement AI’ and includes eight AI Ethics Principles (see Figure 1.1).²³ The framework states that by applying the ethics principles and committing to ethical AI practices, organisations can build public trust, positively influence outcomes from AI, and ensure all Australians benefit from this technology.

Figure 1.1: Australia’s AI Ethics Principles

Source: Department of Industry, Science and Resources, Australia’s AI Ethics Principles.

Use and initiatives

1.11 As of 30 June 2024, 56 Australian Government entities had reported to the ANAO that they had adopted AI (2022–23: 27 entities).²⁴ Most of these entities had adopted AI for research and development activities, IT systems administration and data and reporting. Of the 56 entities adopting AI, 36 (64 per cent) established internal policies specifically governing the use of AI, while 15 (27 per cent) established internal policies regarding assurance over AI use. Use of AI by entities includes: chatbots, virtual assistants and agents in service management; document and image recognition; to support law enforcement; data mapping to geographical areas; and, as noted in Table 1.1, a trial of Copilot for Microsoft 365.

1.12 Table 1.1 presents a timeline of key Australian Government and parliamentary initiatives relating to AI between 1 July 2023 and 31 December 2024.

Table 1.1: Key Australian Government and parliamentary AI initiatives, July 2023 to December 2024

Note a: Copilot for Microsoft 365 is a generative artificial intelligence chatbot/assistant which integrates with Microsoft 365 products. The ANAO participated in this trial.

Source: ANAO analysis.

National framework for the assurance of artificial intelligence in government

1.13 In June 2024, the Australian Government and state and territory governments released the National framework for the assurance of artificial intelligence in government to support Australia’s governments in ‘gaining public confidence and trust in the safe and responsible use of AI’.²⁵ The framework is based on Australia’s AI Ethics Principles with the intention to establish a ‘nationally consistent’ approach to foundations for AI assurance practices across all aspects of government.²⁶

Policy for the responsible use of AI in government

1.14 In August 2024, the government released the Policy for the responsible use of AI in government to position the Australian Government sector ‘as an exemplar under its broader safe and responsible AI agenda’ and ‘to create a coordinated approach to the government’s use of AI’.²⁷ The policy came into effect on 1 September 2024 and requires²⁸ that:

• entities designate ‘accountable officials’ by 30 November 2024²⁹; and
• entities publish AI transparency statements by 28 February 2025 and update these annually or sooner, if they make significant changes to their approach to AI.³⁰

Artificial intelligence at the Australian Taxation Office

1.15 The ATO’s purpose is ‘to contribute to the economic and social wellbeing of Australians by fostering willing participation in the tax, superannuation, and registry systems’.³¹ The ATO states that it uses data and analytics (including AI) to: understand and improve interactions with its ‘clients’; make better, faster and smarter decisions; deliver outcomes with ‘agility’; and support advice to government.³²

1.16 The ATO has stated that its data activities (including its use of AI) must be both lawful and ethical. The ATO’s six data ethics principles (see paragraphs 2.77 to 2.79) set the minimum standards that must be considered when collecting, using, sharing, archiving, and disposing of data. Through these principles, the ATO aims to address the main identified ethical risks that may arise in data activities and ensure that data is used by the ATO in an appropriate way.

1.17 The ATO reported on the way that it uses AI:

The ATO uses AI tools to increase the efficiency and effectiveness of work done by staff enabling us to deliver better services and greater value to the community. The ATO currently uses AI to review large quantities of unstructured data for risk and intelligence purposes, power risk models to identify potential non-compliance for human review, and draft and edit communications. The ATO has human oversight over all uses of AI, and decision making that impacts clients is always made by a human.³³

1.18 The ATO uses AI in a variety of contexts including: to assess risks associated with submitted claims, such as individual tax returns; to help manage its call centre volumes; and to provide its virtual assistant on the ATO website, Alex (see Table 2.1 for an overview of AI in use at the ATO).

1.19 The ATO has participated in government AI-related initiatives, such as: the AI in Government Taskforce; the Copilot for Microsoft 365 trial; the trial of the Australian Government AI assurance framework; and the development of a whole-of-government Data Ethics Framework.

Previous audits and reviews

1.20 In Audits of the Financial Statements of Australian Government Entities for the Period Ended 30 June 2024, the ANAO outlined that an ‘absence of frameworks governing the use of emerging technologies could increase the risk of unintended consequences’. The ANAO reported that during 2023–24, ‘64 per cent of entities that used AI had also established internal policies governing the use of AI (2022–23: 44 per cent). Twenty-seven per cent of entities had established internal policies regarding assurance over AI use’.³⁴

Rationale for undertaking the audit

1.21 AI technologies offer organisations, including public sector entities, a range of opportunities. AI can help public sector entities to drive productivity growth, to improve service delivery and to more effectively deliver on their purposes. On the other hand, issues that need to be managed include the risk of bias, lack of transparency and accountability, privacy, security and legality. This audit provides independent assurance to the Parliament as to whether the ATO has effective arrangements in place supporting its adoption of AI.

Audit approach

Audit objective, criteria and scope

1.22 The objective of the audit was to assess whether the ATO has effective arrangements in place to support the adoption of AI.

1.23 To form a conclusion against the objective, the following criteria were adopted.

• Does the ATO have effective governance arrangements supporting the adoption of AI?
• Has the ATO established effective arrangements to support the design, development and deployment of AI models?
• Is the ATO effectively monitoring, evaluating and reporting on the adoption of AI?

1.24 The audit scope included:

• an examination of the ATO’s governance arrangements applying to its adoption of AI, with a focus on the period from 1 July 2022 to 30 June 2024;
• an examination of the design, development, deployment and monitoring of 14 AI models deployed by the ATO between 1 July 2023 and 14 May 2024 — see Appendix 3 for a list and description of these models; and
• an examination of the ATO’s AI models that it uses in the context of processing and assessing work-related expenses claims — see Appendix 4 for a list and description of these models.

1.25 The audit scope does not include an examination of the ATO’s data governance and data management more broadly.

Audit methodology

1.26 The audit methodology involved:

• examination of entity records, including email records and electronic documentation;
• meetings with ATO officers and external stakeholders;
• walkthroughs of ATO systems and analysis of selected AI models; and
• review of citizen contributions to the audit.

1.27 Appendix 5 provides an overview of the sources (legislation, standards, policies and guidance) that have informed the methodology for this audit.

1.28 The audit was conducted in accordance with ANAO Auditing Standards at a cost to the ANAO of approximately $892,944.

1.29 The team members for this audit were Nathan Callaway, Dr Shannon Clark, Stewart Hafey, Kayla Hurley, Nancy Jin, Kelvin Le, Zhuo Li, Alyssa McDonald, Benjamin Siddans and David Tellis.

2.1 AI governance is an evolving area, with a common theme that there is no one-size-fits-all governance model. The National framework for the assurance of artificial intelligence in government outlines that ‘[g]overnance structures should be proportionate and adaptable to encourage innovation while maintaining ethical standards and protecting public interests’.³⁵

2.2 At the organisational level, AI governance includes: establishing an AI policy and strategy; assigning and communicating roles and responsibilities; establishing risk management arrangements; and integrating ethical considerations into the design and development of AI systems.

Does the ATO have an effective strategic framework supporting the adoption of artificial intelligence?

Artificial intelligence policy

2.3 The ATO’s data management and data governance policies apply to its use of AI.³⁶ The ATO’s Data Management Chief Executive Instruction (data management policy) sets out requirements for managing data throughout the data lifecycle.³⁷ The ATO states that its data governance arrangements incorporate: policy, standards and guidance; organisational structures and oversight mechanisms; culture, ethics and behaviour; people, skills and competencies; and compliance and issues management.

2.4 While the ATO’s data management and data governance arrangements apply to AI, it has identified that these are not fit for purpose for AI. This includes that: existing arrangements do not sufficiently capture AI-specific risks; there is no defined process when undertaking AI-related activities; there are staff capability gaps around AI-specific risks and data management obligations; there is minimal AI-specific governance, with limited enterprise visibility; and there are few controls to manage risks associated with AI activities.

2.5 The ATO was updating or had plans to update several aspects of its data management and data governance arrangements for AI.

• The ATO was developing an AI policy and AI risk management guidance. These were expected to be delivered in 2024, however, as of December 2024, are expected by December 2025. The ATO has identified that a ‘complex consultation process is required to stand up’ the new policy.³⁸
• The ATO’s data management policy mandates the application of the ATO’s data ethics framework and data stewardship model for all data activities, including AI. In August 2024, the ATO updated its ethics assessment processes for analytical models, including AI models (see paragraph 2.81).

2.6 As part of an April 2024 internal review by the ATO, the ATO sought to determine the extent to which AI requires its own governance and management arrangements. The conclusion was that ‘AI requires a more distinct approach since specialist knowledge is required to be able to manage the unique issues and risks that AI poses’. In the review, the ATO assessed that its framework³⁹ for AI had the most significant gaps (compared to data, analytics and automation frameworks) and the lowest level of maturity.⁴⁰ It also noted a lack of a strategy to fill the gaps at the time. The ATO advised the ANAO on 16 September 2024 that ‘prioritisation and resource constraints have slowed planned work to address data governance gaps’.

2.7 The Policy for the responsible use of AI in government outlines that entities should integrate AI considerations into existing frameworks such as those for privacy, protective security, record keeping, cyber and data.⁴¹ As of July 2024, the ATO had not reviewed other relevant ATO frameworks and policies to assess their applicability to AI and to determine if they need to be updated and adapted for AI.

Use of publicly available generative AI policy

2.9 The Australian Government released guidance on the use of publicly available generative AI tools in July 2023, with updated guidance in November 2023.⁴² In December 2023, the ATO finalised a policy for ATO officers on using publicly available generative AI technology.⁴³ The policy sets out processes aimed at supporting the appropriate and responsible use of publicly available generative AI tools by ATO officers. The policy states that ‘publicly available generative AI technologies may be approved for ATO work purposes, on ATO devices and networked computers, where the risk of negative impact is low’. The ATO’s oversight of this policy is discussed at paragraph 2.39 and its approach to assessing the risk of publicly available generative AI technologies is discussed at paragraphs 2.69 to 2.73.

Artificial intelligence at the ATO

2.10 Prior to the release of the Policy for the responsible use of AI in government, the ATO defined AI as a ‘subset of computer science that deals with computer systems able to perform tasks normally requiring human intelligence, such as object recognition, speech recognition, decision-making and language translation’.

2.11 In October 2024, the ATO reported that it has adopted the Organisation for Economic Co-operation and Development (OECD) definition of AI in accordance with the Policy for the responsible use of AI in government. It has also stated that it ‘refers to any application of machine learning, deep learning and generative AI as AI. But does not consider rules-based analytics to be AI, as this form of analytics does not infer how to generate outputs from the inputs they receive’.⁴⁴ Table 2.1 provides an overview of AI in use at the ATO.

Table 2.1: AI in use at the ATO

Note a: The ATO defines AI models as: algorithms designed to mimic or surpass human intelligence and make predictions based on data; and that use mathematical, statistical or machine learning techniques trained on extensive datasets to process and analyse information.
The ATO’s AI models can link to one or more machine learning algorithms. The 43 AI models link to a total of 93 machine learning algorithms.

Note b: One of the work-related expenses AI models — document understanding — is not included in this list as it had not been put into production as of 14 May 2024.

Note c: These were: Microsoft Copilot; GitHub Copilot Visual Studio 2022 Extension for Business; Code Llama; Llama 2; Adobe Creative Cloud; OpenAI ChatGPT Team; IBM Cloud IaaS; and CoPilot for Microsoft 365.

Source: ANAO analysis of ATO documentation.

2.12 The ATO does not have a centralised inventory of all AI uses across the organisation. The ATO advised the ANAO on 16 September 2024 that there were no other uses of AI within the entity other than those listed in Table 2.1. Subsequent to this advice, in October 2024 the ATO reported to the Joint Committee of Public Accounts and Audit that it had also used AI to process 36 million documents to help identify entities of interest and their relationships, commencing in 2016. Although the original AI models used for this work have since been decommissioned, the ATO further advised the ANAO in January 2025 that it currently uses commercial software which incorporates AI for extracting intelligence from high-volume structured and unstructured data.⁴⁵ This use is not included in Table 2.1.

Register and type of AI models

2.13 The National framework for the assurance of artificial intelligence in government (June 2024) states that an entity should maintain a register of when it uses AI, its purpose, intended uses and limitations.⁴⁶

2.14 The ATO’s list of 43 AI models in production included the model name and grouping, and information about the timing of model execution. The list did not include information about each use case such as: the purpose and context of use; the type of model and technology infrastructure being used; the data used in both training and operation; ongoing cost; the results of recent ethical, privacy, legal and other risk assessments; performance information; and the extent of automation or automated decision-making.

2.15 Having oversight over the use of AI in relation to decision-making is important for governance and managing risks.⁴⁷ The ATO advised the ANAO on 10 May 2024 and 22 July 2024 that of the 43 AI models, 30 did not include fully automated actions and 13 did.⁴⁸ Eleven included automated ‘nudge’ messaging⁴⁹; five included other automated messaging; seven included automated case selection for compliance action; and seven included other automated actions.⁵⁰

2.16 Not having a centralised inventory of AI which includes key information impacts the ability of the ATO to effectively oversee and to be transparent and accountable with regard to its use of AI. Having good visibility and clearly defined purposes for AI models and systems is an important component of building fit-for-purpose governance. As at July 2024, the ATO was developing a register of its data and analytics models, including AI models. The ATO expects to finalise the register by March 2025.

2.18 While the ATO does not maintain a register of key information about its AI models, the ATO’s 43 AI models could all be categorised as narrow AI (refer to paragraph 1.6) because they perform specific tasks or functions. The primary function of these models is to help the ATO analyse data in order to evaluate the risks that a taxpayer’s claims are not compliant, or to understand the impact tax agents have on their clients’ tax affairs.

Artificial intelligence strategy

Development and oversight of the strategy

2.19 The ATO commenced development of an automation and AI (A&AI) strategy in September 2020. The ATO did not have a planned approach to the development of this strategy. Consultation occurred across the ATO in developing the strategy — in the strategy, the ATO outlined that 17 business areas across the ATO were consulted to identify 52 potential A&AI use cases.

2.20 The ATO’s Data and Analytics Committee⁵¹ endorsed the A&AI strategy on 7 October 2022, two years after development commenced. In December 2022, the Data and Analytics Committee was dissolved, with its responsibilities transferred to the ATO’s Strategy Committee.⁵² The Strategy Committee makes decisions and provides advice in relation to the ATO’s strategies, oversees the implementation of strategies and monitors operational performance.

Goals of the strategy and linkages to other ATO strategies and plans

2.21 The ATO’s A&AI strategy states that there is a need for an ‘organisational strategy [for A&AI] to fully harness the benefits’. The vision is that ‘by 2030 the ATO is a leader in developing and industrialising ethical, impactful and scalable A&AI solutions, creating immense economic and social benefit for our nation and citizens’. Measurement against the strategy’s goals is discussed further in paragraphs 4.20 to 4.22.

2.22 The A&AI strategy has links to other ATO strategies and plans, including its corporate plan, digital strategy and data and analytics strategy. These strategies and plans incorporate strategic goals and objectives in relation to the adoption of AI.

2.23 Central to the ATO’s A&AI strategy is the delivery of five enterprise A&AI uses cases. The ATO outlined that use cases were prioritised based on value to deliver, feasibility of implementation and demand. The use cases aim to enhance existing or build new AI systems at the enterprise level. An overview of the five use cases is in Table 2.2.

Table 2.2: A&AI strategy use cases

Note a: Use case five is not AI. Rather, it involves the development of a model monitoring system.

Note b: An application programming interface (API) is a software intermediary that allows applications to communicate with each other.

Source: ATO’s A&AI strategy.

Implementation arrangements

2.24 On several occasions throughout the development of the A&AI strategy, the ATO committee overseeing the development of the strategy sought clarity on implementation. The A&AI strategy did not set out implementation arrangements. It stated that the next steps would involve the development of implementation arrangements including: developing a plan; scoping and designing value cases for each enterprise use case; implementing use cases; reporting on value returned; and proceeding with other future use cases.

2.25 Between October 2022 and July 2024, the ATO acknowledged the need to develop overarching implementation arrangements for the A&AI strategy. On 23 July 2024, the ATO provided the ANAO with a document titled ‘A&AI Implementation plan’ dated January 2024. The ATO described this document as ‘a high-level implementation plan designed for an executive audience’ and noted that further detailed planning is needed. Approval of this document was not evident.

2.26 Implementation of the A&AI strategy is the responsibility of the ATO’s Smarter Data area. The Assistant Commissioner Data Science within Smarter Data is nominated as the senior responsible officer for implementation of the strategy. Roles and responsibilities supporting the implementation of the strategy across the ATO have not been clearly defined.

2.27 According to the ATO’s policy on corporate project management, a corporate project is a body of work that requires resourcing or services that cannot be funded or sourced from within a single business line. While this is the case for the A&AI strategy which has funding sources from multiple business areas, implementation of the A&AI strategy is not being managed as a corporate project. The ATO is implementing the A&AI strategy as a business line project.⁵³ The ATO’s guidance for business line projects require a project outline be completed to document how the project will be managed. A project outline had not been developed for the A&AI strategy.

2.28 On 22 May 2024, the Strategy Committee recommended that delivery of the five use cases should continue with current funding and resourcing (this includes a mix of business-as-usual funding and funding from new policy proposals).

Has the ATO clearly defined and communicated roles and responsibilities supporting the adoption of artificial intelligence?

2.31 A part of good governance is ensuring that responsibilities and accountabilities, including decision-making and oversight roles, are clearly defined and communicated.⁵⁴ For AI, ‘existing decision-making and accountability structures should be adapted and updated to govern the use of AI’.⁵⁵ It should be clear who is responsible and accountable for determining and implementing an entity’s overarching approach to AI and for individual AI systems or uses of AI.⁵⁶

Organisational structures

2.32 The ATO advised the ANAO on 19 April 2024 that key AI responsibilities are within the Client Engagement Group, and the Enterprise Solutions and Technology Group.

2.33 Within the Client Engagement Group, the Smarter Data area⁵⁷ has primary responsibility for the ATO’s management and governance of data and analytics, including AI activities. The Data Science branch in Smarter Data provides skills and advice on AI, machine learning, deep learning and natural language processing. In January 2024, the ATO established an AI Governance team within the Data Management branch in Smarter Data. The Deputy Commissioner Smarter Data is responsible for:

• guiding strategic data management policies, practices and procedures;
• ensuring consistent data governance and management across the ATO; and
• the management of two AI-related enterprise risks: the ‘maximising the value of data and analytics’ risk and the ‘misuse of data and analytics’ risk (see Table 2.4).

2.34 Within the Enterprise Solutions and Technology Group, key responsibilities relating to AI relate to technology solutions and ICT architecture, cyber security governance and operations.

Enterprise-level governance committees

2.35 The Commissioner of Taxation (the Commissioner) is the accountable authority of the ATO and is supported by the ATO Executive Committee. The ATO’s Audit and Risk Committee supports the Commissioner by providing independent advice and assurance.⁵⁸ The Commissioner and the ATO Executive Committee are supported by six enterprise-level committees: Finance Committee; People Committee; Risk Committee; Security Committee; Strategy Committee and National Consultative Forum.⁵⁹

2.36 The ATO’s Risk Committee and Strategy Committee have roles in relation to AI.

• The Risk Committee is responsible for oversight of the management of enterprise risks including the ‘misuse of data and analytics’ and the ‘maximising the value of data’ risks.
• The Strategy Committee is responsible for overseeing the implementation of the ATO’s A&AI strategy. It also has responsibilities in relation to data and analytics including ensuring that ATO strategies and programs are data and analytics driven, and overseeing data ethics and data stewardship issues.

2.37 The ANAO reviewed meeting minutes of the ATO Executive Committee, the Audit and Risk Committee, the Risk Committee and the Strategy Committee between 1 July 2022 and 30 June 2024 to understand how these committees have considered the ATO’s adoption of AI.⁶⁰ A summary of this analysis is presented in Table 2.3. AI was discussed at and reported to these committees on a range of occasions, although reporting was not based on defined and agreed arrangements.

Table 2.3: ATO governance committees’ oversight of the ATO’s adoption of AI

Source: ANAO analysis of ATO documents.

AI roles and responsibilities

AI governance committees

2.38 Over time, the ATO has established governance bodies with areas of focus relating to AI. In September 2024, the ATO established the Data and Analytics Governance Committee. The committee’s role is to make decisions and provide to promote the responsible use of data and analytics at the ATO. The charter notes that this role will evolve over time. It reports to the Strategy Committee.

2.39 In establishing this committee, it was noted that ‘continued fragmentation of approaches to [data and analytics] governance and management’ was a risk of not establishing the committee. From October 2024, the following extant groups report to the Data and Analytics Governance Committee.

• The Business Automation Governance Committee (established in June 2021) is to govern and oversee business automation decisions and to ‘fulfil the A&AI strategy’. The Committee’s role in relation to the A&AI strategy is not clearly defined. The ATO advised the ANAO on 8 August 2024 that it has no formal role in relation to the strategy.
• The Data Ethics Review Panel (established in November 2021) is as an escalation point for data ethics issues. The panel is convened as needed. As of April 2024, it had not been convened.
• The Generative AI Senior Executive Service Band 2 Group (Gen AI B2 Group) was established in November 2023 to oversee the ATO’s use of generative AI technology. It is responsible for the ATO’s ‘Use of publicly available generative AI technology policy’. The Gen AI B2 Group⁶¹ reports to the Risk Committee.

Accountable officials and other expertise

2.40 The Policy for the responsible use of AI in government requires that entities designate accountability for implementation of the policy to accountable officials by 30 November 2024.⁶² The responsibilities of accountable officials ‘may be vested in an individual or in the chair of a body. The responsibilities may also be split across officials or existing roles (such as Chief Information Officer, Chief Technology Officer or Chief Data Officer) to suit agency preferences’. In November 2024, the ATO appointed its Chief Data Officer (Deputy Commissioner Smarter Data) as its accountable official.

2.41 The National framework for the assurance of artificial intelligence in government outlines that AI ‘requires a combination of technical, social and legal capabilities and expertise … such as data and technology governance, privacy, human rights, diversity and inclusion, ethics, cyber security, audit, intellectual property, risk management, digital investment and procurement’.⁶³

Responsibilities and accountabilities — system level

2.42 The accountability principle of Australia’s AI ethics principles states that ‘people responsible for the different phases of the AI system lifecycle should be identifiable and accountable for the outcomes of the AI systems, and human oversight of AI systems should be enabled’.⁶⁴

2.43 The ATO’s data stewardship model is about defining the accountabilities and responsibilities around the management and oversight of the ATO’s data assets. The ATO has identified that further work is needed to fully implement stewardship for AI outputs.

Does the ATO have effective arrangements for managing risks in relation to the adoption of artificial intelligence?

2.46 An accountable authority must establish and maintain appropriate systems of risk oversight and management and internal control for their entity.⁶⁵ The ATO is required to comply with the Commonwealth Risk Management Policy. The National framework for the assurance of artificial intelligence in government outlines that ‘risks should be managed throughout the AI system life cycle’ and that ‘monitoring and feedback loops should be established to address emerging risks, unintended consequences or performance issues’.⁶⁶

Enterprise risk management — AI-related enterprise risks

2.47 The ATO’s Risk Management Chief Executive Instruction⁶⁷ and Enterprise Risk Management Framework (ERMF) set out the ATO’s enterprise-wide approach to risk management. The ATO has a register of its enterprise risks.⁶⁸

2.48 The ATO advised the ANAO on 12 March 2024 that two of its enterprise risks are related to AI due to their focus on data and analytics (see Table 2.4).⁶⁹ This section focusses on these two risks.

Table 2.4: The ATO’s AI-related enterprise risks

Note a: Controls are assessed as: effective; partially effective; ineffective; and insufficient evidence.

Note b: The risk level is based on the consequence if the risk is realised and the likelihood that the risk will be realised. The ATO’s risk levels categories are: very low; low; medium; high; very high and extreme. The ATO updated its risk level and consequence category names in August 2024.

Note c: The tolerance rating is either within tolerance or above tolerance. Above tolerance means that the risk rating is higher than what is acceptable.

Note d: Target risk refers to the level of risk willing to be tolerated for a risk.

Source: ATO documentation.

2.49 The ATO’s risk appetite statement states that it is willing to accept higher levels of risk where there is a clear opportunity to realise benefits and where risks can be controlled to acceptable levels. It is less willing to accept risk where it is not clear that benefits will be realised or where risks are unable to be controlled to acceptable levels. The risk appetite statement provides a sound basis to guide its approach to managing the AI-related enterprise risks. The ATO has identified that failing to appropriately treat the ‘maximising the value of data and analytics’ risk could result in ‘sub-optimal outcomes or opportunities lost’ and failing to appropriately treat the ‘misuse of data and analytics’ risk could result ‘in actual harm’.

Responsibilities

2.50 Roles for managing the AI-related enterprise risks are defined in accordance with requirements. Under the ERMF, risk owners of enterprise risks are ‘personally accountable’ for risk management, provide direction on relevant risk management activities and oversee the status of risks, controls and treatment strategies. The Deputy Commissioner Smarter Data is the risk owner for the two AI-related enterprise risks. In accordance with the ATO’s ERMF, each of these risks also has a risk manager, and control and treatment owners.

Risk assessment

2.51 The ‘misuse of data and analytics’ risk was categorised as an enterprise risk in March 2023. The ‘maximising value of data and analytics’ risk was updated in May 2023 to include ‘analytics’ as part of its scope. The following paragraphs examine the management of these risks.

Maximising the value of data and analytics risk

2.52 There were nine controls allocated to the ‘maximising the value of data and analytics’ risk. Overall, these controls were rated as ‘partially effective’. One of the ATO’s controls had a direct link to the ATO’s adoption of AI and was about the implementation of the five A&AI strategy use cases. This control was identified as supporting the ATO to leverage investment across the organisation by providing access to A&AI solutions more efficiently and quickly.

2.53 The ATO had not assessed the effectiveness of this control due to ‘insufficient evidence’.⁷⁰ The ATO’s assessment indicated a lack of progress in implementing the use cases of the A&AI strategy.

2.54 Overall, the risk was rated as high. The tolerance for this risk (target risk level) was ‘medium’ which means that this risk was above tolerance. As controls were ‘partially effective’ and the risk was tracking above tolerance, the ATO has allocated a risk treatment which focusses on the ATO’s Data and Analytics Strategy, including delivering the A&AI strategy. The treatment owner for this risk was the Deputy Commissioner Smarter Data. The due date for this treatment was listed as 31 December 2024 in the enterprise risk register and 31 December 2027 in the risk assessment and treatment plan. The ANAO sought clarification from the ATO about the due date for this treatment. The ATO advised the ANAO on 16 September 2024 that it anticipates that both enterprise risks will be within tolerance by the completion of the next Data and Analytics Strategy which is planned for 2029.

Misuse of data and analytics risk

2.55 There were 17 controls allocated to the ‘misuse of data and analytics’ risk. These controls were rated as ‘partially effective’ overall. One control was directly relevant to the ATO’s adoption of AI — the implementation of a model ethics assessment process for models containing automation, AI or machine learning. It was described as aiming to detect the possible misuse of data and analytics, including AI, in the design and implementation of models. This control was assessed as having ‘insufficient evidence’ to be able to be assessed as the model ethics assessments were in pilot phase at the time of assessment. See paragraph 2.81 for further discussion on the model ethics assessment process.

2.56 Overall, the ‘misuse of data and analytics’ risk was rated as medium. The tolerance for this risk was ‘low’ which means that this risk was above tolerance. As controls were ‘partially effective’ and the risk was tracking above tolerance, the ATO has allocated a treatment for the risk — the same treatment as for the ‘maximising the value of data and analytics’ risk which relates to the implementation of the ATO’s Data and Analytics Strategy and the A&AI strategy.

Shared risks

2.57 The ATO has categorised the two AI-related enterprise risks as shared⁷¹ with other Australian Government entities that receive or use its data or analytical outputs. According to the ATO’s guidance, shared risks require shared oversight and management between entities.⁷² Arrangements for managing these shared risks have not been established.

Monitoring and review

2.58 The ATO’s risk management guidance states that risks should be monitored on an ongoing basis and periodically reviewed. Monitoring and review should be planned, with clearly defined responsibilities. The ‘maximising the value of data and analytics’ risk should have been reviewed in August 2024 and the ‘misuse of data and analytics’ risk in February 2024. Both risks were reviewed in December 2024.

2.59 Commonwealth entities are required to establish processes for identifying, managing and escalating emerging risks.⁷³ The ATO has identified that of its data and analytics activities⁷⁴, AI activities have the highest overall risk, with a particularly high risk for the ‘misuse of data and analytics’ risk. This is due to the combination of autonomy and adaptivity and lack of controls. This highlights the importance of ongoing monitoring and review of the two AI-related enterprise risks as they are likely to continue to evolve.

2.60 ATO risk management guidance outlines that reporting on risks is ‘an integral part of governance’.

• There was no structured and regular reporting on the two AI-related enterprise risks to the risk owner. The ATO advised the ANAO in January 2025 that since June 2024 the risk manager for the two AI-related risks ‘generally meets with the risk owner on a fortnightly basis’.
• In November 2023 and September 2024, the risk owner for the two AI-related risks provided an update to the Risk Committee, including an overview of the risks, and work underway to mature controls and treatment strategies. The September 2024 reporting provided more information about the environmental factors and drivers of the risks, the controls in place and the treatments needed to bring the risks in tolerance.
• The ATO’s Audit and Risk Committee receives reporting on all enterprise risks, including the two AI-related enterprise risks. This reporting has improved over time.

Arrangements for managing AI business risks

2.63 Under the Commonwealth Risk Management Policy, ‘risk management must be embedded into the decision-making activities of an entity’.⁷⁵ The National framework for the assurance of artificial intelligence in government outlines that the use of AI should be risk based.⁷⁶ According to the OECD:

[o]rganisations should devise, adopt and disseminate a combination of risk management policies that articulate an organisation’s commitments to trustworthy AI. These policies should be embedded into an organisation’s oversight bodies.⁷⁷

2.64 At the ATO, business risks are all other risks (i.e. not enterprise) associated with day-to-day operations. Business risks may be at the group, business line, team or project level. There is no central register of business risks, although these risks can be added to the enterprise register. The ATO advised the ANAO on 12 April 2024 that there are no AI-related business level risks in the enterprise risk register. The ATO also advised that risks may be managed at the project level. It did not identify any such risks.

Risk management throughout the AI system lifecycle

2.65 The ATO’s organisational risk appetite statement notes that it is willing to accept higher risk if there are clear opportunities and where risk can be controlled to an acceptable level. Except for the process to assess publicly available generative AI tools for use by ATO officers (see paragraphs 2.69 to 2.73), as of July 2024, the ATO did not have a defined process for assessing the risk level of AI models and systems.⁷⁸

2.66 The ATO has identified that its current arrangements ‘do not sufficiently capture AI specific risks’. As at July 2024, the ATO was developing AI risk management guidance which is planned to include an AI risk assessment approach. The guidance will aim to assist ATO officers assess whether proposed AI systems are within risk tolerance and whether governance is being applied proportionate to the risk. The ATO’s desired future state includes that governance and processes for AI will be tailored and based on assessed risk.

2.67 As at July 2024, the ATO had a variety of processes that capture or assess risks related to AI. This includes: data and analytics ethics assessments⁷⁹; privacy impact assessments; security risks assessments; and risk assessments as part of project management and model design documentation.

2.68 In chapters 3 and 4, the ANAO examines the ATO’s processes for designing, developing, deploying and monitoring AI models. Risks should be assessed and managed throughout the AI system lifecycle.⁸⁰ As of July 2024, the ATO was also considering how it can integrate risk management into existing processes.

Use of publicly available generative AI

2.69 The ATO has identified the use of generative AI by its officers as a business risk. In September 2023, the ATO conducted a cyber security risk assessment of two generative AI platforms. This risk assessment focussed on the security implications of using publicly available generative AI, and noted that neither of the platforms ‘are recommended as secure for use within the ATO or with ATO data (including the registration of accounts with ATO email addresses)’.

2.70 The ATO commenced drafting a risk assessment and treatment plan for this risk in September 2023. It assessed that the elements of this risk related to security, potential for biased or inaccurate outputs from platforms, impact on accurate decisions and lack of transparency and explainability of outputs. As at July 2024, the assessment had not been finalised.

2.71 As discussed at paragraph 2.9, the ATO has a risk-based policy and process for approving the use of publicly available generative AI by its officers. Under the policy, ATO officers may apply to have generative AI tools approved for work purposes where the risk of negative impact is low.

• Officers must assess the benefits and risks of their proposals. The assessment of risk is around: privacy; technology; data; transparency; and cost.
• Applications are to be sponsored by an SES Band 1 officer, reviewed by the Generative AI Band 1 Working Group and then either approved or rejected by the Gen AI Band 2 Group.

2.72 As of 18 June 2024, there were eight publicly available generative AI tools approved for use in the ATO (see Table 2.1).

2.73 While the ATO did not finalise its risk assessment and treatment plan for the generative AI-related business risk, this risk warrants ongoing monitoring and review. The charters for the Generative AI Band 1 Working Group and the Generative AI Band 2 Group specify responsibilities for these groups to monitor the risks related to the ATO’s use of generative AI.

Does the ATO have effective arrangements to support the implementation of artificial intelligence that aligns with ethical principles?

2.75 Developing and using AI in an ethical manner is an important part of building public trust in the use of AI by public sector entities. Australia’s AI Ethics Principles (see Figure 1.1) provide a framework to support the ethical design, development, deployment and use of AI.

The ATO’s data ethics framework

2.76 The ATO has developed a data ethics framework to support the delivery of data activities, including AI, that meet ethical principles. Since January 2022, the framework has been mandatory for all ATO data activities⁸¹, including AI developed in house or procured by the ATO. The data ethics framework outlines the ‘systems of practices and procedures that help guide our actions’ and ‘demonstrat[e] adherence to our data ethics principles’ and includes: principles; ethics threshold and impact assessments; guidance; and governance.

Principles

2.77 The data ethics principles (see Box 2) were endorsed by the ATO Executive Committee in November 2020 after internal and public consultation. The ATO’s six data ethics principles provide guidance on the behaviours expected when conducting data and related activities.

Source: ATO, How we use data and analytics, 2022, available from https://www.ato.gov.au/about-ato/commitments-and-reporting/information-and-privacy/how-we-use-data-and-analytics [accessed 23 May 2024].

2.78 Aspects of Australia’s AI Ethics Principles which are not addressed by the ATO’s data ethics principles include reproducibility (part of the reliability and safety principle⁸²) and auditability (part of the accountability principle⁸³) of AI system outputs.

2.79 Australia’s AI Ethics Principles do not have a specific principle related to legality.⁸⁴ Principle 2 of the ATO’s data ethics principles is ‘uphold privacy, security and legality’. This principle states that the ATO ‘ensures that the individual and community information we hold is kept safe, protected and shared securely as authorised by law.’ ATO officers are to assess whether AI activities are ‘permissible by law’.

Ethics threshold assessment and impact assessment

2.80 As part of the data ethics framework, ATO officers are required to assess ethical considerations for data activities. Since January 2022, all new projects and any new or significantly changed data activities must complete a data ethics assessment. There are four components to the ATO’s ethical assessments:

• the data ethics threshold assessment (DETA) and the data ethics impact assessment (DEIA) — see Box 3; and
• the model ethics threshold assessment (META) and the model ethics impact assessment (MEIA) — see Box 4.

2.81 In August 2021, the ATO identified that the DETA and DEIA were not sufficient for analytical models, including AI, due to the lack of processes and guidance to support the assessment of ethical considerations for AI models. In April 2022, the ATO commenced piloting a supplementary process to assess ethical considerations for modelling, AI and algorithmic activities (models ethics assessments). The pilot finished in August 2023. The ATO released a revised data ethics assessment on 26 August 2024 (the data and analytics ethics assessment), which includes the four components of the ethical assessments (DETA, META, DEIA and MEIA) in a single assessment template.

Guidance

2.82 The ATO’s Data Ethics Behavioural Guidelines aim to assist ATO officers apply the data ethics framework, including by providing examples of relevant behaviours and guidance on how to address the data ethics questions.

2.83 In August 2024, with the release of the data and analytics ethics assessment (see paragraph 2.81), the ATO released a revised guide (the Data and Analytics Ethics Assessment User Guide) to support officers assess ethics considerations for its data and models. The user guide notes that it is a ‘living document’ that will be updated as advice changes or new content needs to be included.

Governance

2.84 Key roles and responsibilities set out in the data ethics framework include the following.

• The data ethics framework is owned by the Deputy Commissioner Smarter Data.
• Senior Accountable Officers, also known as data leaders, own ethics risks and are responsible for risk mitigation strategies and making decisions about these risks.
• The data ethics team in Smarter Data⁸⁵ has an advisory and escalation role. ATO officers can request advice on ethics assessments and related risk management.
• The Data Ethics Review Panel is to be convened if data ethics issues cannot be resolved or warrant further consideration or escalation.

2.85 The ATO advised the ANAO in July 2024 that several learning and development products support implementation of the data ethics framework. These products include: ‘Working in the ATO’; ‘Safe, Secure and Inclusive’; and ‘Data Ethics Intermediate Training’.⁸⁶

2.86 Similar to ‘privacy by design’⁸⁷ and ‘secure by design’⁸⁸ processes, ethical considerations should be embedded into the design of AI models and systems. The ATO aims to ensure that data ethics is ‘built into corporate processes’.

• In February 2024, the ATO included references to the data ethics framework in corporate project management processes.
• For business line projects, ATO guidance suggests that the ethics assessments should be completed and approved prior to deployment of an AI model.

Completion of ethics assessments — AI models

2.87 As outlined in Table 2.1, as of 14 May 2024, the ATO had 43 AI models in production. For these 43 AI models, as of August 2024, the ATO had finalised:

• two data ethics assessments (covering 11⁸⁹, or 26 per cent of, AI models) — 32 (74 per cent) models did not have an assessment; and
• two model ethics assessments (covering two AI models — five per cent).⁹⁰

2.88 The ANAO requested that the ATO provide an explanation as to why data ethics assessments have not been completed for AI models in production.

• As stated in paragraph 2.80, data ethics assessments have been required for ‘new’ and ‘significantly changed’ data activities and projects since January 2022.
• Of the 43 models in production, 22 were introduced after January 2022 and were ‘new’. Of these 22, 10 (45 per cent) had data ethics assessments.
• Of the other 21 models in production, the ATO advised the ANAO on 21 October 2024 that it considered that two of these models had ‘significantly changed’ since January 2022. Neither of these models had a data ethics assessment. The ATO’s assessment that two of the 21 models have significantly changed is not consistent with ATO guidance which states that ‘significantly changed’ includes ‘changes or updates to the data, model or model suite’. On this aspect, the ATO advised that not all updates to data will result in a significant change. Rather, changes that have the potential to impact identified data ethics risks, or to introduce new risks are considered significant.

Monitoring and assurance arrangements

2.89 Monitoring and assurance arrangements should be designed to assess whether AI is being delivered ethically.⁹¹ Under the ATO’s data ethics framework, there are to be arrangements for the ‘assurance and confidence in the application [of the framework]’.

2.90 As of July 2024, the ATO has not established monitoring and assurance arrangements over its data ethics framework for its AI models, including over the completion and approval of assessments. The data ethics team maintains a register of data ethics assessments that have commenced or have been completed. It does not monitor whether assessments have been completed in accordance with requirements.

2.91 The National framework for the assurance of artificial intelligence in government advises that governments should ‘test and verify the performance of AI systems’ and that the ‘use of AI is continuously monitored and evaluated to ensure its operation is safe, reliable and aligned to ethics principles’.⁹²

3.1 The concept of an AI system lifecycle refers to the phases involved in turning a business problem into an AI solution. Appendix 6 depicts two AI system lifecycles. The Organisation for Economic Co-operation and Development states that organisations implementing AI systems must:

take measures [to] ensure their AI systems are trustworthy — i.e. that they benefit people; respect human rights and fairness; are transparent and explainable; and are robust, secure and safe. To achieve this, actors need to govern and manage risks throughout their AI systems’ lifecycle — from planning and design, to data collection and processing, to model building and validation, to deployment, operation and monitoring.⁹³

Has the ATO established effective arrangements for the design of artificial intelligence models?

Policies and procedures

3.2 The ATO has not developed a framework, including policies and processes, for designing its AI models. The ATO advised the ANAO on 10 May 2024 of the steps it takes to design models (see Box 5). Some of these steps are based on the ATO’s IT Delivery Method.⁹⁴

Note a: ‘Business’ refers to the ATO officers who will be using the outputs of the AI model.

Note b: This is different to a data ethics impact assessment outlined in Box 3.

Source: ATO advice and documentation.

3.3 In addition to the advice in Box 5, AI models may be developed as part of ATO project management methodologies. The ATO requires that its IT Delivery Method is used for projects which have an IT component.

3.4 The steps described in Box 5 provide aspects of a design process. The ANAO compared the ATO’s advice with a range of relevant standards, frameworks, guidelines and policies and observed the following.

• The ATO has not defined the minimum documentation requirements of the design phase, including both technical and non-technical design documentation, and record keeping arrangements for design documentation.⁹⁵
• The ATO has not defined how the ethics principles are to be considered throughout the design phase and other phases of the AI system lifecycle.⁹⁶
• Risk management could be better embedded throughout the design phase.⁹⁷
• Assurance and decision-making responsibilities are not clearly defined.⁹⁸

Privacy and security — platform for model development

3.5 The ATO’s Advanced Analytics Platform (AAP) is a platform for the ATO’s data scientists to develop, train and execute data and analytics models including machine learning, deep learning, neural networks, forecasting and pattern matching. Since December 2022, the AAP has operated in a cloud environment.⁹⁹ The AAP environment is separated from other on-premises and cloud environments used by the ATO, however, it can connect to selected sources of data which host sensitive information.

3.6 Given the role of the AAP, the Protective Security Policy Framework (PSPF) requires that the Commissioner of Taxation, the Chief Information Security Officer or their delegate provide an authorisation to operate prior to use.¹⁰⁰ The ATO refers to this authorisation as a Security Authorisation to Operate (SATO). The ATO has also assessed that obligations under the Privacy Act 1998 to protect personal information collected by the ATO are relevant to the use of the AAP.

3.7 The SATO for the AAP was approved on the basis that activities would be completed to confirm the effectiveness of security controls relied upon by the SATO. These activities were also relevant to confirm the effectiveness of controls addressing the ATO’s Privacy Act 1988 obligations.¹⁰¹ The ATO did not complete all of these activities prior to bringing the AAP into use, and as at June 2024, they had not been completed or planned.

3.8 The PSPF also requires that entities manage security risks for ICT systems on an ongoing basis.¹⁰² The ATO has not consistently monitored the effectiveness of security controls while the AAP has been in operation. The ATO receives regular reports from its cloud provider attesting to the sufficiency of the providers security controls during the reporting period and any identified exceptions to these. These reports were not reviewed by the ATO for a period of two years.¹⁰³

3.9 The ATO developed a system security plan describing how the AAP addresses relevant security controls of a 2019 version of the Information Security Manual. It has not reviewed the system security plan since then to confirm if controls remain appropriate.¹⁰⁴ Regular review of security plans and service provider reports is important to ensure that controls remain effective and appropriate as circumstances change.

Designing models in practice

3.10 The ANAO examined the design practices for the 14 AI models deployed between 1 July 2023 and 14 May 2024. Overall, the ANAO found mixed practices with regards to the planning, governance and design of AI models. Table 3.1 provides a summary of the ANAO’s assessment (as of June 2024).

Table 3.1: Assessment of the planning, governance and design of the ATO’s AI models

Note a: The components in this table were informed by: the AS ISO/IEC 42001:2023 Information technology — Artificial intelligence — Management system standard; Australia’s AI Ethics Framework; and the National framework for the assurance of artificial intelligence in government.

Note b: Models may have more than one PL191.

Note c: To meet the Privacy Act 1988, a privacy impact assessment (PIA) is to be conducted for all high privacy risk activities (where determined through a PIA threshold assessment).

Note d: Three privacy impact threshold assessments were completed to assess 10 models in the same project.

Source: ANAO analysis of ATO documentation.

3.11 Table 3.2 provides a summary of the ANAO’s assessment of the assurance, decision-making and record keeping of the design phase for the 14 AI models deployed between 1 July 2023 and 14 May 2024. The ANAO found that there are areas requiring improvement.

Table 3.2: Assurance, approval and record keeping of the ATO’s design of AI models

Note a: The components in this table were informed by: the AS ISO/IEC 42001:2023 Information technology — Artificial intelligence — Management system standard; Australia’s AI Ethics Framework; and the National framework for the assurance of artificial intelligence in government.

Note b: Models may have more than one PL191.

Source: ANAO analysis of ATO documentation.

Work-related expenses artificial intelligence models

3.12 Work-related expenses (WRE) are expenses incurred by taxpayers, and not reimbursed, to earn their income or when performing their work.¹⁰⁵ WRE claims are the single largest contributor to the tax gap¹⁰⁶ for individuals not in business.¹⁰⁷

3.13 In July 2020, the ATO commenced the Streamlining Substantiation of Deductions project to address the tax gap related to WRE claims by individuals not in business. The project includes the development of AI models to predict and support the treatment of risks associated with WRE claims. Throughout the audit report, these are referred to as the WRE AI models. Appendix 4 provides an overview of the four WRE AI models.

3.14 The ANAO examined aspects of the design of the WRE AI models, with a focus on how the ATO defined the business problem and selected model attributes, and how it has assessed ethical considerations, including privacy and legality.

Defining the business problem and selection of AI model attributes

3.15 AI models should be designed in collaboration with the project owner, or business area, to establish required technical and functional attributes for model design.¹⁰⁸ Case study 1 outlines the business problem and selection of model attributes for the WRE AI models.

Note a: ATO website, Latest estimates and trends.

Ethical AI assessments, including privacy and legality

3.16 To help maintain public trust in the use of AI, Australia’s AI Ethics Principles should be applied throughout the AI system lifecycle to support the ethical design, development and use of AI, including by respecting and upholding rights and data protection.¹⁰⁹ The ATO’s six data ethics principles ‘outline the minimum standards that must be considered’ when conducting data or related activities to ‘address the main identified ethical risks that may arise’. AI can present additional privacy and legal risks. Case study 2 discusses ethics assessments, including assessments of privacy and legality, for the WRE AI models.

Has the ATO established effective arrangements for the development of artificial intelligence models?

Policies and procedures

3.17 The ATO has not developed a framework, including policies and procedures, to support the development of its AI models.¹¹⁰ The ATO advised the ANAO on 10 May 2024 that for model development:

Data Science teams design, build, evaluate and document their models. Version Control is through GitLab.¹¹¹ Models are documented using the AP455 Modelling Solution Report, which is part of the ATO IT Delivery Method documentation suite.

3.18 Box 6 sets out the components of the standardised modelling solution reports (AP455s) that relate to the development of AI models.

Note a: Assumptions refer to the source currency, accuracy and if there are any missing fields.

Source: ATO documentation.

3.19 The ATO’s approach to developing models does not include the following.

• While the ATO has data governance and data management arrangements that relate to data quality, the ATO does not have a process for documenting whether data is fit for purpose for specific AI models. There was also a lack of documented process regarding reproducibility of model outputs and the testing and validation of models.¹¹²
• The ATO does not have a standard approach for controls to segregate duties in such a way that changes to AI models are adequately reviewed before being deployed. The ANAO observed instances where ATO officers made changes to code and reviewed their own changes.¹¹³ These control deficiencies extend to the approval and version control of AP455s which in some instances can be edited by ATO officers with edit access to the project without requiring approval.

Developing models in practice

3.20 The ANAO examined the development practices for the 14 AI models deployed between 1 July 2023 and 14 May 2024. Analysis identified that there was scope to better define considerations relating to data, bias and fairness, peer review and finalisation and approval. Table 3.3 provides a summary of the ANAO’s assessment (as of June 2024).

Table 3.3: Assessment of the development of the ATO’s AI models

Note a: The components in this table were informed by: the AS ISO/IEC 42001:2023 Information technology — Artificial intelligence — Management system standard; Australia’s AI Ethics Framework; and the National framework for the assurance of artificial intelligence in government.

Source: ANAO analysis of ATO documentation.

Work-related expenses artificial intelligence models

3.21 The ANAO examined aspects of the development of the WRE AI models including: considerations of bias and fairness; and data quality, reliability and reproducibility.

Considerations of bias and fairness

3.22 Australia’s AI Ethics Principles include three principles relating to bias and fairness. These are: human, societal and environmental wellbeing (principle 1); human-centred values (principle 2); and fairness (principle 3).¹¹⁴ The National Institute of Standards and Technology states that:

As model implementation progresses and is trained on selected data, the effectiveness of bias mitigation should be evaluated and adjusted. During development, the organization [sic] should periodically assess the completeness of bias identification processes as well as the effectiveness of mitigation.¹¹⁵

3.23 Case study 3 outlines the ATO’s consideration of bias and fairness for the WRE AI models.

Data quality and reliability and model reproducibility

3.24 According to the National framework for the assurance of artificial intelligence in government, ‘the quality of an AI model’s output is driven by the quality of its data’.¹¹⁶ The International Organization of Supreme Audit Institutions states that ‘data quality and reliability is central to the performance of [machine learning] models’ and that ‘reproducibility [is] a mandatory condition for reliability’.¹¹⁷ Reproducibility is the ability to reproduce the model and model outcomes. Reproducibility includes the retention of data sources for reuse.¹¹⁸

3.25 Case study 4 outlines data and reproducibility considerations in the development of the WRE AI models.

Note a: The document understanding model is not included in this assessment as it was not in production.

Has the ATO established effective arrangements for the deployment of artificial intelligence models?

Policies and procedures

3.26 The ATO has not developed a specific framework, including policies and processes, for deploying its AI models. The ATO advised the ANAO in May and July 2024 the steps it takes to deploy AI models. Key aspects are set out in Box 7.

Source: ATO advice and documentation.

3.27 In addition to the advice set out in Box 7, the ATO’s IT change enablement policy applies to all applications, systems or services and to all production, test and development environments. The policy aims to ensure that all IT changes are implemented, managed and governed under a structured framework. This policy is supported by process guidance for deploying models.

3.28 The ANAO reviewed the ATO’s advice about deployment against relevant standards, frameworks, guidelines and policies and observed the following. The ATO’s advice does not specify a deployment planning process, including establishing technical performance criteria to be met prior to deployment.¹¹⁹ Without pre-defined performance criteria, the ATO’s processes do not establish when models should continue in production, be decommissioned or be re-deployed.¹²⁰

Deploying models in practice

3.29 The ANAO examined the deployment practices for the 14 AI models deployed between 1 July 2023 and 14 May 2024. Overall, the ANAO found that there was a lack of clearly defined criteria to be met prior to deployment and there were varying processes for deploying models. Table 3.4 provides a summary of the ANAO’s assessment (as of June 2024).

Table 3.4: Assessment of the deployment of ATO’s AI models

Note a: The components in this table were informed by AS ISO/IEC 2001:2023 Information technology — Artificial intelligence — Management system, and the ATO’s IT change enablement policy.

Note b: A standard change is a low-risk, predictable and repeatable change.

Note c: Segregation of duties requires that developers building and testing software cannot have a role in deploying or promoting their own code to production environments.

Source: ANAO analysis of ATO documentation.

4.1 Entities should document AI objectives¹²¹ and ‘commit to implementing robust AI system performance monitoring and evaluation, and to ensuring each system remains fit for purpose’.¹²² Evaluation helps to ‘confirm that expected results are being achieved and risks are being managed’.¹²³

4.2 Entities should also monitor and evaluate the performance and effectiveness of its arrangements for managing AI. This includes setting out: what needs to be monitored and evaluated; the methods and timing for monitoring and evaluation; and how results will be documented.¹²⁴

Is the ATO effectively monitoring, evaluating and reporting on the implementation of artificial intelligence models?

Policies and procedures

4.3 The ATO has not developed a framework of policies and processes to support monitoring, evaluation and reporting for its AI models. The ATO advised the ANAO on 10 May 2024 and 3 July 2024 of the steps it undertakes in the ‘execution and monitoring’ and ‘decommissioning’ phases (see Box 8).

Note a: The ATO advised the ANAO in December 2024 that it decommissioned two AI models between 1 July 2023 and 14 May 2024.

4.4 The ANAO compared the ATO’s advice with relevant standards, frameworks, guidelines and policies.

• The ATO’s approach does not include developing model operating and monitoring plans for system repairs, updates, and support and corrective actions.¹²⁵
• The ATO does not have processes for the assessment of the ongoing utility of an AI system, including for model drift¹²⁶, which is a key risk to model reliability and safety.¹²⁷ It does not have a defined approach to setting out the measures which should lead to model changes (such as retraining) or decommissioning.
• The ATO does not have a standardised approach for establishing performance metrics and benchmarks.¹²⁸ The ATO is developing a performance metrics library under its automation and AI strategy (A&AI strategy).
• The ATO does not have a consistent approach for training ATO officers who use AI.¹²⁹

4.5 In May 2024, the ATO issued a direction to staff in relation to: implementing peer review; regular performance monitoring; model refresh; and six-monthly review cycles. In relation to this direction, the ATO developed guidance for reviewing models prior to deployment, and created arrangements to support performance monitoring and review of models.

Monitoring and evaluation of models in practice

4.6 The ANAO examined the monitoring and evaluation practices for the 14 AI models deployed between 1 July 2023 and 14 May 2024. Overall, the ANAO found there was a lack of monitoring and evaluation. Table 4.1 provides a summary of the ANAO’s assessment (as of June 2024).

Table 4.1: Assessment of the monitoring of the ATO’s AI models

Note a: The components in this table were informed by: AS ISO/IEC 42001:2023 Information Technology — Artificial intelligence — Management system; Australia’s AI Ethics Principles and the ATO’s Benefits Management Framework.

Note b: Unsupervised learning uses machine learning algorithms to analyse unlabelled data sets.

Note c: At the time of this assessment (June 2024), all models had been in production for less than 12 months.

Source: ANAO analysis of ATO documentation.

4.7 The Policy for the responsible use of AI in government outlines that entities should monitor AI use cases to assess for unintended impacts. The policy also requires that entities publish AI transparency statements by 28 February 2025 that include ‘information on measures to monitor [the] effectiveness of deployed AI systems’.¹³⁰

Work-related expenses artificial intelligence models — benefits and outcomes

4.9 The ANAO examined the ATO’s monitoring and evaluation of the work-related expenses (WRE) AI models (see Appendix 4). Case study 5 describes the ATO’s process of defining benefits and measuring outcomes for the models.

Note a: Under the Commonwealth Evaluation Policy, entities are to ‘plan to conduct fit for purpose monitoring and evaluation activities before beginning any program or activity. This includes identifying time-frames, resources, baseline data and performance information’. Department of the Treasury, What is evaluation, Treasury, available from https://evaluation.treasury.gov.au/toolkit/what-evaluation [accessed 23 October 2024].

Is the ATO effectively monitoring, evaluating, and reporting on the enterprise-wide adoption of artificial intelligence?

Enterprise-wide monitoring and evaluation of the adoption of AI

4.10 As outlined in Chapter 2, the ATO is developing AI-specific governance arrangements including an enterprise AI policy and AI risk assessment. The ATO’s desired ‘future state’ for AI governance recognises the need for additional monitoring, evaluation and reporting.

Monitoring the use of AI

4.11 The ATO does not have centralised and standardised monitoring and reporting arrangements relating to its use of AI. Under use case five of the A&AI strategy, the ATO is developing arrangements to support a consistent approach to the monitoring and evaluation of its AI models and other data and analytics models.

4.12 Planning documents for use case five (enterprise learning feedback loop) outline that ‘there is no centralised and maintained repository of model metadata, or registry models in production’ (see Table 2.2). As such, it involves manual, ‘costly and time intensive’ processes to extract information about models. Further, the lack of a centralised registry ‘presents itself during the model maintenance phase where models may not have clearly defined performance monitoring structures established. It becomes difficult to assess which models are operating effectively, and which models need to be retrained or retired’.

4.13 Full implementation of the model register and performance monitoring components of use case five is planned for December 2026.

Use of publicly available generative AI

4.14 Under the ATO’s ‘use of publicly available generative AI technology policy’, ATO officers are required to comply with a range of ATO policies, such as policies on data management; privacy and confidentiality; proper use of information technology equipment; security; and social media.

4.15 The Generative AI SES Band 1 Working Group and Generative AI Band 2 Group are to monitor and report on the policy, including any breaches of the policy. The Generative AI Band 1 Working Group met 17 times between 15 September 2023 and August 2024. The Generative AI SES Band 2 Working Group met nine times between 28 September 2023 and August 2024. Both groups review applications and associated risk assessments from ATO officers to use publicly available generative AI tools. There was no reporting to these governance bodies in relation to compliance (or non-compliance) with the ‘use of publicly available generative AI technology policy’.

4.16 The ATO advised the ANAO that more than 2000 ATO staff are using approved generative AI tools. As of December 2024, six breaches of the ATO’s generative AI policy had been reported — all six breaches were self-reported by the relevant officer. The six breaches involved the unauthorised use of a generative AI technology (either an unapproved tool was used, or an approved tool was used in an unapproved way).

Artificial intelligence strategy

4.17 The ATO’s A&AI strategy was approved in October 2022 and is focussed on the delivery of five enterprise A&AI use cases (see paragraphs 2.19 to 2.28).

Monitoring implementation and reporting

4.18 Between October 2022 and January 2024, there was no regular internal monitoring and reporting on the implementation of the A&AI strategy.

4.19 In February 2024, the ATO developed a monthly report to monitor the implementation of the five use cases of the A&AI strategy. The reports are provided to the Deputy Commissioner Smarter Data, and relevant Senior Executive Service Band 1 officers across the ATO. Table 4.2 provides a summary of the status of the five A&AI use cases as at June 2024.

Table 4.2: Reporting status of five A&AI strategy use cases, as at June 2024

Note a: Green means ‘on track’; amber means that ‘impacts/mitigations are under review’; and red means ‘action required’.

Source: ATO documentation.

Measurement and evaluation

4.20 The vision of the A&AI strategy is that ‘by 2030 the ATO is a leader in developing and industrialising ethical, impactful and scalable A&AI solutions, creating immense economic and social benefit for our nation and citizens’. The A&AI strategy includes five statements of success:

• AI ethics is at the heart of everything;
• there is enterprise-wide client personalisation;
• there is auto content extraction and understanding for most documents;
• pre-emptive action and real-time interaction is the norm; and
• human and AI are interacting partners through feedback integration and Application Programming Interfaces.

4.21 The ATO has not developed an evaluation approach against the A&AI strategy.

Related strategies and business plans

4.24 As outlined at paragraph 2.22, the ATO has other strategies and plans that include components relating to AI. The ATO does not have overarching monitoring and reporting frameworks for its digital strategy and its data and analytics strategy.

Continuous improvement

4.25 The AS ISO/IEC 42001:2023 Information technology — Artificial intelligence — Management system standard highlights that organisations should ‘continually improve the suitability, adequacy and effectiveness of the AI management system’.¹³¹ The Policy for the responsible use of AI in government states that entities should review ‘on an ongoing basis the internal policies and governance approaches to AI to ensure they remain fit for purpose’.¹³²

4.26 The ATO did not have a structured approach to continuous improvement, although improvement activities had occurred. Activities include: an ATO Executive initiated review of AI governance — this was closed without being finalised (November 2022); an internal paper on the application of the ATO’s data governance and management arrangements to AI (April 2024); and that continuous improvement forms part of the A&AI strategy.

4.27 Internal audits can form part of an entity’s continuous improvement arrangements. Between 1 July 2022 and 12 March 2024, the ATO did not undertake any internal audits on its enterprise-wide adoption of AI. The ATO’s internal audit team reported to the ATO’s Audit and Risk Committee in March 2022 that it had provided guidance to Smarter Data on appropriate governance for AI, and that it had identified gaps in the ATO’s AI governance arrangements. The ATO was unable to locate this advice.

Information management

4.29 The management of information is fundamental to sound public sector administration.¹³³ The National Archives of Australia has noted that record keeping obligations are ‘technology and format neutral’ and that records in relation to AI should document research, development, use and management.¹³⁴ Maintaining reliable data and information assets is important for the development and maintenance of ethical AI by supporting transparency, explainability and accountability.¹³⁵

4.30 The ANAO identified information management issues throughout the course of the audit including poor records of: governance, planning and oversight; risk management; activities throughout the AI system lifecycle; and key decisions. These issues have impacted the ability of the ATO to demonstrate transparency and accountability with respect to its adoption of AI.¹³⁶ Addressing these issues will be important to support additional transparency requirements and expectations, such as the requirement to produce transparency statements as part of the Policy for the responsible use of AI in government.

Appendix 1 Entity response

Appendix 2 Improvements observed by the ANAO

1. The existence of independent external audit, and the accompanying potential for scrutiny improves performance. Improvements in administrative and management practices usually occur: in anticipation of ANAO audit activity; during an audit engagement; as interim findings are made; and/or after the audit has been completed and formal findings are communicated.

2. The Joint Committee of Public Accounts and Audit (JCPAA) has encouraged the ANAO to consider ways in which the ANAO could capture and describe some of these impacts. The ANAO’s Corporate Plan states that the ANAO’s annual performance statements will provide a narrative that will consider, amongst other matters, analysis of key improvements made by entities during a performance audit process based on information included in tabled performance audit reports.

3. Performance audits involve close engagement between the ANAO and the audited entity as well as other stakeholders involved in the program or activity being audited. Throughout the audit engagement, the ANAO outlines to the entity the preliminary audit findings, conclusions and potential audit recommendations. This ensures that final recommendations are appropriately targeted and encourages entities to take early remedial action on any identified matters during the course of an audit. Remedial actions entities may take during the audit include:

• strengthening governance arrangements;
• introducing or revising policies, strategies, guidelines or administrative processes; and
• initiating reviews or investigations.

4. In this context, the below actions were observed by the ANAO during the course of the audit. It is not clear whether these actions and/or the timing of these actions were planned in response to proposed or actual audit activity. The ANAO has not sought to obtain assurance over the source of these actions or whether they have been appropriately implemented.

• The ATO was developing an enterprise-wide AI policy and AI risk management guidance (paragraph 2.5).
• The ATO updated its data ethics framework to incorporate a model ethics assessment process (paragraph 2.5).
• In January 2024, the ATO established an AI Governance team within the Data Management branch in Smarter Data (paragraph 2.33).
• The ATO established a Data and Analytics Governance Committee (paragraph 2.38).
• The ATO reviewed its two AI-related risks in December 2024 (paragraph 2.58).
• The ATO completed a data ethics assessment for the work-related expenses AI models in December 2024 (case study 2).
• The ATO was developing a performance metrics library under its automation and AI strategy (paragraph 4.4).
• The ATO developed guidance for reviewing models prior to deployment, and created arrangements to support performance monitoring and review of models (paragraph 4.5).
• The ATO was developing a register and standardised performance monitoring and reporting framework for its AI models (paragraph 4.11).
• In February 2024, the ATO developed a monthly status report to monitor the implementation of the automation and AI strategy (paragraph 4.19).

Appendix 3 AI models deployed by the ATO between 1 July 2023 and 14 May 2024

1. The ATO deployed 14 AI models between 1 July 2023 and 14 May 2024. The ANAO assessed the design, development, deployment and monitoring of these models — a list of the models and their descriptions is provided in Table A.1.

Table A.1: AI models deployed by the ATO between 1 July 2023 and 14 May 2024

Source: ATO documentation.

Appendix 4 Overview of the ATO’s work-related expenses AI models

1. Table A.2 provides an overview of the ATO’s four AI models in use for work-related expenses (WRE) claims.

Table A.2: WRE AI models, as of 14 May 2024

Note a: Nexus refers to the requirement for taxpayer’s WRE claims to directly relate to their occupation.

Note b: The ATO advised the ANAO on 16 September 2024 that ‘WRE Nexus was ‘productionised’ in June 2022 to enable two case officers to view the outputs and provide verbal feedback to the technical team. It has not been used to inform or determine case outcomes’.

Note c: NNA assumes the average behaviour of a population is compliant.

Source: ANAO analysis of ATO documentation.

Appendix 5 Audit methodology — key sources

1. This audit assessed the ATO’s governance arrangements supporting the adoption of AI — with a focus at both the organisational level and the AI system level.

2. The methodology for this audit was informed by a range of sources from: the Australian Government sector; other Australian jurisdictions; and frameworks from other jurisdictions. Throughout the audit, references are specified where relevant. Table A.3 lists the key sources that underpinned the ANAO’s methodology.

Table A.3: Sources supporting the ANAO’s methodology for this audit

Source: As specified in table.

Appendix 6 AI system lifecycles

1. The following figures present depictions of AI system lifecycles — one from the Organisation for Economic Co-operation and Development and the other from the Government Accountability Office.

Figure A.1: AI system lifecycle — Organisation for Economic Co-operation and Development

Source: Organisation for Economic Co-operation and Development, OECD AI Principles overview, OECD, available from https://oecd.ai/en/ai-principles [accessed 21 October 2024].

Figure A.2: AI system lifecycle — Government Accountability Office

Source: Government Accountability Office, GAO-21-519SP Artificial Intelligence Accountability Framework, GAO, p. 17, available from https://www.gao.gov/assets/gao-21-519sp.pdf [accessed 2 June 2024].