Introduction

1. The Australian Taxation Office (ATO) is responsible for administering Australia’s taxation and superannuation systems. It seeks to build confidence in its administration through helping people understand their rights and obligations, improving ease of compliance and access to benefits, and managing non‑compliance with the law. In 2012–13, the ATO collected $311.7 billion in net cash from taxpayers, incurred operating expenses of $3.5 billion, and had around 25 000 staff.¹

2. As an Australian Government agency, the ATO is expected to use its resources in an efficient and effective manner and to demonstrate to the Parliament and Australian community that its activities are promoting taxpayer compliance and dealing with non‑compliance. As the Organisation for Economic Co‑operation and Development (OECD) has observed, it is a key requirement that revenue agencies responsible for tax compliance ‘are able to understand the impacts achieved and effectiveness of their risk treatment strategies.’² However, measuring the impacts of compliance activities is not straightforward, and revenue agencies around the world grapple with this issue, described by the ATO as ‘challenging and evolving’.³

3. Over many decades, parliamentarians and parliamentary committees have been interested in how the ATO measures the effectiveness of its compliance activities and the related calculations of the ‘tax gap’—the difference between the tax that is actually paid and the tax that should be paid if all taxpayers complied with their legal obligations. The ATO calculates the tax gap for the Goods and Services Tax and the luxury car tax, but not for any other taxes. In February 2014, the ATO advised the House of Representatives Standing Committee on Tax and Revenue that it was researching international practice and consulting with the private sector with a view to developing a position on a suitable approach to tax gaps. At that meeting, the ATO also noted the use of ‘participation indicators’ as a high‑level measure of tax compliance.⁴

4. The challenge of better measuring the effectiveness of compliance strategies is longstanding. In late 2006 the ATO established a project to research, design and implement a methodology to measure the effectiveness of the ATO’s compliance activities.⁵ This approach, known as the Compliance Effectiveness Methodology (CEM), aimed to support the ATO’s business intent of optimising voluntary compliance. The methodology was seen to be an invaluable tool for improving compliance and risk mitigation strategies, resource allocation, and accountability to government and the community.⁶

5. The CEM is based on two key elements: identification of measurable compliance objectives; and articulation and treatment of the risks to achieving them. The CEM involves four phases:

• Phase 1—articulate the risk and align it to the ATO’s business intent of optimising voluntary compliance, making payments under the law and building community confidence;
• Phase 2—define successful outcomes and develop compliance strategies to achieve these outcomes;
• Phase 3—identify and test indicators of success to be used during the compliance strategies; and
• Phase 4—use these indicators to evaluate and report on the extent of the effectiveness of the compliance strategies in the short, medium and long term.

6. The ATO conducted two pilot projects before deciding in July 2007 to implement the CEM within the Compliance Group, and other Groups were requested to consider developing better effectiveness measures relevant to their activities.⁷ From 1 July 2009, the CEM became part of ‘business as usual’ within the Compliance Group.

7. The ATO has taken a devolved approach to implementing the CEM, with Compliance business and service lines (BSLs) being responsible for applying the methodology to their priority compliance risks. As a result, the BSLs have adopted differing approaches to resourcing and implementing the CEM. As at February 2014, the CEM had been applied to 80 identified compliance risks across the eight Compliance BSLs. Under established procedures, risk owners provide direction, context and sign‑off as appropriate in relation to the risks under their control. Risk managers are predominantly tasked with conducting the CEM process. The ATO has established a number of advisory groups to provide assistance to BSLs undertaking CEM evaluations. These include the Centre of Expertise, the Community of Practice and the Compliance Effectiveness Leadership Forum.

8. The ATO initiated an internal review of the CEM process in November 2011. This review found that the CEM had not been fully implemented by the Compliance Group and that little progress had been made in evaluating effectiveness across the ATO. To address these concerns, in January 2012, the ATO initiated an Evaluating Compliance Effectiveness project. Also, in November 2012, the ATO commenced an Embedding Effectiveness Thinking work program, to integrate effectiveness thinking across the agency (at all levels) and to enable it to assess, demonstrate and improve its effectiveness. This work is ongoing.⁸

Audit objective and criteria

9. The objective of the audit was to examine the application of the ATO’s Compliance Effectiveness Methodology in evaluating the effectiveness of key compliance activities and shaping the development of strategies to promote voluntary compliance.

10. To form an opinion against this audit objective, the ANAO adopted the following high level criteria:

• sound management arrangements were in place to support the development and implementation of the CEM;
• CEM evaluations were conducted effectively;
• the results of CEM evaluations were used to shape future compliance strategies; and
• the Evaluating Compliance Effectiveness project and the Embedding Effectiveness Thinking work program were implemented effectively.

Overall conclusion

11. In 2006, the ATO developed and implemented the Compliance Effectiveness Methodology (CEM) to better measure the effectiveness of its compliance activities undertaken to improve voluntary compliance by Australian taxpayers and to build community confidence in the administration of the tax system. The intent was to evaluate the effectiveness of tax compliance strategies in order to improve compliance strategies, treatments, resource allocation and external accountability. At the time, the ATO also hoped to extend the lessons of, and thinking behind, effectiveness evaluations to the whole office. The CEM is a ‘bottom‑up’ approach—evaluating discrete, priority compliance risks—with the results of these evaluations being considered in conjunction with higher‑level approaches such as tax gap analysis and indicators of participation in the tax system, to provide an overall picture of the ATO’s compliance effectiveness. As at February 2014, the CEM had been applied to 80 compliance risks across all Compliance BSLs.

12. The CEM is a sound evaluation methodology for assessing the effectiveness of the ATO’s compliance activities, and has been recognised by the OECD as an innovative approach that provides practical methods for systematically evaluating the impacts of compliance risk treatment strategies.⁹ The four phases provide for the specification of compliance risks and strategies, design and validation of indicators, and reporting of evaluation findings and proposed outcomes.

13. The ATO applied generally sound project management practices and administrative frameworks in developing the CEM and preparing for its implementation within the eight BSLs of the Compliance Group. However, in implementing the CEM as an ongoing business process from 1 July 2009, there has been a lack of strategic focus and direction on its application and outcomes, and on fully embedding the methodology into the ATO’s core processes, including risk management. As a consequence, the full benefits to be gained from the CEM as a means of improving the ATO’s compliance strategies and treatments, resource allocation and external accountability have not been realised. Realising these benefits will require a reenergised approach by the ATO to assessing the effectiveness of its compliance activities.

14. There has always been an intention to apply the CEM to high priority compliance risks. However, without a common understanding across the Compliance Group BSLs of which risks are high priority, the methodology has been applied to a large number of risks of varying nature.¹⁰ The conduct of compliance evaluations¹¹ has also been inconsistent across the Compliance BSLs, particularly during the latter phases of the CEM (Phases 3 and 4) when there is a requirement to design indicators and report evaluation findings. The more thorough evaluations examined by the ANAO¹² (Project Wickenby, Lodgment and Transfer Pricing) had indicators that were clearly linked to specific success goals, desired outcomes and compliance strategies. The reports for these evaluations also contained a clear conclusion about compliance effectiveness that was defensible on the basis of documented evidence and analysis.

15. A significant proportion of the CEM evaluations examined in detail by the ANAO were either incomplete (particularly in their coverage of the non‑compliant population¹³), had insufficient data to support conclusions, or made little reference to ATO compliance activities. Also, nearly all the evaluations resulted in a positive conclusion, and had little influence in encouraging the development of strategies to further promote voluntary compliance. Only four of the 16 evaluations examined in detail had suggested changes to compliance strategies and treatments.¹⁴ These results indicate that the ATO has missed opportunities to use the CEM to substantially strengthen existing risk management processes, and consequently it has not achieved an important element of the overall intent of the program—improved compliance strategies and treatments.

16. More broadly, the CEM program has also not notably improved reporting to the Parliament and Australian community about the effectiveness of the ATO’s compliance strategies. External reporting has mainly involved presenting effectiveness case studies on the ATO website, or similar coverage in annual reports and compliance program reports, rather than presenting a comprehensive picture based on completed compliance effectiveness evaluations. To this end, there would be merit in the ATO taking a more strategic approach to selecting high priority risks for evaluation, and aggregating the results of the CEM evaluations of these risks. Aggregation of CEM results could then be considered in conjunction with a more systematic approach to measuring the tax gap, which the ATO has advised the Standing Committee on Tax and Revenue that it will be considering over 2014.

17. Since 2012, the ATO has sought to extend the focus on effectiveness beyond the Compliance Group to the other two Groups. This work is still in its early stages, and the ATO recognises that further work is required to embed the new effectiveness methodology throughout the organisation. In this regard, it is important that the ATO applies the lessons learnt from developing and implementing the CEM. These lessons include: providing centralised direction and oversight of evaluations; reaching clear conclusions about effectiveness and proposing improved strategies and treatments where appropriate; monitoring the outcomes being achieved; and aggregating results to enable an integrated view of the ATO’s effectiveness.

18. The ANAO has made two recommendations with the aim of the ATO taking a more strategic approach to selecting the compliance risks to be evaluated; and improving the conduct of compliance effectiveness evaluations and reporting of their results.

Key findings by chapter

Developing and Implementing the Compliance Effectiveness Methodology (Chapter 2)

19. The ATO put in place a sound process to develop the CEM in 2006 and 2007, involving a dedicated project team, extensive literature review, two pilot projects and a business case to transition to business as usual. Implementation arrangements were extensive and also generally sound, based on an action plan arising from the business case. A project closure report was produced in June 2009, which concluded that all major project deliverables, such as the establishment of the Effectiveness Centre of Expertise, had been delivered. However, the report did not evaluate whether the project outcomes had been achieved. Also, no processes were established to monitor emerging risks over the course of the program.

20. Despite establishing an extensive administrative framework, there were some shortcomings in managing the CEM as business as usual in the Compliance Group from 1 July 2009, particularly in providing strategic direction in the use of the methodology and achieving the intent of the program. In November 2011, an internal review identified that the implementation of the CEM had been inconsistent. To promote greater consistency, the ATO advised in April 2014 that the existing Effectiveness Centre of Expertise would provide central direction and oversight of the CEM. These roles are intended to ensure that the CEM will: be applied to appropriate risks; be conducted consistently; and achieve its overall objectives.

21. As the CEM is not budgeted or costed, and with devolution of responsibility to the BSLs for staffing and resourcing, the levels of CEM resourcing between BSLs vary depending on availability of funding and local management decisions. This has contributed to some BSLs implementing and integrating the methodology more effectively than others.¹⁵ It has also led to a lack of understanding of the cost of undertaking CEM activity, which has limited the capacity of the ATO to manage the CEM consistently.

22. Training of staff in the CEM is left to the discretion of individual risk managers. CEM training is done though an iLearn package, knowledge‑sharing and on‑the‑job training at the BSL level. As it has been around five years since the original CEM training was provided, and the ATO has identified possible skill and knowledge gaps, there would be merit in the ATO reviewing the adequacy of current training and support for staff applying the methodology, including with respect to statistical and evaluation techniques.

Conduct of Compliance Effectiveness Methodology Evaluations (Chapter 3)

23. Considerable guidance material is available to Compliance BSLs to support staff in conducting CEM evaluations. Nevertheless, the BSLs have taken differing approaches to conducting these evaluations, many of which have recast or extended prevailing risk assessments.

24. Rather than being high priority as envisaged in the CEM guidance, many of the risks originally selected for compliance effectiveness evaluation were of a varying nature.The decisions to select risks were often made by BSLs’ risk management committees or executives, in the absence of clear guidance on determining which risks were high priorities to evaluate, and with the concept of priority not being clearly specified within existing risk management processes.¹⁶ Since the introduction of the Enterprise Risk Management Framework¹⁷ in 2012,the ATO has revised its compliance risks, with a number being downgraded, removed or consolidated with other risks. Accordingly, many operational and tactical risks have been incorporated into higher‑level risks for CEM evaluations.

25. The ANAO’s analysis of the 80 compliance risks that have been subject to a CEM evaluation found that BSLs have not systematically either applied or documented key elements of the methodology outlined in key guidance documents—namely, that workshops were undertaken, indicators developed and evaluations completed. Of the 80 risks, in 62 cases at least one of these three key aspects of the methodology (workshops) was not undertaken or documented. The analysis highlighted considerable problems with record keeping, as many BSLs were unable to provide CEM documentation for important phases, such as workshops.

26. In‑depth analysis of 16 CEM evaluations from the eight Compliance BSLs found that Phases 1 and 2 (to articulate risks and define outcomes) were generally conducted in line with guidance. In contrast, a number of key aspects of Phases 3 and 4 (to design indicators and report results) were not always conducted in line with procedures or effectively overall. The ATO considered that separately conducting Phases 1 and 2 has been useful in encouraging evaluators, who were often drawn from risk teams, to focus on the effectiveness of the compliance strategy rather than deliverables (such as compliance revenue targets) as they had done in the past. However, a number of BSL staff advised that completing the required CEM documentation was burdensome, such as workbook templates that could run to 100 pages when completed. The Compliance Group could now draw more explicitly on existing risk processes for these two phases, with major additional work conducted on an exceptions basis.

27. The ATO has provided direction to its staff on designing indicators¹⁸ in a number of guides, which include extensive discussion of the theory and application of indicator design. Consistent with this guidance, 15 of the 16 evaluations proposed indicators for each success goal.¹⁹ However, only five of the evaluations had incorporated testing of the indicators against the SMART criteria. The lack of indicator analysis reduces the level of assurance that the indicators will measure the outcome of the success goals.

28. All 16 evaluations included indicators that were relevant and reliable²⁰, with more than 93 per cent of the 197 specific indicators satisfying each of the four specific characteristics for these two criteria—that they are focused, understandable, measureable and free from bias. However, six of the 16 evaluations had an incomplete set of indicators, mainly because they did not cover the extent of non‑compliant behaviour, or to a lesser extent, the results of active compliance activities. Measuring the extent of non‑compliance requires information to indicate the magnitude of the non‑compliant taxpayer population and the revenue at risk. Also, the results need to be attributed to the ATO’s activities. CEM guidance provides examples of statistical methods that can be used to attribute observed results to compliance activities, such as randomised control trials which exclude other variables.²¹ In the 16 sets of indicators analysed, no such methods were applied through the CEM process.

Outcomes of Compliance Effectiveness Methodology Evaluations (Chapter 4)

29. Based on ANAO analysis, nine of the 16 evaluations reached defensible conclusions. That is, there was sufficient evidence of detailed analysis and sound judgements to support the ATO’s conclusions about the effectiveness of compliance strategies. Of the seven that did not reach defensible conclusions, five had incomplete sets of indicators (largely because they did not adequately address the non‑compliant population), and two did not have sufficient evidence of the analysis undertaken to reach the conclusion. As identified in the ATO’s November 2011 review of the CEM, the conduct of these evaluations could be improved by strengthening the evaluation capability, and encouraging risk managers and evaluators to address identified deficiencies, such as data inadequacies, attribution of results to the ATO compliance activities and covering the non‑compliant population.

30. According to CEM guidance, the final report is an important means of communicating the outcomes of the CEM so that continuous improvement can occur based on an improved understanding of the risk.²² The ATO was only able to provide evidence of a final report for 53 of the 80 completed compliance evaluations. Of those, 50 had recorded a conclusion, of which 46 were positive, one was neutral, and three were negative—that is, concluding that the existing compliance strategies were not effectively addressing compliance risks. This high level of positive conclusions can partly explain the relatively small proportion of proposals to significantly change compliance strategies and treatments. A notable exception was the fringe benefits tax (FBT) evaluation completed in 2011. This evaluation influenced the Government’s decision to make a law change to the FBT living away from home allowance, effective from 1 July 2012, which limited access to the tax exemption and strengthened the substantiation requirements.

31. While the ATO has published a number of case studies in its recent annual reports and compliance programs, there has been no meaningful aggregation of the results of evaluations to help provide an integrated view of the ATO’s effectiveness. Accordingly, it is not evident that the ATO has achieved its program objectives for the CEM, particularly to improve compliance strategies and treatments, and external reporting of effectiveness.

Embedding Effectiveness Thinking in the Australian Taxation Office (Chapter 5)

32. The ATO’s 2011 internal review identified scope to better integrate the CEM—both into key corporate processes, and throughout the ATO—and provided the impetus for developing the Evaluating Compliance Effectiveness project in January 2012 and a comprehensive blueprint and roadmap for the Embedding Effectiveness Thinking work program in November 2012. A number of deliverables of the blueprint had been achieved as at April 2014, although progress was slow in some areas, such as integrating the CEM into the ATO’s Health of the System Assessments (HoTSAs)²³ and other key corporate processes.

33. The ATO has developed simplified approaches for ATO staff in support areas to consider when evaluating the effectiveness of their activities. Staff undertaking effectiveness work outside the Compliance Group can choose from a suite of support tools, depending on the complexity of work being undertaken, while work is continuing on a modified version of the compliance effectiveness methodology for use in non‑compliance work. It is too early to assess the success of the project to embed effectiveness thinking, particularly outside the Compliance Group.

34. The ATO has identified barriers to achieving its objective of making effectiveness an integral part of its work, both within the Compliance Group and across the agency. Some of the identified barriers are significant, such as the need to develop the ATO’s evaluation capability. Other barriers not explicitly identified by the ATO relate to addressing some of the key lessons learned from implementing the CEM process. Addressing these issues will support the ATO to better integrate effectiveness thinking into the ATO’s decision making processes and demonstrate its effectiveness to the Parliament and Australian community.

Summary of agency response

35. The ATO provided the following summary comment to the audit report:

The ATO welcomes this audit and the recognition by the ANAO that our Compliance Effectiveness Methodology is a sound evaluation method for assessing the effectiveness of the ATO’s compliance activities. The ATO continues to pursue approaches to understanding and demonstrating the impact of our activities.

The ATO recognises the opportunities highlighted by this audit to strengthen and further develop our capacity to undertake strategic evaluations, including the need to better aggregate the results of evaluations to help provide an integrated view of the ATO’s effectiveness.

Over recent years the ATO has increased the publication of case studies and results based on compliance effectiveness evaluations in the Commissioner’s annual reports and compliance program publications. We have also published improved performance information aggregated by tax product, market segment and key compliance obligations. We will continue to build a comprehensive picture of our effectiveness in our performance reporting to the Parliament and the community.

36. The ATO’s full response is included at Appendix 1.

Recommendations

Footnotes

[1] Commissioner of Taxation, Annual Report 2012‒13, pp. i–vi.

[2] OECD Forum on Tax Administration, Evaluating the effectiveness of compliance risk treatment strategies, p. 5, available from <http://www.oecd.org/tax/administration/46274278.pdf> [accessed 25 March 2014].

[3] Commissioner of Taxation Michael D’Ascenzo, Speech to CEDA Trustees Roundtable, Measure for measure: The four pillars of compliance, 25 February 2011, available from <https://www.ato.gov.au/Media‑centre/Speeches/Measure‑for‑Measure‑‑The‑Four‑Pillars‑of‑Compliance/> [accessed 28 May 2014].

[4] House of Representatives Standing Committee on Tax and Revenue, 2013 Annual Report of the Australian Taxation Office, First Report, March 2014. Participation in the system is seen as a good indicator of the level of compliance with tax obligations generally, and the participation indicators are linked to the OECD’s four pillars of compliance—correct registration; on‑time lodgment; complete and accurate reporting; and on‑time payment of tax.

[5] The Compliance Effectiveness Project Business Case noted that while the ATO performs well against key performance measures and indicators, ‘it is acknowledged that these are largely activity based and efficiency focused, rather than indicators of effectiveness that gauge the impact our compliance activities have on compliance behavior’.

[6] ATO, Measuring Compliance Effectiveness—Our Methodology, p. 3. Available from <http://www.ato.gov.au/About‑ATO/About‑us/In‑detail/Key‑documents/Measuring‑compliance‑effectiveness‑‑‑our‑methodology/> [accessed 13 January 2014].

[7] At the time, Compliance was one of five sub‑plans within the ATO. Since 2012, the ATO has changed to a three‑group structure: Compliance; People, Systems and Services; and Law Design and Practice.

[8] For convenience, the ‘Evaluating Compliance Effectiveness’ project and the ‘Embedding Effectiveness Thinking’ work program are referred to collectively in this report as Embedding Effectiveness Thinking.

[9] OECD Forum on Tax Administration, Evaluating the effectiveness of compliance risk treatment strategies, pp. 5 and 9, available from <http://www.oecd.org/tax/administration/46274278.pdf> [accessed 25 March 2014].

[10] Under the ATO’s risk management guidance, there are four levels of risk: strategic risks are linked to the ATO’s strategic planning and priorities; enterprise risks relate to a core or enabling business function or process; operational risks are a component or a part of an enterprise risk; and tactical risks are usually associated with localised events or activities such as transactions, incidents and cases.

[11] Compliance evaluations refer to the application of the CEM to compliance risks.

[12] The ANAO examined 80 CEM evaluations at a high level and 16 CEM evaluations from the eight Compliance BSLs in more detail.

[13] For any compliance risk, there will be a segment of the taxpayer population, generally small, likely to be non‑compliant. It is important that evaluations attempt to understand the nature and key characteristics of this population, and if possible estimate its size in terms of number of taxpayers and revenue at risk.

[14] There was also little documentation to indicate whether these strategies had been implemented, and their impact, in subsequent compliance evaluations.

[15] Based on information provided by BSLs, the ANAO estimated that the total direct employee costs captured across the Compliance Group for CEM‑related activities in 2013–14 was around $1.8 million.

[16] While the ATO’s annual Health of the System Assessment process asks BSLs to identify and rate their compliance risks and develop strategies to mitigate those risks, the process does not have an agreed understanding of what is a high priority risk.

[17] The ATO uses the Enterprise Risk Management Framework as a foundation for managing risks across the office. It uses the Framework to record, categorise and manage all ‘enterprise risks’. The ATO Corporate business line has overall responsibility for the Framework.

[18] The methodology requires the development of a suite of indicators to paint a defensible picture of effectiveness and also advocates use of the SMART (Specific, Measurable, Achievable, Relevant and Timed) criteria in formulating and validating indicators.

[19] Success goals are linked to positive, sustainable changes in behavior and/or community confidence. An example of a success goal is sustained improvement in the population of late lodgers (that is, those that do not lodge their returns by the due date).

[20] The ANAO examined whether the indicators used were relevant, reliable and complete, using an approach developed for assessing Australian Government key performance indicators. This approach is applicable to CEM indicators, which also need to be relevant, reliable and complete in order to build a defensible argument regarding compliance effectiveness.

[21] In a randomised control trial, taxpayers are randomly selected from the population and separated into a target group and a control group. In the example in the CEM guidance, the target group is used to test the effectiveness of an advisory letter on their compliance behavior. The control group is isolated from the treatment. The difference in behavior between the groups is then used as an indicator of the effectiveness of the strategy.

[22] ATO, Measuring compliance effectiveness—Evaluating effectiveness, p. 25, available from http://www.ato.gov.au/uploadedFiles/Content/ATP/downloads/COR25789n73580.pdf [accessed 17 April 2014].

[23] Annually, each BSL is required to prepare a HoTSA that addresses its major areas of risk, establish plans to manage those risks within anticipated funding levels and justify how these positions were reached. Resources are allocated according to these risk assessments. The Superannuation BSL has integrated the CEM into its 2013 HoTSA.