1.1 In November 2005 then Aboriginal and Torres Strait Islander Social Justice Commissioner, Tom Calma, presented his Social Justice Report 2005 to the Attorney-General, which made a series of recommendations regarding establishing a human rights based approach to addressing Aboriginal and Torres Strait Islander² health inequality, including:

That the governments of Australia commit to achieving equality of health status and life expectation between Aboriginal and Torres Strait Islander and non-Indigenous people within 25 years… [and]

That benchmarks and targets for achieving equality of health status and life expectation be negotiated, with the full participation of Aboriginal and Torres Strait Islander peoples, and committed to by all Australian governments.³

1.2 In response to the report, a coalition of peak Aboriginal and Torres Strait Islander and non-Indigenous health bodies and human rights organisations established the ‘Close the Gap’ campaign in March 2006.

1.3 In July 2006 the Council of Australian Governments (COAG), as a component of its national reform agenda, agreed that ‘a long-term, generational commitment is needed to overcome Indigenous disadvantage’, and noted ‘the importance of significantly closing the gap in outcomes between Indigenous Australians and other Australians’.⁴ This commitment, formalised two years later through the November 2008 National Indigenous Reform Agreement (Closing the Gap) (NIRA), came to be known as ‘Closing the Gap’.

The Closing the Gap framework

1.4 The NIRA introduced a framework of objectives, outcomes, outputs, performance measures and targets that the Australian, state and territory governments committed to achieving to close the gap in Aboriginal and Torres Strait Islander disadvantage.⁵ While no direct funding was provided through the NIRA, the Closing the Gap framework was underpinned by eight Indigenous-specific national partnership agreements, which committed funding of around $8.9 billion over ten years for services and programs for Aboriginal and Torres Strait Islander peoples, as well as other mainstream agreements. It also provided for overarching bilateral Indigenous plans to be developed between the Australian Government and state and territory governments to support the implementation of the framework. The NIRA, and the communique from COAG’s November 2008 meeting, stated that it would be a ‘living document’ that would be refined over time based on the effectiveness of reforms and to reflect changes to underpinning agreements.⁶

1.5 The NIRA contained the six Closing the Gap targets that COAG had previously committed to in meeting communiques during 2007 and 2008, covering life expectancy, childhood mortality, education and employment outcomes. COAG subsequently committed to two further targets in 2014 and 2015 (which have not been included in the NIRA but form part of the Closing the Gap framework): a school attendance target and a revised early childhood education target (following the expiry of the previous remote early childhood education target). The eight Closing the Gap targets are outlined in Table 1.1.

Table 1.1: COAG’s Closing the Gap targets

Note a: As life expectancy data is published every five years, following the Census, mortality rates are used as a proxy target in other years.

Note b: Reporting of writing results was discontinued in 2012 due to a testing change that affected the comparability of results over time.

Source: COAG Communiques.

1.6 The NIRA also outlined seven ‘building blocks’ — early childhood; schooling; health; economic participation; healthy homes; safe communities; and governance and leadership — focus areas that COAG recognised were interlinked and foundational to achieving the targets. In addition, it specified six ‘service delivery principles’, which all governments should apply when designing and delivering services for Aboriginal and Torres Strait Islander peoples (see Table 1.2).

Table 1.2: COAG’s service delivery principles

Source: COAG, National Indigenous Reform Agreement (Closing the Gap), September 2012, pp. A21–22.

1.7 Chapter 2 of this report examines the evolution of the Closing the Gap framework, focussing on the governance arrangements that were established to oversee implementation and monitoring of the framework, and mechanisms for stakeholder engagement.

Monitoring and reporting progress towards Closing the Gap

1.8 Through the NIRA and overarching bilateral Indigenous plans, all states and territories committed to annually reviewing their contribution towards the Closing the Gap targets. While the Australian Government did not make an explicit performance reporting commitment under the NIRA, in April 2008, before the NIRA was finalised, the Prime Minister committed to reporting to the Australian Parliament on the first sitting day of each year on progress towards Closing the Gap. Successive Prime Ministers have upheld this commitment by publishing an annual Closing the Gap Prime Minister’s Report. The content of the report has varied over the past decade, but since 2010 it has included high-level analysis of progress towards the Closing the Gap targets.

1.9 In addition, the COAG Reform Council was tasked under the NIRA with preparing an annual independent assessment of progress in each jurisdiction against the Closing the Gap targets and indicators: the NIRA Performance Assessment.⁷ The Reform Council was established in 2006 to report on COAG’s reform agenda and comprised former politicians, senior public servants, academics and business leaders. It produced five performance assessments between 2010 and 2014. After the Reform Council was dissolved in 2014 as part of broader reforms to the COAG council system, COAG asked the Productivity Commission to produce a one-off performance assessment in 2015.

1.10 Since 2016 there has not been an independent performance assessment. The Closing the Gap Prime Minister’s Report, which is currently prepared by the Department of the Prime Minister and Cabinet (PM&C), has consequently been the primary mechanism for the annual assessment of progress against the targets.

1.11 For the duration of the Closing the Gap framework, the annual progress assessments (in both the Closing the Gap Prime Minister’s Report and the NIRA Performance Assessment) have been supported by the Productivity Commission, with oversight from the Steering Committee for the Review of Government Service Provision⁸ (the Steering Committee). Each year it has compiled and published the underlying data for each indicator in the NIRA Performance Information report, drawing on datasets managed by the Australian Bureau of Statistics, Australian Curriculum, Assessment and Reporting Authority and Australian Institute of Health and Welfare.⁹ The Productivity Commission and Steering Committee also produce two other key Aboriginal and Torres Strait Islander reports for COAG on a two to three yearly basis:

• Overcoming Indigenous Disadvantage: Key Indicators — which includes detailed analysis and reporting against a broader range of indicators than Closing the Gap; and
• Indigenous Expenditure Report — which provides aggregated information on Australian, state and territory government expenditure on services for Aboriginal and Torres Strait Islander peoples.

1.12 Figure 1.1 provides an overview of these reports, noting the years they have been published.

Figure 1.1: National reporting on Aboriginal and Torres Strait Islander services and outcomes

 

 

Note a: Department of Families, Housing, Community Services and Indigenous Affairs (FaHCSIA).

Source: ANAO.

1.13 Chapter 3 of this report examines the processes that have been established for collating the underlying data for the Closing the Gap indicators and reporting of progress towards the Closing the Gap targets.

Closing the Gap ‘Refresh’

1.14 In December 2016, as the ten-year anniversary of the establishment of the Closing the Gap framework was approaching and four targets were due to expire in 2018, COAG agreed to refresh the Closing the Gap framework. It stated that the refresh process would be conducted ‘with a renewed emphasis on collaborative effort, evaluation and building on what works in each jurisdiction’.¹⁰

1.15 PM&C commenced a stakeholder consultation process in November 2017 with the aim of developing revised targets by June 2018. At its February 2018 meeting COAG extended the refresh timeframe, committing to agree to a new Closing the Gap framework, national and state targets, performance indicators and accountabilities by 31 October 2018. In September 2018, following the change of Prime Minister, the COAG meeting planned for October was cancelled, with consideration of the Closing the Gap Refresh deferred to the December meeting.

1.16 In December 2018 COAG issued a statement on the Closing the Gap Refresh that committed to finalise new targets, develop Australian, state and territory government action plans, review the NIRA, and establish a three-yearly independent review process by mid-2019.

Closing the Gap timeline

1.17 A timeline outlining the key activities and milestones related to the establishment and implementation of the Closing the Gap framework is at Figure 1.2.

Figure 1.2: Timeline of Closing the Gap

 

 

Source: ANAO analysis.

Australian Government Aboriginal and Torres Strait Islander programs

1.18 In 2004 the Aboriginal and Torres Strait Islander Commission was abolished and responsibility for programs and services for Aboriginal and Torres Strait Islander peoples was transferred to ‘mainstream’ Australian Government entities. When the Closing the Gap framework was established in 2008, FaHCSIA was the lead agency responsible for Aboriginal and Torres Strait Islander Affairs. At that time, other mainstream portfolios that significantly contributed to the Closing the Gap targets were: Health and Ageing; and Education, Employment and Workplace Relations (EEWR). As shown in Figure 1.3 below, these three portfolios were collectively responsible for between 82 to 85 per cent of Australian Government direct Indigenous-specific expenditure in 2008–09, 2010–11 and 2012–13 (the periods covered by the first three COAG Indigenous Expenditure Reports).

Figure 1.3: Direct Indigenous-specific expenditure by portfolio, 2008–09 to 2015–16

Note a: Data is from Australian Government submissions for the 2010, 2012, 2014 and 2017 Indigenous Expenditure Reports, which covered the 2008–09, 2010–11, 2012–13 and 2015–16 financial years, and is the most recent reported data. Reported figures are nominal direct Indigenous-specific expenditure by the Australian Government, which does not include indirect (transfers to other governments) or mainstream expenditure on Aboriginal and Torres Strait Islander peoples.

Source: ANAO analysis of unpublished Australian Government data submissions for the Indigenous Expenditure Report.

1.19 In September 2013 responsibility for Aboriginal and Torres Strait Islander Affairs was transferred to PM&C through a machinery of government change. The introduction of the Indigenous Advancement Strategy in 2014 saw 27 programs consisting of 150 administered items, activities and sub-activities from eight separate entities moved into PM&C. The 27 programs were consolidated into five broad programs under a single outcome, with $4.8 billion initially committed over four years from 2014–15. In 2015–16 (the period covered by the 2017 Indigenous Expenditure Report), PM&C was responsible for 49.3 per cent of Australian Government direct Indigenous-specific expenditure; the Department of Health was responsible for an additional 25.7 per cent of such expenditure through the Indigenous Australians’ Health Programme.

1.20 In addition to direct Indigenous-specific expenditure, like all Australians, Aboriginal and Torres Strait Islander peoples are recipients of mainstream government services. In 2015–16 the proportion of Australian Government mainstream direct expenditure attributed to Aboriginal and Torres Strait Islander peoples amounted to $11.3 billion; most of which was spending on healthcare subsidies and support services, social security support, and general government services and defence. This mainstream expenditure represented 77 per cent of total Australian Government direct expenditure on Aboriginal and Torres Strait Islander peoples.

1.21 Chapter 4 of this report focusses on arrangements for monitoring and evaluating the contribution of Australian Government programs towards the Closing the Gap framework.

Rationale for undertaking the audit

1.22 Aboriginal and Torres Strait Islander Affairs is a policy area with a high level of Parliamentary and community interest. In particular, there is strong interest in ensuring funding provided through government programs achieves intended outcomes. The Australian Parliament and COAG rely on the monitoring and reporting arrangements led by PM&C and the Productivity Commission to understand the level of progress that is being made towards achieving the Closing the Gap targets.

1.23 The audit was undertaken to provide assurance that the Closing the Gap framework is appropriately governed, and that monitoring and reporting arrangements are being managed effectively and draw on accurate and appropriate data. It also sought to provide assurance that PM&C has maintained an appropriate leadership role in managing the framework and ensuring Australian Government programs contribute to Closing the Gap objectives. As the audit was undertaken at an early stage of the Closing the Gap Refresh process, its findings can inform the development of new implementation and monitoring arrangements for the refreshed framework.

Audit approach

Audit objective, criteria and scope

1.24 The objective of the audit was to assess the effectiveness of arrangements for monitoring, evaluating and reporting progress towards Closing the Gap in Aboriginal and Torres Strait Islander disadvantage.

1.25 To form a conclusion against the audit objective, the Australian National Audit Office (ANAO) adopted the following high level criteria:

• Have appropriate governance arrangements for implementing and monitoring the Closing the Gap framework been maintained? (Chapter 2)
• Have appropriate processes been established for reporting progress towards the Closing the Gap targets? (Chapter 3)
• Have effective processes been established to monitor and evaluate the impact of Australian Government programs on Closing the Gap? (Chapter 4)

1.26 The audit examined the Australian Government’s arrangements for monitoring, evaluating and reporting progress towards Closing the Gap, focusing on PM&C’s role as the lead agency for Aboriginal and Torres Strait Islander policy co-ordination and the Productivity Commission’s role in collating and reporting information on outcomes for Aboriginal and Torres Strait Islander peoples.

1.27 The audit did not examine the appropriateness of the original Closing the Gap targets or the new targets being proposed through the Closing the Gap Refresh. It did not directly examine the effectiveness of programs to address the targets.

Audit methodology

1.28 The audit methodology included:

• examining and analysing documentation relating to the monitoring, evaluating and reporting progress towards Closing the Gap;
• interviews with key officials in PM&C, the Productivity Commission and other Australian Government entities involved in the delivery of programs for Aboriginal and Torres Strait Islander peoples or the collection and analysis of data for Closing the Gap; and
• consultation with key Aboriginal and Torres Strait Islander stakeholders, including peak Aboriginal and Torres Strait Islander bodies and human rights and health organisations.

1.29 The audit was conducted in accordance with ANAO Auditing Standards at a cost to the ANAO of $480,000.

1.30 The team members for this audit were Daniel Whyte, Iain Gately, Mary Huang and Deborah Jackson.

Have oversight arrangements been maintained?

2.1 Two key components of the Closing the Gap framework were the commitments made by the Australian, state and territory governments through the NIRA to:

1. a comprehensive, integrated approach to achieving the Closing the Gap targets through coordinated program and service delivery, including regular reviews of whether policy settings are effective in achieving the intended outcomes; and
2. an annual assessment of progress against the Closing the Gap targets.

2.2 At the commencement of the Closing the Gap framework, two primary oversight bodies that reported to COAG were tasked with managing these two components:

• Working Group on Indigenous Reform (WGIR) — which was established to drive the Aboriginal and Torres Strait Islander reform agenda through engagement on COAG agreements specific to Closing the Gap; and
• COAG Reform Council — which was tasked with undertaking the annual independent assessment of progress.

2.3 These bodies were disbanded in 2013 and 2014 respectively as part of broader COAG council reforms. Figure 2.1 shows the oversight arrangements in place in 2010 and 2018, noting the high-level functions of certain oversight bodies. This section examines whether effective oversight has been maintained following the dissolution of the WGIR and COAG Reform Council.

Figure 2.1: Oversight arrangements for the Closing the Gap framework, 2010 and 2018

 

 

Note a: The NIRA Performance Information Management Group provides advice in relation to selected aspects of data collation to the Steering Committee for the Review of Government Service Provision on request.

Source: ANAO analysis of departmental documentation.

Oversight of coordinated program and service delivery

Oversight arrangements at the commencement of the Closing the Gap framework

2.4 The WGIR was established in 2008 with the goal of driving Aboriginal and Torres Strait Islander reform and achieving the Closing the Gap targets. It was chaired by the Australian Government minister with responsibility for Aboriginal and Torres Strait Islander Affairs, and its membership included senior officers from Australian, state and territory government entities. The WGIR met nine times from 2009 to 2013, following an annual work plan agreed by members at the beginning of each year. It reported directly to COAG, and was supported by the Indigenous Reform Coordination Sub-Group and the NIRA Performance Information Management Group.

2.5 In addition, the development of overarching bilateral Indigenous plans was intended to establish oversight mechanisms under the Closing the Gap framework. In June 2009 the Australian Government commenced the negotiation of bilateral plans with each state and territory. While these plans were expected to commence in 2010, the majority took longer to negotiate, with the last plan finalised in August 2012 (see Table 2.1).

2.6 The bilateral plans established working groups between the Australian Government and each state or territory, and were envisaged as forums to coordinate the implementation of the Closing the Gap framework. Although the working groups provided some coordination functions, no arrangements were established for providing advice to COAG on bilateral plan implementation.

COAG Reform Council’s lessons for federal reform

2.7 The COAG Reform Council identified shortcomings in the implementation arrangements for the Closing the Gap framework in its 2013 report to COAG, Lessons for federal reform, which stated:

While there is evidence that much of the [Closing the Gap] framework has been implemented, there remain areas in which implementation appears incomplete. For instance, it was suggested in relation to the NIRA‘s schedules that implementation has been patchy and that they are seldom a significant driver of government activity.¹¹

2.8 The report attributed these implementation gaps to a lack of leverage available to Closing the Gap working groups within government. It noted that further progress could have been achieved through more consistent leadership and regular meetings of senior officials to create a foundation of shared commitment, which could then be communicated across governments to drive cultural change.

Oversight arrangements since 2013 COAG council reforms

2.9 In late 2013 COAG announced it would undertake significant reforms to its council system to minimise ‘bureaucracy and red tape’, reducing the number of COAG councils from 22 to eight.¹² At the same time, it dissolved the WGIR and introduced Aboriginal and Torres Strait Islander Affairs as a standing agenda item for the remaining eight COAG councils.¹³

2.10 With the dissolution of WGIR, bilateral working groups established under the overarching bilateral Indigenous plans also ceased meeting. While the bilateral plans were nominally due to expire between 2015 and 2017, all of the plans were removed from the Council on Federal Financial Relations’ website in late 2014.¹⁴

2.11 The ANAO examined meeting communiques published by COAG councils over the period of 2014 to 2018 and found only one reference to the Closing the Gap framework, by the Health Council in March 2017. In addition, COAG council consideration of matters relating to Aboriginal and Torres Strait Islander Affairs over the period was limited.

• The Health Council referenced items relating to Aboriginal and Torres Strait Islander Affairs in six of its thirteen communiques over the period (46 per cent).
• Four councils (Education Council, Disability Reform Council, Law, Crime and Community Safety Council, and Industry and Skills Council) referenced items relating to Aboriginal and Torres Strait Islander Affairs in fewer than 30 per cent of communiques.
• Three councils (Energy, Federal Financial Relations, and Transport and Infrastructure) did not make any references to items relating to Aboriginal and Torres Strait Islander Affairs.

2.12 The COAG Senior Officials Meeting¹⁵ has more regularly considered matters relating to Aboriginal and Torres Strait Islander Affairs. The ANAO examined its meeting papers relating to Aboriginal and Torres Strait Islander Affairs over the period of 2014 to 2018 and categorised the items considered by topic (see Table 2.2).

2.13 The Senior Officials Meeting’s focus from 2014 to 2016 was primarily on economic participation and schooling (in particular, school attendance, which was introduced as a Closing the Gap target in 2014). From 2017, its focus has been solely on the Closing the Gap Refresh. It has not provided a forum for the regular reviews envisaged under the NIRA of ‘the contributions of existing and announced initiatives to meeting the targets’ or to ‘identify areas where programs may need to be adjusted to help meet the targets’.¹⁶

Table 2.2: COAG Senior Officials Meeting consideration of Aboriginal and Torres Strait Islander Affairs, 2014–2018a

Note a: Figures reported in the table are the number of items relating to each building block considered in meeting papers during the year. Items are categorised by Closing the Gap building block.

Source: ANAO analysis of departmental documentation.

2.14 In 2017 COAG established a time-limited Ministerial Council for Indigenous Affairs to oversee work on the Closing the Gap framework and other Aboriginal and Torres Strait Islander priorities. The Council’s terms of reference focus primarily on the Closing the Gap Refresh process (discussed in paragraphs 1.14 and 1.15). In early 2019 the Council will be reviewed as part of a scheduled COAG council review.

Oversight of reporting of progress against Closing the Gap targets

2.15 Oversight of monitoring and reporting on progress under the Closing the Gap framework was originally the responsibility of the COAG Reform Council. The Reform Council had responsibility for reporting annually on the performance of all jurisdictions in achieving outcomes against the Closing the Gap targets and associated performance indicators. As noted in paragraph 1.9, it was considered by COAG to be an independent body, with appointed members from outside of government, and it produced five NIRA Performance Assessments between 2010 and 2014.

2.16 The primary focus of the NIRA Performance Assessments was reporting against the Closing the Gap targets and indicators at both a national and state and territory level. Until its 2012 report, it also included reporting by relevant entities against national partnership agreements that underpinned the Closing the Gap framework. From 2012 it ceased including this material, noting a concern with such reporting:

We cannot link the activity reported to the outcomes and objectives in the [NIRA]. Reports on National Partnerships generally provide information on the activity without evidence of the effect the activity has on outcomes.¹⁷

2.17 The Reform Council was dissolved in early 2014 following COAG’s decision to streamline and refocus the COAG council system. As an interim measure, COAG asked the Productivity Commission to provide an independent assessment of progress against the Closing the Gap targets in 2015. In 2016 the COAG Senior Officials Meeting considered rationalising national Aboriginal and Torres Strait Islander reporting, by consolidating the Overcoming Indigenous Disadvantage report, Indigenous Expenditure Report and NIRA Performance Assessment, but no decision was taken at this time.

2.18 The required annual independent assessment of jurisdictional progress under the Closing the Gap framework has not been undertaken since 2015.

2.19 The Closing the Gap Prime Minister’s Report, which has been released in February each year since 2009, has also included reporting against the Closing the Gap targets. From 2016 it has been the primary reporting mechanism for the national assessment of progress against the targets.¹⁸ However, as discussed further in Chapter 4, it does not provide an independent assessment of jurisdictional performance.

2.20 Submissions and feedback received by the ANAO noted a deterioration of the reporting and oversight arrangements over time and a focus on reporting against the targets without corresponding oversight of and accountability for implementation of the framework.¹⁹ For example, one peak body stated that:

… the process has come to focus on measuring progress against the Closing the Gap targets and producing reports, rather than providing analysis of the effectiveness of interventions, appropriateness of expenditure and where to direct resources, and measuring incremental progress to highlight more clearly any changes towards achieving the targets.

NIRA Performance Information Management Group

2.21 From July 2009 the NIRA Performance Information Management Group has played the lead role in advising on Closing the Gap data issues. It has provided advice to the WGIR, the Steering Committee for the Review of Government Service Provision and other COAG councils and working groups on reporting against the Closing the Gap targets, and facilitated cross-jurisdictional agreement on data specifications. As shown in Figure 2.1, it is the only Closing the Gap oversight body that has been maintained for the duration of the framework. The Group is currently chaired by a senior officer from the Department of the Prime Minister and Cabinet (PM&C) with representation from data agencies and Australian, state and territory government Aboriginal and Torres Strait Islander Affairs and mainstream social policy agencies.²⁰

2.22 Over the duration of the Closing the Gap framework, the Group has undertaken tasks such as annually updating the specifications for NIRA datasets, and monitoring progress on various data quality improvement projects. For example, following the introduction of the new target for early childhood education in December 2015 (95 per cent of Indigenous four-year-olds enrolled in early childhood education by 2020), the Group provided advice to finalise technical specifications for the target and determine an appropriate methodology for measuring it. The Group’s work on administering protocols for Closing the Gap data is discussed in more detail in Chapter 3.

2.23 Following the dissolution of the WGIR, there has been no formal governance structure for the Performance Information Management Group, although it has continued to meet regularly and has provided advice on request to the Steering Committee for the Review of Government Service Provision.

2.24 The Productivity Commission has continued collating and publishing data for the annual assessment of progress towards the Closing the Gap targets. This work has continued to feed into the Closing the Gap Prime Minister’s Report and COAG Report on Performance. In June 2017 the Performance Information Management Group noted the data collation task was originally intended to support the COAG Reform Council’s independent assessment of progress against the Closing the Gap targets, and its purpose is unclear following the dissolution of the Reform Council.

Oversight of the refreshed Closing the Gap framework

2.25 PM&C has noted the need for oversight of the refreshed Closing the Gap framework, and this also emerged as a key message from PM&C’s stakeholder consultation. In October 2018 a group of 14 Aboriginal and Torres Strait Islander peak bodies wrote to the Prime Minister, Premiers and Chief Ministers to express concern around the oversight of the refreshed framework, noting that it must include agreement on a governance mechanism that will allow joint monitoring, evaluation and reporting on progress.

2.26 To ensure a program or framework is effective, appropriate governance arrangements should be established to oversight its implementation and ongoing management. COAG’s December 2018 communique committed to establishing a new Ministerial Council on Closing the Gap ‘with Ministers nominated by jurisdictions and representation from Aboriginal and Torres Strait Islander peoples’.²¹ The communique and a separate COAG statement on the Closing the Gap Refresh state that the arrangements for this formal governance partnership will be established by the end of February 2019. In order to ensure momentum is sustained for the refreshed framework, it will be important to maintain the governance arrangements established through this partnership for the duration of the framework.

Has the National Indigenous Reform Agreement been updated to reflect changes to the framework?

2.27 The NIRA states that:

[This agreement], like other National Agreements, is a living document subject to enhancement over time to reflect additions and changes to existing and new National Agreements and National Partnership agreements. As COAG agrees to additional reforms to Close the Gap in Indigenous disadvantage, these will be reflected in this Agreement.²²

2.28 Responsibility for maintaining and updating the NIRA was vested in the WGIR. The NIRA was first updated in June 2009 to incorporate additional elements such as information on baselines and trajectories for the Closing the Gap targets and the National Framework for Reporting Expenditure on Services to Indigenous Australians. It was updated again in February 2011 and November 2012 to include various changes, such as revisions to performance indicators.

2.29 The NIRA has not been updated since November 2012 and does not reflect a number of changes that have occurred within the Closing the Gap framework since 2012 (see Table 2.3).

Table 2.3: Elements that are out of date or not reflected in the NIRA

Source: COAG, National Indigenous Reform Agreement (Closing the Gap), 2012.

2.30 A November 2017 internal PM&C briefing for the Closing the Gap Refresh stated that the NIRA did not support the Australian Government’s ambitions for modernising COAG arrangements in its current form, and it would need to be shortened, language clarified and expired schedules removed.

2.31 As the NIRA is out of date, it should be revised or replaced with a new agreement to support the refreshed Closing the Gap framework. To avoid the new or revised agreement becoming out of date in the future, PM&C should work through the COAG system to ensure that mechanisms are established that enable it to be regularly updated to reflect changes in the underlying framework.

2.32 In December 2018 COAG committed to review the NIRA by mid-2019.

Have Aboriginal and Torres Strait Islander stakeholders been engaged in governance of the framework?

Engagement in implementation of the Closing the Gap framework

2.33 As noted in paragraph 1.2, the Closing the Gap framework had its genesis in a campaign initiated by Aboriginal and Torres Strait Islander stakeholders. Despite this, there was limited consultation with Aboriginal and Torres Strait Islander stakeholders in the development of the original Closing the Gap framework.

2.34 Under the NIRA, COAG committed to work in partnership with Aboriginal and Torres Strait Islander peoples to achieve the Closing the Gap reforms. This was reflected in the NIRA’s service delivery principles (see Table 1.2); in particular, the Indigenous engagement principle stated that ‘engagement with Indigenous men, women and children and communities should be central to the design and delivery of programs and services.’²³

2.35 Nevertheless, working groups established to oversight the framework, such as the WGIR, did not provide for Aboriginal and Torres Strait Islander participation. The NIRA Performance Information Management Group has noted that its lack of formal Aboriginal and Torres Strait Islander representation has limited its ability to provide advice on barriers to data development and Aboriginal and Torres Strait Islander data priorities. In 2017 it agreed that any future terms of reference should expand its membership to include Aboriginal and Torres Strait Islander representatives.

2.36 Stakeholders consulted by the ANAO noted the need for greater clarity on engagement mechanisms and genuine engagement with Aboriginal and Torres Strait Islander peoples under the refreshed Closing the Gap framework.

National Congress of Australia’s First Peoples

2.37 In 2008 the Australian Government requested that the Aboriginal and Torres Strait Islander Social Justice Commissioner convene an independent steering committee to develop a preferred model for a national representative body for Aboriginal and Torres Strait Islander peoples. Following widespread consultation, the National Congress of Australia’s First Peoples was announced by the Commissioner in May 2010 as the national representative body. The Australian Government allocated $6 million to establish the National Congress and a total of $38.2 million to support its operation from January 2011 to June 2017.

2.38 The overarching Indigenous bilateral plans established between the Australian Government and states and territories recognised the National Congress as the peak representative body for Aboriginal and Torres Strait Islander engagement, and noted that it would be consulted where appropriate. However, the ANAO reviewed the minutes of the bilateral plan working groups and found no evidence that they engaged with the National Congress.

2.39 In 2014 the Government withdrew the remaining $15 million in funding that the National Congress had been allocated from 2014–15 to 2016–17. It was subsequently provided $3 million over three years in the 2016–17 Budget to deliver specific outcomes for the Government, which have included facilitating consultation on the Closing the Gap Refresh (discussed below).

Indigenous Advisory Council

2.40 The Prime Minister’s Indigenous Advisory Council was established by the Government in 2013. The Council plays a key advisory role to the Government, and its members are directly appointed by the Prime Minister. The Council’s original terms of reference did not include any reference to the Closing the Gap framework.

2.41 Following the change of Prime Minister in late 2015 and the expiry of the previous Council’s term, a new Indigenous Advisory Council was appointed in 2016. Its revised terms of reference noted that it would perform a key role in Closing the Gap, including through providing advice on the Closing the Gap Refresh.

Engagement in Closing the Gap reporting

2.42 The NIRA states that jurisdictions will work in partnership with Aboriginal and Torres Strait Islander peoples by ‘sharing information on performance against indicators in implementation plans and agreements, and reports on progress towards the COAG targets’.²⁴

2.43 The ANAO examined the extent to which Aboriginal or Torres Strait Islander stakeholders were engaged in the preparation of the 2018 Closing the Gap Prime Minister’s Report and found there was limited engagement prior to its final release.

• PM&C consulted internally with staff that identify as Aboriginal or Torres Strait Islander through a Senior Indigenous Reference Group and the PM&C Aboriginal and Torres Strait Islander Employee Network.
• Through the PM&C Regional Network, engagement with Aboriginal or Torres Strait Islander communities focussed on asking for ‘good news stories’ and obtaining permission for the use of photos in the report.
• The Indigenous Advisory Council was briefed on the proposed approach to the 2018 report in October and December 2017 and provided advice on messaging.

Engagement in the Closing the Gap Refresh

2.44 PM&C’s Closing the Gap Refresh taskforce developed an engagement strategy in 2017 to facilitate collaboration with Aboriginal and Torres Strait Islander peoples during the Refresh process. The strategy proposed conducting consultation sessions from late 2017 until mid-2018. Engagement was targeted at national representative bodies, peak organisations, regional organisations, academics, specialists, practitioners, key Aboriginal and Torres Strait Islander leaders and community members.

2.45 PM&C held 18 meetings across Australia between November 2017 and April 2018, co-hosted with the National Congress, and included a Special Gathering of prominent Aboriginal and Torres Strait Islander peoples that coincided with the first COAG meeting of 2018. These meetings were attended by approximately 460 stakeholders representing a wide range of organisations and bodies. These meetings were intended to gain an understanding of issues of importance to Aboriginal and Torres Strait Islander peoples and key priorities. In addition, in December 2017 a discussion paper was released for public comment. Submissions closed in April 2017 and PM&C received 107 organisation and 66 individual submissions.

2.46 Following the meetings and public consultation, PM&C hosted two technical workshops in May and June 2018. These workshops focused on developing targets and indicators based on the priority areas raised in the meetings and other consultations, and were attended by academics, practitioners, other experts and Australian, state and territory government officials.

2.47 An additional 11 meetings were held across Australia in August 2018 to provide stakeholders that had previously been engaged through the Refresh process the opportunity to provide feedback on the draft target areas that had been developed through the technical workshops. The information gathered from these consultations fed into the draft Closing the Gap targets that were released by COAG in its December 2018 statement on the Closing the Gap Refresh.

2.48 COAG’s December 2018 communique committed to establishing a formal partnership between COAG and Aboriginal and Torres Strait Islander representatives by the end of February 2019, which would include a new Ministerial Council on Closing the Gap. COAG committed to working jointly through this formal partnership to undertake the following actions by mid-2019:

• finalise the draft targets;
• review the NIRA;
• develop an independent, Aboriginal and Torres Strait Islander-led, three-yearly evaluation and review of progress, working with the Productivity Commission; and
• co-design the development of Commonwealth, state and territory action plans to achieve the refreshed targets.

2.49 In addition, COAG’s statement outlined a set of implementation principles. Engagement with Aboriginal and Torres Strait Islander stakeholder was addressed by the Shared Decision-Making principle, which stated:

Shared Decision-Making — Implementation of the Closing the Gap framework, and the policy actions that fall out of it, must be undertaken in partnership with Aboriginal and Torres Strait Islander peoples. Governments and communities should build their capability to work in collaboration and form strong, genuine partnerships in which Aboriginal and Torres Strait Islander peoples can be an integral part of the decisions that affect their communities.²⁵

2.50 COAG’s commitment to a formal partnership with Aboriginal and Torres Strait Islander peoples to design, implement and monitor the refreshed Closing the Gap framework is an opportunity to establish a governance framework that facilitates ongoing engagement and collaboration. It will be important to ensure this partnership arrangement is maintained for the duration of the refreshed framework.

Have robust processes been implemented to administer Closing the Gap data?

3.1 Since 2009 the National Indigenous Reform Agreement (NIRA) Performance Information Management Group has maintained oversight of Closing the Gap data protocols. Two key aspects of its oversight role have been:

• annual updates to Closing the Gap data specifications; and
• monitoring progress on data quality improvement projects.

Closing the Gap data specifications

3.2 Under the Closing the Gap framework, there are currently 15 performance indicator sets (see Table 3.1), seven of which relate to the Closing the Gap targets. These indicator sets draw on source data from 16 separate national datasets, which are managed by the Australian Bureau of Statistics (ABS), Australian Curriculum, Assessment and Reporting Authority (ACARA) and Australian Institute of Health and Welfare (AIHW).

3.3 As shown in Table 3.1, ABS is the data custodian²⁶ for the majority of Closing the Gap datasets, providing data for 13 of the 16 datasets. AIHW is the custodian for one dataset and ACARA is responsible for two datasets. Unlike ABS and AIHW, ACARA acts as a clearinghouse that redistributes data received from the states and territories, rather than as a data custodian, for the two datasets it manages.

Table 3.1: Data responsibilities under the Closing the Gap framework

Note a: Performance indicator sets related to Closing the Gap targets are bold.

Note b: These indicator sets have separate specifications for the use of Census and survey data.

Source: ANAO analysis of AIHW’s Metadata Online Registry (METeOR) data specifications.

3.4 The NIRA Performance Information Management Group provides advice on proposed changes to data specifications for Closing the Gap performance indicators.²⁷ Data specifications include:

• type, status, description, rationale and related outcome area;
• quality statement — describing any data quality issues relating to the indicator set;
• computation — how it is calculated, including specifying the numerator and denominator;
• disaggregation —for example, by state/territory and/or remoteness area; and
• data source.

3.5 Each year AIHW and the Productivity Commission manage a process to update the data specifications for indicator sets, after which AIHW publishes them on METeOR²⁸ (see Figure 3.1 below). After the specifications have been formally endorsed by the Performance Information Management Group, they are provided to the Steering Committee for the Review of Government Service Provision for endorsement. The process is generally undertaken in June to July each year.

Figure 3.1: Closing the Gap data specification process

 

 

Source: ANAO analysis.

3.6 Based on a review of agency internal documentation and meeting papers and minutes of oversight bodies, the ANAO found the annual data specification process has been conducted consistently since 2009, with regular oversight from the Performance Information Management Group and Steering Committee. It is good practice to publish specifications for performance indicators at the level of detail that has been demonstrated for the Closing the Gap indicators, and this process should be retained for the refreshed Closing the Gap framework.

Data quality improvement projects

3.7 Under Schedule F of the NIRA, ABS and AIHW were tasked with undertaking a variety of data development activities to improve the quality of Aboriginal and Torres Strait Islander health and welfare data in order to better monitor the COAG Closing the Gap targets. Oversighting this work has been a key responsibility of the NIRA Performance Information Management Group. In November 2010, all members of the Group agreed to provide written updates on data development activities.

3.8 Improvements that resulted from this work include:

• publication in 2012 of national best practice guidelines for data linkages relating to Aboriginal and Torres Strait Islander peoples;
• completion of a feasibility study in 2012 that led to the initiation of an ongoing enhanced mortality database project to improve estimates of Aboriginal and Torres Strait Islander life expectancy;
• development and release in 2015 of a best practice model for improving Aboriginal and Torres Strait Islander identification in the medical general practice sector; and
• enhancements to the Perinatal National Minimum Data Set to capture data on antenatal care and the Indigeneity of babies.

3.9 Although Schedule F of the NIRA expired in June 2013 (as noted in Table 2.3), members have continued to provide an annual update on progress against the schedule to the Performance Information Management Group. In June 2017 the Group noted some concerns with the annual update report, including that:

• while ABS, AIHW and some jurisdictions had provided some updates for the report, on the whole changes were minimal and there were areas in the report that had not been updated for a number of years; and
• funding to oversee data development activities was provided by a national agreement that had since ceased.

3.10 Following the June 2017 meeting, AIHW was tasked with reviewing the reporting on data development activities. In April 2018 NIRA Performance Information Management Group papers listed this work as closed as it will be superseded by new arrangements anticipated to come out of the Closing the Gap Refresh.

Are effective arrangements in place to ensure the quality and timeliness of data?

3.11 The preparation of data for annual reporting under the Closing the Gap framework occurs across three key stages:

1. data agencies (ABS, ACARA and AIHW) create relevant national datasets;
2. data agencies prepare data for the Closing the Gap indicators in accordance with agreed specifications; and
3. the Productivity Commission and PM&C collate and publish the data in the NIRA Performance Information report and Closing the Gap Prime Minister’s Report.

National dataset creation

3.12 ABS, ACARA and AIHW are responsible for administering the 16 national datasets used for reporting under the Closing the Gap framework (see Table 3.1 above). Depending on the dataset, data is either collected directly by the agency or obtained from state and territory data owners. In either case, the agency conducts a validation process to assess the integrity and quality of the data, which can include both automated and manual data validation procedures. In the case of datasets managed by AIHW and ACARA, approval for validated data is sought from state and territory data owners before the national datasets are created and locked.

3.13 The ANAO examined whether there was evidence that data agencies validated datasets used for reporting under the Closing the Gap framework.²⁹ All agencies were able to provide evidence of the processes they undertake.

• ABS has established a number of mechanisms to plan and monitor data validation processes for its datasets. These include strategic decision points throughout the data creation and validation process, and automated validation checks undertaken using SAS and SuperCROSS.³⁰ ABS has also established additional validation processes for Census data involving Aboriginal and Torres Strait Islander peoples.
• ACARA disseminates national data standards to guide schools and education agencies in collecting data for the National Student Attendance Collection and NAPLAN Report. For student attendance data it has an automated validation process that generates exception reports, which are provided to states and territories to correct identified discrepancies. Data validation processes for the NAPLAN Report are conducted by the contractor that compiles the dataset.
• AIHW uses a system called ‘Validata’ for the National Perinatal Data Collection, which identifies anomalies and discrepancies with submitted data and allows states and territories to correct and re-upload their data. After jurisdictions have addressed issues identified by Validata, AIHW runs manual data validation checks and liaises with states and territories to address any additional issues.

Preparation and collation of Closing the Gap data

3.14 Following the annual data specification process outlined in paragraph 3.4 and 3.5, the Productivity Commission prepares a series of data request templates for each indicator set, reflecting any changes to the specifications. It provides these templates and data specifications to ABS, ACARA and AIHW in July, which then extract data from the datasets for which they are responsible and populate the templates. ACARA undertakes limited data extraction, as it provides the entire National Student Attendance Collection and NAPLAN Report datasets directly to the Productivity Commission and PM&C.³¹

3.15 Although the request process undertaken by PM&C for the Closing the Gap Prime Minister’s Report is different to the Productivity Commission’s data collation request, it predominantly relies on the same datasets.

3.16 The data collation process undertaken for Closing the Gap reporting, including the quality assurance activities to be conducted by data and reporting agencies, is outlined in Figure 3.2. The collation process generally occurs from July to December each year, with the timing of data delivery depending on the availability of data.

Figure 3.2: Preparation and collation of Closing the Gap data process

 

 

Source: ANAO analysis of entity guidelines.

Quality assurance of data extracted from national datasets

3.17 The ANAO examined whether ABS and AIHW had formal quality assurance processes for Closing the Gap data extraction and whether they followed these processes for the 2017 Closing the Gap data collation process (see Table 3.2).

Table 3.2: Quality assurance of data extracted for Closing the Gap reporting

Note a: For these tests, the rating is based on an aggregated assessment across multiple datasets. For ABS, the ANAO examined processes for five indicators (linked to Closing the Gap targets) extracted from ten datasets; and for AIHW, three indicators extracted from one dataset.

Note b: Under the Census and Statistics Act 1905, ABS is not required to seek approval from state and territory data owners to release data compiled from jurisdictional sources.

Source: ANAO analysis of department quality assurance procedures and processes.

3.18 AIHW had a standard operating procedure outlining quality assurance checks and approval processes for extracting data from the National Perinatal Data Collection, which aligned with the tests in Table 3.2. AIHW maintained adequate records that it followed the required steps in accordance with its procedure.

3.19 ABS’s internal Policy and Legislation Manual contains high-level principles outlining the need for statistical teams to define and apply quality assurance and approval processes for the release of data, and document their compliance with these processes for audit purposes. While there was evidence that ABS undertook quality assurance checks for data extracted for the 2017 process, well defined procedures for extracting data from its datasets for Closing the Gap reporting were not consistently in place and quality assurance checks and internal approvals were not adequately documented.

Quality assurance of data for the NIRA Performance Information report

3.22 The Productivity Commission does not have an overarching quality assurance framework. As outlined in Figure 3.2, for the NIRA Performance Information report, it manually reviews data return spreadsheets received from data agencies to ensure there are no obvious anomalies. Where anomalies are identified, it liaises with the data agency to resolve issues and confirm there are no errors in the data. Once it is satisfied the data is correct, it transfers the data from the data return spreadsheets to the report. The Productivity Commission informed the ANAO that, prior to its release, two staff members review the accuracy of the data in the report, but it did not maintain records of these checks for the 2017 NIRA Performance Information report.

3.23 The ANAO independently checked the accuracy of data presented in the 2017 NIRA Performance Information report relating to the seven Closing the Gap targets and did not find any errors. While this suggests the Productivity Commission’s quality assurance was effective, it should document its processes to ensure they are consistently followed and reporting is accurate.

Quality assurance of data for the Closing the Gap Prime Minister’s Report

3.24 PM&C undertook quality assurance checks of the accuracy of reported data for each of the Closing the Gap targets reported in the 2018 Closing the Gap Prime Minister’s Report. It maintained registers to track quality assurance checks for two targets. For the other targets, email correspondence provided evidence of quality assurance checks. PM&C did not maintain a central register to track quality assurance processes for the report as a whole.

3.25 The ANAO checked the accuracy of data presented in the 2018 Closing the Gap Prime Minister’s Report relating to the seven Closing the Gap targets and did not find any errors in the reported figures, but identified one instance where the presentation of data was unclear.

• The 2018 Closing the Gap Prime Minister’s Report states that ‘Between 1998 and 2016, the Indigenous infant mortality rate declined significantly by around 66.7 per cent (from 13.5 deaths per 1,000 live births in 1998 to 6 in 2016)’.³² The decline from 13.5 to 6 is 55.6 per cent. The reported decline of 66.7 per cent was calculated using linear regression, but the method of calculation was not communicated in the report. PM&C indicated that it planned to include a footnote in the 2019 Closing the Gap Prime Minister’s Report to clarify this point.

3.26 In response to the ANAO’s findings during the audit, PM&C improved its approach to systematically managing quality assurance on data presented in the 2019 Closing the Gap Prime Minister’s Report, using a central register to record data sources and checks undertaken.

Timeliness of Closing the Gap data

3.27 Although the NIRA Performance Information report and Closing the Gap Prime Minister’s Report are produced annually, the datasets they draw on were not created for annual Closing the Gap reporting and are updated at different intervals. For example, the National Aboriginal and Torres Strait Islander Social Survey provides data for four Closing the Gap indicator sets, but has only occurred twice within the duration of the Closing the Gap framework.³³ The disparity in timeframes between datasets means some indicator sets are not updated each reporting cycle. In addition, for data that is updated annually, due to differences in the timing of data collection and availability, reported data can be from the previous year or two years prior.

3.28 Timeframes for the NIRA performance information reporting cycle are agreed in May by the Steering Committee for the Review of Government Service Provision. Following the annual confirmation of Closing the Gap data specifications (discussed in paragraphs 3.2 to 3.6), the Productivity Commission sends out data requests in late July. A data deadline is set as late September, although much of these data are not available until after this date with the latest currently available in December. The draft NIRA Performance Information report is provided to PM&C in late December, and then published in February.

Does reporting draw on appropriate information, analysis and interpretation?

3.29 The NIRA included a commitment to annual reporting against COAG’s Closing the Gap targets. This has been a central component of the Closing the Gap framework since 2010, when baseline data collected in 2009 was first reported in the NIRA Performance Assessment and Closing the Gap Prime Minister’s Report.

3.30 Since 2011 progress towards the targets has been determined by comparing annual results for each target indicator with indicative target trajectories, developed based on an assessment of the level of progress that would be needed each year to meet the target. If the result is on or better than the target trajectory in a given year, progress is determined to be ‘on track’, otherwise progress is determined to be ‘not on track’. See Table 3.3 for a summary of progress determinations between 2011 and 2018.

Table 3.3: Progress towards Closing the Gap targets, 2010–2018^a

Note a: Years are reporting years; depending on the target, the results are either the previous year or two years earlier.

Note b: Productivity Commission.

Note c: Since 2012, due to a change in the NAPLAN writing test, progress has only been assessed for the reading and numeracy targets.

Source: ANAO analysis of NIRA Performance Assessments and Closing the Gap Prime Minister’s Reports.

3.31 The COAG Reform Council was initially responsible for reporting progress towards the Closing the Gap targets in the NIRA Performance Assessment. Following the Reform Council’s dissolution, the Productivity Commission published an independent assessment in 2015. Since 2016, there has not been an independent assessment, so the Closing the Gap Prime Minister’s Report (which has been published since 2009) has been the primary means for reporting progress against the targets at a national level.

3.32 The ANAO examined whether information, analysis and interpretation included in the 2018 Closing the Gap Prime Minister’s Report was appropriate, focussing on determinations of progress against the Closing the Gap target trajectories. As noted in paragraph 2.18, the report also includes case studies and descriptions of certain programs and services for Aboriginal and Torres Strait Islanders; the appropriateness of this information as a form of performance reporting is discussed in Chapter 4.

Determinations of progress against target trajectories

Development of target trajectories

3.33 The NIRA Performance Information Management Group developed target trajectories at a national, state and territory level in December 2010. The trajectories were developed by calculating the ‘gap’ at the baseline year, then determining the rate of change that would be necessary to close or halve the gap by the target year. For the two targets that use mortality data (life expectancy/mortality and child mortality), the Group subsequently updated the target trajectories in 2014 to take account of population estimate updates from the 2011 Census. The Group developed target trajectories for the school attendance and early childhood education targets when they were introduced in 2014 and 2015.

3.34 Five of the current targets (life expectancy/mortality, child mortality, employment outcomes, student attendance and early childhood education) have straight line trajectories. Two targets (reading and numeracy, and Year 12 or equivalent attainment) have non-linear trajectories, which were selected to anticipate the impact of the roll-out of education programs. Unlike the data specifications for Closing the Gap performance indicators, the methodologies for calculating the target trajectories are not published.

3.35 For the life expectancy/mortality and child mortality targets, ABS calculates variability bands around the annual results to assist in understanding the variability in the data using a methodology agreed by the Performance Information Management Group in 2010.³⁴ In addition, since the COAG Reform Council’s 2011 NIRA Performance Assessment, a variability band has been applied to the target trajectory when determining whether a target is on track. While the methodology for calculating variability bands for the annual results is published in the METeOR data specifications, the methodology for applying the variability band to the target trajectory has not been published.

3.36 The trajectories for the seven current targets, including variability bands for the mortality and child mortality targets and results from the 2018 Closing the Gap Prime Minister’s Report, are presented in Appendix 2.

2018 progress determinations

3.37 In the 2018 Closing the Gap Prime Minister’s Report, four of the Closing the Gap targets were determined to be ‘not on track’ and three were determined to be ‘on track’. The ANAO reviewed the underlying data and found the determinations, along with the supporting analysis and interpretation in the report, were generally appropriate (see Table 3.4).

Table 3.4: ANAO assessment of 2018 progress determinations

Source: ANAO analysis; quotes are from PM&C, Closing the Gap Prime Minister’s Report 2018.

3.38 The ANAO’s analysis of the child mortality data in Case Study 1 and Appendix 3 was based on unpublished papers and data obtained over the course of the audit. It would be difficult to replicate this analysis without access to this unpublished information. Given the high level of stakeholder interest in the Closing the Gap targets, there is a need for greater transparency about the methodologies used for determining trajectories and making determinations about whether targets are on track. PM&C should work through the COAG system to publish these methodologies in sufficient detail to allow replication of the calculations.

Have new programs been explicitly linked to the Closing the Gap framework?

4.1 In line with the National Indigenous Reform Agreement (NIRA) priority principle, any Australian Government programs for Aboriginal and Torres Strait Islander peoples developed since 2008 should exhibit clear links to the Closing the Gap framework. To assess whether new programs have been linked to Closing the Gap, the ANAO examined the extent to which national partnership agreements and Indigenous Budget measures introduced between 2008 and 2018 had explicitly stated links to the Closing the Gap framework. The ANAO also reviewed links between the Australian Government’s two major Indigenous-specific programs: the Department of the Prime Minister and Cabinet’s (PM&C) Indigenous Advancement Strategy; and the Department of Health’s Indigenous Australians’ Health Programme. These programs did not involve any new funding, so were not reported as new Indigenous-specific measures in Budget papers.

National partnership agreements

4.2 National partnership agreements, negotiated under the 2009 Intergovernmental Agreement on Federal Financial Relations, facilitate payments from the Australian Government to states and territories to support the delivery of improvements in service delivery or reforms. They can be supported by implementation plans that outline key deliverables and milestones. Figure 4.1 shows national partnership agreements implemented between 2008 and 2018 with direct links to the Closing the Gap framework.

Figure 4.1: National partnership agreements with links to Closing the Gap, 2008–2018

 

 

Source: ANAO analysis.

4.3 When the NIRA was last updated in 2012, it was underpinned by eight Indigenous-specific national partnership agreements, which had been negotiated during 2008–09 and 2009–10, as well as eight other mainstream agreements that had direct links to the Closing the Gap framework (see Figure 4.1). Many of the agreements that underpinned the NIRA expired between 2012–13 and 2013–14, or were superseded by other agreements. These changes to the underlying structure of the Closing the Gap framework were not incorporated into the NIRA.

4.4 As Figure 4.1 illustrates, the majority of agreements that reference the Closing the Gap framework were implemented in its first five years. Only three additional national partnership agreements have been developed since 2012 that reference the Closing the Gap framework:

• National Partnership Agreements on Universal Access to Childhood Education — four consecutive agreements, implemented in 2013, 2015, 2016 and 2018, which include the revised Closing the Gap target for early childhood as a performance measure;
• National Partnership on Northern Territory Remote Aboriginal Investment (2016–2022) — which references the Closing the Gap targets; and
• National Partnership on Remote Housing (2016–2018) — which states that it supports the outcomes of the NIRA.

4.5 There have been other national partnership agreements implemented since 2012, including agreements targeting Aboriginal and Torres Strait Islander peoples, that have not included direct links to the Closing the Gap framework. An example of an agreement that did not link to the framework is the National Partnership on Essential Vaccines (2017–2021), which includes an outcome to minimise the incidence of vaccine preventable diseases in Aboriginal and Torres Strait Islander people for diseases with vaccines listed under the National Immunisation Program, but, unlike the 2009–2017 Essential Vaccines agreement, there is no reference to the NIRA or Closing the Gap.

Aboriginal and Torres Strait Islander Budget measures

4.6 Between 2008 and 2013 the Department of Families, Housing, Community Services and Indigenous Affairs (FaHCSIA) coordinated the development of a Single Indigenous Budget Statement that compiled Indigenous-specific new policy proposals for consideration by the Government. A key step in this process was a self-assessment by proponent agencies against the NIRA’s service delivery principles (listed in Table 1.2). From 2014 the Single Indigenous Budget Statement process ceased, and there was no longer a requirement that policy proposals include an assessment of their contribution towards Closing the Gap targets.

4.7 In 2014 PM&C sought to establish an overarching whole-of-government framework for Aboriginal and Torres Strait Islander Affairs. This was intended to address program duplication through a review of all Australian Government Indigenous-specific programs and mainstream programs with a significant Aboriginal and Torres Strait Islander client base, including an assessment of the level of alignment with the Closing the Gap targets. Following delays collating information from entities and concerns about the utility of the information received, implementation of the framework was not completed.

4.8 In 2016 the Government established an Indigenous Policy Committee to promote a whole-of-government approach to Aboriginal and Torres Strait Islander policy decision-making and program implementation.

4.9 Since 2016, there have been two new measures, which were included in the 2017–18 Budget, with explicit links to the Closing the Gap framework:

• Closing the Gap — English language learning for Indigenous children trial ($5.9 million over four years); and
• Closing the Gap — Employment services ($54.3 million over four years).

4.10 Table 4.1 shows a breakdown of Indigenous-specific Budget measures announced between 2008–09 and 2018–19 where explicit reference was made in the description of the measure to Closing the Gap. The majority (88 per cent) of Budget measures with direct links to Closing the Gap framework were announced in the 2008–09 and 2009–10 Budgets.

Major Aboriginal and Torres Strait Islander programs

Indigenous Advancement Strategy

4.11 The introduction of the Indigenous Advancement Strategy (IAS) in 2014 saw 27 programs consisting of 150 administered items, activities and sub-activities from eight separate entities moved into PM&C. The IAS created five broad-based programs: Jobs, Land and Economy; Children and Schooling; Safety and Wellbeing; Culture and Capability; and Remote Australia Strategies. As at 31 July 2018, approximately $7.0 billion had been allocated through IAS grants since its inception.

4.12 While the Auditor-General’s performance audit of the IAS (Auditor-General Report No.35 2016–17) found PM&C did not effectively implement the strategy, the report did note that the IAS’s program structure broadly reflected the Government’s five key priority areas for investment.³⁵ The audit report included four recommendations, which PM&C accepted, focusing on improving IAS grant administration processes, performance measurement and the operation of PM&C’s Regional Network.

4.13 The Closing the Gap framework is not referenced in the original policy proposal for the IAS (2014), the implementation plan (2014) or the current IAS grant guidelines (March 2016). Nevertheless, there is a degree of alignment between IAS program objectives and the Closing the Gap building blocks and targets. The IAS programs broadly align with five of the seven Closing the Gap building blocks (the two not aligned are health and healthy homes), and program outcomes have been linked to five of the seven targets (the two not linked are life expectancy/mortality and child mortality).

Indigenous Australians’ Health Programme

4.14 On 1 July 2014 the Australian Government established the Indigenous Australians’ Health Programme (IAHP), consolidating four existing funding streams administered by the Department of Health. Since 2015 grants for primary healthcare under the IAHP totalling approximately $1.44 billion have been awarded, with approximately 85 per cent of this funding going to Aboriginal Community Controlled Health Organisations (ACCHOs). As at 1 April 2018, more than 230 organisations (including around 140 ACCHOs) were funded under the IAHP. Of these, around 160 organisations and 130 ACCHOs were funded to deliver primary health care.

4.15 The Auditor-General’s performance audit of primary healthcare grants under the IAHP (Auditor-General Report No.50 2017–18) found the Department of Health’s design and implementation of this IAHP component has been partially effective as it ‘has not implemented the planned funding allocation model and there are shortcomings in performance monitoring and reporting arrangements.’³⁶ The audit report included three recommendations, which the Department of Health accepted, focusing on improving value for money assessments, risk management of data collection processes and performance measurement.

4.16 The IAHP program guidelines explicitly refer to the Closing the Gap framework, referencing the two health related targets: life expectancy and childhood mortality. The guidelines also explicitly reference the NIRA service delivery principles, stating that the Department of Health’s administration of the IAHP will comply with these principles.

Has an Australian Government implementation plan been developed for Closing the Gap?

4.17 When developing a whole-of-government strategic policy framework, it is good practice to prepare an implementation plan prior to commencement to set the direction for implementation processes.³⁷

4.18 For the Closing the Gap framework, as discussed in paragraph 2.5, it took nearly four years from the framework’s commencement in 2008 for overarching bilateral Indigenous plans to be negotiated with all states and territories. While six of the eight Indigenous-specific national partnership agreements that underpinned the framework had implementation plans, no provision was made for an overarching Australian Government Closing the Gap implementation plan.

4.19 The Council of Australian Government’s (COAG’s) December 2018 statement about the implementation of the Closing the Gap Refresh committed to developing Australian, state and territory government action plans. It also included draft Closing the Gap targets, with lead responsibility attributed to different levels of government. As PM&C is the lead entity for Aboriginal and Torres Strait Islander Affairs, it will need to work with other Australian Government entities to implement COAG’s commitment.³⁸ Drawing on better practice performance measurement principles, the Australian Government action plan should identify the links between program inputs, outputs, intermediate outcomes and higher-level outcomes and targets.³⁹

Has the Australian Government’s contribution to Closing the Gap been effectively monitored?

4.25 In 2008, under the NIRA accountability principle, COAG agreed that programs and services should have regular and transparent performance monitoring, review and evaluation. In addition, all jurisdictions committed to an annual review of progress, which would include an assessment of the impact of initiatives being undertaken through COAG and other mechanisms on the targets. These annual reviews were intended to be a mechanism for ongoing accountability and for adapting and adjusting programs to ensure they were on track to meet the targets.

4.26 As discussed in paragraph 2.6, intergovernmental oversight bodies to facilitate these annual reviews of progress were not maintained. Nevertheless, in accordance with the accountability principle, the Australian Government, through relevant departments, should have established monitoring arrangements to measure their contribution to the Closing the Gap framework. In line with this, the ANAO examined the extent to which the Australian Government has maintained such arrangements over the duration of the framework.

Australian Government monitoring arrangements 2008–2013

4.27 Over the first five years of the Closing the Gap framework, when FaHCSIA was the lead agency for Aboriginal and Torres Strait Islander Affairs, the primary whole-of-government mechanisms for monitoring the Australian Government’s contribution towards Closing the Gap were:

• the Indigenous Budget Statement — an annual public Ministerial statement released on Budget night outlining Indigenous‐specific and relevant mainstream programs;
• Australian Government Indigenous expenditure (AGIE) — reported annually by all entities through Portfolio Budget Statements; and
• regular reports provided to government between 2010 and 2012 on the implementation of Australian Government Aboriginal and Torres Strait Islander programs.

4.28 The Auditor-General’s 2012 performance audit of Australian Government Coordination Arrangements for Indigenous Programs (Auditor-General Report No.8 2012–13) was critical of the effectiveness of these performance monitoring arrangements. Key findings from the report are outlined in Box 1. The audit report included a recommendation, which was agreed by FaHCSIA, to ‘include a greater focus on outcomes in its overall reporting and enhance its financial oversight of mainstream and Indigenous-specific Australian Government Indigenous expenditure’.⁴⁰

Source: Auditor-General Report No.8 2012–13 Australian Government Coordination Arrangements for Indigenous Programs, pp. 25-26.

Australian Government monitoring arrangements since 2014

4.29 After responsibility for Aboriginal and Torres Strait Islander Affairs was transferred from FaHCSIA to PM&C in late 2013, all of the whole-of-government monitoring mechanisms discussed in paragraph 4.27 ceased with the exception of the AGIE, which was last reported in the 2015–16 Portfolio Budget Statements. No additional mechanisms have been developed since that time. Consequently, other than reporting on program activity through the Closing the Gap Prime Minister’s Report (discussed below), there has not been any consolidated monitoring of the Australian Government’s contribution towards the Closing the Gap framework since 2015.

4.30 Over the duration of the Closing the Gap framework, individual entities (including PM&C, Department of Health and Department of Education and Training in recent years) have adopted specific Closing the Gap targets as their performance criteria in Portfolio Budget Statements and Corporate Plans. The Department of Finance’s guidance for developing performance criteria states that criteria should be focused on the impact of programs and the contribution they make to overall outcomes. In all cases, agencies had adopted the national Closing the Gap targets (which represent outcomes that many agencies across all levels of government are responsible for) as their performance criteria, rather than seeking to measure the specific impacts or contributions of their programs on these national targets.

Performance reporting in the Closing the Gap Prime Minister’s Report

4.31 In addition to its analysis of progress towards the Closing the Gap targets, discussed in Chapter 3, the Closing the Gap Prime Minister’s Report has also included information about Australian, state and territory government programs and services for Aboriginal and Torres Strait Islander peoples and other initiatives deemed relevant to the Closing the Gap framework. Over the last ten years, the number of initiatives and case studies highlighted in the reports has ranged from 212 in 2013 to 10 in 2014 (see Figure 4.2).

Figure 4.2: Performance information in Closing the Gap reports, 2009–2018

Source: ANAO analysis

4.32 In 2012, Auditor-General Report No.8 2012–13 noted that the Closing the Gap Prime Minister’s Report was primarily focussed on inputs and outputs and did not generally include analysis of the outcomes of initiatives or their contribution to meeting the Closing the Gap targets. The ANAO’s analysis of performance reporting over the period 2009 to 2018 found 12.3 per cent of program descriptions or case studies referenced measured outcomes, and five per cent made an explicit link to the Closing the Gap targets or indicators. Information presented in the reports has also been overwhelmingly positive or neutral, with only one description out of 975 referencing delivery challenges or setbacks.

4.33 The original intention of the Closing the Gap Prime Minister’s Report was to provide a high-level summary of national progress towards the Closing the Gap targets and achievements since the last report. The report was not intended, and has not functioned, as an accountability document for the Australian Government.

Have evaluation frameworks been established to measure the impact of programs on Closing the Gap targets?

4.42 The NIRA accountability principle notes the importance of ‘evaluating programs and services from multiple perspectives including from the client, Indigenous communities and government perspectives and incorporating lessons into future program and services design’.⁴¹

4.43 Over the period the NIRA has been in place, an overarching evaluation strategy or framework to measure the impact of programs on Closing the Gap targets has not been developed. Stakeholders consulted by the ANAO over the course of the audit commented on the lack of evaluation linked to the Closing the Gap framework, noting a need for more and better evaluation to inform policy and program development.

Strategic Review of Indigenous Expenditure

4.44 In 2009 the Strategic Review of Indigenous Expenditure, conducted by Department of Finance to assess how well the Government’s Aboriginal and Torres Strait Islander policy co-ordination arrangements and programs were placed to achieve the Closing the Gap targets, noted:

The strong commitments to evidence-based policy made by the Prime Minister and other Heads of Government are not matched by the quality of the evidence currently available in the Commonwealth’s own domain. Even basic information on the number of distinct Indigenous-specific programs and their key characteristics was a challenge to discover and assemble… Moreover, a major constraint on the work of the Review has been the lack of robust evidence on the performance and effectiveness of many Indigenous programs.⁴²

4.45 The report recommended that the Productivity Commission be requested to conduct a review of the adequacy of arrangements for evaluating Aboriginal and Torres Strait Islander programs, at both an Australian Government and state and territory level, but this recommendation was not taken up by the Government at that time.

Closing the Gap Clearinghouse

4.46 In 2009, following a tender process, the Closing the Gap Clearinghouse was initiated with funding of $5.2 million over five years from the Australian, state and territory governments. Jointly delivered by the Australian Institute of Health and Welfare and Australian Institute of Family Studies, its key objective was to improve access to evidence by analysing research and evaluation findings to identify what works to address Indigenous disadvantage. While the Clearinghouse maintained a register of government research and evaluations (which contained 1,249 items as at March 2014), it was not tasked with establishing an overarching whole-of-government evaluation strategy or framework.

4.47 Through analysis of key research and evaluation reports from Australia and overseas, the Clearinghouse published 50 evidence synthesis reports (37 resource sheets and 13 issues papers) and three ‘what works’ summary reports. A common issue identified in the ‘what works’ reports was a lack of robust evaluations to inform program development. In particular, the reports noted there was a lack of high-quality quantitative social policy research in the Australian and Aboriginal and Torres Strait Islander contexts.

Closure of the Clearinghouse

4.48 A review of the Clearinghouse in 2013 found stakeholders were ‘overwhelmingly positive about [its] existence’ and there was strong support for continuing its research synthesis function. However, it also found there was scope for the Clearinghouse to take a more active role in providing a quality assurance function for evaluation services and commissioning research to fill evidence gaps.

4.49 In briefing the Minister for Indigenous Affairs on the review, PM&C noted there was insufficient new research and evaluation evidence being produced in Australia for the Clearinghouse to continue with its previous rate of publication. In 2014 the minister gave in principle agreement to fund a smaller-scale Clearinghouse for three years from 2014–15, pending states and territories matching the Australian Government funding commitment. In late 2014, after PM&C was unable to negotiate matching commitments from all jurisdictions, the Minister decided to discontinue the initiative. Funding for the Clearinghouse ceased in 2014, but until 2016 it continued to publish synthesis reports that had previously been commissioned.

Productivity Commission’s 2015 NIRA performance assessment

4.50 In November 2015 the Productivity Commission’s NIRA Performance Assessment (covering the 2013–14 period) stated that there was extensive focus at a whole-of-government level on monitoring broad outcome targets relating to Indigenous disadvantage, but little evidence of what works to bridge outcome gaps. The report outlined three options for COAG to reinvigorate evaluation by:

• conducting an overarching review of Indigenous evaluation (as had been recommended in the 2009 Strategic Review of Indigenous Expenditure);
• committing to evaluating policy settings in a target area (such as education); and/or
• adding a procedural, evaluation-focused Closing the Gap target.⁴³

4.51 The ANAO examined COAG meeting papers from 2015 to 2018 and found no evidence that it considered these options. As noted in paragraph 2.18, this was the last independent assessment of progress towards Closing the Gap.

2017 Aboriginal and Torres Strait Islander research and evaluation package

4.52 In late 2016, in the context of considering the forward agenda for Aboriginal and Torres Strait Islander reform, the Government agreed to a greater focus on evidence, measuring impact and external oversight in Aboriginal and Torres Strait Islander policy. Noting the inconsistent quality of Aboriginal and Torres Strait Islander program evaluations and the lack of a structured requirement to undertake evaluations, in early 2017 the Government agreed to a package of three measures to address these gaps:

1. $2.9 million over four years for an expanded role for the Productivity Commission in Aboriginal and Torres Strait Islander evaluation;
2. $40 million over four years to strengthen evaluation of the IAS; and
3. $10 million over three years for an Aboriginal and Torres Strait Islander research fund that would be co-designed with experts in research, data and analysis.

Expanded role for Productivity Commission

4.53 The Productivity Commission will develop a whole-of-government evaluation strategy for policies and programs that affect Aboriginal and Torres Strait Islander peoples, to be reported against by all Australian Government agencies. This work will be overseen by a new commissioner with experience in this area.

4.54 Funding of $2.9 million over four years was reprioritised from the IAS in the 2017–18 Budget to support the Productivity Commission’s enhanced role in evaluation, including the new commissioner. The measure required an amendment to the Productivity Commission Act 1998 to allow for the appointment of an additional commissioner and to specify additional skill requirements, which was enacted in April 2018. On 11 December 2018 the appointment of the Indigenous evaluation commissioner was announced, with a commencement date in April 2019. The Productivity Commission advised the ANAO that work on developing a whole-of-government Aboriginal and Torres Strait Islander evaluation strategy had not formally begun pending the commencement of the commissioner.

Strengthening evaluation of the IAS

4.55 In early 2014 PM&C had committed to developing an Evaluation and Performance Improvement Strategy for the IAS by June 2014. The strategy was not formalised. In February 2017 the Minister for Indigenous Affairs announced funding of $40 million over four years from 2017–18 to strengthen evaluation of the IAS. The following year, in February 2018, PM&C released the Indigenous Advancement Strategy Evaluation Framework and 2017–18 Annual Evaluation Work Plan. The Auditor-General is conducting a separate performance audit, due to table during 2019, of the effectiveness of PM&C’s design and implementation of this framework.

4.56 The IAS evaluation framework notes that it is intended to align with the Productivity Commission’s future whole-of-government strategy, but it does not include any references to the Closing the Gap framework.

4.57 The ANAO also examined the 32 evaluations completed by PM&C from 2015 to 2018 and found that 31 contained between zero and five substantive references⁴⁴ to the Closing the Gap framework. The exception was the Closing the Gap—Retrospective Review, finalised in February 2018 but not published as at December 2018, which concluded that:

Overall we would have to conclude that the framework was not well implemented, and so there has been a great deal of lost opportunity… but there is plenty of potential to shift the way government and Indigenous Australian communities work together to Close the Gap.⁴⁵

Indigenous Research Exchange

4.58 In September 2017 PM&C briefed the Minister for Indigenous Affairs on potential models for establishing an Aboriginal and Torres Strait Islander research fund, which had been announced in the 2017–18 Budget, and indicated its preferred model was ‘an Indigenous Research Hub that would improve on the previous Closing the Gap Clearinghouse approach by emphasising Indigenous involvement’. In June 2018 the Australian Institute of Aboriginal and Torres Strait Islander Studies was contracted to establish and manage an Indigenous Research Exchange. On 3 December 2018 the Institute announced the establishment of the Exchange on its website, with an invitation for potential partners and collaborators to participate in the Exchange.

Appendix 1 Entity responses

Formal responses received by ANAO following circulation of the draft report are reproduced in Appendix 1.

Responses were received from:

• Department of the Prime Minister and Cabinet;
• Productivity Commission;
• Australian Bureau of Statistics;
• Australian Curriculum, Assessment and Reporting Authority;
• Australian Institute of Health and Welfare; and
• Department of Health.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix 2 Closing the Gap target trajectories and results

Figure A2.1: Close the gap in mortality rates by 2031

 

 

Figure A2.2: Halve the gap in mortality rates for Indigenous children under five by 2018

 

 

Figure A2.3: Halve the gap for Indigenous students in reading and numeracy by 2018 (students meeting national minimum standards, %)

 

 

Figure A2.4: Halve the gap in Indigenous employment outcomes by 2018

 

 

Figure A2.5: Halve the gap for Indigenous people aged 20-24 in Year 12 or equivalent attainment rates by 2020

 

 

Figure A2.6: Close the gap between Indigenous and non-Indigenous school attendance by 2018

 

 

Figure A2.7: 95 per cent of all Indigenous four-year-olds enrolled in early childhood education by 2025

 

 

Source: PM&C internal documentation, based on data available at the time of preparation.

Appendix 3 Technical detail on analysis of child mortality target

Variability bands are calculated for mortality rates using the same formula used to calculate a 95 per cent confidence interval for a sample proportion. For child mortality, this effectively treats each cohort of 0 to 4 year-olds as a sample drawn from the population of potential cohorts. The formula used is:

 

 

The methodology used to assess whether the childhood mortality indicator is on track is effectively a one-tail hypothesis test, with ‘on-track’ the null hypothesis and using a 95 per cent confidence interval. This means the likelihood of not being on track needs to be lower than 2.5 per cent to assess progress as ‘not on track’.

Another way to approach the progress assessment would be to calculate the probability (p-value) that the ‘true rate’ for a year, given the ‘observed rate’ for that year, is equal to or lower than the target trajectory. The formula below can be used to calculate the Z-score and determine the probability from a normal probability distribution table.

 

 

Using this formula for the 2016 rate yields a Z-score of -1.45, which indicates the probability that the 2016 rate is equal to or lower than the target trajectory is 7.3 per cent. Figure A3.2 provides a graphical illustration of this one-tail hypothesis test.

Figure A3.2: One-tail hypothesis test

 

 

Source: ANAO.

The ANAO calculated the probability that the Indigenous child mortality rate was equal to or lower than the target rate (or ‘on track’), based on annual results published from 2012 to 2018. As shown in the table below, while the probability of being on track was relatively high between 2012 and 2014, since 2015 it has fluctuated between 0.7 per cent and 8.5 per cent. Based on these results, it is unlikely that the child mortality target will be achieved.

Table A3.1: Child Mortality target - Probability of on-track/off-track determinations, 2012-2018

Source: ANAO analysis.