Introduction

1. The purpose of the Australian Government’s overseas aid program is to promote Australia’s national interests through contributing to economic growth and poverty reduction.¹ Australia’s aid program focuses on the Indian Ocean Asia Pacific region, while also providing assistance to countries in Africa, the Middle East, Latin America and the Caribbean. The Department of Foreign Affairs and Trade (DFAT) is responsible for administering the Australian aid program. Australia’s total Official Development Assistance (ODA) is budgeted at $5.32 billion in 2014–15.

2. The aid program is delivered within the context of a new aid framework, released in June 2014. Australian aid: promoting prosperity, reducing poverty, enhancing stability outlines Australia’s aid policy, describing the purpose of the aid program and Australia’s aid investment priorities. Making Performance Count: enhancing the accountability and effectiveness of Australian Aid provides the framework against which the performance of the aid program is assessed.

Context and development challenges

3. Delivering aid programs in any developing country is challenging, with inherent risks. The development environment exists within the context of often competing political and implementation objectives. It may also take considerable time for the benefits from aid programs to be realised and the number of variables affecting the programs and the environment within which they are delivered may make the assessment of impact difficult. Often Australia’s role is limited to influencing and encouraging change rather than direct program management.

4. While Vanuatu is viewed by many Australians as a holiday destination, it is a developing country and implementing successful development initiatives can be challenging. Vanuatu was ranked 131 (of 187 countries; with a value of 0.616) on the Human Development Index in 2013², its progress towards the Millennium Development Goals³ is mixed and it is unlikely to meet most of the targets by 2015. One of the country’s challenges is maintaining political stability, and one third of its population lack access to multiple basic services, such as education, health services and safe water.

5. Vanuatu is a culturally diverse country. The geography of Vanuatu means that the small population of approximately 250 000 ni-Vanuatu (people from Vanuatu) is dispersed over 65 of the country’s 83 islands and speak 113 distinct languages and numerous dialects. Vanuatu lacks natural resources that could contribute to revenue generation, except for tourism, and the formal economy is small. Infrastructure on many of the islands is very limited, resulting in difficulties in travel, communication and disbursement of funds. Political instability and the operational and management capacity of the Government of the Republic of Vanuatu (GoV) contribute to a difficult delivery environment. In addition, the state has limited reach outside of the capital, Port Vila. Vanuatu has a strong kastom system⁴ and chiefs, as well as island and family allegiances, play an important role in all aspects of society.

Tropical Cyclone Pam

6. On 13 and 14 March 2015, Tropical Cyclone Pam passed over Vanuatu. The category five cyclone is the most powerful on record to have impacted Vanuatu and is considered to be the worst natural disaster in the country’s history. It claimed lives and caused extensive damage across the country. In response, Australia provided a package of assistance that included: funding to Australian non-government organisations, the Australian Red Cross and United Nations organisations; deployment of humanitarian supplies and specialist equipment; and deployment of specialist personnel. The cyclone will result in changes to the development environment and, consequently, will impact on Australia’s investment in aid programs in Vanuatu.

Australian aid investment in Vanuatu

7. Australia is the largest donor of aid to Vanuatu, providing around half of total aid. In 2014–15, the budget for Australian ODA to Vanuatu is $60.4 million, the majority of which ($41.9 million) is bilateral aid managed by DFAT.⁵ The remaining ODA to Vanuatu is managed by other government departments or is delivered through DFAT’s regional or global programs. In 2014–15 bilateral aid is allocated across the program’s strategic objectives as follows:

• education—$11.2 million (26.8 per cent of bilateral ODA);
• health—$4.1 million (9.8 per cent);
• economic governance—$4.7 million (11.3 per cent);
• infrastructure—$13.0 million (31.1 per cent);
• law and justice—$5.6 million (13.3 per cent); and
• disaster response—$10 000 (0.0 per cent).⁶

8. In 2013–14, Australia had 15 aid investments operating in these six areas. The four largest investments are: Education Support Program ($39.3 million over five years to June 2017), Transport Sector Support Program ($27.0 million over four years to June 2016), Port Vila Urban Development Project ($26.5 million over five years to June 2017) and Health Sector Support Program ($26.0 million over six years to March 2016).

Audit objective and criteria

9. The objective of the audit was to assess the effectiveness of the Department of Foreign Affairs and Trade’s management of Australian aid to Vanuatu.

10. To form a conclusion against the audit objective, the ANAO adopted the following high-level criteria:

• a sound strategic approach to planning Australia’s aid investments in Vanuatu has been developed and effectively implemented;
• Australia’s aid investments in Vanuatu are effectively managed, monitored and evaluated; and
• administrative arrangements facilitate the cohesive delivery of Australia’s aid investments in Vanuatu.

11. When undertaking this audit, the ANAO carried out fieldwork in Canberra and three islands in Vanuatu (Efate, Malekula and Tanna). The audit focused on Australia’s bilateral program in Vanuatu. The audit team examined the following five initiatives in detail:

• Vanuatu Education Support Program (VESP);
• Vanuatu Transport Sector Support Program (VTSSP) Phase 2;
• Port Vila Urban Development Project (PVUDP);
• Vanuatu Health Sector Program (Health); and
• Vanuatu Women’s Centre (VWC) Phase 6.

12. The audit examined Australia’s investment in aid programs in Vanuatu before Tropical Cyclone Pam. The audit did not assess Australia’s aid program in Vanuatu following the cyclone or Australia’s response to the cyclone.

Overall conclusion

13. The Republic of Vanuatu, a tropical holiday destination for many Australians, is a developing country struggling with poverty and a lack of access to many basic services, including education and health services. Australia has been providing development assistance to Vanuatu for 40 years, including $60.4 million in 2013–14.⁷ Managing development programs is challenging, with inherent risks. Aid delivery may be impacted by competing political and implementation objectives, it may take a long time for benefits to be realised, and those benefits may be difficult to quantify. In 2013–14, DFAT assessed the performance of the Vanuatu aid program against the five strategic objectives that were current at the time⁸, and reported that progress was as expected in three areas but was somewhat less than expected in two areas. The strategic objectives of the program are presently being reconsidered. Against this background, DFAT’s management of the Australian aid program in Vanuatu has been generally effective.

14. DFAT’s new Aid Investment Plan for Vanuatu, currently in draft, is consistent with Australia’s aid policy and the priorities of the Government of Vanuatu as well as being sufficiently detailed to provide an understanding of the key elements of the program. DFAT has a well-defined process for designing individual aid initiatives, and investment designs are generally comprehensive. Also, DFAT has developed and maintained good relationships with key stakeholders, including representatives of the Vanuatu Government but acknowledges that mapping the large number of stakeholders and development programs being undertaken in Vanuatu would assist it to identify program gaps and overlaps, and guide the development and maintenance of key stakeholder relationships. There is, however, scope to strengthen DFAT’s approach to planning and managing the program in some areas, principally its management of risk, and the monitoring and oversight of individual initiatives.

15. The management of risks would be improved if the assessment of significant risks, as well as risk treatments, responsibilities and timeframes, were appropriately documented. In relation to the oversight of individual initiatives, DFAT currently adopts a variety of monitoring and evaluating approaches, including visits, progress reporting and mid-term reviews. Documenting its approach in a monitoring and evaluation plan specific to each aid initiative would assist DFAT to improve its monitoring of the performance of delivery partners, as well as the progress of each initiative. There would also be benefit in DFAT developing, as part of the Aid Investment Plan process, performance indicators that allow it to measure and publicly report on the extent to which its objectives are being achieved.

16. To assist DFAT to improve its management of the Australian aid program in Vanuatu, the ANAO has made two recommendations relating to DFAT’s strategic approach to aid investment planning and management, and its monitoring and oversight of individual initiatives.

17. The ANAO recognises that, following Tropical Cyclone Pam, Australia’s aid program in Vanuatu will change in response to the needs of the Government of Vanuatu and the population as they rebuild after the cyclone. The findings and recommendations from this audit remain relevant for DFAT’s ongoing management of Australia’s aid investments and, specifically, to the Vanuatu program as it develops and changes in the post-cyclone environment.

Key findings by chapter

Planning the Vanuatu Aid Program (Chapter 2)

18. DFAT is currently drafting an Aid Investment Plan (AIP) for Vanuatu that covers the period 2014–15 to 2017–18 and describes the strategic priorities and rationale for aid investment in Vanuatu, and the mutual obligations of the Australian and Vanuatu Governments. The strategic direction embodied in the draft AIP for the Vanuatu aid program is consistent with the Australian Government’s new aid policy. Australia’s development priorities in Vanuatu are also generally consistent with Vanuatu’s policy agenda. At the level of individual aid initiatives, the objectives and priorities of the five initiatives examined, while more narrowly focused, are in line with the broader priorities of Australia’s aid program and the Government of Vanuatu (GoV).

19. The draft AIP meets the requirements of DFAT’s Aid Programming Guide, and is sufficiently detailed to provide an understanding of the key elements of Australia’s aid program in Vanuatu. However, the program’s performance indicators, described as performance benchmarks in the Aid Program Performance Report, measure only a small portion of the Vanuatu aid program for a period of one year. DFAT has an opportunity to create a broader framework for measuring the success of the Vanuatu aid program by developing performance indicators and targets, to be included in the AIP, that better reflect the Vanuatu aid program’s objectives and provide a firm basis for assessing progress against those objectives.

20. DFAT staff have developed good individual relationships with external stakeholders, including within the GoV, and actively participate at key meetings, such as program steering committees. Stakeholders advised the ANAO that DFAT staff were open, accessible and responsive. However, at present, stakeholder management is largely the responsibility of the DFAT officer managing an initiative. As such, there is a risk that relationships with stakeholders will be adversely impacted by changes in staff within DFAT, the GoV or other stakeholders. Developing a more structured approach to stakeholder management, in the context of finalising the AIP, including mapping the large number of stakeholders and development programs being undertaken in Vanuatu, would help to mitigate this risk and guide the development and maintenance of key stakeholder relationships. It would also allow DFAT to identify program gaps and overlaps.

21. The Vanuatu aid program risk register, including the individual risk registers relevant to the five initiatives reviewed, was updated in April 2015. Despite a requirement to update risk registers bi-annually, prior to the recent update the register not been reviewed since November 2013. The latest iteration of the register includes 74 risks. However, the majority of implementation dates for risk treatments are not specific. Of the 30 risks with risk treatment implementation dates, 20 are described as ‘ASAP’ or ‘ongoing’. Moreover, the risk treatment implementation dates for eight risks have passed and the treatments for two of the risks do not have implementation dates. In addition, 10 of the 14 whole-of-program risks do not include an assessment of the acceptability of the current risk rating, despite a risk rating of high or very high for seven of the 10 risks, or identified possible risk treatments for these risks. The registers do not include risks to aid management, such as the impact of changes in aid policy, budget or resourcing. In some cases responsibility for implementing risk mitigation strategies is not specific and there is also inconsistency across the registers with respect to assigning responsibilities.

22. The post and sector/investment risk registers include fraud as a key risk. In terms of potential financial loss, DFAT reports that external fraud in the Vanuatu aid program is low, but the number of new cases reported in Vanuatu regularly places the program in the top six country programs with the highest incidence of reported fraud. Of the total 1280 fraud cases recorded by DFAT since 2009, 48 (3.75 per cent) were in Vanuatu. DFAT estimates the value (actual or potential financial loss) of the 48 cases is $102 831.

Designing Aid Initiatives (Chapter 3)

23. DFAT has a well-defined process for designing aid initiatives that includes concept notes, design documents, peer reviews and consultation with a range of stakeholders. The design and strategy documents resulting from the design process are comprehensive, include a rationale for the project and a consideration of value-for-money, and comply with the intention of DFAT’s guidance and Investment Design Quality Criteria. However, the design of the Port Vila Urban Development Project (PVUDP) did not follow the standard process. A concept document was not developed, and the technical design is ongoing, three years after commencing the project.

24. More than 50 per cent of bilateral aid to Vanuatu was channelled through GoV systems, including the majority of aid funding in the education and health sectors, by the end of 2012. In 2013, DFAT undertook assessments of public financial management systems at the national and sector levels in Vanuatu to inform the extent to which aid might be provided through partner government systems in the future.⁹

25. In response to the assessments in the education and health sectors, which resulted in similar findings, different aid management models have been implemented in each of the sectors. A managing contractor has been engaged to manage the majority of Australian aid funds in the education sector, while, in the health sector, DFAT continues to deliver a significant portion of the program through government systems. A mixed approach has been adopted for the Vanuatu Transport Sector Support Program (VTSSP) Phase 2, with 58 per cent of Australian funding provided through government systems and 42 per cent paid to the contractor. Nevertheless, the Vanuatu Education Support Program (VESP) and VTSSP Phase 2 agreement and contracts, which were signed after the public financial management assessments were undertaken, do incorporate increased fiduciary controls, including arrangements to oversight the management of donor funds.

26. While the design process is reasonably comprehensive and DFAT adhered to the required process for developing designs, there are limitations to the process as it relates to monitoring and evaluation frameworks and risk management plans. For example, frameworks against which investments will be monitored and evaluated are developed after an investment has been approved, are usually a contract deliverable for the delivery partners, and are not always timely. The monitoring and evaluation plan for VESP, for instance, was due in February 2014, but was not completed until December 2014.

27. In addition, poor design can impact on the successful implementation of a project. The design of the Vanuatu Education Road Map (VERM) program was overly ambitious and placed too much reliance on the GoV to effectively implement the program. Consequently, the results of VERM were not commensurate with expectations and the level of investment, that is, the program was not considered by DFAT to be value-for-money. With respect to VTSSP, the Phase 2 design was predicated on assumptions and baseline information concerning road quality and lengths achieved in Phase 1 that were subsequently found to be inaccurate. The VTSSP Phase 2 tender documents and contract were based on the flawed assumptions and data inaccuracies contained in the design document. As a result, after one and a half phases of the project, there is ongoing discussion between the stakeholders, including the GoV, DFAT and the contractor, about the required quality of roads to be rehabilitated under the program.

28. In 2014, DFAT streamlined the design process to some extent. There are opportunities, however, to further improve the process, particularly for established, mature or straightforward initiatives. The Vanuatu Women’s Centre, for example, is a successful program that Australia has been supporting since 1994 and is now in the sixth phase of funding. Future designs for such a mature program could be simplified, reducing the time and resources dedicated to the process.

Monitoring Progress and Evaluating Performance (Chapter 4)

29. DFAT adopts a variety of methods to monitor the performance of delivery partners, including: conducting monitoring visits; requiring reports on performance; undertaking annual Aid Quality Checks; and conducting periodic evaluations. Some of these approaches are incorporated into agreements with delivery partners, and include requirements for accountability and reporting. However, DFAT has not documented its approach to monitoring and evaluation in a plan (for each initiative) as part of its ongoing management of the initiatives. Developing initiative plans that focus on DFAT’s management and monitoring of the performance of delivery partners, as well as the progress of the initiative, would assist DFAT to take a more risk-based approach to monitoring.

30. While DFAT staff in Vanuatu are now encouraged to undertake monitoring visits, and have been provided with training and tools to assist with these visits, only a limited number of visits have been undertaken to date. Progress reporting is a feature of four of the five initiatives. Generally, the progress reports were of a satisfactory quality, provided an appropriate level of detail about progress and issues, and met DFAT’s monitoring and evaluation requirements. Each of the initiatives is also subject to an annual financial audit. Audits have been completed in each of the five programs, but have not always been completed in a timely manner.

31. DFAT assessed annually the quality of the implementation of each investment (now referred to as Aid Quality Checks) as required. The resulting reports provided a sound overview of the status of each initiative and included actions that DFAT intended to take to address issues. The actions covered a continuum from high to low priority and from simple to complex requirements. However, DFAT has not addressed or completed many of the actions identified.¹⁰ DFAT also requires the performance of advisors and contractors to be assessed annually. In 2013–14, only one of the five required contractor performance assessments for the Vanuatu bilateral program was completed.

32. DFAT has responded to issues identified as a result of its monitoring of aid investments. For example, DFAT and its donor partners were sufficiently informed to be concerned about the implementation and progress of VERM, implementing actions to minimise the risk to donor funds while redesigning the program. With respect to PVUDP, by January 2015 the initiative was reported to be 26 months behind schedule. DFAT raised its concerns about the progress of PVUDP with its delivery partner, the Asian Development Bank, on several occasions, encouraging the Bank to increase its presence in Vanuatu in order to more effectively monitor progress. However, issues have not always been highlighted through monitoring processes. Weaknesses in DFAT’s monitoring of VTSSP Phase 1 were not known until Phase 2, and DFAT was not aware of these discrepancies until notified by the contractor.

Administrative Arrangements (Chapter 5)

33. Responsibility for the Vanuatu bilateral aid program is shared between DFAT staff in Canberra, Australia, and at the Australian High Commission in Port Vila, Vanuatu. The roles of the Canberra and Vanuatu teams are well understood by all parties and there is frequent communication between the two teams about upcoming activities and the status of current projects.

34. The majority of Australian-based aid officers located in Vanuatu did not feel that they were adequately prepared for their posting to Vanuatu, particularly with respect to establishing, negotiating and managing large contracts. While one branch within DFAT’s head office is responsible for coordinating and managing learning and development, delivery of the program is decentralised and multiple areas within DFAT provide their own training courses. However, DFAT has not maintained accurate and complete records, for its Australian-based or locally engaged staff, of all training provided by each area within DFAT. Without reliable training records, it is difficult for DFAT to assess the adequacy, currency or completeness of the training undertaken by each officer.

35. DFAT’s internal reporting for management purposes is not specific to the Vanuatu bilateral aid program. Vanuatu is generally referred to on an exception basis. Nevertheless, internal reporting is sufficient to provide DFAT executive with reasonable oversight of the program. Externally, DFAT reports on the progress and performance of the Australian aid program via several channels. Aside from a section on DFAT’s website and occasional references in the Annual Report and Performance of Australian Aid, external reporting about the Vanuatu program centres on the annual Vanuatu Aid Program Performance Report. The 2013–14 report rates the Vanuatu program’s performance against its five strategic objectives as ‘amber’ for two objectives and ‘green’ for three objectives.¹¹ The details provided in the report were soundly based, but provided without context. Additionally, the report does not provide a consolidated view of the effectiveness of the program overall. There would be benefit in DFAT developing, as part of the AIP process, performance indicators and targets that allow it to measure and publicly report on the extent to which its strategic objectives and the objective for the Vanuatu bilateral program as a whole are being achieved.

Summary of entity response

36. DFAT’s summary response to the proposed report is provided below, with the full response at Appendix 1.

DFAT welcomes the ANAO’s findings that the Vanuatu aid program is generally effective and is consistent with Australia’s aid policy and the Vanuatu Government’s priorities. The report’s recognition of the complex and challenging operating environment in which Australia delivers aid in Vanuatu is also welcome. We are pleased that the ANAO has recognised the program’s relationship management strengths, its awareness of and compliance with DFAT’s fraud policy, the well-defined processes for designing aid initiatives and the range of methods adopted to monitor and manage performance. The ANAO has highlighted some areas requiring improvement, particularly in documenting risk and planning of monitoring and evaluation. DFAT welcomes the ANAO’s recommendations in support of these improvements and has commenced addressing the issues raised.

Recommendations

Australia’s overseas aid program

1.1 The purpose of the Australian Government’s overseas aid program is to promote Australia’s national interests through contributing to economic growth and poverty reduction.¹² Australia’s aid program focuses on the Indian Ocean Asia Pacific region, while also providing assistance to countries in Africa, the Middle East, Latin America and the Caribbean. The Department of Foreign Affairs and Trade (DFAT) has been responsible for administering the aid program since the Australian Agency for International Development (AusAID) was abolished on 1 November 2013 and its functions integrated with DFAT.¹³ DFAT administers the program under Outcome 1:

The advancement of Australia’s international strategic, security and economic interests including through bilateral, regional and multilateral engagement on Australian Government foreign, trade and international development policy priorities.¹⁴

1.2 In 2014–15, the total budget for Outcome 1 is $5.57 billion—$4.66 billion in administered expenses and $0.91 billion in departmental expenses.¹⁵ Australia’s total Official Development Assistance (ODA)¹⁶ is budgeted at $5.32 billion in 2014–15.

1.3 The aid program is delivered within the context of a new aid framework, released in June 2014. Australian aid: promoting prosperity, reducing poverty, enhancing stability outlines Australia’s new aid policy. The policy describes the purpose of the aid program and Australia’s aid investment priorities, which are:

• infrastructure, trade facilitation and international competitiveness;
• agriculture, fisheries and water;
• effective governance: policies, institutions and functioning economies;
• education and health;
• building resilience: humanitarian assistance, disaster risk reduction and social protection; and
• gender equality and empowering women and girls.

1.4 Making Performance Count: enhancing the accountability and effectiveness of Australian Aid provides the framework against which the performance of the aid program is assessed. The framework includes 10 strategic targets, as well as requiring performance benchmarks at the country, regional and partner program level and quality systems at the project level. The 10 strategic targets are listed in Figure 1.1.

Figure 1.1: Ten strategic targets for the aid program

Source: ANAO representation of Commonwealth of Australia, Making Performance Count: enhancing the accountability and effectiveness of Australian, DFAT, June 2014.

1.5 More broadly, Australia’s aid program is guided by the Millennium Development Goals. The goals are agreed targets set by the signatories to the Millennium Declaration 2000, including Australia, to reduce extreme poverty and disadvantage by 2015 (see Figure 1.2). The United Nations is currently facilitating the post-2015 development agenda, including the development of sustainable development goals for the future.

Figure 1.2: Millennium Development Goals

Source: <http://www.un.org/millenniumgoals/> [accessed 16 June 2014].

1.6 The aid program is also informed by the Government’s commitment to the:

• 2005 Paris Declaration on Aid Effectiveness—commits donor governments and developing countries to work together to improve aid effectiveness and outlines five core principles for making aid more effective¹⁷;
• 2008 Accra Agenda for Action—seeks to strengthen and deepen implementation of the Paris Declaration; and
• 2011 Busan Partnership for Effective Development Cooperation—building on the 2005 and 2008 documents, the Busan Partnership sets out principles, commitments and actions that offer a foundation for effective co-operation in support of international development.

Australian aid to Vanuatu

Context and development challenges

1.7 The Republic of Vanuatu is an independent archipelagic nation covering over 12 000 square kilometres in the Pacific Ocean.¹⁸ The three official languages are English, French and Bislama. The population of approximately 250 000 live across 65 of the country’s 83 islands. Around 70 per cent of Ni-Vanuatu (people from Vanuatu) live in rural areas, where subsistence farming and fishing are the main sources of livelihood. However, the country is rapidly urbanising, with larger cities such as the capital Port Vila, on the island of Efate, and Luganville, on the island of Espiritu Santo, attracting people seeking education and employment opportunities.

1.8 Vanuatu has a 52-member Parliament elected for a four-year term. The head of the Government is the Prime Minister and the Constitutional Head of State is the President of the Republic. Vanuatu has a history of frequent changes of government, often as the result of no-confidence motions. Since the last national elections on 30 October 2012, the country has had three Prime Ministers. The state, however, has limited reach outside Port Vila; at the provincial and rural level customary and informal institutions exercise a greater level of influence.

1.9 The country has a stable economy, however Tropical Cyclone Pam has caused significant disruption to economic activity. In April 2015, the International Monetary Fund projected that real GDP would decline by two per cent in 2015, in contrast to the pre-cyclone forecast of about 3.5 per cent grown, and then increase to around 5 per cent in 2016.¹⁹ Vanuatu’s economic growth is driven largely by tourism and construction. Tourism and tourism-related services account for approximately 40 per cent of gross domestic product and one third of people in formal employment. Australians account for around two thirds of long-stay tourist arrivals.

1.10 Generally, designing and delivering aid programs in a developing country is challenging, with inherent risks. The development environment exists within the context of often competing political and implementation objectives. It may also take considerable time for the benefits from aid programs to realised and the number of variables affecting the programs and the environment within which they are delivered may make the assessment of impact difficult. Often Australia’s role is limited to influencing and encouraging change rather than direct program management.

1.11 While Vanuatu is viewed by many Australians as a holiday destination, it is a developing country struggling with high levels of poverty, and one third of its population lack access to multiple basic services, such as education, health services and safe water. Vanuatu was ranked 131 (of 187 countries; with a value of 0.616) on the Human Development Index²⁰ in 2013, its progress towards the Millennium Development Goals is mixed and it is unlikely to meet most of the targets by 2015 (see Figure 1.4).

Figure 1.3: Vanuatu’s progress towards the Millennium Development Goals

Source: Pacific Islands Forum Secretariat, 2013 Pacific Regional MDGs Tracking Report, August 2013, p. 26.

1.12 Implementing successful development initiatives in Vanuatu can be challenging. It is a culturally diverse country. The geography of Vanuatu means that the small population are dispersed over a large number of islands and speak 113 distinct languages and numerous dialects. Vanuatu lacks natural resources that could contribute to revenue generation, except for tourism, and the formal economy is small. Infrastructure (including transport, electrical and financial) on many of the islands is very limited, resulting in difficulties in travel, communication and disbursement of funds. Political instability and the operational and management capacity of the Government of Vanuatu (GoV) contribute to a difficult delivery environment. In addition, there is a strong kastom system²¹ and chiefs, as well as island and family allegiances, play an important role in all aspects of society.

Australian aid investment in Vanuatu

1.13 In 2014–15, the budget for Australian ODA to Vanuatu is $60.4 million, the majority of which ($41.9 million) is bilateral aid managed by DFAT.²² The remaining ODA to Vanuatu is managed by other government departments or is delivered through DFAT’s regional or global programs. Bilateral aid is allocated across the program’s strategic objectives, as shown in Table 1.1. Appendix 3 lists Australia’s bilateral aid investments in Vanuatu in 2013–14.

Table 1.1: Allocation of bilateral aid to Vanuatu, 2013–14 and 2014–15

Source: Department of Foreign Affairs and Trade, Aid Program Performance Report 2013–14, Vanuatu, September 2014, p. 3, and correspondence from DFAT, 10 April 2015.

Note 1: Other includes: Vanuatu Churches Partnership Program; Pacific Women’s Initiative; Vanuatu Land Program; and Won Smolbag Theatre Partnership.

1.14 Australia is the largest donor of aid to Vanuatu, providing around half of total aid to the country. Table 1.2 shows the top six donors of gross ODA to Vanuatu in 2012–13.

Table 1.2: Top six donors of aid to Vanuatu, 2012–13 average

Source: Vanuatu: Aid at a Glance, Organisation for Economic Co-operation and Development.

1.15 Like in many developing countries, China is an emerging donor to Vanuatu. Historically, there has been a lack of reliable data on the level or distribution of Chinese aid. However, the Lowy Institute for International Policy recently undertook a survey of Chinese-funded aid projects in the Pacific Islands, which noted that Vanuatu is attracting increased interest from Chinese companies and total Chinese aid to Vanuatu since 2006 amounts to over $200 million.²³

Managing the Vanuatu aid program

1.16 DFAT manages the aid program on behalf of the Australian Government. Responsibility for the Vanuatu bilateral aid program is shared between DFAT staff in Canberra, Australia, and at the Australian High Commission in Port Vila, Vanuatu. Generally, Canberra-based officers (collectively referred to as ‘desk’) are responsible for policy development, operational coordination with the Australian High Commission and coordination with other Australian agencies. DFAT officers in Vanuatu (collectively referred to as ‘post’) are either Australian-based or locally engaged staff and are responsible for operational delivery of the aid program.

Tropical Cyclone Pam

1.17 On 13 and 14 March 2015, Tropical Cyclone Pam passed over Vanuatu. The category five cyclone is the most powerful on record to have impacted Vanuatu and is considered to be the worst natural disaster in the country’s history. It claimed lives and caused extensive damage across the country. Reports suggest that around 70 to 80 per cent of the population was displaced, over 80 per cent of structures in the worst affected areas were damaged or destroyed, and essential infrastructure, including hospitals, roads, bridges, communications, and water and sewerage systems, sustained damage.

1.18 In response, Australia initially provided a package of assistance that included:

• funding to Australian non-government organisations, the Australian Red Cross and United Nations organisations;
• deployment of humanitarian supplies and specialist equipment; and
• deployment of personnel, including DFAT officers, medical assistance teams, and urban search and rescue assessment teams.

1.19 The cyclone will result in significant changes to the development environment in Vanuatu and, consequently will impact on Australia’s investment in aid programs. Following the cyclone it will be necessary for DFAT to evaluate the impact on the aid program and initiate the necessary changes to Australia’s investments in Vanuatu.

Audit objective, criteria and scope

1.20 The objective of the audit was to assess the effectiveness of the Department of Foreign Affairs and Trade’s management of Australian aid to Vanuatu.

1.21 To form a conclusion against the audit objective, the ANAO adopted the following high-level criteria:

• a sound strategic approach to planning Australia’s aid investments in Vanuatu has been developed and effectively implemented;
• Australia’s aid investments in Vanuatu are effectively managed, monitored and evaluated; and
• administrative arrangements facilitate the cohesive delivery of Australia’s aid investments in Vanuatu.

1.22 The audit focused on Australia’s bilateral program in Vanuatu. The audit team examined five initiatives in detail (see Table 1.3 and Appendix 4 for more detail about each of the initiatives). While the audit focused on DFAT’s current investments in the five areas, where necessary it also examined previous phases, including the Vanuatu Transport Sector Support Program Phase 1 and the Vanuatu Education Road Map (VERM)²⁴, as well as requirements under the previous Direct Funding Agreement for the Health Sector Program (Health).

Table 1.3: Five initiatives examined by the ANAO

Source: ANAO analysis of DFAT documents.

1.23 The audit examined Australia’s investment in aid programs in Vanuatu before Tropical Cyclone Pam. The audit did not assess Australia’s aid program in Vanuatu following the cyclone or Australia’s response to the cyclone. However, the findings and recommendations from this audit remain relevant for DFAT’s ongoing management of Australia’s aid investments and, specifically, to the Vanuatu program as it develops and changes in the post-cyclone environment.

Audit methodology

1.24 The audit team conducted fieldwork in Canberra and Vanuatu. The ANAO reviewed DFAT files and documentation and interviewed key DFAT personnel and relevant stakeholders, including representatives of the GoV, other donor countries, contractors, and multilateral and non-government organisations (NGOs). During its visits to Vanuatu, the audit team visited three islands (Efate, Malekula and Tanna) conducting site visits to initiatives receiving Australian aid. The fieldwork assisted the audit team to gain an appreciation of the issues facing DFAT and the impact of the environment on aid delivery, including the complexity of the relationships between all stakeholders.

1.25 This audit was conducted in accordance with the ANAO Auditing Standards at a cost to the ANAO of $472 000.

Report structure

1.26 This report comprises five chapters, as follows:

Introduction

2.1 Australian aid to Vanuatu is delivered within the context of a complex policy and operational environment, and under the auspices of Australian government policy and agreements with the GoV. Therefore, to maximise the impact of Australian investments, a sound strategic approach is required when planning aid initiatives in Vanuatu.

2.2 To assess whether DFAT has a sound strategic approach to planning Australia’s aid investment in Vanuatu, the ANAO examined:

• the relevant policies and priorities of the Australian and Vanuatu Governments and the extent to which priorities are aligned;
• the development and content of Australia’s Aid Investment Plan for Vanuatu, including DFAT’s management of key stakeholders; and
• DFAT’s management of risk, including the risk of fraud to the aid program in Vanuatu.

Aid policies and priorities

2.3 Australia’s aid investment in Vanuatu is delivered within a complex and changing policy environment. The relationship between the Australian and Vanuatu Governments is supported by a Memorandum of Understanding and a Partnership for Development (PFD). The Memorandum of Understanding was enacted in December 2005 and outlines arrangements with respect to protocols and facilitation of Australia’s development assistance to Vanuatu. The PFD establishes the shared vision, principles and commitments of the two governments as they relate to aid and development, and provides detail about how development would be focused and measured in five specified priority areas—education, health, infrastructure, economic governance, and law and justice.

2.4 The PFD was signed in May 2009 and is out-of-date in some areas. At the March 2013 Partnership for Development Talks between the Australian and Vanuatu Governments it was agreed that the PFD would be updated, with a new version of the agreement signed by the end of 2013. As a result of the 2013 Federal Election in Australia and changes implemented by the new Government, including integration of AusAID and DFAT, this did not occur.

2.5 The Government has introduced a new approach to framing Australia’s aid program and its relationships with partner governments. Under the new approach, Aid Investment Plans (AIPs) will be developed in consultation with partner governments and will set out the direction for a country or regional program. They will be supported by a framing paper that outlines Australia’s national interest in the country or region.

2.6 DFAT has prepared the framing paper and is currently drafting the AIP for Vanuatu. The draft AIP describes the Australian Government’s initial position regarding the aid program in Vanuatu and is subject to discussion and confirmation with the GoV. The draft AIP covers the period 2014–15 to 2017–18 and describes the proposed strategic priorities and rationale for Australia’s aid investments in Vanuatu, and the proposed mutual obligations of the Australian and Vanuatu Governments.

2.7 The strategic direction embodied in the draft AIP for the Vanuatu aid program is consistent with the Australian Government’s new aid policy; that is, the draft AIP identifies three priority areas that are consistent with Australia’s investment priorities, as outlined in the aid policy (see Figure 2.1). Australia’s development priorities in Vanuatu are also generally consistent with Vanuatu’s policy agenda, as communicated in Government of the Republic of Vanuatu Priorities and Action Agenda.²⁵

Figure 2.1: Australia’s proposed strategic priorities for the Vanuatu aid program—comparing the draft Aid Investment Plan with Australia’s aid policy and Vanuatu’s strategic priorities

Source: ANAO analysis of DFAT and GoV documents.

2.8 At the level of individual aid initiatives, there should also be a clear alignment between an aid initiative and Australia’s aid policies and strategies, and the priorities of the GoV. The ANAO examined, for five initiatives, the relevant Australian and Vanuatu government policy documents and strategies, design documents, and agreements between the Australian and Vanuatu Governments and with relevant third parties. In all cases the objectives and priorities of the aid initiatives, while more narrowly focused, are in line with the broader priorities of Australia’s aid program and the GoV.

2.9 The Australian aid policy states that the aid program will invest in better quality education and that Australia’s investments in education will primarily focus on supporting changes to the systems and policies that deliver these services. The GoV’s mission, outlined in its education strategy, includes providing good quality student-centred education and a well-managed and accountable education system. Australia’s current education strategy in Vanuatu, VESP, is consistent with these goals, but is more narrowly focused on early education, specifically kindergarten to school year three. This focus is outlined in the Vanuatu AIP and is reiterated in the program design documents for VESP and relevant agreements.

2.10 With respect to infrastructure, one of GoV’s strategic priorities addresses economic infrastructure and support services, and includes objectives to ‘ensure the provision of competitively priced, quality infrastructure, utilities and services…’ and ‘ensure economic infrastructure and support services are available to other sectors’. The Australian aid policy includes a commitment to tackling infrastructure bottlenecks in the region to help create the right conditions for the private sector and to expand trade by, among other things:

• investing in infrastructure that enables private sector and human development, such as transport infrastructure and water systems; and
• working with multilateral organisations that have significant expertise in innovative solutions to deliver enabling infrastructure.

Therefore, Australia’s two infrastructure projects in Vanuatu, VTSSP and PVUDP, which focus on road transport infrastructure and urban development (including road networks, drainage and sanitation systems) are consistent with the GoV’s priorities and broader Australian policy.

Developing the Vanuatu Aid Investment Plan

2.11 The Vanuatu program is the first bilateral program to develop an AIP. The drafting process commenced in early 2014 and has been lengthy. In April 2015, DFAT advised the ANAO that it expects to commence formal consultations with the GoV in mid-2015. Following consultations, DFAT anticipates finalising the AIP and a successor partnership to the PFD by the end of 2015.

2.12 DFAT’s Aid Programming Guide (the Guide)²⁶, issued in July 2014, contains direction for the development of AIPs, including the required contents and the approval process. As noted above, DFAT commenced drafting the Vanuatu AIP in early 2014, prior to issuing the Guide. DFAT staff working on the Vanuatu program did well to develop a draft AIP in the absence of guidance and clarity about the purpose or status of the AIP, or the process to be followed in its development.

2.13 Consistent with the Guide, the draft AIP includes commentary about Australia’s strategic priorities, proposed approaches for implementing aid investments, the mutual obligations agreed to by the Australian and Vanuatu Governments, program management and monitoring arrangements, and the performance indicators and targets (described as ‘performance benchmarks’), that will be used to assess progress against the strategic objectives. The appendices to the final AIP are expected to include an outline of program funding, future investment evaluations, a risk register and a performance assessment framework. Generally, the draft AIP (as at November 2014) meets the requirements of the Guide, and is sufficiently detailed to provide an understanding of the key elements of Australian’s aid program in Vanuatu.

2.14 The final AIP is expected to also include performance indicators (or benchmarks), to be developed in consultation with the GoV, and is anticipated to include targets for a five year period. When developing the performance benchmarks, it will be important that they reflect the Vanuatu aid program’s objectives and provide a firm basis for assessing progress against those objectives. As mentioned above, the current partnership agreement between Australia and Vanuatu, the PFD, is out-of-date in some areas, including the performance benchmarks and targets. For example, the PFD pre-dates a key infrastructure project, the PVUDP, and includes targets for infrastructure projects until June 2012. Agreed outcomes for the future are discussed during annual PFD talks between the Australian and Vanuatu Governments and benchmarks are included in the annual Aid Program Performance Report (APPR), but these focus on targets for the following year only (see Table 2.1).

Table 2.1: 2014–15 performance benchmarks

Source: Department of Foreign Affairs and Trade, Aid Program Performance Report 2013–14, Vanuatu, September 2014, p. 20.

2.15 While relevant, the benchmarks in the APPR measure only a small portion of the Vanuatu aid program for a limited period of time. For example, the only benchmark addressing education relates to the development of a monitoring tool. Assessing performance against this benchmark will not provide a complete picture of progress towards the achievement of the objective to ‘support increased access to and quality of education for all boys and girls, and equip them with skills and knowledge’.²⁷ Similarly, the health objective is to ‘strengthen health services and accelerate progress towards health MDGs’.²⁸ The only health benchmark is limited to measuring the number of nurses and midwives trained annually.

2.16 DFAT has informed the ANAO that performance benchmarks are not intended to capture the entire aid program. However, when developing performance indicators and targets to be included in the AIP, DFAT’s has an opportunity to develop a broader framework for measuring the success of the Vanuatu aid program. Comprehensive and up-to-date performance indicators that reflect the program’s objectives would provide a firm basis for assessing progress against the objectives, and should be consistent with the benchmarks and targets approved in investment design documents and agreements with GoV and other contractors.

Strategic stakeholder management

2.17 Effective stakeholder management is an essential element in the strategic planning and management of an aid program. In practice, building and maintaining relationships and networks is largely the responsibility of the DFAT officer managing an initiative and the effectiveness of relationships is reliant on the skills, knowledge and attitude of that individual.

2.18 Generally, DFAT staff have developed productive relationships with stakeholders, including the GoV and its donor and delivery partners, and actively participate at key meetings, such as program steering committees. Stakeholders advised the ANAO that DFAT staff were open, accessible and responsive.²⁹ However, as stakeholder management is dependent upon the individual, there is a risk that relationships with stakeholders will be adversely impacted by changes in staff within DFAT, GoV or other stakeholders. When an experienced DFAT program manager, who has invested time building relationships with GoV or other agency representatives, changes role or leaves the office, maintenance of those relationships is dependent upon effective personnel transition arrangements. However, this does not always occur.³⁰ For an officer new to a sector, it takes time to build relationships, networks and the depth of knowledge necessary to effectively manage Australia’s aid investments. Changes in DFAT staff in Vanuatu can also change the dynamics between DFAT and partner governments.

2.19 As the largest donor in Vanuatu, Australia also has a responsibility to take a leading role with fellow donors and delivery partners. DFAT participates in, and at times leads, donor coordination meetings within relevant sectors, including health and education and is in regular, even daily, contact with delivery partners. Also, in October 2014 DFAT reinstated meetings with team leaders implementing Australian aid programs (such as NGOs and private contractors). These meetings, held quarterly until late 2013, provided a forum for development partners to discuss current programs, challenges and successes, and future directions. The October 2014 meeting was the first in a year. DFAT used the meeting to inform participants about Australia’s new aid policy and performance framework—four months after the current approach was announced.

2.20 The decreasing Australian aid budget has resulted in tension with some donors on specific projects. For example, DFAT was involved in early discussions about the refurbishment of the Lepatasi International Multipurpose Wharf in Port Vila. With the refocusing of the aid program, DFAT does not have the available funds to contribute to the project and, as such, Australia’s involvement in, and funding to, the project has decreased.³¹ The project is being undertaken with the assistance of the Japan International Cooperation Agency.

2.21 At a sector and initiative level, aspects of stakeholder management are captured in various documents. For example, the Aid Program Risk and Fraud Management Plan 2013–14 for Vanuatu included, for each sector, an overview of key stakeholders and/or partners and how DFAT Vanuatu will engage with them about risks and risk management. The design documents for current phases of VTSSP and VWC covered the proposed programs’ consistency with existing Australian and other donor programs, and relevant key stakeholders. Similarly, the Delivery Strategy for the Vanuatu Health Sector included a table of the main agencies involved in the health sector and the programs those agencies are funding. However, relationships and the role of key stakeholders are not features of all design documents. The VESP design document, for example, does not include a similar analysis, only referencing development partners as necessary throughout the document.

2.22 In summary, on an individual basis, relationship management is a strength of the bilateral Vanuatu program. Nevertheless, a more structured approach to stakeholder management (that is periodically reviewed) would be useful for aid program management, and is particularly important in a country like Vanuatu where Australia is the lead donor and there are a large number of stakeholders, including other DFAT-managed regional and global programs. In April 2015, DFAT advised the ANAO that:

Stakeholder engagement is at the heart of the Vanuatu program’s work, and is crucial to the success of the initiatives and our broader bilateral relationship. … We agree … that capturing and documenting key relationships is important for efficient and effective delivery of any aid program. Although not required by the Aid Programming Guide, the Vanuatu aid program is taking steps to map stakeholders ahead of our AIP consultations with the Vanuatu Government and other key players.

2.23 Mapping stakeholders, including partner governments, other donors, delivery partners and internal stakeholders, as well as other aid programs operating in Vanuatu, in the context of finalising the AIP, would help to guide DFAT’s development and maintenance of effective and productive relationships with stakeholders and would inform its approach to planning and managing aid investments within Vanuatu. It would also assist DFAT to identify program gaps and overlaps and contribute to continuity in the aid program in the event of staff changes.

Strategic risk management

2.24 Effective risk management is another important factor in the successful management of an aid program. DFAT manages risk at several levels, including at the division, country and initiative level. At each of these levels, DFAT has risk management plans and/or registers that are intended to capture key risks and mitigation treatments.

Pacific Division risk management

2.25 DFAT’s Pacific Division, within which management of the Vanuatu program sits, maintains a risk register that is reviewed and updated regularly. The current register identifies 11 risks in four categories. Six of the risks apply across the division, three risks are specific to the Papua New Guinea program and two address a specific risk in each of the Nauru and Tonga programs. The risk register does not include any risks specific to Vanuatu, but does include general risks that are relevant to the Vanuatu program, including:

• delays to programming and expenditure resulting in under-expenditure;
• reduced use of evidence to inform decisions on aid management;
• failure to ensure compliance with Australian Aid’s Child Protection policy and implement controls;
• monies paid through trust accounts or partner systems not spent effectively or accountably; and
• financial loss and reputational damage due to fraud.

Vanuatu program and initiative risk management

2.26 At the country level, the Guide prescribes the minimum risk management documentation required. These are post (for example, DFAT Vanuatu) risk and fraud management plans, and country risk registers and initiative or sectoral registers.³² For individual initiatives, registers should be developed as part of the design process and identify the risks to achieving the objectives of the investment. The Guide advises that reliance on partner risk documentation is not sufficient, as partners are not in a position to assess important or relevant risks from DFAT’s perspective.

2.27 For the Vanuatu aid program, the Aid Program Risk and Fraud Management Plan 2013–14 was signed in November 2013. It covers internal and external factors that might impact on risks, key stakeholders and partners and provides a brief description of how DFAT will engage with those stakeholders on risks and risk management, and how DFAT Vanuatu will monitor risks. The plan specifically identifies fraud risks, against a standard template, and provides an indicative risk rating for each of the 16 risks.³³

2.28 DFAT Vanuatu also maintains a Vanuatu risk register. In April 2015, in response to the ANAO’s preliminary audit findings, DFAT updated the risk register and addressed a number of shortcomings in the previous version. The new Vanuatu risk register includes risk registers for each of the key sectors in which the Australian aid program is operating³⁴, as well as a new whole-of-program risk section that addresses:

• contextual risks—such as risks relating to political instability and natural disasters;
• programmatic risks— such as risks relating to financial management capacity and the non-performance of managing contactors; and
• institutional risks—such as risks relating to fraud, theft and conflict of interests.

Table 2.2 shows the distribution of the 74 risks across the individual registers.

Table 2.2: Distribution of risks in the Vanuatu risk register, as at April 2015

Source: ANAO analysis of DFAT data.

2.29 The Aid Program Risk and Fraud Management Plan states that the risk register should be updated bi-annually. However, prior to the April 2015 update, the risk register had not been reviewed since November 2013.

2.30 Weaknesses remain in the new version of the risk register. For example, the register does not consider the impact on the aid program of several strategic aid management risks, including changes in aid policy, adjustments in the bilateral aid budget, or resourcing changes. It is important that these risks are considered at the bilateral program level given their potential to impact on the effective delivery of the aid program. Furthermore, for 10 of the 14 whole-of-program risks, an assessment has not been made regarding the acceptability of the current risk rating, despite a risk rating of high or very high for seven of the 10 risks and moderate for the remaining three risks, or identified possible risk treatments for these risks. A further risk in one of the sectoral registers (education) notes that the risk is unacceptable, but the suggested risk treatment is incomplete.

2.31 As illustrated in Figure 2.2, two of the 74 risks rated as requiring treatment do not have implementation dates, the implementation dates for some risks are out-of-date and the majority of implementation dates for risk treatments are not specific.

Figure 2.2: Vanuatu risk register implementation dates

Source: ANAO analysis of DFAT documents.

Note 1: The 30 risks with implementation dates include 18 risks that were described as acceptable but also included additional risk treatments.

2.32 In some cases responsibility for implementing risk mitigation strategies is not specific³⁵ and there is also inconsistency across the registers with respect to assigning responsibilities. For example, responsibility for all the risks in the Health register are assigned to DFAT, while the majority of infrastructure risks are assigned to entities external to DFAT. While, in the latter initiative, there are contactors in place to share the responsibility for risk, DFAT is accountable for Australian aid funding and, as such, is ultimately responsible for documenting and managing risk to the aid program.

2.33 DFAT is in the process of changing its approach to risk management at the country program level. In future, post risk and fraud management plans are expected to be phased out once AIPs, which are to include consideration of risk, are in place. The Vanuatu risk register is included, as an appendix, in the draft AIP for Vanuatu.

2.34 Overall, while the April 2015 version of the Vanuatu risk register addresses a number of shortcomings identified by the ANAO in the previous iteration, weaknesses remain. There would be benefit in DFAT continuing to enhance its documentation of risk in the Vanuatu aid program by assessing all significant risks, as well as identifying and documenting risk treatments, responsibilities and timeframes.

Fraud risk

2.35 One of the key risks to the aid program is the risk of fraud, which is the subject of one of the 10 strategic performance targets identified in Making Performance Count: enhancing the accountability and effectiveness of Australian aid (Target 10: Combatting corruption). DFAT has a policy of zero tolerance of fraud or corruption and DFAT’s guidance states that all instances of suspected fraud are to be investigated. Fraud policy and the management of cases is overseen centrally by DFAT’s Audit Risk Management and Fraud Control Branch. The Guide outlines the processes for detecting, managing and reporting fraud, as well as links to additional information, including DFAT’s Fraud Policy Statement and relevant administrative circulars.

2.36 Within the Vanuatu bilateral aid program, fraud is recognised as a key risk and is incorporated into the post and sector/investment risk registers. The ANAO observed that there was a high level of awareness of fraud management procedures and responsibilities among the DFAT staff in Vanuatu. In terms of potential financial loss, known external fraud in the Vanuatu program is low, but the number of new cases reported in Vanuatu regularly places the program in the top six country programs with the highest incidence of reported fraud. Vanuatu accounted for five per cent of the 298 cases recorded by DFAT in 2013–14 (see Figure 2.3).

Figure 2.3: Number of fraud cases, 2013–14

Source: ANAO analysis of DFAT documents.

2.37 During the six months to December 2014, five new cases of external fraud were reported in Vanuatu, compared with 23 cases in Indonesia, 15 in Papua New Guinea, nine in Solomon Islands, six in Cambodia, and five in Timor Leste.³⁶ Of the total 1280 fraud cases recorded by DFAT since 2009, 48 (3.75 per cent) were in Vanuatu. DFAT estimates the value (actual or potential financial loss) of the 48 cases is $102 831. As at 31 December 2014, eight fraud cases were classified as active in the Vanuatu program, with an actual or potential value totalling $40 549.

2.38 To address the high levels of fraud in certain Pacific country programs, in September 2013 the (then) AusAID Director General issued a direction to increase fiduciary controls on funds administered through partner government systems in such countries. The direction required that Australian funds were not to be exposed to sole authority by partner government officials in these countries; that is, that all funding through partner government systems must include ex-ante review and sign off by an Australian Government delegate. In terms of the Vanuatu bilateral program, the directive was implemented in each of the five initiatives reviewed by the ANAO (if similar fiduciary controls were not already in place). For example, additional fiduciary controls regarding spending proposals and school grants were embedded in the agreement between the Australian and Vanuatu Governments for VESP and the contract with the managing contractor. During 2013, DFAT also conducted an assessment of the Vanuatu Government’s national systems and public financial management reviews in several sectors. These reviews are discussed in more detail in Chapter 3 and DFAT’s response to concerns about, among other things, partner government financial management is described in Chapter 4.

Conclusion

2.39 DFAT’s development of a new Aid Investment Plan for Vanuatu is timely given the limited currency of the Partnership for Development. The draft Aid Investment Plan is consistent with the new Australian aid policy, as well as the priorities of the GoV. It was one of the first drafted and is sufficiently detailed to provide an understanding of key elements of Australia’s aid program in Vanuatu. Generally, DFAT has developed and maintained good relationships with stakeholders, participating in a variety of formal and informal meetings, and relationship management is a strength of the Vanuatu program on an individual basis. However, while risk management registers exist at several levels, including at the country and investment level, in a number respects they could be improved substantially.

2.40 The development of a new Aid Investment Plan for the Vanuatu bilateral program provides an opportunity for DFAT strengthen its strategic approach to planning Australia’s aid investment by developing performance indicators and targets for measuring progress that more completely reflect the Vanuatu aid program’s objectives. In addition, developing a more structured approach to stakeholder management, in the context of finalising the AIP, would help to guide the development and maintenance of key stakeholder relationships and identify program gaps and overlaps. While it is impossible to eliminate risk, it is important that DFAT documents how it identifies, evaluates and determines the appropriate treatment of the risks to the Vanuatu bilateral program. The management of risks would be improved if risk registers included an assessment of all significant risks, as well as appropriate risk treatments, responsibilities and timeframes.

Recommendation No.1

2.41 To strengthen its strategic approach to aid investment planning and management, the ANAO recommends that the Department of Foreign Affairs and Trade reflects, in the Vanuatu country and investment risk register, an assessment of all significant risks and identifies and documents appropriate risk treatments, responsibilities and timeframes.

DFAT’s response: Agreed.

2.42 DFAT is updating the Vanuatu country and sector risk registers; is finalising the risk matrix annex of the draft Aid Investment Plan (which, as per the Aid Programming Guide, replaces the Risk and Fraud Management Plan); has included risk as a formal standing agenda item at regular aid program staff meetings; and has nominated a staff member as ‘risk champion’ to oversee these processes and report to management on their implementation.

Introduction

3.1 Effective design should provide a sound basis upon which an aid investment is approved and implemented. It includes the development of appropriate design documents that set out the rationale for selecting the aid activity and whether it is considered value-for-money, the extent to which aid might be provided through partner government systems, and the basic elements for implementing the proposed activity.

3.2 The ANAO reviewed:

• DFAT’s processes for designing aid initiatives;
• key considerations such as stakeholder engagement, value-for-money, and the use of partner government systems to deliver aid;
• the limitations in the design process, including the impact of poor design; and
• potential improvements to the design process.

The design process

3.3 Aid initiatives are expected to be designed in accordance with DFAT guidance and standards.³⁷ The key outputs of the design process are:

• investment concept—considers the development issue and rationale for investment, proposes outcomes and alternative approaches and options for delivering the intended outcomes, and provides a justification for the recommended investment; and
• investment design document—describes the purpose of the aid investment, what the investment will achieve and how it will be implemented, and forms the basis for financial approval to implement the investment initiative.

3.4 The design document must comply with the principles outlined in DFAT’s Investment Design Quality Criteria, meeting eight criteria: relevance; effectiveness; efficiency; monitoring and evaluation (M&E); sustainability; gender equality; risk management and safeguards; and innovation and private sector.³⁸ Investment designs are expected to be subject to internal and, where relevant, external quality assurance. The form of the quality assurance may differ between investments, but should usually involve a peer review and independent appraisal of the design document. The concept may also be subject to peer review. At the time the five investments reviewed by the ANAO were designed, a Quality at Entry report was also mandatory for the majority of aid investments. This report focused on the key design quality issues and was reviewed at the design peer review meetings. Quality at Entry ratings were assigned against seven criteria.³⁹

3.5 Of the five initiatives examined, DFAT undertook a design process and produced a design document for three—VESP, VTSSP, and VWC. For Health, DFAT produced a design strategy, which is similar to a design document and its development followed a similar process. The key stages and dates in the design of the four initiatives are shown in Table 3.1. The ongoing design of PVUDP is discussed separately in Figure 3.2.

Table 3.1: Key design stages and dates

Source: ANAO analysis of DFAT documents.

Note 1: A design process was also undertaken for the Strengthening Early Childhood Care and Education portion of the education investment in mid-2013. The design was finalised in September 2013.

Figure 3.1: Investment Design Documents

Figure 3.2: Designing the Port Vila Urban Development Project

The design of PVUDP, a partner-led design, did not follow the standard process. While DFAT did not develop a concept document as required, it did prepare a Design Summary and Implementation Document, which is a requirement for partner-led aid investments designs. The document summarises the proposed project, including the outputs to be achieved and funding arrangements, as proposed in design documents drafted by Australia’s delivery partner (investment manager), the Asian Development Bank (ADB), in November 2011. This was followed by preliminary and interim design reports in April and June 2012 respectively.

As a result of the delayed signing of the financing agreements (funding agreements in relation to the project were signed in December 2012) and difficulty recruiting and positioning consultants, physical work on the road and sanitation projects had not commenced by late 2013. GoV engaged a design, supervision and capacity development consultant in September 2013. The consultant was tasked to review the technical engineering design documents, formulate quality criteria for the design and construction of road and drainage works, and prepare bid documentation and cost estimates for the rehabilitation works. The consultants have produced a number of technical design reports (a Preliminary Design Report in April 2014 and a 70% Design Review Report in August 2014) to provide stakeholders with updates about the progress of the design stage, which describe progress, key events and issues to date.

Following a review mission in January 2015, the ADB reported that the overall progress of the project is delayed by approximately 26 months and estimates that project progress remains at 13 per cent against an overall elapsed project period of 51 per cent. The delays have caused DFAT to rephase the funding. Australian investment in the PVUDP was originally $31 million—a project specific grant of $26.5m and an additional $4.5 million to be sourced from remaining monies held in an ADB managed trust fund. However, the delays and consequent rephasing mean that outlays to date have been less than anticipated.

Note 1: Once expensed, the additional funding of $4.5 million was removed from the budget. As such, the December 2012 and March 2014 budgets only include the project specific grant of $26.5 million.

Note 2: Amounts are exclusive of 10 per cent GST (if applicable).

While some preliminary work has been undertaken (for instance, data collection and surveys, and consultation) the technical design has not been finalised and construction work has not started, over three years after commencing the project.

Source: ANAO analysis.

Key considerations when designing an aid initiative

3.6 Design documents are the foundation upon which an investment is built. As such, designs need to be robust, accurate and complete. The factors considered in the design process, and reported in the design documents, include stakeholder consultation, value-for-money and the use of partner government systems.

Stakeholder consultation

3.7 The design process should include consultation within DFAT and with the partner government and other stakeholders at several stages (such as, consultation on the concept note and design document). For the initiatives reviewed, DFAT consulted with a range of relevant internal and external stakeholders. For example, during the design of VTSSP Phase 2, 17 people from DFAT and the GoV attended the concept peer review meeting and the design team consulted with around 20 key stakeholders during the design mission to Vanuatu. During the VWC Phase 6 design, a mature program that was only minimally changed from the previous phase, 12 people attended the design peer review, including eight DFAT representatives.

Value-for-money

3.8 ‘Ensuring value-for-money’ is one of the 10 key targets of the new aid performance framework, Making Performance Count: enhancing the accountability and effectiveness of Australian. At the initiative level, DFAT guidance directs that investment designs include an analysis of value-for-money and how it will be achieved. Elements to consider when determining whether an investment represents value-for-money include whether there is a sound rationale and justification for the program, the cost of the program and process for allocating funding, how risk and finances will be managed and how the investment will be monitored.

3.9 The VESP, VTSSP Phase 2, Health, and VWC design documents each referenced value-for-money. For example, the VWC design document included a robust rationale for the program, including references to past reviews of VWC describing its successes, the benefits of the program and its impact on Vanuatu society (for example, the impact on poverty and economic development). A detailed breakdown of costs by desired outcome, and sustainability of the results of the program were also outlined. The VESP and VTSSP design documents contain similar analysis. Discussion of value-for-money in the Health strategy included a brief discussion of alternative delivery options and the rationale for adopting a direct financing approach, and an overview of the financial and human resources required.

Using partner government systems

3.10 In line with the principles of the Paris Declaration on Aid Effectiveness and the Accra Agenda for Action, one of the key considerations when designing a bilateral aid program is the extent to which aid might be provided through partner government systems. That is, whether aid will be channelled directly through the financial management systems of recipient governments. By the end of 2012, more than 50 per cent of bilateral aid to Vanuatu was channelled through GoV systems, including the majority of aid funding in the education and health sectors.

3.11 VERM was one of the first initiatives to work through government systems and was considered to be a prototype for this approach. At the time VERM was designed, stability within the Ministry of Education and strong leadership from the Minister and within the Ministry resulted in a decision to channel donor funds through the GoV systems. However, maintaining stability within the Ministry of Education became more challenging within 12 months of the start of the program. As a result, VERM was not implemented as planned and a decision was taken to redesign the program, including reviewing the appropriate level of funding that should be channelled through GoV systems for programs in the education sector.

3.12 The Guide states that if a bilateral program is considering using partner government systems as a potential funding mechanism, DFAT must undertake an assessment of public financial management and procurement systems at the national and, where necessary, sector levels. At the national level, the purpose of an assessment of national systems is to examine the strengths and weaknesses of a partner government’s financial management systems and to identify potential fiduciary risks for the Australian aid program. At the sector level, assessments cover fiduciary risks associated with using the public financial management and procurement systems of a sector or agency. These assessments are intended to inform decisions about the future use of government systems during the development of Aid Investment Plans and at the initiative design stage.

3.13 An assessment of national systems in Vanuatu was completed in June 2013 and public financial management reviews of several sectors in Vanuatu, including education, health and infrastructure, were also undertaken in 2013. The assessments of the Ministries of Education and Health made similar recommendations, while the assessment carried out in the infrastructure sector, specifically of the systems of the Ministry of Infrastructure and Public Utilities, came to a different conclusion.⁴⁰

3.14 DFAT advised the ANAO that sector level assessments inform a risk based approach to the use of partner systems across sectors, which will potentially result in different approaches to the delivery of aid within a country. Nevertheless, in response to the assessments in the education and health sectors, which resulted in similar findings, different aid management models have been implemented in each of the sectors. For example, a managing contractor has been engaged to manage the majority of Australian aid funds in the education sector; while in the health sector, DFAT continues to deliver a significant portion of the program through the government’s systems, signing a new funding agreement with GoV in May 2014. In the infrastructure sector a mixed approach has been adopted for VTSSP Phase 2, with 58 per cent of Australian funding provided through government systems and 42 per cent paid to the contractor.

3.15 It should be noted, however, that the public financial management assessments postdate the design of the VESP, VTSSP Phase 2 and Health initiatives. The VESP and VTSSP Phase 2 contracts, which were signed after the assessments, do incorporate increased fiduciary controls, such as the appointment of financial specialists. The funding agreements between the Australian and Vanuatu Governments in relation to VESP and VTSSP also include procurement and financial controls. However, the requirements for ex-ante review and sign-off by an Australian Government delegate, as required by the September 2013 directive⁴¹, has not been written into the agreements. For the Health program, a Health Resource Mechanism has been established, including the recruitment of an Audit and Finance Officer whose responsibilities include improving public financial management relating to the expenditure of donor funds (including co-authorising donor funds) and analysing and reporting on the expenditure of donor funds to stakeholders.

3.16 DFAT recognises that building capacity and strengthening government systems is a crucial element of its aid program in Vanuatu. Under the Governance for Growth program Australia is working with Vanuatu’s Prime Minister’s Office and Ministry of Finance and Economic Management to implement economic and financial reforms. In addition, each of the three strategic priority areas outlined in the draft AIP, and most of the individual initiatives, include this element as an objective.⁴²

3.17 However, with a contractor in place, the risk is that opportunities to build capacity within the GoV are limited. The VESP contract with the managing contractor lists capacity development of the Ministry of Education as one of five implementation strategies. While capacity building priorities need to consider program implementation and fiduciary risks, it will, nevertheless, be incumbent upon the managing contractor and DFAT to make sure that sufficient opportunities for systems improvement and reform are provided within the Ministry of Education. With respect to VTSSP, the key objective of Phase 2 is to build the capacity of Vanuatu’s Public Works Department—that is, to enable the GoV to demonstrate capacity to responsibly prioritise, plan, build and maintain road transport infrastructure within available national and donor resources—and the program is designed to facilitate this objective.

Limitations in the design process

3.18 While the design process is reasonably comprehensive and DFAT adhered to the required process for developing designs, there are limitations to the process, particularly in relation to monitoring and evaluation frameworks and risk management plans. DFAT requires that all aid investments meet the Investment Design Quality Criteria, which includes M&E (the M&E framework must be appropriate for measuring progress towards expected outcomes). In practice, however, these frameworks have been developed after an investment has been approved and have usually been a contract deliverable for the delivery partners.

3.19 For VESP, the managing contractor had six months from the signing of the contract in August 2013 to develop a M&E plan. In this case, the M&E Advisor commenced later than anticipated, delaying drafting of the plan. The plan was not completed until December 2014.⁴³ For VTSSP Phase 2, the contractor was required to develop a M&E plan within three months of mobilisation in July 2013. The VTSSP plan was finalised in May 2014. In addition, risk management plans are included in the design documents but, as discussed in Chapter 2, do not include some key aid management risks. Therefore, key elements required to manage initiatives and assess their progress and effectiveness were not comprehensive and, for the two initiatives discussed above, were not completed in a timely manner.

3.20 Poor design can also impact on the successful implementation of a project. For example, in the VESP design documents and approvals DFAT recognised that VERM, as designed, was overly ambitious and placed too much reliance on GoV, which did not have the necessary levels of leadership and capacity to implement the program successfully. DFAT has stated that the results of VERM were not commensurate with expectations and the level of investment, that is, the program was not considered to be value-for-money. Consequently, the subsequent VESP program is being implemented in partnership with a managing contractor.

3.21 With respect to VTSSP, the Phase 2 design was predicated on assumptions and baseline information about the road quality and lengths achieved in Phase 1 that were subsequently found to be inaccurate. In particular, early reporting in Phase 2 revealed discrepancies in road completion data from Phase 1, including double counting of roads reported as improved. These discrepancies were not disclosed until after the Phase 2 design was finalised.

3.22 Weaknesses in design documents can also flow through to subsequent agreements with implementing partners. The VTSSP Phase 2 tender documents and subsequent contract were based on the flawed assumptions and data inaccuracies contained in the design document. As a result, after one and a half phases of the project, there is ongoing discussion between the stakeholders, including GoV, DFAT and the contractor, about the required quality of roads to be rehabilitated under the program.

Potential improvements to the design process

3.23 In general, the complexity of an investment design process should be informed by an assessment of risk, including the type of investment, delivery partners, and proposed delivery options. Also, the time taken to design an aid investment can be lengthy. With the introduction of the Guide in July 2014, the design process has been streamlined to some extent. Concept and investment design documents are now not required for investments valued at less than $3 million and, for DFAT-led designs, low risk investments valued between $3 million and $10 million are not required to go through the concept stage. There are opportunities, however, to further streamline the process, particularly for established, mature or straightforward initiatives.

3.24 The VWC program is one example for which the design process could be streamlined. It is a successful program that Australia has been supporting since 1994, and is now in the sixth phase of funding. Design of VWC Phase 6 complied with the requirements current at that time, including the production of a concept note and detailed design document in October 2011 and June 2012 respectively. However, on 2 November 2011, the then Parliamentary Secretary for Pacific Island Affairs and the then Minister for the Status of Women jointly announced that Australia would fund VWC for a further four years and the amount of that funding, anticipating the outcome of the Phase 6 design process by eight months. Future designs for such mature programs could be simplified, reducing the time and resources dedicated to the process. Future design documents could build on past experience and lessons learnt over time, focus on potential changes to the program and limit the amount of background detail and program justification.

Conclusion

3.25 DFAT has a well-defined process for designing aid initiatives that includes concept notes, design documents, peer reviews and consultation with a range of stakeholders. The resulting design outputs are generally comprehensive, include a rationale for the project and a consideration of value-for-money, and recognise successes and failings of prior programs and future challenges. Furthermore, while the design documents predate DFAT’s current Aid Programming Guide, each complies with the intention of the guidance and the department’s Investment Design Quality Criteria. However, there is scope to improve the design process, particularly as M&E frameworks and risk management plans were not completed in a timely manner in some cases. For mature or straightforward initiatives, there may be opportunities to streamline the process.

3.26 The planning for one initiative in particular, PVUDP, has not been efficient. To minimise the risk that the project will not be able to achieve its aims within the approved timeframe, DFAT should work with the contracted investment manager and the GoV to finalise and implement the design as soon as practicable.

3.27 DFAT has undertaken reviews of partner government systems to ascertain the extent to which Australian aid might be delivered via those systems and has responded to identified weaknesses by increasing fiduciary controls in the VESP, VTSSP Phase 2 and Health programs. Building the capacity of government systems is also an element of aid investments in Vanuatu. However, for sectors where public financial management review findings were similar, DFAT’s approach to using Vanuatu government systems was not consistent. Contractors have been engaged for some programs, such as VESP, while significant amounts of funding continues to be delivered through government systems in other programs, such as Health.

Introduction

4.1 To help ensure that progress is satisfactory and desired outcomes are being achieved, it is important that aid initiatives are actively managed throughout their life. This includes monitoring progress and evaluating results. The Aid Programming Guide, and the DFAT monitoring and evaluation (M&E) standards⁴⁴, provide DFAT staff with direction about managing, monitoring and evaluating aid investments.

4.2 The ANAO reviewed:

• the M&E frameworks for the five initiatives selected for the audit;
• DFAT’s approach to monitoring the progress of these initiatives and evaluating their outcomes;
• DFAT’s approach to assessing the performance of advisors and contractors; and
• how DFAT responded to M&E findings.

Monitoring and evaluation frameworks

4.3 M&E frameworks, also referred to as M&E plans, are required for each aid investment. They describe how progress and performance will be measured and assessed. Satisfactory M&E plans have been developed for four of the five programs (VESP⁴⁵, VTSSP Phase 2, PVUDP and VWC Phase 6). However, development of the VESP, VTSSP Phase 2 and PVUDP plans was not timely (as outlined in Table 4.1).

Table 4.1: Monitoring and evaluation plan deadlines and delivery dates

Source: ANAO analysis of DFAT documents.

4.4 The performance assessment framework developed for the Health program is brief and does not include interim targets or discuss how progress against outcomes will be monitored. As such, it is not an adequate replacement for a robust M&E system.

DFAT’s monitoring of progress

4.5 M&E plans are usually developed by delivery partners and focus on the initiative; not on the performance of the program deliverer. DFAT adopts a variety of methods to monitor the performance of delivery partners and some of these are incorporated into agreements with delivery partners, including requirements for accountability and reporting. For example, requirements for progress reporting are specified in DFAT’s agreements with the VESP and VTSSP contractors, the Ministry of Health, and VWC. However, DFAT has not documented, in one plan or strategy, its approach to monitoring and evaluating each initiative. Developing M&E plans for each initiative that focus on DFAT’s management and monitoring of the performance of delivery partners, as well as the progress of the initiative, would assist DFAT to take a more risk-based approach to monitoring.

4.6 In terms of the performance of delivery partners and progress of individual initiatives, the approaches DFAT adopts to monitor and evaluate its aid investments include:

• conducting monitoring visits;
• requiring reporting on performance, including progress reports and financial reports;
• engaging with delivery partners through meetings and other contact;
• performing annual Aid Quality Checks;
• undertaking periodic evaluations, such as mid-term reviews (MTRs); and
• evaluating the performance of technical advisors and contractors.

4.7 DFAT and other stakeholders may also commission reviews of elements of an investment, or require them as deliverables to an agreement, on an ad hoc basis. The key tools that DFAT has used to monitor and evaluate the five initiatives reviewed by the ANAO are discussed below.

Monitoring visits

4.8 Visits, planned or unplanned, to Australian-funded projects are useful to gain first-hand insight into the performance of the implementing agency and progress of the initiative. They also provide an understanding of the context and environment within which the initiative is operating. Although monitoring visits are the subject of one of the M&E standards, DFAT has not mandated a particular approach to visits, with the standard providing one option as an example only.

4.9 Until 2013, formal documented monitoring visits were not conducted in Vanuatu. DFAT staff in Vanuatu are now encouraged to undertake monitoring visits, and have been provided with training and tools to assist with these visits via the Evaluation Capacity Building Program (ECBP) (see Chapter 5 for more details about ECBP). A limited number of visits have been undertaken to date.

4.10 In October 2014, DFAT officers visited a number of schools in Torba provence to review the awareness and use of school grants and review implementation of the Early Childhood Care and Education pilot program.

Figure 4.1: Schools in Tanna, Malekula and Efate

Source: ANAO.

4.11 The visit reports record that DFAT has discussed issues identified during these visits with the Early Childhood Care and Education contractor but not with the Ministry of Education. DFAT’s education advisor also undertakes periodic visits to Vanuatu to review the progress of VESP. Following these visits, the advisor provides a report to DFAT highlighting areas of progress and concern, and recommended actions to address identified issues.

4.12 For VTSSP Phase 2, DFAT advised that its intention is to visit each of the islands involved in the program twice a year, although a schedule of site visits has not been established. Between August and October 2014, DFAT’s program manager and the infrastructure advisor visited the four islands where the program is currently operating.⁴⁶ Following the visits, the infrastructure advisor provided a report to DFAT containing his main findings, conclusions and recommendations. DFAT has not documented discussions with either the managing contractor or GoV about the findings arising from these visits. However, in April 2015 DFAT informed the ANAO that the advisor has consolidated his findings into one report that would be discussed with GoV and other stakeholders when finalised.

4.13 DFAT has not developed a schedule of visits to VWC sites, but has undertaken three to date, in April 2013, March 2014 and April 2014. The latter two visits were documented, but the reports of these visits are incomplete.

Figure 4.2: Vanuatu Women’s Centre materials on display in Tanna

Source: ANAO.

4.14 Feedback about the visits was not discussed with VWC management until October 2014, when DFAT offered further assistance to VWC regarding the quality of education and community awareness materials and agreed that it would continue to follow-up with VWC over the next several months.

4.15 Site visits are a useful and effective way to monitor a project. In the context of the Vanuatu program, they would be more effective as a management tool if the findings and observations were documented and discussed with relevant stakeholders in a timely manner, particularly if opportunities for improvement are identified.

Performance reporting

4.16 Delivery partners are generally required to provide a range of reports in accordance with relevant agreements and contracts. These include progress reports, financial reports and audits. Progress reports, the primary report that DFAT uses to monitor an investment, are a useful source of information about the progress of an initiative and any challenges or issues that might be inhibiting the successful achievement of objectives. One of the M&E standards covers progress reporting and outlines the required features of a progress report. Further guidance is also provided in the ECBP materials.

Progress reports

4.17 Progress reports are required from the Ministry of Education and the managing contractor (for VESP), the VTSSP Phase 2 contractor, VWC and on request from ADB (for PVUDP). The frequency with which these reports are to be submitted, and their required content, are outlined in the relevant contract or agreement. The ANAO’s assessment of reporting is set out in Figure 4.3.

Figure 4.3: Progress reporting in each of the initiatives

Source: ANAO analysis.

4.18 In summary, most progress reports were of a satisfactory quality, provided an appropriate level of detail about progress and issues, and met the M&E standards. However, progress reporting is not a feature of all the initiatives.

Financial reports

4.19 In addition to the financial information provided in progress reports, the financial performance of two initiatives, Health and VWC, is also presented in financial acquittals and reporting. The Ministry of Health provides DFAT with details about expenditure against sub-projects within the Health initiative. Reporting has been sporadic in the past, with reports provided at irregular intervals and covering periods from one month to several months, but has improved since mid-2013 and is now monthly. Nevertheless, the reports include limited detail about progress or issues and are not adequate for the purpose of monitoring the progress of the program or assessing performance against expected outcomes. As mentioned in Figure 4.3, the current health sector review provides an opportunity for DFAT to strengthen its approach to monitoring the Health program.

4.20 VWC provides annual financial acquittal reports to DFAT as per its agreement. The two Phase 6 reports, covering 1 July 2012 to 31 August 2013 and 1 July 2013 to 31 July 2014, are detailed and include explanations about the distribution of funds under the program. In contrast, the requirements in the VTSSP Phase 2 contractor and ADB (PVUDP) contracts to provide financial reports have not been met.

Financial audit reports

4.21 Each of the initiatives is also required to be subject to an annual financial audit. Table 4.2 summarises the audits completed for each of the initiatives.

Table 4.2: Summary of annual financial audits for each of the initiatives

Source: ANAO analysis of DFAT documents.

Note 1: Two audits of VERM covering 2009 to the first half of 2012 identified significant weaknesses across the Ministry of Education, including control weaknesses and non-compliance with internal controls.

Note 2: The Health program was not audited in 2012–13. The 2013–14 Health audit report has been accepted by DFAT and the Government of Vanuatu, but has not been finalised.

Engaging with delivery partners

4.22 A valuable monitoring tool at DFAT’s disposal is its relationship with delivery partners, fostered through meetings and other contact, and participation at program committee meetings.⁴⁷ The VESP Steering Committee, for example, has met monthly since June 2014, and irregularly prior to that.⁴⁸ The meeting is attended by all key stakeholders, including DFAT, covers a range of topics, and includes discussion of issues arising under VESP. Similarly, stakeholders of the VTSSP initiative have met monthly since late 2013 and irregularly prior to that, to discuss progress and issues, as well as meeting periodically (approximately quarterly) at a Steering Committee meeting. As a result of satisfactory reporting (discussed above) and more regular key meetings, monitoring of VESP and VTSSP Phase 2 has improved over the past year. The VWC Program Coordination Committee, which is scheduled to meet twice-yearly, has met twice during Phase 6, in October 2013 and August 2014. Discussion at these meetings is frank and covers a wide range of relevant topics.

4.23 DFAT also meets delivery partners on a regular or as necessary basis. The department meets monthly with representatives from GoV and the contractors to discuss PVUDP and during the Asian Development Bank’s periodic (bi-monthly since September 2014) missions to review PVUDP. DFAT also meets with VWC representatives several times a year for general discussion or to address specific issues. Program managers are also in regular (for some, daily) email and telephone contact with delivery and donor partners.

4.24 However, as noted in Chapter 2, the strength of relationships with delivery and donor partners is reliant on individual DFAT officers. While DFAT has established a good working relationship with delivery and development partners, developing a more structured stakeholder management approach would assist DFAT to more effectively manage these relationships.

Aid Quality Checks

4.25 DFAT assesses and reports internally on the performance of individual investments in an Aid Quality Check, formerly referred to as a Quality at Implementation (QAI) report. These reports, which are DFAT’s main mechanism for assessing the performance of individual aid investments, are expected to be prepared annually for the preceding 12 month period using data gathered from progress reporting, monitoring visits, reviews and evaluations. Investments that are rated as unsatisfactory against the two criteria (effectiveness and efficiency) are classified as investments requiring improvement and must demonstrate improvement within 12 months or be cancelled.⁴⁹ The most recent round of annual assessment and reporting for investments in Vanuatu pre-dates the change from QAIs to Aid Quality Checks. As such, the following discussion refers to QAI reports.

4.26 In February 2014, DFAT completed QAIs for 14 initiatives in Vanuatu. The average rating for these initiatives was 4.2 (with a range from 3.0 to 5.0). The results for the five initiatives reviewed by the ANAO are consistent with those of the Vanuatu aid program more broadly, and are shown in Table 4.3.

Table 4.3: Quality at Implementation ratings, February 2014

Source: ANAO representation of DFAT documents.

Note 1: VESP was not evaluated in this round of QAI assessments. The VERM QAI was a final report.

Note 2: The six ratings denote the following: 1 = very poor quality, needs major overhaul; 2 = poor quality, needs major work to improve; 3 = less than adequate quality, needs work to improve in core areas; 4 = adequate quality, needs some work to improve; 5 = good quality, needs minor work to improve in some areas; and 6 = very high quality, needs ongoing management and monitoring only.

4.27 DFAT’s ratings are consistent with the ANAO’s findings. VERM and PVUDP rated poorly against a number of criteria; while VWC received the highest ratings. Overall, QAIs have been completed as required and provide a good overview of the status, progress and challenges of each initiative.

4.28 The QAI reports included the actions that DFAT intended to take to address the issues raised. The five QAI reports (referred to in Table 4.3 above) recommended over 20 actions covering a continuum from high to low priority and from simple to complex requirements including:

• analysing the evaluations produced during VERM and whether recommendations have been implemented;
• for VTSSP, following up with GoV to confirm roadwork sections to be funded by loans from a Chinese bank;
• completing a performance audit of the GoV’s Public Works Department (as stipulated in the original funding conditions for PVUDP);
• requiring quarterly performance reports as part of the tranche release requirements for the Health sector; and
• requesting quarterly risk matrix updates from VWC.

4.29 By December 2014, DFAT’s progress on these, and other, actions was variable. A review of VERM evaluations was completed in June 2014, and details about the roadwork to be funded by Chinese bank loans have been received from the Public Works Department. A performance audit of the Public Works Department had not been undertaken⁵⁰, quarterly performance reports had not been received from the Ministry of Health, and quarterly risk matrix updates had not been requested from VWC (also see ‘Responding to monitoring and evaluation findings’ below).

Periodic evaluation of investments

4.30 Independent evaluations of aid investments are intended to assist decision-making and conclusions about the impact of an initiative. DFAT requires that independent evaluations be completed for high risk initiatives and those with a total budget greater than $10 million.⁵¹ If required, an independent evaluation must be conducted at least once during the life of the program and timed to maximise its value (for example, as a mid-term review [MTR] or at the completion of the investment). The requirement to conduct independent evaluations applied to four of the five initiatives reviewed by the ANAO. VWC Phase 6 is valued at $5.6 million and, as such, an independent evaluation is not required.

4.31 DFAT has not completed evaluations of the current phases of the initiatives. However, evaluations have been completed for the previous phases of two of the initiatives and the department is planning or conducting reviews of three initiatives (see Table 4.4).

Table 4.4: Completed and planned reviews of the five initiatives

Source: ANAO analysis of DFAT documentation.

Note 1: Periodic evaluations are not required to be conducted at the midpoint of the initiative; this column is included to provide context only.

Note 2: VWC Phase 6 is valued at $5.6 million and, as such, an independent evaluation is not required.

4.32 The ANAO reviewed the quality and findings of the two MTRs that have been completed for VTSSP Phase 1 and VWC Phase 5. The VTSSP Phase 1 MTR identified several issues and made eight recommendations for the ongoing management of the initiative. A number of these issues were still applicable at the end of Phase 1 and have continued into Phase 2, such as a shortage of personnel at the Public Works Department and delays in procurement processes. However, as previously noted, progress reporting during Phase 2 has improved. The VWC Phase 5 MTR was a positive report, and the five recommendations related to future improvements that could build upon the current management of the initiative. Both reviews complied with the intent of the M&E standards.

4.33 DFAT completed the Vanuatu Health Sector Strategy Update in October 2012. Although referred to as an MTR, it was not independent and did not report on the performance or impact of the current strategy. Instead the Strategy Update focused on why a new strategy was needed and the options for the new model to be discussed at the Partnership for Development talks in early 2013. DFAT completed a review of Australia’s investment in Health in Vanuatu in March 2015 and is currently considering its response to the review’s findings. The PVUDP initiative commenced in December 2012 and has not been evaluated. Given the delays and complexity of the project, DFAT should carefully consider the optimal timing for an independent evaluation of the PVUDP initiative.

4.34 In summary, the mid-term reviews that have been completed for two initiatives were adequate, providing an overview of the success of the initiative. While not all initiatives have been subject to a mid-term review, three reviews are underway or planned for 2015 and early 2016.

Other evaluations

4.35 DFAT or external stakeholders may also commission reviews of certain elements of an investment, or require reviews as agreement deliverables. Recent evaluations include:

• a socio-economic and gender impact study of the effect that VTSSP-improved road transport links had on local communities, completed in February 2014, which found that the initiative had a mixed impact; and
• the ‘Vanuatu National Survey of Women’s Lives and Family Relationships’, May 2011, which noted the positive impact of the VWC initiative, reporting a decrease in physical and sexual violence against women in areas where the VWC is more active.

4.36 DFAT can use these reports to understand the impact of its investments and inform future changes. However, it is important that evaluations are appropriately targeted. The education sector in Vanuatu has been evaluated often, with at least 37 reports produced between 2010 and 2013 on various areas, including: school construction; school grants; school-based management; budget management; and literacy and numeracy levels. Five of the 37 reports were commissioned by Australia. A DFAT commissioned review of the 37 reports concluded that several recommendations were contradictory, many reports were overly complex, some reports were written with no reference to previous work in the same area and there was little institutional memory about the reports. As such, the value of these reviews is limited.

4.37 DFAT’s Office of Development Effectiveness is responsible for producing high-level evaluations of aid program policies, strategies and approaches to common development issues. It has conducted one evaluation of DFAT’s activities in Vanuatu in the past three years. Vanuatu was one of the seven countries selected as part of an evaluation of Australian aid in support of women’s economic empowerment. The report, published in August 2014, found that despite sound policy settings, the Australian aid program is not yet delivering consistent and strong outcomes for women in its economic investments. It made four recommendations, which were not specific to a country program or initiative, to improve the way Australian aid can support women’s economic empowerment.

Evaluating technical advisor and contractor performance

4.38 While DFAT staff may be experienced development professionals, they may not be experts in their specific areas of responsibility. As such, DFAT relies on technical advisors and contractors to provide expertise in specific areas. As stated in DFAT’s guidance, ‘the manner in which DFAT staff, advisers and other contractors behave over the life of a contract has a significant influence on the quality of outcomes and on the quality of aid (for aid program agreements) delivered through commercial contracts’. It is necessary, therefore, for DFAT to periodically assess the performance of advisors and contractors.

4.39 Assessments are required annually and on completion for engagements longer than one year, and on completion for shorter engagements. The completed advisor and contractor performance assessments are managed centrally by DFAT’s Contracting and Services Branch. The Vanuatu bilateral program used both technical advisors and contractors in 2013–14. However, only one of the five required contractor performance assessments was completed for that period.⁵² Given the significance of advisors and contractors to the management of aid in Vanuatu, it is important that DFAT complete annual performance assessments as required.

4.40 In addition to the contractor performance assessments, a Strategic Performance Assessment report is produced centrally for the 10 highest-value contractors across the Australian aid program. This report assists DFAT to review the performance of contractors involved in several bilateral and regional programs. At posts where any of the contractors on the 10 highest-value contractor list are involved in programs valued over $10 million, the post is required to complete a Strategic Performance Assessment questionnaire to assist the development of the consolidated Strategic Performance Assessment report. Two of these contractors are delivering aid on DFAT’s behalf in Vanuatu, and DFAT Vanuatu submitted the required questionnaire for each of the contractors. Both reports were generally positive, with one contractor receiving an overall average rating of four (satisfactory) and the other contractor was rated as five (very good).

Responding to monitoring and evaluation findings

4.41 When implemented effectively, performance reporting, together with effective communication, site visits and lessons learnt from mid-phase evaluations have assisted DFAT to monitor the progress of initiatives and address issues as they arise. DFAT’s response to issues is made within the context of the aid delivery environment and the feasibility of implementing the actions required to address concerns. For example, responsibility for implementing many of the initiatives lies with the GoV and Australia’s role is to advise and support Vanuatu’s ministries to effectively manage the programs. However, DFAT is accountable for Australian aid funding. As such, it is important that DFAT establishes and maintains effective relationships with the GoV, exercises its influence to improve public administration and takes appropriate action in a timely manner.

4.42 With respect to the education program, DFAT and other donor partners were sufficiently informed about VERM to be concerned about the implementation and progress of the initiative. As a result, they implemented actions to minimise the risk to donor funds while redesigning the program. The redesign of the education program was initiated in early 2012, following agreement with GoV during partnership talks in November 2011 that a change of approach was necessary. The subsequent new initiative (VESP) design was approved in October 2012 and an agreement signed with a managing contractor in August 2013. In the interim, DFAT and its donor partners⁵³, responded to concerns, as well as the findings of the public financial management reviews and the September 2013 directive, by:

• encouraging the Ministry of Education to hold meetings;
• requiring additional fiduciary controls, including that all transactions involving donor pool partner funds be signed-off by an independent person appointed by the donors;
• reinforcing that payment of tranches was conditional upon meeting certain conditions, such as compliance with financial acquittal and reporting requirements; and
• delaying, freezing or cancelling the payment of tranches (tranches were delayed and/or reduced in each year from 2010 to 2013).

4.43 A similar approach was taken in the Health initiative. As a result of concerns about political stability, the Ministry of Health’s management capacity and capability, financial management practices and the adequacy of reporting, funding tranches were delayed and/or reduced in 2011, 2012 and 2013. Also, DFAT more narrowly focused its support for the program, prioritising funding to specific programs, including malaria, village health workers program, health information systems support and human resource management support. However, as discussed, DFAT has not implemented the recommendation of the Health QAI that quarterly performance reporting is a condition of tranche release.

4.44 As previously discussed, the PVUDP has experienced ongoing and increasing time slippages. The risk assessment and management plan developed by ADB in 2011 included a number of risks that might impact on the project’s timetable. By January 2015 the ADB reported that the overall progress of the project is delayed by approximately 26 months. Some of the delay is due to factors anticipated in the risk assessment and management plan, with slippage attributed to delays in the GoV’s spending pipelines, signing of agreements, recruiting and fielding consultants and the lack of availability of suitable personnel at key stages, as well as changes to the design. The February 2014 QAI report for PVUDP recommended that a performance audit of the Public Works Department be undertaken in accordance with the co-financing agreement. To date, this has not occurred.⁵⁴

4.45 DFAT has raised its concerns about the progress of PVUDP with the ADB on several occasions. For example, DFAT discussed with ADB the need to increase and improve communication and closely monitor contractors, and encouraged ADB to increase its presence in Vanuatu in order to more effectively monitor progress.⁵⁵ DFAT also developed an issues register that will be updated and used as a basis for discussing issues with ADB. While the development of such a register is a positive move, the timeframes for responding to the majority of the issues are described as ‘ongoing’. The register would be more effective if it included precise actions to be taken by specified deadlines. The ANAO considers that DFAT’s monitoring of PVUDP is adequate for the stage of the project (that is, the design phase) but, given the significant and increasing delays in the project, it will be important for DFAT to strongly encourage its partners to complete the project as scheduled.

4.46 For at least one initiative, program issues were not uncovered via DFAT’s monitoring. Weaknesses in DFAT’s monitoring of VTSSP Phase 1 were not known until Phase 2. Early reporting by the Phase 2 contractor revealed discrepancies in the road completion data from Phase 1. DFAT was not aware of these discrepancies until notified by the contractor. Analysis of Phase 1 reporting by the ANAO found inaccuracies, including double counting of roads reported as improved during Phase 1.

4.47 The existence of inaccuracies regarding the length and quality of roads rehabilitated during Phase 1 suggests that DFAT did not verify the information it received in Phase 1 reporting.⁵⁶ Had it done so, DFAT would have been aware of concerns with Phase 1 progress and been in a position to take appropriate and timely action. As a consequence of the need to revise the condition of Phase 1 roads, a review of some of the essential elements of the program was initiated, including clarification about the program objective and the expected quality of roads rehabilitated under the program. It is important that key information provided in Phase 2 reports is verified by requesting evidence to support key data and/or undertaking visits to a selection of work sites.

Figure 4.4: Roads in Malekula and Tanna

Source: ANAO.

4.48 At times, DFAT has delayed its involvement in issues when it considers that it would be more beneficial for the partner government to address the issue in the first instance. For example, a NGO was contracted by the GoV in January 2014 to provide management and implementation services for a portion of VESP (Strengthening Early Child Care and Education). The contract did not reference the Strengthening Early Child Care and Education design document. Instead, it detailed how the NGO’s approach would be used to deliver the desired outcomes. However, GoV’s expectation was that the program would be implemented in accordance with the Strengthening Early Child Care and Education design document. The existence of two approaches (the Strengthening Early Child Care and Education design document and the NGO’s approach) has led to confusion between the NGO and the Ministry of Education about the appropriate approach to be implemented, and the need to clarify aspects of the program has delayed implementation.

4.49 This issue illustrates the complexity of the aid delivery environment. The program is funded by Australia and New Zealand and managed by Vanuatu’s Ministry of Education. DFAT advised that it is working with the Ministry and NGO to amend the contract and with Ministry staff to improve their contract management capacity. DFAT has been hesitant to involve itself too early in program issues as the program is the responsibility of the partner government. However, as DFAT is accountable for Australian aid funding, it is important that appropriate steps are taken where there is a risk that agreed outcomes will not be achieved.

Conclusion

4.50 Delivery partners have developed satisfactory monitoring and evaluation plans for four of the five initiatives examined in detail. While these plans result in useful information that may be reported to DFAT, their purpose is to monitor a program from the delivery partner’s perspective. DFAT has not documented, for each initiative, how it will monitor the performance of delivery partners. In practice, DFAT adopts a variety of monitoring methods, including visits, progress reporting and mid-term reviews. However, the consistency and robustness of these processes was variable across the aid initiatives examined by the ANAO. Taking a more risk-based approach to monitoring, and documenting that approach in a monitoring and evaluation plan for each initiative, would assist DFAT to improve its monitoring of the performance of delivery partners, as well as the progress of the initiative.

4.51 DFAT has responded adequately to issues identified through monitoring or evaluation processes. These actions have included maintaining or increasing pressure on delivery partners, increasing fiduciary controls, delaying or cancelling the payment of tranches, and redesigning programs. However, a number of actions identified in Quality at Implementation reports have not been implemented.

4.52 In addition, DFAT’s monitoring has not always highlighted issues or weaknesses. Reporting from VTSSP Phase 2 suggests that DFAT did not adequately manage Phase 1. To mitigate the risk of this recurring, DFAT should assure itself that key information provided in reports and discussed at meetings is timely and supported by evidence. While the level of influence Australia may exercise can vary depending upon the amount of funding, delivery method and phase of a project, when Australia is funding aid initiatives it is appropriate for DFAT to act in a timely manner if issues are impacting on the investment’s performance or the achievement of outcomes.

Recommendation No.2

4.53 To better monitor and evaluate the Vanuatu bilateral aid program, the ANAO recommends that the Department of Foreign Affairs and Trade develops and implements a risk-based monitoring and evaluation plan for each initiative.

DFAT’s response: Agreed.

4.54 DFAT is taking steps to fully implement the new Aid Programming Guide released in July 2014, including by building on the monitoring and evaluation processes already established through the Evaluation Capacity Building Program and the application of DFAT’s Monitoring and Evaluation Standards. We will continue to work with the relevant areas within DFAT and with external stakeholders to ensure there are adequate risk-based monitoring and evaluation planning and systems in place for each initiative.

Introduction

5.1 The Australian aid program operates a decentralised management model, with DFAT staff in Canberra and the specific country working towards common development goals and reporting on progress towards these goals. Understanding roles and responsibilities and providing appropriate guidance and training to DFAT staff so they can perform their roles is fundamental to the success of the Australian aid program. Furthermore, to support accountability, it is important that the program’s performance is accurately and transparently reported. In this context the ANAO examined DFAT’s:

• roles and responsibilities for staff involved in the bilateral aid program to Vanuatu;
• support for its staff in Canberra and Vanuatu; and
• internal management and external reporting arrangements.

Roles and responsibilities

5.2 With the integration of AusAID and DFAT in 2013–14, the aid program was consolidated with the foreign policy and trade activities of the broader DFAT. As a result, the Head of Mission in each of Australia’s Embassies and High Commissions became responsible for the financial and operational aspects of the aid program delivered within their country. Each country desk became responsible for the aid activities relating to the represented country, as well as foreign policy and trade operations. The ANAO observed that the roles of the Canberra and Vanuatu teams are well understood by all parties and there is frequent communication between the two teams about upcoming activities and the status of current projects.

Roles and responsibilities in Vanuatu

5.3 The Head of Mission in Vanuatu, with responsibility for the financial and operational aspects of the aid program, is the Australian High Commissioner to Vanuatu. The Counsellor Development Cooperation is responsible for managing the aid program, and reports to the Australian High Commissioner.⁵⁷ DFAT staff in Canberra and Vanuatu spoke positively of the new role of the Heads of Mission and the changed lines of reporting.

5.4 The in-country monitoring and implementation of the Vanuatu bilateral aid program is principally performed by locally engaged staff (LES) program managers, under the oversight of an Australian-based (A-based) First Secretary Development Cooperation. DFAT’s organisational chart for the aid program in Vanuatu is shown at Figure 5.1.

Figure 5.1: Organisational chart of DFAT’s aid program in Vanuatu, June 2014

Source: ANAO analysis of DFAT documentation.

Note: The position of Australian High Commissioner to Vanuatu is a Senior Executive Service Band 1, the positions of Counsellor Development Cooperation and Director, Governance for Growth, are APS Executive Level 2, and the position of First Secretary Development Cooperation is an APS Executive Level 1.

5.5 The extent of the involvement of DFAT staff in the management of an aid initiative is dependent on the way in which the initiative is delivered, the proficiency of the delivery partner and the maturity of the initiative. For example, the VWC is currently in its sixth phase of funding and is aware of DFAT’s requirements regarding performance and progress reporting. Alternatively, DFAT officers are more involved in the ongoing management of the PVUDP than originally intended, due to the initiative’s delayed progress and the ADB’s limited presence in Vanuatu.

5.6 There are times when the First Secretary or program manager can resolve an issue without escalating to senior management. However, due to the climatic, development and political challenges of delivering aid in Vanuatu there have been occasions when it was appropriate for a DFAT officer to escalate the resolution of an issue to senior management. For instance, a weather event requiring a humanitarian response and a request to subsidise the Ministry of Health’s budget were escalated. DFAT officers consulted by the ANAO had a shared understanding of the appropriate timing and reasons for engaging senior management and adhered to these in practice.

Support for departmental staff

5.7 It is important that DFAT officers are provided with sufficient support to assist them to perform their duties. Support may take the form of guidance (both written guidance and assistance from internal specialist areas), as well as a program of training. It is also important that transitional arrangements during times of personnel change are supported and result in the effective transfer of knowledge.

Written guidance

5.8 DFAT’s current aid management guidance, released in July 2014, is the Aid Programming Guide (the Guide).⁵⁸ The Guide is a summary document of all the mandatory steps and processes for managing and delivering aid programs. The Guide covers the areas of policy and direction setting, planning and design, procurement, implementation and monitoring, performance management, risk management and fraud control, and evaluations.

5.9 The Guide includes links to additional documents that provide supplementary guidance about specific aid management processes. These include investment design quality criteria, M&E standards, and Secretary’s Instructions. However, it is not always clear whether the additional documents are mandatory or not. For instance, the Guide states that investment design documents ‘must meet’ the investment design quality criteria, whereas the M&E standards ‘guide’ staff managing investments on good practice. DFAT advised that the officer responsible for a particular activity determines the applicability of the supplementary guidance for their situation based on their assessment of their activity’s risk, cost and political interest. To assist staff to comply with all intended mandatory guidance, there would be benefit in DFAT clarifying the status of guidance which is attached to, but not located within, the Guide.

Internal specialist areas

5.10 Throughout DFAT, there are several specialist areas, including risk management, fraud and contract management, that provide guidance to desk officers and DFAT Vanuatu if requested. The Pacific Division also has a small team of sector specialists who can provide assistance, if requested, in the areas of public financial management, economics, infrastructure, health and gender. The number of specialists in the Pacific Division has decreased in recent years and the relatively modest scale of the Australian bilateral program in Vanuatu means that there is competition from the larger bilateral programs, such as the Papua New Guinea program, for the scarce resources of DFAT’s sector specialists. However, to date, the Vanuatu program has involved sector specialists in peer reviews of the design and evaluation stages of each of the aid initiatives.

Learning and development

5.11 DFAT’s Learning and Development Section, located in Canberra, has overall responsibility for the learning and development of DFAT officers.⁵⁹ However, the training program is decentralised and multiple areas within DFAT organise and provide their own training courses. For example, the Aid Management and Performance Branch are responsible for aid management capability and training, including on topics such as design, program logic, M&E and AidWorks.⁶⁰ Sections within DFAT responsible for risk management, contract management, public financial management and fraud control also provide specialised training.

5.12 The ANAO assessed the training records of five A-based DFAT aid officers in Vanuatu and the three officers on the Vanuatu desk in Canberra.⁶¹ The ANAO’s analysis of the training records is provided in Table 5.1.

Table 5.1: Training courses recorded as undertaken by A-based aid officers in Vanuatu and Vanuatu desk officers in Canberra, September 2014

Source: ANAO analysis of DFAT documents.

Note: The above analysis is based on data provided by DFAT in September 2014. DFAT also provided course data in April 2015. The latter data differed slightly from, but was generally consistent with, the data provided in September 2014.

5.13 As shown in Table 5.1, while the five A-based DFAT officers had an average of 16.8 courses recorded, the number of courses undertaken varied from one to 29 courses. For example, one officer had only a single training record listed against their name, a Fraud Control course from September 2012. This officer advised the ANAO of several training courses completed prior to deployment. DFAT advised the ANAO that the training records may not be complete because of the difficulty in extracting information from AusAID’s legacy database. Without reliable training records, it is difficult for DFAT to assess the adequacy, currency or completeness of the training undertaken by each officer.

5.14 Given the responsibilities of A-based officers and the steep learning curve at the commencement of an international posting, particularly for officers on their first overseas assignment, pre-deployment training is a crucial component of preparing A-based staff for that posting. The majority of the A-based aid officers in Vanuatu did not feel that they were adequately prepared, particularly with respect to establishing, negotiating and managing large contracts.⁶² However, without full training records it is not possible to assess the completeness of the training received by individuals. In light of staff views concerning the adequacy of some training it may be prudent for DFAT to review the requirements for the aid management training offered. DFAT advised, in April 2015, that a greater emphasis is now being placed on pre-posting training and that the procurement and contract administration training has been upgraded and redesigned and a new course on contract and agreement management is being developed.

5.15 Training opportunities are also provided for LES and in-country A-based personnel by the High Commission or Embassy, and all officers are able to access training provided by DFAT in Australia and regional centres.⁶³ Subject matter areas within DFAT Head Office often provide training while overseas conducting other activities. For instance, members of the fraud control team provided training in August 2014 while visiting Vanuatu to assess outstanding fraud cases. However, several teams (including risk management and contract management) have reduced their regional presence recently due to budget constraints. DFAT Vanuatu, which is responsible for maintaining training records for any course it funds, was not able to provide training records for its officers. This information was subsequently provided by DFAT Canberra. However, similar to the records for A-based and desk officers, the LES officer training records appear to be incomplete.

5.16 A training course which was spoken highly of by both LES and A-based officers in Vanuatu was the Evaluation Capacity Building Program (ECBP), described in Figure 5.2.

Figure 5.2: Evaluation Capacity Building Program

Source: ANAO analysis of DFAT data.

5.17 At the time of audit fieldwork, there had been limited analysis of the impact that ECBP had made to the quality of DFAT’s M&E activities in Vanuatu. There would be benefit in reviewing the effect that ECBP has had on the quality, credibility and use of M&E products in Vanuatu. Such analysis would support the continued investment in, and possible expansion of, this program.

Transitional arrangements

5.18 Individual DFAT officers have an important role in the management of the aid program and the maintenance of effective relationships with key stakeholders. It is therefore necessary that appropriate steps are in place during times of personnel change to facilitate program continuity.

5.19 DFAT Vanuatu has benefitted from retention of its LES program managers for a number of years.⁶⁴ Transition arrangements have been largely unnecessary due to the stability in the LES staff, which has contributed to continuity in the management of the aid initiatives. The consistency of DFAT officers on the Vanuatu desk has also enabled continuity in the strategic planning of the Vanuatu aid program, even during integration. The A-based personnel to Vanuatu are generally deployed on three year postings. DFAT postings are arranged with a long lead time to allow for pre-deployment training and to prepare staff and their families for their new role and environment. The long lead time also allows sufficient time for the outgoing officer to prepare for the turnover, including preparing an effective handover for the officer that will replace them.

5.20 The ANAO observed that DFAT Vanuatu did not have a systematic handover process; it was reliant on the departing staff member to prepare briefing notes for their replacement. To minimise the potential disruption caused by staff turnover, there would be merit in DFAT documenting the minimum requirements, such as requiring handover briefing notes, and expectations for familiarisation activities for A-based staff beginning a posting.

Reporting arrangements

5.21 Effective internal management reporting, which senior executives rely on to monitor a program of activities and support decision-making processes, should be consistent with external reporting. Furthermore, external reporting should be transparent and allow the Parliament, the public and key stakeholders to assess the effectiveness of an agency’s operations and how public funds are being used to achieve government policy objectives.

Internal management reporting

5.22 High-level internal reporting on the whole of the aid program centres on the Aid Operations Report. The report is produced quarterly and is the primary means by which DFAT executive are informed of the policy alignment, effectiveness and efficiency of the Australian aid program. The Aid Operations Report was reinstated following a hiatus from March 2013 to June 2014. The Vanuatu bilateral program is referred to in the Aid Operations Report on an exception basis. For example, the October 2014 report noted that the draft AIP for the Vanuatu program had been considered by the Aid Investment Committee since the previous report in June 2014. The ANAO considers the re-introduction of reporting on the Australian aid program to be a positive step for the executive oversight of the program.

5.23 At the Pacific regional and country program level, progress is captured in a monthly dashboard report.⁶⁵ The monthly report collates data held in AidWorks and refers to the Vanuatu bilateral aid program as necessary. For example, with respect to the Vanuatu program, the September 2014 report included the timeframes relating to the AIP and Aid Program Performance Report (APPR), expense commitments as a proportion of budget and percentage that could be considered Aid for Trade expenses, QAI ratings, and the change in the number of initiatives in the past quarter.⁶⁶

5.24 The monthly dashboard reports are prepared by the division’s Performance Quality & Risk team for the monthly divisional aid operations meeting, which is chaired by the First Assistant Secretary Pacific Division. The monthly meetings are attended by the Assistant Secretaries, and via teleconference the Heads of Mission and senior aid personnel in-country. DFAT considers the dashboard reporting and monthly meetings to be better practice and, as a result, a further two divisions have adopted the dashboard reporting process. In addition, the key audience for the report, the Executive of the Pacific Division, advised the ANAO that they consider the dashboard reporting to be informative and useful. The current internal reporting arrangements provide DFAT executive with reasonable oversight of the program.

External reporting

5.25 In the current aid policy, Australian aid: promoting prosperity, reducing poverty, enhancing stability released in June 2014, DFAT committed to high standards of transparency and accountability in the management of the Australian aid program. DFAT has several methods for externally reporting on progress and performance of the Australian aid program, including via its website, the Performance of Australian Aid report, Portfolio Budget Statements and its Annual Report.

5.26 The aid policy noted that DFAT would give effect to the commitment to transparency and accountability through publishing information on the aid program on the DFAT website, including policies, plans, results, evaluations and research. DFAT’s website contains a section on the Vanuatu aid program. The section includes the total expenditure of Australian aid to Vanuatu (including funding for regional and global programs), and activities and achievements in each development sector. However, the website now contains fewer and less current program specific documentation than prior to May 2014. For example, the Direct Funding Agreement with the GoV for the Health program and the Health Delivery Strategy are no longer available on the website.

5.27 In February 2015, DFAT published its first annual Performance of Australian Aid report.⁶⁷ The report, which covered the 2013–14 year, described the performance of country and regional programs according to four geographic groups, with Vanuatu located in the Pacific group. The report did not comment specifically on the performance of the Vanuatu country program.

5.28 Similarly, due to the comparatively small size of the Vanuatu aid program⁶⁸ it is not specifically mentioned in DFAT’s Portfolio Budget Statements. The Vanuatu aid program sits within Program 1.6: Official Development Assistance—Papua New Guinea and Pacific. However, additional information was provided in the form of budget highlights, posted to DFAT’s website, and the release of The 2014-15 development assistance budget: a summary in June 2014, which included a section on Vanuatu outlining where aid funds would be directed in 2014–15.

5.29 DFAT’s Annual Report 2013–14 summarised, on one page, details of the Vanuatu bilateral aid program, including program finances and international development measures (for instance, gross domestic product per capita). The yearly activities and achievements of each development sector were presented, and correlate with the ANAO’s findings. However, the report did not provide an assessment of the overall performance of the Vanuatu aid program and the achievements were presented without context so it was not possible to assess performance against set targets. For instance, the Annual Report mentioned that DFAT assisted 3737 women survivors of violence to access counselling and support services through the VWC.⁶⁹ However, the report did not include results for previous years, making it difficult to compare and assess the initiative’s trend performance.

Aid Program Performance Report

5.30 Annually, DFAT also produces an APPR for each country and regional program, including the Vanuatu bilateral aid program. The APPRs are intended to strengthen program management, demonstrate accountability and improve effectiveness. The Vanuatu APPR includes key messages, Vanuatu context and yearly expenditure, and detailed the progress on initiative objectives, program quality and management (including risks) and management responses.

5.31 Since the Partnership for Development (PFD) was signed in May 2009, the five Vanuatu APPRs have all reported progress against the five objectives outlined in the Partnership for Development.⁷⁰ The ANAO analysed these progress ratings in the APPRs from 2009 to 2013–14 (see Table 5.2).

Table 5.2: Aid Program Performance Report progress ratings for the Partnership for Development objectives, from 2009 to 2013–14

Source: ANAO analysis of the Vanuatu APPRs from 2009 to 2013–14.

Note 1: Rating definitions:

 • Green: Progress is as expected for this point in time and it is likely that the objective will be achieved. Standard program management practices are sufficient.

 • Amber: Progress is somewhat less than expected for this point in time and restorative action will be necessary if the objective is to be achieved. Close performance monitoring is recommended.

 • Red: Progress is significantly less than expected for this point in time and the objective is not likely to be met given available resources and priorities. Recasting the objective may be required.

Note 2: The wording of the objective has changed from the PFD objective of ‘enhancing access to, and quality of, legal and police services to the Government and citizens of Vanuatu’, but the intent is unchanged.

5.32 The ratings in Table 5.2 correlate with the ANAO’s findings. As previously discussed, concerns about the support provided in the education sector prompted a redesign of the program, which has resulted in improved ratings. The Health initiative was consistently rated as underperforming (‘amber’). Although the recommended management response to an ‘amber’ rating is close performance monitoring, as discussed in Chapter 4, the Health sector has not been closely monitored.

5.33 While the report assesses progress against the individual objectives, it does not provide a consolidated view about the effectiveness of the program overall. Developing, as part of the AIP process, performance indicators and targets would allow DFAT to measure and publicly report on the extent to which its strategic objectives and the objective for the Vanuatu bilateral program as a whole are being achieved.

5.34 The ANAO considers that the details provided in the 2013–14 APPR were soundly based. However, similar to the Annual Report, material in the APPR was provided without context. Furthermore, it is not possible to reconcile all the statements made in the Annual Report with those made in the APPR. For example, the Annual Report statement on the number of women survivors of violence with access to counselling and support services as a result of the VWC cannot be reconciled with the APPR 2013–14 as the APPR reports on the total number of people (including men) who were provided with counselling and support services by the VWC.

5.35 In terms of risk reporting, the maturity of the risk management monitoring is rated in the 2013–14 APPR as ‘mature’ for the each objective and the program as a whole. The definition of a ‘mature’ rating is when risks are discussed at least monthly and the risk registers are updated at least quarterly. As discussed in Chapter 2, the ANAO found that, prior to April 2015, DFAT’s risk registers for the reviewed investments had not been updated since November 2013.

5.36 In summary, although DFAT has publicly committed the management of the Australian aid program to a high standard of transparency and accountability, this is yet to be fully realised given the reduction in current initiative specific documentation available on the DFAT website. The APPRs are an adequate mechanism for reporting on the progress of the Vanuatu bilateral program. However, the lack of publicly available targets means that the useability of DFAT’s external reporting to assess the performance of the Vanuatu bilateral aid program is reduced, and the data reported is not always up-to-date.

Conclusion

5.37 The division of responsibilities between DFAT staff in Canberra and Vanuatu, while not formally documented, is well understood and implemented by staff. Similarly, escalation procedures were generally understood and adhered to in practice. However, DFAT does not effectively coordinate learning and development opportunities or maintain complete staff training records. DFAT could strengthen its administrative arrangements by reviewing the requirements and adequacy of the training provided and better coordinating learning and development activities, including maintaining accurate training records. Furthermore, while the Evaluation Capacity Building Program was well received by all DFAT staff in Vanuatu, evaluating the impact of the course on the quality of DFAT’s monitoring and evaluation of initiatives would add support to the continuation of the program.

5.38 The internal reporting by the Pacific Division is considered better practice by DFAT and has been adopted by other Divisions, and the re-instatement of the Aid Operations Report is a positive step for the executive oversight of the Australian aid program. However, while external reporting is intended to allow the Parliament and public to assess the effectiveness of the Australian aid program, the usefulness of DFAT’s external reporting is limited as it does not provide sufficient context within which to assess the performance of the program. Furthermore, limitations in performance benchmarks and the currency of data reported by DFAT reduces the usefulness of external reporting for assessing the performance of the Vanuatu aid program.

Appendices

Please refer to the attached PDF for the Appendices:

• Appendix 1: Entity Response
• Appendix 2: Map of Vanuatu
• Appendix 3: Australia’s Bilateral Aid Investments in Vanuatu, 2013–14
• Appendix 4: Five Initiatives Examined by the ANAO

Abbreviations