Background

Australian government assistance to East Timor

Over the past four years, Australia has provided substantial assistance to the restoration and maintenance of security and the economic and social recovery of East Timor. This assistance has been a significant part of the international response to the destruction of property and the displacement of the East Timorese people after the August 1999 ballot on separation from Indonesia.

The main components of Australian government assistance have been security assistance, aid and policing assistance. The aid component, estimated to cost $235 million over the five years 1999–2000 to 2003–04, is managed by the Australian Agency for International Development (AusAID).

Audit objective

The objective of the audit was to assess AusAID's planning for, and management of, the delivery of aid to East Timor.

Initial response to the 1999 crisis

The violence that followed the August 1999 ballot created an immediate need for humanitarian assistance, particularly food, water, shelter and medical supplies. AusAID's pre-crisis monitoring of developments in East Timor placed it in a good position to respond quickly and appropriately to this need.

At the time of the response there were no specific, current AusAID guidelines for responding to emergency and humanitarian crises. However, AusAID was able to rely on the expertise of key staff, who were familiar with United Nations disaster coordination arrangements, to ensure that appropriate steps were taken. AusAID has since issued guidelines for emergency and humanitarian assistance.

Australia's contribution of $37 million in 1999–2000 was the largest ever AusAID contribution to a humanitarian crisis. Most of this contribution was provided through four assistance packages, which were approved by AusAID and by the Minister for Foreign Affairs between September and November 1999. Delivery of the assistance was primarily through funding the activities of United Nations agencies.

In addition to providing a substantial financial contribution, AusAID actively facilitated the international response, providing significant support for planning, coordination, administration and implementation. Stakeholders confirmed the important part played by AusAID in the response to the crisis.

The international response concluded in mid 2000, with a transition to reconstruction and development assistance. The ANAO did not seek to assess the effectiveness of the international response as this involved a range of donors. However, two external reviews indicated that the response was effective and well coordinated. Stakeholders consulted by the ANAO supported this view.

Post-crisis country strategy

Strategy development and focus

AusAID developed an interim country strategy for East Timor through extensive and effective consultations with Australian and overseas stakeholders over a six-month period during 2000. The ANAO found that the timing of the preparation of the strategy was appropriate. It followed an easing in the need for emergency relief and enabled AusAID to take into account assistance proposals of other donors, which became clearer during 2000.

AusAID's East Timor country strategy articulated Australia's aid objectives and provided the framework for the management and delivery of aid during 2001 and 2002. It identified East Timor's development needs and recognised its limited capacity to manage assistance and sustain aid outcomes. It also utilised lessons learned from previous post-conflict situations.

The strategy acknowledged the role of non-government organisations (NGOs) in providing Australian aid to East Timor, but did not explicitly address the role of other Commonwealth and State government agencies or how the Commonwealth contribution would be coordinated. It would be appropriate to ensure that future strategies address such issues.

In developing the strategy, AusAID considered the appropriate balance between different means of providing assistance which would promote Australian aid objectives in East Timor. The main choice was between ‘bilateral aid', that is, aid provided directly by Australia, and aid channelled to East Timor trust funds, and similar arrangements, managed by multilateral agencies (‘multilateral aid'). Following an initial focus on multilateral aid, bilateral aid is now the main focus of Australian assistance to East Timor.

The strategy provided a sound and comprehensive basis for planning and managing bilateral aid at the sectoral level. The choice of priority sectors—governance; education and training; water supply and sanitation; rural development; and health—and specific priorities within each sector, were based on expert assessments of East Timor's development needs.

Risk management

The East Timor interim country strategy set out a risk management matrix. The matrix identified strategic risks to the aid program, such as the potential lack of East Timorese capacity and possible lack of donor coordination, and addressed ways of managing those risks. However, it was not supported by an assessment of each risk in terms of its likelihood and the severity of its consequences, to assist in prioritising risks and their treatment. In addition, the matrix had not been updated since the finalisation of the interim strategy in October 2000.

AusAID's strategic risk management framework for country programs is underpinned by operational plans for activity monitoring, referred to as Country Program Risk Assessment and Monitoring Plans (CPRAMPs). However, a CPRAMP was not prepared for East Timor until March 2003, more than two years after the country strategy was in place.

Notwithstanding the need for AusAID to address these shortcomings in the risk management process, the ANAO found that AusAID has put considerable emphasis on the day-to-day management of risks associated with East Timor aid. The main means of monitoring risks included project site visits; independent project reviews; and regular reporting by Australian contractors.

Performance management

The East Timor interim country strategy included a performance monitoring plan, which set out program objectives and linked them to what are described as ‘outputs' and ‘performance indicators'. However, this plan had some limitations, which affected its usefulness in monitoring program performance. For example, outputs (goods and services) to be delivered were not clearly stated for each program objective. Where outputs were identified, they were not specified in measurable terms.

In practice, AusAID did not use the performance monitoring plan to assess the strategy's effectiveness. The focus of AusAID's monitoring was on the individual components of the country program. However, the lack of overall performance monitoring at the country program level is a shortcoming, which has meant that AusAID is not well placed to demonstrate clearly that the strategy has achieved the overall desired aid objectives.

The ANAO recognises that it would have been difficult, at least initially, to establish performance measures and targets for some outputs and outcomes, given the timeframe and context within which the interim strategy was developed. However, the ANAO considers that it would have been beneficial to add appropriate measures and targets subsequently as the program components were being designed and implemented. This would have provided a more robust and well articulated performance monitoring plan, which would have enhanced AusAID's capacity for performance management.

Review of the interim country strategy

AusAID commissioned a team of consultants to review the interim country strategy in October 2002. The review had a number of strengths—including its independence from AusAID and involving a team with appropriate knowledge and experience of East Timor. AusAID provided the team with pre-departure briefings and documentation on the East Timor aid program. However, there were some limitations to the review process, including limited time for the team to absorb a substantial part of the briefing material, which could have been avoided by earlier preparation by AusAID.

The review team highlighted the significant contributions of Australian aid to capacity-building in East Timor government administration. It also highlighted AusAID's contribution to peacebuilding, reconciliation and the development of NGO capacities.

The review team found that AusAID's bilateral activities aimed at the restoration of basic services had achieved some success. However, it noted the slow pace of implementation of some of these activities and the considerable challenges facing aid delivery in some sectors. The review team also found that, on balance, the multilateral trust fund mechanism made an important contribution during East Timor's transition to independence.

International donor coordination

The international donor community recognised that effective coordination between donors would be crucial to the success of the international response to the East Timor crisis. Australia was an active participant in international donor conferences, convened between December 1999 and May 2002. It hosted one of these meetings, in Canberra, in June 2001. These meetings facilitated donor consultation on East Timor's recovery and transition to independence, as well as donor approval and monitoring of multilateral reconstruction projects.

The ANAO found that AusAID's strategic preparation for these donor meetings was well managed. AusAID identified objectives to be pursued at each donor conference and prepared briefing material on key topics for the Australian delegation. AusAID also maintained a consistent team of senior officials to represent Australia's interests at these donor meetings. On occasions, AusAID would have benefited from further background preparatory work and more detailed reporting back on some issues discussed at the meetings.

The ANAO found that AusAID actively advanced Australia's views on East Timor aid and development issues through these forums. AusAID's dialogue with other stakeholders made a positive contribution to successful outcomes in relation to its broad aims of encouraging wide international support for East Timor and continued donor engagement and aid coordination. On some specific issues, Australia played an important role in influencing developments.

Multilateral assistance

The Trust Fund for East Timor (TFET) was established by the World Bank for East Timor's reconstruction and development. Australia has contributed some US$12 million to this multilateral trust fund. The World Bank and the Asian Development Bank have managed individual TFET projects.

TFET donors have been provided with six-monthly reports containing summary information on TFET, and information on individual TFET projects. However, the ANAO found that AusAID would be in a better position to review TFET performance if it could obtain enhanced project reporting from the development banks. In particular, an overall assessment (rating) of the status of each project could be included in the reports. In response to the draft audit report, the World Bank advised that it has now improved project reporting to TFET donors.

AusAID actively participated in a wide range of joint project supervision missions, through which donors and other stakeholders jointly assessed the TFET and other aid activities. These missions have been a valuable mechanism for providing Australia and other donors with information on the performance of TFET projects. AusAID has also used mission reports for monitoring sectoral developments and identifying future aid opportunities.

AusAID has assessed the performance of the World Bank and the Asian Development Bank in managing TFET. AusAID canvassed some concerns regarding the Asian Development Bank's assistance to the roads and power sectors with the banks and other stakeholders through donor coordination processes. The Asian Development Bank advised the ANAO that these concerns had been brought to the Bank's attention at appropriate times and this had helped resolve some of the issues in these sectors. Following further TFET and other donor assistance, some performance improvements have been achieved in the power sector. AusAID has assisted with the provision of expert advice on the main issues of concern.

The TFET trust fund arrangements produced major benefits, including the mobilisation and consolidation of considerable donor resources into a common reconstruction fund. Australian assistance through TFET contributed to East Timor's social and economic recovery within a relatively short timeframe, particularly through the reconstruction of schools, health facilities and transportation infrastructure.

TFET was successful in preparing projects for implementation more rapidly than is the norm for World Bank projects. However, project management and implementation difficulties were greater than anticipated by the development banks, particularly in the early phase of East Timor's reconstruction. TFET financial disbursements and project progress have been much slower than originally planned.

Bilateral assistance

Bilateral assistance has comprised both shorter-term transitional assistance activities and medium-term development assistance activities. The former were designed and implemented rapidly in 1999 and 2000 to meet East Timor's immediate needs. The latter were designed more methodically, under less pressing time constraints, from 2001 onwards. The ANAO sampled 14 bilateral activities covering each of AusAID's key aid sectors.

Activity planning and design

The ANAO found that sampled transitional activities were prepared and approved within shorter than normal design timeframes, to enable AusAID to deliver assistance more rapidly. This meant that they did not go through AusAID's full design processes. Reduced design processes included limited stakeholder consultation and risk management planning.

One consequence was that outputs/outcomes and performance indicators were not specified for most activities examined. While this was understandable at the design stage of these activities, AusAID did not take the opportunity to subsequently identify and use suitable outcomes/outputs and performance indicators. This occurred notwithstanding that some activities were substantial and implemented over more than a year.

These limitations hampered AusAID's ability to assess and manage performance. It would have been beneficial for AusAID to have addressed this issue early in the implementation of transitional activities.

The ANAO found that sampled medium-term activities generally met the attributes of good planning and design. Performance indicators were specified for all of these activities. However, the indicators had a number of shortcomings. For example, they did not adequately measure or assess the quality of outputs or the effectiveness of activities, which are critical for assessing performance. Most indicators did not incorporate targets; as a result it was unclear what would constitute good performance.

The ANAO acknowledges that the performance measurement of aid activities is challenging. However, the ANAO identified several ways in which the indicators employed could have been strengthened to better monitor the performance of East Timor activities. This is an aspect of activity management that warrants further attention by AusAID for future projects.

Monitoring of activity implementation and performance

The ANAO found that there was regular, and generally effective, communication and exchange of information between AusAID staff in Canberra and in Dili during the delivery of aid activities. In addition, contractors provided regular reports on activity progress and performance, although there is a need for AusAID to strengthen its efforts to ensure that contractor reporting requirements are more focused on performance achieved.

AusAID regularly monitored activity expenditure against budget. It also monitored activity progress, using qualitative information. However, it made only limited use of performance indicators in assessing activity performance, partly reflecting the design problems noted above. AusAID's activity monitoring should incorporate preferably quantitative performance measurement, to enable more rigorous assessment against planned activity outcomes and outputs.

Risk management and fraud awareness

AusAID prepared a risk management plan for each medium-term activity. In the case of most transitional activities, risks were identified in other activity documentation. However, the ANAO found risk management strategies in plans, or other related documentation, were not systematically reviewed or updated during the implementation of the majority of activities.

AusAID is aware of the potential for fraud in aid activities in East Timor and has explicitly addressed fraud risk in activity monitoring. Two instances of suspected fraud were identified and investigated by AusAID. These involved grants for relatively small amounts under the East Timor Community Assistance Scheme (ETCAS). AusAID now has tighter procedures for assessing ETCAS applications and for monitoring activity progress.

Activity achievements

Audit evidence indicated that, overall, Australia's bilateral aid has made an important contribution to East Timor's reconstruction and development. East Timor government stakeholders supported this conclusion. They indicated to the ANAO that Australia's bilateral assistance program has been of high quality and has delivered positive outcomes for East Timor in a number of high priority sectors and activities.

Weaknesses in AusAID's performance information, for both transitional and medium-term activities, limited the ANAO's capacity to assess activity performance. Notwithstanding this, the ANAO considers that the individual bilateral aid activities have had positive results and mostly achieved planned outcomes and outputs, in a difficult operating environment.

Sustainability of activities

AusAID requires that bilateral aid activities have a sustainability strategy for ensuring that the benefits of the activity are maintained to the maximum extent possible, when Australian assistance ends. The ANAO found that a sustainability strategy was included in the designs of most medium-term activities. Sustainability issues were also generally considered in the designs of most transitional activities examined.

Notwithstanding the attention given to sustainability issues in activity designs, the ANAO observed that the continued sustainability of many activities examined was problematic. Likely sustainability of some 30 per cent of activities in the audit sample was rated by AusAID as ‘weak' or ‘marginally satisfactory'. The extent of this risk was also evident in ANAO visits to activities and discussions with stakeholders in East Timor. Reviews of East Timor aid activities commissioned by AusAID have also consistently identified sustainability as an area of relative weakness.

Overall conclusion

The ANAO concluded that AusAID made a significant and timely contribution to the international response to the humanitarian crisis that followed the ballot on separation from Indonesia. In addition to providing a substantial financial contribution, AusAID provided key planning and logistical support elements that helped to make the international response timely and effective.

AusAID's humanitarian response was followed by a well planned and targeted interim strategy for the delivery of post-crisis assistance to East Timor. This interim strategy took into account stakeholder views, East Timor's development needs and the activities of other donors. However, there were some administrative shortcomings.

In particular, risk management did not include an assessment of the likelihood and consequences of individual risks, to assist in prioritising risks and their treatment. Identified risks were not regularly updated to reflect changing conditions in East Timor. In addition, limitations in performance management at the country program level hampered AusAID's ability to assess whether overall desired aid objectives have been met.

AusAID has played an active role in the successful coordination of international assistance to East Timor, and had adequate mechanisms to assess whether Australia's financial contributions to the Trust Fund for East Timor have been used effectively. On the basis of these assessments and audit examination, the TFET trust fund arrangements produced major benefits for AusAID and other stakeholders. However, unanticipated difficulties and delays were encountered by the development banks in the implementation of TFET projects.

AusAID's management of most aspects of the design and implementation of individual bilateral aid activities has been sound, although the ANAO identified limitations in some supporting structures. In particular, the quality of performance indicators and associated targets can be improved to better monitor the performance of East Timor activities. In addition, the relatively high inherent risk in delivering aid to East Timor warrants stricter adherence to AusAID requirements for preparing and updating risk management plans for activities.

Australian bilateral aid has made an important contribution to East Timor's reconstruction and development, by providing assistance in a number of high priority sectors. Individual aid activities have mostly achieved planned outcomes and outputs. However, the sustainability of many of these activities is at risk.

Recommendations and AusAID response

The ANAO made four recommendations aimed at strengthening AusAID's risk management and performance management in its delivery of aid to East Timor. AusAID agreed with all four recommendations in the report.

AusAID response

AusAID welcomes the ANAO review findings of aid to East Timor which confirm the Australian aid program's significant and timely contribution not only to the humanitarian crisis following the ballot in 1999, but also to East Timor's subsequent reconstruction and development. Importantly, the ANAO recognises that despite the difficult operating environment, AusAID actively facilitated the international response and provided significant support for planning, coordination and implementation of international assistance for East Timor's reconstruction.

AusAID agrees with the audit report recommendations and acknowledges the importance of risk and performance management in administration of the aid program. The report reinforces the direction of work under way to improve risk and performance management systems, thus contributing to stronger development outcomes and greater sustainability.