Introduction

1. The Department of Veterans’ Affairs (DVA) is part of the Defence portfolio and is responsible for developing, implementing and administering government policy and programs to fulfil Australia’s obligations to the veteran community.¹ The department’s day-to-day activities are directed by two Commissions—the Repatriation Commission² and the Military Rehabilitation and Compensation Commission.³ The department advises the commissions on policies and programs for beneficiaries, and administers these policies and programs. It also conducts commemorative programs to acknowledge the service and sacrifice of Australian servicemen and women.

2. DVA is administering $12 billion of expenses in 2011–12 across three outcomes: Compensation and Support, Health and Commemorations.⁴ At 30 June 2011, there were 348 929 Veterans’ Entitlements Act 1986 beneficiaries and 15 469 Military Rehabilitation and Compensation Act 2004 and Safety, Rehabilitation and Compensation Act 1988 beneficiaries who had received benefits in the past two years.⁵

3. Each year DVA undertakes millions of transactions with both DVA beneficiaries and the general public.⁶ These transactions are conducted through telephone, letters, emails, and face to-face contact with DVA staff across Australia. All staff in DVA are responsible for accepting, recording and dealing with complaints, compliments and suggestions for improvement that are received by the department⁷, with 2167 complaints and 940 compliments recorded in 2010–11. Overall, the number of complaints recorded by DVA is relatively low when compared to the volume of transactions.

4. Errors, misunderstandings, client dissatisfaction and unexpected problems occur in all administrative systems. Effective complaints handling assists in resolving problems before they become worse, providing a remedy to clients who have suffered disadvantage, and nurturing good relations between government agencies and the public. Complaints, and suggestions for improvement, also provide agencies with information about potential program weaknesses and service delivery faults. Conversely, compliments provide information about potential program and service delivery strengths, which can be applied more broadly in an agency, and used to motivate staff. Good administration involves regular review of existing programs, and the lessons learned from client feedback can be used to continually improve processes and the standard of services provided to clients.

5. In 2010, DVA revised its complaints and feedback management policy and implemented new complaints handling arrangements following a number of reviews⁸ which identified that the department was not capturing and reporting a large proportion of the complaints and compliments received. The revised policy has been publicised among ex-service organisations and a link to the policy is available on the home page of the department’s web site.⁹

6. For reasons of client confidentiality, complaints and compliments which are received by the Veterans and Veterans Families Counselling Service (VVCS) are recorded and managed separately from complaints and compliments for the department as a whole.¹⁰ Only VVCS staff have access to VVCS-related feedback data.

Audit objectives and scope

7. The audit objective was to assess the effectiveness of the Department of Veterans’ Affairs management of complaints and other feedback to support service delivery. The audit criteria were that DVA has:

• a well-designed framework for managing complaints and other feedback;
• effective processes and practices to manage complaints; and
• appropriately analysed complaints to inform service delivery.

8. The audit considered guidance on effective complaints handing arrangements provided in the Commonwealth Ombudsman’s Better Practice Guide to Complaint Handling of April 2009 and in Australian Standard AS ISO 10002—2006, Customer Satisfaction—Guidelines for complaints handling in organisations.

9. Complaints to the Veterans’ Review Board and Defence Service Homes Insurance are handled separately from complaints to DVA and the VVCS and their management was not included in this audit.

Overall conclusion

10. DVA has recently revised its complaints and feedback management framework in response to external review and in recognition that complaints, compliments and suggestions are useful sources of information which can contribute to the improvement of client services.

11. DVA has adopted a well-designed framework for managing complaints and other feedback, which took effect from 1 July 2010. A new unit has been given responsibility for the overall management of complaints and other feedback and the development of a revised Complaints and Feedback Management Policy. The revised policy has been brought to the attention of stakeholders and is readily accessible on DVA’s website. The department has implemented effective processes to handle complaints and other feedback, with the revised management framework supported by an upgraded computer application for recording complaints and compliments (the Complaints and Feedback Management System (CFMS)). DVA staff have been provided with appropriate guidance and training on the revised policy, processes and CFMS, as a means of raising awareness and bedding-down the framework.

12. To date, DVA has undertaken little analysis of its feedback data to identify and address the main causes of complaints and further improve client service. While DVA is planning to conduct such analysis in the near future, the effectiveness of the process will be limited by the data currently recorded on the CFMS, which does not indicate the main cause of the complaint or its exact nature. There would be benefit in DVA reviewing the recording of complaints and other feedback data on the CFMS, to better support the information needs of DVA business groups.

13. Further opportunities to improve the administration and integrity of the framework could be considered by DVA to overcome the continued level of under-reporting of complaints and to better protect the privacy of complaints records in the CFMS. While there has been a significant increase in the level of awareness among DVA staff of the department’s complaints management policy and the importance of recording complaints and compliments, not all instances are recorded in the CFMS.¹¹ There would be benefit in estimating the level of under-reporting, as a first step in considering appropriate remedial action to improve record keeping.¹²

14. DVA has also emphasised to staff the importance of privacy protection and the need for a legitimate business reason to access records in the CFMS. While DVA considers that the risk of privacy breaches is low, there is scope to improve the protection of complaints records in the CFMS by introducing an effective audit trail of staff access to records, which is monitored in accordance with DVA’s information technology protocols.

15. To maintain the integrity of the framework, DVA performs quality assurance checks on data on completed complaints, which have identified a high error rate of 53 per cent of CFMS records in 2010–11. However, DVA’s internal reporting does not address the reasons for the errors appearing in CFMS records or aggregate the results for each month as an aid to management. Information on these matters would assist DVA to more readily identify the prime causes of errors and so enable better targeting of training needs. There would also be benefit in DVA’s quality assurance process examining DVA’s recording of whether complainants were satisfied with the process, as the ANAO’s analysis found that these satisfaction rating assessments were not accurate or reliable.¹³

16. DVA’s business processes and support to clients are enhanced by ensuring that the CFMS is both robust and accessible. Improving the search and reporting capabilities of the CFMS would provide DVA business groups with more flexibility to search the database and reduce the need to maintain separate local correspondence tracking systems (in the form of spreadsheets) to assist in tracking complaints. DVA business groups also indicated that there would be benefit in CFMS providing more detailed information on the service being complained about, particularly regarding the performance of contracted service providers, as an aid to monitoring performance and managing related contracts.

17. The ANAO has made three recommendations directed at supporting DVA’s service delivery by improving the department’s administration of the framework for managing complaints and compliments and the efficiency of the CFMS.

Key findings by chapter

Complaints and Feedback Management: Framework and Application (Chapter 2)

18. In deciding to redevelop its complaints and feedback management framework in 2009, DVA created a Veterans’ Standards and Complaints Management Team to lead policy development and support the overall management of complaints and other feedback within the department.¹⁴

19. A revised complaints and feedback management policy supports the implementation of the revised framework by setting out standard definitions of complaints and compliments, establishing the cultural norm that DVA welcomes complaints and other feedback as a means of enhancing service delivery, describes the options available to complainants if they are dissatisfied with DVA's responses and establishes timeframes for responding to complaints. DVA has also developed detailed administrative arrangements for handling complaints and other feedback that are documented in written guidance to staff that is readily available on the DVA intranet.¹⁵ The revised framework is further supported by training provided to approximately one-third of DVA staff.

20. All DVA business units consulted during the course of the audit considered that their staff had a heightened awareness of the complaints and other feedback management policy and procedures following the implementation of the new policy. However, to maintain awareness by its staff of the complaints and other feedback management arrangements and of the importance of recording all complaints and compliments, there would be benefit in DVA implementing ongoing awareness training for new staff and for relevant staff who have still to undertake the training.

21. There is ready access by clients and other persons to DVA’s complaints and other feedback policy though the DVA website and there are many channels available (in-person, by telephone, in writing, through an intermediary, such as a Member of Parliament or the Commonwealth Ombudsman) to provide feedback. There is also a dedicated complaints hotline, with calls to this hotline referred to the responsible business area for attention. Most feedback is provided by telephone.

22. DVA’s policy requires that complaints be acknowledged within one day (except in the case of complaints which are received by post, when they must be acknowledged within five days) and be resolved within a period of 28 days from the date of the complaint (except for ministerial correspondence which can exceed the 28 day standard). This standard is based on DVA’s service charter, which states that DVA will strive to action or acknowledge correspondence within 28 days. In 2010–11, 76 per cent of DVA complaints and 52 per cent of VVCS complaints were recorded as having been resolved within the 28 day timeframe for resolution of complaints.

23. ANAO examined a sample of 40 overdue responses to assess the reasons for the delay. For 23 (or 43 per cent of) responses, it was not possible to determine the reasons for the delay as there was insufficient documentation. The reasons for the delay in other cases included delays in recording matters as finalised in the system or in obtaining information from the client, provider or other parts of DVA. To improve the accuracy of the information held by DVA and to help identify the reasons why complaints may not be finalised within the 28 day timeliness standard, there would be benefit in DVA considering amendments to the CFMS to require reasons for delayed responses to complaints to be recorded and to indicate when a response has been provided, as distinct from when a complaint record has been closed.

24. Recognising that the expected level of service will not be met on all occasions, some organisations, such as Medicare Australia, set performance targets based on the standard; that is, they seek to meet the target in a specified percentage of cases. This target can be reviewed regularly and adjusted as the organisation’s performance against the standard improves, so helping the organisation improve its service delivery performance. There would be merit in DVA considering such an approach, to assist it in further refining its complaints management system and promoting continuous improvement.

Analysis and Reporting of Complaints and Other Feedback (Chapter 3)

25. Complaints and other feedback provide a potentially valuable source of information to an agency on the performance of their service delivery and can identify areas of possible weakness for future improvement.

26. DVA has undertaken little analysis of the complaints data to assist it to identify ways of improving its services, and DVA business groups have indicated that they would welcome more detailed information about complaints, such as the main areas of their responsibilities subject to complaint. Collecting information of this sort would require more detailed fields to be included in the CFMS database. While major changes to the CFMS may not be warranted until DVA has made a decision on the future of the system¹⁶, there would be benefit in DVA considering the type of information it would need to collect on the reasons for complaints and compliments in order to meet the preferences of business groups.

27. Monthly reports are provided to DVA’s Executive Management Group on the numbers of complaints and compliments received during the month and on key aspects of these feedback data, such as the timeliness of resolving complaints. Information is also included in the department’s annual reports to the Parliament, although these reports have to date not included information on complaints to the VVCS. DVA has indicated that it will include VVCS complaints in its 2011–12 annual report, and there would also be benefit in considering the inclusion of information on the timeliness of resolving complaints.

Complaints and Feedback Management System (Chapter 4)

28. The CFMS is a relatively simple database, which only records data on complaints and compliments received by DVA. Suggestions relating to the improvement of DVA services or policy matters are not currently recorded in the CFMS. This limits DVA’s ability to analyse possible areas for service improvement or the policy implications arising from them. There would be benefit in DVA considering amendments to its complaints and feedback management policy and processes to include such suggestions for improvement in the CFMS.

29. DVA has emphasised to its staff that they must have a legitimate business reason to access records in the CFMS. However, the fact that DVA staff members have access to all DVA-related complaints records¹⁷ and the absence of an audit trail to identify staff accessing the system, raises privacy issues. In the course of implementing the revised feedback handling arrangements, DVA considered including an audit trail in the CFMS (which it was advised would cost around $17 500), but determined that the overall risks associated with the CFMS not having an audit capability were low, and that no changes should be made to the system at that time. CFMS has been in use since mid-2010, and there would be benefit in reviewing the merits of including an audit trail, having regard to experience in the use of the system.

30. To improve the accuracy of CFMS records, DVA has implemented quality assurance protocols, under which a random sample of completed records is quality assured each month. DVA identified errors in 53 per cent of CFMS records for 2010–11. Forty-six per cent of records had errors relating to registration details, such as incorrect recording of the activity to which the feedback related, or recording an incorrect satisfaction rating. In two per cent of cases the records incorrectly contained sensitive information, such as the name of a staff member about whom a complaint had been made.¹⁸

31. DVA’s quality assurance data does not enable more detailed information on the reasons for errors to be derived, and DVA does not aggregate the monthly results. The availability of such information would assist DVA to more readily identify the causes of errors and so enable better targeting of training needs and improvements in service delivery.

32. The ANAO undertook an analysis of the quality of CFMS records using a sample of 77 cases. While the analysis found that follow-up action by DVA staff was generally thorough, it also indicated that:

• in 48 per cent of cases sampled, the satisfaction assessment was not strongly supported by the documented information¹⁹, raising questions about the accuracy of the CFMS satisfaction ratings recorded by DVA staff; and
• in 18 per cent of cases, details of the complaint were unclear from the record.²⁰

33. The ANAO also found that DVA could improve its quality assurance arrangements by improving its monitoring of duplicate records and including input controls in the CFMS to ensure that DVA staff complete all the required fields in the database.

34. To improve the accuracy of satisfaction outcome assessments, and the clarity of DVA’s records, the ANAO considers that there would be benefit in:

• DVA separately asking complainants whether they are satisfied with the outcome of their complaint and the way it was handled. These aspects are currently combined in the one question which does not enable a complainant to comment on each individual element;
• asking staff members to record in the database reasons for the satisfaction assessment (such as advice from the complainant); and
• reminding business groups of the need to maintain the integrity of CFMS records by including all relevant information, even in cases where business groups elect to use local internal correspondence tracking systems to manage complaints.²¹

DVA’s response to the audit

35. DVA agrees in principle with the three recommendations of the ANAO report.

36. The audit of DVA’s feedback system and processes was timely as the enhanced feedback system, revised policy and associated processes became operational in July 2010. DVA will use the audit’s findings, observations and recommendations to ensure the management processes continue to improve feedback mechanisms for DVA clients. The audit will also assist DVA to consider how the management of complaints and other feedback can be used effectively in association with the recent organisational restructure within DVA. The restructure has focused attention on the client experience and an important element will be the analysis of feedback data to develop strategies for future service delivery opportunities.

37. DVA agrees that feedback data, and the ability to analyse this information, can enhance aspects of service delivery that will improve interactions with the client. With this in mind, DVA will also examine its feedback system capabilities as noted in the audit and develop opportunities to improve the scope of current recording to reflect business needs.

Footnotes

[1]   DVA provides a range of services to veterans, war widows and widowers, current and former defence force members and their families and eligible members of the Australian Federal Police with overseas service.

[2]   Established in 1917, the Repatriation Commission is responsible for providing support to veterans under the Veterans’ Entitlements Act 1986 and delegates its powers to DVA to grant pensions and benefits, and treatment and other services to eligible clients. The department’s formal role in supporting the Repatriation Commission has evolved over the years in order to meet the needs of subsequent generations of veterans.

[3]   The Military Rehabilitation and Compensation Commission was established in 2004 to support the more recent generations of serving and ex‑serving members.

[4]   Department of Veterans’ Affairs, Budget Portfolio Statements 2011–12, Budget Related Paper No.1.5B, p. 75.

[5]   Department of Veterans’ Affairs, Annual Report 2010–11, pp. 14-15.

[6]   By way of illustration, in 2010–11, DVA received 3.6 million calls to its general enquiries phone numbers, provided around 2.8 million allied health services, and arranged almost 800 000 visits to eligible service providers.

[7] Department of Veterans’ Affairs, Complaints and Feedback Management Policy, <http://www.dva.gov.au/contact_us/Pages/feedback.aspx#policy> [accessed 14 November 2011].

[8]   These included ANAO Audit Report No. 28, 2008-09 Quality and Integrity of the Department of Veterans’ Affairs Income Support Records, which found that there was inaccurate recording and reporting of complaints and compliments in DVA’s former Feedback Management System.

[9]   DVA consulted the Office of the Commonwealth Ombudsman in redeveloping its complaints and feedback management policy.

[10]   VVCS is a counselling service administered by DVA which is kept operationally separate from the department to preserve client confidentiality.

[11]   Benchmarking against other Australian Government agencies—the Department of Human Services (Centrelink and child support programs) and the Australian Taxation Office—indicated to DVA that it could expect around 2500 to 3000 complaints per year, which is respectively 15 to 38 per cent higher than the 2167 recorded complaints in 2010–11.

[12]   Remedial action could include reinforcing to DVA staff the importance of recording all complaints and compliments in the CFMS.

[13]   DVA records indicate that 61 per cent of complainants were recorded as satisfied with the handling and outcome of their complaints in 2010–11, two per cent were dissatisfied and, in the remaining 37 per cent of complaints, it was not known whether the complainant was satisfied or dissatisfied. The ANAO found that only 52 per cent of complaints records supported the satisfaction rating.

[14]   The section is located in Sydney, and following a departmental restructure from 1 March 2012 is part of the Client and Commemorations Division. The section has an allocation of three staff members.

[15]   The VVCS used these guidelines as a basis for its handling of VVCS-related complaints in 2010-11, but has since adapted the guidelines to meet its own needs.

[16]  DVA is planning to examine the future of the CFMS in the context of a larger project―the Choice & Maintainability in Veterans’ Services project―through which DVA is upgrading its information technology systems, primarily to provide greater opportunities for DVA clients to access services from the department online.

[17]   With the exception of VVCS records, all DVA staff are expected to record complaints and compliments in the CFMS and for this reason have access to the system.

[18]   Where a person complains about a particular DVA staff member, DVA requires that the name of that staff member not be recorded in the CFMS.

[19]   Based on the ANAO assessment and the sample size of 77, this means that there is a 90 per cent level of confidence that between 39 per cent and 58 per cent of the CFMS satisfaction ratings will be incorrect.

[20]   Based on the ANAO assessment and the sample size of 77, this means that there is a 90 per cent level of confidence that between 12 per cent and 27 per cent of records do not contain clear details of the nature of the complaint.

[21]   Some business groups indicated that they continue to use their own internal correspondence tracking systems to handle complaints because the search facility in the CFMS is difficult to use.