Chapters 201 to 204

201. Contract out audits

Background

201.1 Under section 27 of the Auditor-General Act 1997 (Cth) (A-G Act), the Auditor-General, on behalf of the Commonwealth, may engage any person under contract to assist in the performance of any Auditor-General function.¹ For various reasons, the ANAO engages firms to undertake some performance audits or discrete parts of performance audits on his behalf.

Guidance

201.11 The mandatory requirements governing ANAO audits need to be made known to contractors via tender and contract documentation.

201.12 ANAO policy is to comply with the requirements of APES 110. Those requirements apply equally to contractor firms and their staff. ANAO policy also adds to these professional requirements; for example, ‘Provision of other services by ANAO Contractors to ANAO auditees’ includes additional prohibited services to the requirements of APES 110.

201.13 To help ensure contractors comply with ANAO Policy, paragraph 201.2 of this manual requires tenders and contracts for audit services to make provision for the ANAO Engagement Executive to make the contractor aware of policies and procedures the ANAO requires be followed on the audit. The procedural steps relevant to performance audits are included in the PASG Workflow, which contains corporate templates to be used during an audit.

201.14 The ANAO Engagement Executive is formally the ‘lead assurance practitioner’ under the ASAEs. In practice, when the ANAO contracts out an audit, or part of an audit, some of the duties of the lead assurance practitioner (e.g. supervising the conduct of fieldwork) may be fulfilled by the contactor partner (or equivalent).

201.15 In accordance with ASQM1, the ANAO’s review policies and procedures (including the ANAO Audit Manual and PASG Workflow) are determined on the basis that the work of less experienced team members is reviewed by more experienced team members.

201.16 The Engagement Executive shall comply with the requirement under the ANAO Auditing Standards to be satisfied that sufficient appropriate audit evidence has been obtained to support the conclusions reached and auditor’s report to be issued. The Engagement Executive’s review covers critical areas of judgement, especially those relating to difficult or contentious matters identified during the course of the engagement, significant risks and other areas the Engagement Executive has identified as important.

201.17 The audit contractor is responsible for ensuring that audit work undertaken on behalf of the Auditor-General is performed in accordance with professional standards and to have in place quality management policies and assurance procedures to be employed throughout the audit engagement.

201.18 Whereas financial audit work conducted by the Auditor-General is broadly similar to the assurance work performed by the external auditing profession in Australia, performance auditing is the almost exclusive domain of Auditors-General. While based upon the same assurance framework as compliance, controls and similar assurance work commonly performed by the private sector, there are extensive differences. The ANAO Engagement Executive should be conscious of their level of familiarity with the ANAO performance audit methodology in determining the extent of their involvement in the performance audit.

201.19 The expected milestone dates should be communicated to the contract partner at the commencement of the audit. Milestones usually include:

• preparation and approval of the AWP;
• entry meeting conducted with the auditee;
• regular progress review meetings with the relevant ANAO Engagement Executive during the period of fieldwork;
• progress review briefing to the ANAO Executive during the finalisation of fieldwork;
• provisions of draft Report Preparation Papers (RPPs) to the ANAO for clearance prior to issuance of RPPs to auditee(s) for comment;
• exit meeting conducted with the auditee;
• preparation of a formal draft audit report (section 19 report) and workshop/clearance of the report with the ANAO Executive prior to issuance to auditee(s) for comment;
• amendment of the section 19 report (as appropriate) for auditee responses, followed by provision of a final draft report to the Auditor-General for consideration/clearance;
• the final performance audit report to be available for tabling date; and
• audit wrap-up and lessons learnt meeting.

202. Role and responsibilities of Group Executive Directors

Guidance

202.2 The GEDs manage the ANAO’s Performance Audit Services Group (PASG) and engage in key elements of the conduct of a performance audit.

202.3 The strategic priorities of the GED are set out in the Service Group plan.

202.4 The procedural steps relevant to GEDs are included in the PASG Workflow, which contains corporate templates for staff to use during an audit.

202.5 For high risk audits, the GED is expected to have greater involvement in audit planning and execution, and to review significant matters arising during the audit.

203. Roles and responsibilities of the Engagement Executive

Background

203.1 The Engagement Executive is the ‘lead assurance practitioner’ for the purposes of the Standards on Assurance Engagements. The Engagement Executive is the head of a single administrative Branch in the Service Group that is responsible for delivering multiple performance audits in a financial year that support the delivery of the ANAO’s Annual Audit Work Program.

Guidance

203.4 The responsibilities in the auditing standards which this policy places on Engagement Executives include the following:

• the overall quality on each audit engagement to which the Engagement Executive is assigned;
  
  1. through their actions and appropriate messages to the engagement team, the Engagement Executive should emphasise the importance of compliance with ANAO Auditing Standards and quality management policies and procedures. In addition, the engagement team should have the ability to raise concerns without fear of reprisals. Quality is essential in performing audit engagements and the overall quality of the audit will be helped by ensuring that the audit is performed in a manner consistent with ANAO standard methodology;
• the engagement team’s compliance with relevant ethical requirements including APES 110;
  
  1. these include the principles of integrity, objectivity (including independence), professional competence and due care, confidentiality and professional behaviour. The Engagement Executive should remain alert, through observation and making enquiries as necessary, for evidence of breaches of relevant ethical requirements by members of the engagement team. If matters come to the Engagement Executive’s attention through the ANAO’s system of quality management or otherwise that indicate that members of the engagement team have breached relevant ethical requirements, the Engagement Executive, in consultation with others in the ANAO, shall determine the appropriate action²;
• the engagement team’s compliance with Chapter 5 Professional, ethical and independence requirements of the ANAO Audit Manual – Shared Content;
• the appropriate planning of the engagement consistent with ASAE 3500 Performance Engagements;
• being satisfied that those persons who are to perform the engagement collectively have the appropriate competence and capabilities, including having sufficient time to perform the engagement in accordance with relevant standards and applicable legal and regulatory requirements and enable an assurance report that is appropriate in the circumstances to be issued³;
• the direction, supervision and performance of the engagement consistent with professional and auditing standards and regulatory and legal requirements. The Engagement Executive should document the extent and timing of their reviews. Refer to the policy Direction, Supervision and Review (ANAO Audit Manual - Shared Content, paragraphs 8.2-8.4) for further guidance;
• that sufficient and appropriate audit evidence exists and is documented to support the conclusions reached and for the auditor’s report to be issued;
  
  1. in line with the ANAO policy Audit Documentation (ANAO Audit Manual - Shared Content, paragraphs 9.2-9.16) the only work that should be done after the issuing of the final report is that of an administrative nature.
• following appropriate procedures for consultations and differences of opinion and in particular ensure compliance with the ANAO policy on Differences of Opinion (ANAO Audit Manual - Shared Content, paragraphs 8.64-8.68);
• determine that an EQR has been appointed, as required by the ANAO Auditing Standards and ANAO policy (Refer to the Engagement Quality Review (ANAO Audit Manual - Shared Content, paragraph 8.42); and
• enough involvement in the audit engagement at appropriate stages throughout the engagement including attendance at key meetings, discussions with the engagement team, EQR and ANAO Executive and the completion of review of the planning and completion procedures at the appropriate stages of the audit.

203.5 Engagement Executives are assisted by the Audit Manager and team allocated to the engagement in fulfilling these responsibilities, including assisting in:

• consideration of whether the audit should be delivered through in-house or externally contracted resources;
• ensuring all audit related documentation is filed;
• planning the scope of work to produce the audit deliverables in the agreed timeframe;
• planning the time taken to prepare for, conduct and close the engagement;
• delivering in accordance with agreed timeframes;
• monitoring the costs associated with the audit, including recommending a variation to the budget if required;
• ensuring the quality of the audit deliverables;
• resourcing the audit to ensure that the audit team has the requisite skills to undertake the audit;
• the direction, review and supervision of audit team members;
• communication within the ANAO, among audit team members, and the entity being audited, and more broadly other interested parties;
• assessing and managing the operational and engagement risks associated with the audit;
• the procurement of any specialist resources and any associated contract management;
• documenting the agreed scope, timeliness and quality assurance arrangements in respect of any services required from SADA, FSASG or PSASG to contribute to a performance audit; and
• ensuring all persons engaged in the audit complete the required independence documentation and action is taken to manage any declared conflicts as required.

203.6 Audit Managers are expected to regularly monitor progress against established audit milestones and complete the actual date that audit milestones are achieved in the ANAO’s time monitoring systems in a timely manner. The data held in those systems forms the basis of reports to ANAO senior executives.

203.7 When engaging with the entity being audited, especially on difficult or contentious matters, the Engagement Executive and audit team should ensure that a professional and productive approach is taken. This includes, for example, trying to understand the audited entity’s circumstances, operating environment and point of view.

203.8 The Engagement Executive should be aware of any risks to audit timeliness and budget, and escalate these as soon as practicable.

203.9 The procedural steps involving the Engagement Executive are included in the PASG Workflow. As outlined in the PASG Workflow, the level of responsibility differs for audits with different risk ratings.

204. Role and responsibilities of the SADA Executive

Background

204.1 This policy sets out the responsibilities of the SADA Executive in a performance audit or any other PASG assurance engagement that engages the specialist skills of the SADA Group.

204.2 A SADA Executive is allocated to a performance audit or other assurance engagement consistent with ANAO Audit Manual - Shared Content, paragraph 6.4.

Guidance

204.4 The responsibility for the direction, supervision and performance of the SADA component of the engagement includes:

• emphasising the importance of audit quality on each engagement;
• tracking the progress and quality of the SADA component of the engagement;
• considering the competence and capabilities of the SADA audit team assigned to the engagement;
• ensuring that appropriate SADA procedures are planned and performed;
• addressing significant SADA matters arising during the engagement and the impact on the planned approach;
• ensuring the SADA work performed supports the conclusions reached and is appropriately documented; and
• agreeing with the performance Engagement Executive any SADA services to be provided in a performance audit, the scope, timeliness and quality assurance arrangements for those services, as well as ensuring those services are appropriately resourced.

204.5 IT can pose specific risks to an entity’s internal control. Some examples of IT risks which may be relevant to performance audits are:

• reliance on systems or programs that are inaccurately processing data, processing inaccurate data, or both;
• unauthorised access to data that may result in destruction of data or improper changes to data, including the recording of unauthorised or non-existent transactions, or inaccurate recording of transactions. Particular risks may arise where multiple users access a common database;
• the possibility of IT personnel gaining access privileges beyond those necessary to perform their assigned duties thereby breaking down segregation of duties;
• unauthorised changes to data in master files;
• unauthorised changes to systems or programs;
• failure to make necessary changes to systems or programs;
• inappropriate manual intervention; and
• potential loss of data or inability to access data as required.

204.6 Attendance at key meetings should be determined by the nature of the audit and the level of SADA involvement. If SADA involvement has been minimal and/or there are no material findings arising from SADA work performed, it may not be necessary for the SADA Executive to attend entry/exit meetings with the auditee, progress reviews or the section 19 workshop. However, attendance at key meetings would be expected if there was extensive SADA involvement and/or material findings arising from SADA work performed.

Chapters 205 to 209

205. Understanding the entity and activity subject to audit

Guidance

205.2 The collection of information about the entity and activity to be audited is a key element of initial planning for an audit.

205.3 Obtaining an understanding of the activity and its context is an essential part of planning and conducting a performance audit. It includes gaining knowledge of the entity that is responsible for the activity, and where relevant, the broader program of which the activity is part.

205.4 This provides the auditor with a framework to:

• distinguish between the activity and its control systems;
• develop and assess the suitability of criteria;
• target sources of relevant evidence;
• identify performance audit operational risks;
• identify engagement risks and assess materiality;
• identify whether there is the need for specialist skills or the work of an expert; and
• estimate resource requirements.

Types of information to be collected

205.5 The types of information that it may be appropriate to collect about the entity subject to audit includes:

• objectives of the entity;
• external accountability relationships — who the stakeholders and clients are and what their interests are in the entity;
• internal accountability relationships — such as organisational arrangements, delegations and committee structures;
• resources — the physical, financial, human and information resources available to the entity;
• applicable legal and policy frameworks / requirements for the activity
• management processes including:
  • governance arrangements;
  • performance criteria used by management;
  • assessment of performance by management;
  • the nature and frequency of reporting on performance;
  • the use of performance results to assess entity operations and performance;
  • the systems and controls in place for controlling the entity’s resources and ensuring appropriate client service;
  • risk assessments used by management; and
  • the role of internal audit in performance auditing;
• performance goals — their consistency with the entity’s legislation and governance framework;
• methods of program delivery — intended outputs of programs and outcomes, delivery methods and constraints on effective delivery;
• the external environment — factors that influence the entity’s operations, over which the entity may have little control, such as economic, social and political influences, with a particular focus on changes to that environment; and
• other publicly available information on the program.

205.6 The types of information to be collected for a cross-entity audit may focus more directly on the subject matter of the activity to be examined in the audit rather than entity-wide information. For example, the subject matter could include fraud control arrangements, the corporate planning framework, project and contract management, internal audit operations, recordkeeping or human resource management.

Sources of information

205.7 As a starting point, information already available within the ANAO should be reviewed and discussions undertaken, as necessary, with other ANAO staff who have knowledge of the entity or activity to be audited. Background information may have already been collected as part of the ANAO’s annual planning process, as part of another performance audit, by FSASG as part of the audit of an entity’s financial statements, or by PSASG as part of the audit of an entity’s performance statements.

205.8 Background information can also be derived from public or other sources external to the entity such as:

• the entity’s website;
• enabling and program-specific legislation;
• Cabinet submissions and decisions;
• the entity’s Corporate Plan, Annual Report and Portfolio Budget Statements;
• audit committee papers;
• Senate Estimates and other Parliamentary hearings, parliamentary committee reports, and Second Reading Speeches;
• media reports, Ministerial and entity press releases, newspaper and journal articles, and television and radio reports;
• central entity policies, standards, directives and guidelines;
• Government and review tribunal hearings and reports, such as those by the Commonwealth Ombudsman;
• reports of other external scrutineers, such as the Inspector-General of Taxation;
• the entity’s planning documents and organisation charts; and
• Australian or overseas material from entities that have similar programs or experiences.

205.9 In addition, the audit team should hold discussions with the entity, including Internal Audit, and obtain and review documentation, including:

• relevant policies, plans and procedures; and
• reports on any evaluations or reviews.

205.10 Sources of information for cross-entity audits, in relation to the subject matter rather than specific entity information, include:

• policies or guidance promulgated by central agencies;
• audits conducted by the ANAO and other audit offices within Australia or overseas;
• internal audits and reviews conducted by the entities included within the scope of the audit; and
• professional and community organisations and standard-setting bodies such as Standards Australia.

205.11 The audit team may also hold discussions with one or more of the entities to be included in a cross-entity audit to gain a practical understanding of the subject matter of the audit. The decision to undertake such discussions and collect documents from entities will need to be decided on a case-by-case basis. Some factors to consider when making the decision include:

• whether it is likely that the particular policy or practice is widely understood and is being implemented; and
• the need for information to assist in developing a suitable audit objective, criteria and approach.

205.12 The information-gathering powers in the A-G Act can be used to obtain information and documents required for planning an audit. However, in practice, the information-gathering powers are used as ‘reserve’ powers and access to required information is almost always obtained through cooperation with entities. If an entity is not cooperating, consult with the Engagement Executive and responsible GED in the first instance to escalate the request. If necessary, they will consider possible use of either section 33 or requesting that the Auditor-General uses section 32 to obtain the necessary information. ANAO Legal Services can also assist with informing entities about the nature of the information-gathering powers.

206. Materiality and risk assessment and management

Guidance

Materiality

206.9 ASAE 3500 defines materiality as variations in performance of an activity evaluated against the identified criteria which have the potential to affect the economy, efficiency, effectiveness and/or ethics (added by virtue of paragraph 6(a) of the ANAO Auditing Standards) of the activity and be reasonably expected to influence relevant decisions of the intended users or the discharge of accountability by the responsible party or governing body of the entity.

206.10 As such, findings are considered to be material if they, individually or in the aggregate, could reasonably be expected to influence relevant decisions taken by intended users on the basis of the auditor’s report. The Parliament is considered to be the primary user of the ANAO’s performance audit reports. The audit team’s consideration of materiality is a matter of professional judgement. The audit team needs to also consider the aggregate effect of individually insignificant findings.

206.11 Materiality can also be understood as the relative importance of a matter to change or influence the decisions of users of the report, such as legislatures or the executive. Materiality is considered in the context of quantitative and qualitative factors, such as relative magnitude, the nature and effect on the subject matter and the interests expressed by intended users or recipients. In addition to monetary value, materiality includes issues of social and political significance, compliance, transparency, governance and accountability. Materiality can vary over time and can depend on the perspective of the intended users and responsible parties (GUID 3910 – Central Concepts for Performance Auditing (GUID 3910), issued by INTOSAI).

206.12 Guidance in respect of applying materiality to findings to form a conclusion is at Forming the Audit Conclusion.

Materiality and audit risk

206.13 Materiality and audit risk need to be considered together as considerations of materiality will consequently impact performance engagement risk.

206.14 Audit risk includes performance engagement risk, which is the risk of expressing an inappropriate conclusion based on evidence that is not soundly based. This may include evidence that is improper or incomplete as a result of inadequacies in the evidence gathering process, misrepresentation or fraud. Audit risk also includes operational engagement risk, which is the risk that an audit will not be completed in accordance with the approved budget and timeframe and to the required quality (GUID 3910).

206.15 Assessing and applying the concept of materiality throughout the audit addresses performance engagement risk by driving examination of:

• material areas where the performance engagement risk is high; and
• material areas where the performance engagement risk is low, but any significant variations or deficiencies could have a material effect on the economy, efficiency, effectiveness and/ or ethics of the activity/subject matter.

206.16 Determining materiality and performance engagement risk as a team, with Engagement Executive involvement, is vital to avoiding under, or over-auditing. This can result where team members have different views on materiality and performance engagement risk.

206.17 Audit risk is assessed in planning and throughout the conduct of a performance audit. The assessment of audit risk requires the audit team to:

• understand the entity and its environment;
• assess risks to the audit conclusion; and
• design and conduct audit procedures to reduce engagement risk to an acceptably low level.

Materiality considerations

206.18 The consideration of materiality is relevant to all aspects of performance audits. Therefore, the auditor needs to consider materiality when selecting the audit topics, determining the audit objective(s), questions and scope, defining the criteria, evaluating the evidence, documenting the findings and developing the conclusions and recommendations.

206.19 Professional judgement about materiality is made in light of surrounding circumstances but is not affected by the level of assurance. Materiality for a reasonable assurance engagement is the same as for a limited assurance engagement because materiality is based on the information needs of intended users.

206.20 Quantitative materiality factors relate to audit findings that are expressed or evaluated numerically. Generally, audit findings are more material where they relate to relatively larger numbers or values in the context of the audit. There is no standard threshold for quantitative materiality in performance audits and in each case the auditor exercises their professional judgement in determining what numerical value represents a finding that is important in the measurement of the activity’s performance. For example, an audit finding that $1 million out of $2 million in spending activity lacked documentation is likely to be material, given the relative value of the finding to the subject matter under audit. On the other hand, a finding that a high proportion of low value transactions contained an error might not be considered quantitatively material if those errors related only to rounding errors that were within the audited entity’s accepted error rates. The auditor needs to consider the aggregate effect of individually insignificant findings.

206.21 Qualitative factors affecting materiality may include such things as:

• the number of persons or entities affected by the matter being audited;
• the interaction between, and relative importance of, various components of the activity when it is made up of multiple components, such as a report that includes numerous performance indicators;
• the wording chosen with respect to the activity that is expressed in narrative form;
• the nature of a finding; for example, the nature of findings in respect of a control when the assurance report includes a statement that the control is effective;
• whether a finding affects compliance with law or regulation;
• in the case of periodic reporting on an activity, the effect of an adjustment that affects past or current activities or is likely to affect future activities;
• whether a finding is the result of an intentional act or is unintentional;
• whether a finding is significant having regard to known previous communications to users, for example, in relation to the expected outcome of the audit;
• whether a finding relates to the relationship between the ANAO and the auditee, or their relationship with other parties;
• when a threshold or benchmark value has been identified, whether the result of the procedure deviates from that value;
• whether a particular aspect of the program or entity is significant with regard to the nature, visibility and sensitivity of the program or audited entity;
• whether the health or safety of citizens is affected; and
• whether a finding relates to transparency or accountability.

Performance engagement risk considerations

206.22 In the context of a performance audit, risk assessment is the identification and analysis of the key risks to the achievement of objectives concerning economy, efficiency, effectiveness and ethics, thus forming a basis for developing potential audit questions and determining the potential audit scope.

206.23 Performance engagement risks have the potential to adversely affect the auditee in some way. As well as failure to deliver a policy or program economically, efficiently or effectively, other performance engagement risks could include exposure to financial loss, loss of reputation of the entity or its Minister(s), or concerns regarding national security or commercial confidence.

206.24 Indicators that there is a high level of engagement risk may include:

• highly complex entities with multiple programs or functions;
• deficiencies in corporate governance;
• significant business risks which impact on the economy, efficiency, effectiveness or ethics of a program or the entity as a whole;
• poorly controlled, or changing, processes and systems;
• frequent changes in key personnel, systems or programs which are not well managed; and
• previous performance engagements may have reported on significant findings.

206.25 Relevant factors for consideration when determining performance engagement risk include:

206.26 The risk assessment should be informed by the auditor’s understanding of the entity and activity subject to audit. Based on this understanding, risks could be identified and analysed by answering the following questions:

• What can go wrong?
• What assets are at risk and from what sources?
• With whom does the risk lie?
• What factors are/can be constraining performance (economy, efficiency, effectiveness, ethics)?
• What could be the cause (including weaknesses in controls)?
• What could be the consequences or the impact, including on the entity’s reputation?
• How could this risk be managed?

206.27 The risks identified should be closely examined in order to decide on the ones that are key (significant and relevant). The risk level of the key risks should be determined by assessing the likelihood and potential impact of each risk using the risk matrix in the Risk Assessment Template.

206.28 Audit responses to address the key risks identified should then be documented in the Risk Assessment Template.

206.29 The Risk Assessment Template provides a repository for all established information on the risks and serves as a resource when communicating risk information to stakeholders.

207. The Audit Work Plan

Background

207.1 The Audit Work Plan (AWP) documents the planning activities for the audit which are required by the Auditing Standards (ASAE 3500).

Guidance

207.18 Audit planning varies according to the size, nature and complexity of the audit. The resources to be used in planning an audit should be commensurate with the nature and complexity of each audit and the assessment of the risks to the audit.

207.19 The preparation of an AWP necessitates understanding the entity and activity subject to audit. Per section 205.12 of this manual, the information-gathering powers in the A-G Act can be used to obtain information and documents required for planning an audit.

207.20 An AWP template and instructions on planning are available as part of the PASG Workflow, which contains corporate templates for staff to use during an audit.

207.21 Each AWP should contain sufficient information to allow the GED and the Executive to make a fully informed decision on the conduct of the proposed performance audit. Once approved, the AWP provides the authority to conduct the audit.

207.22 In preparing the AWP, the audit team should refer to the criteria, approach, budget and timeframes for similar audits considered previously and as outlined in their AWPs.

207.23 In a small number of cases, it may not be possible to finalise elements of the AWP until the delivery phase of the audit has commenced. In these circumstances, the audit team should clearly outline in the AWP those aspects of the audit plan that may be subject to change and set a timeframe to confirm any changes. Approval will be needed to proceed with the audit while the details of the AWP are still being finalised. The audit team is responsible for progressing the audit in parallel with the refinement of the audit plan during the delivery phase.

207.24 The following paragraphs outline considerations for preparing specific elements of the AWP.

Rationale for undertaking the audit

207.25 The AWP outlines the rationale for conducting the audit. Usually, the rationale for undertaking a particular audit has been identified in the course of preparing the Annual Audit Work Program. This rationale will be included in the audit report in accordance with paragraph 224.4 of this manual, and hence should be clear and understandable to a broad audience, with reference to the relative importance of the subject matter from a stakeholder perspective. The following table illustrates examples that should be incorporated in a rationale.

Background to the audit

207.26 Each AWP should include background information regarding the entity, program or function to be audited. This background information reflects and generally builds on the material for the particular audit that was included in the Annual Audit Work Program.

Audit objective

207.27 The audit objective, outlined in the AWP, is a key statement that is intended to define the intention of the audit. The objective of a performance audit is to provide an assessment of specified elements of an entity’s operations. The assessment should address one or more of the following terms: effectiveness; efficiency; economy; or compliance.

These terms are defined in ASAE 3100 and ASAE 3500 as follows:

207.28 ’Ethics’ is defined in the ANAO Auditing Standards as the performance principle relating to the extent to which the proposed use of public resources is consistent with the core beliefs and values of society. Where a person behaves in an ethical manner it could be expected that a person in a similar situation would undertake a similar course of action. For the approval of proposed commitments of relevant money, an ethical use of resources involves managing conflicts of interests, and approving the commitment based on the facts without being influenced by personal bias. Ethical considerations must be balanced with whether the use will also be efficient, effective, and economical. See ‘Methodology Guidance: Audits of Ethics’ for further guidance about conducting audits focussed on ethics.

207.29 The audit objective and the audit scope (see paragraph 207.37) are interrelated and should be considered together. The audit objective needs to be realistic and achievable and give sufficient understanding to the entity and other relevant parties about the focus of the audit. The audit objective also provides the basis for developing the audit criteria and the audit approach.

Audit criteria

207.30 Audit criteria are the specific measures used to assess the performance of the activity. In accordance with ASAE 3500 paragraph 16(d), the criteria are benchmarks used to evaluate the underlying subject matter. Audit criteria are reasonable and attainable standards of performance against which the extent of effectiveness, efficiency, economy, ethical or compliance aspects of an entity’s programs or activities can be assessed.

207.31 Audit criteria are important because they provide:

• a common understanding between the audit team, the ANAO Executive and the entity regarding the standards against which the entity is to be assessed; and
• a structure for the evidence-gathering phase of the audit.

207.32 Suitable criteria are those that are relevant to the subject matters being audited and appropriate to the circumstances. As outlined in ASAE 3500 the characteristics of suitable criteria include:

• relevance: relevant criteria contribute to conclusions that assist decision-making by the intended users;
• completeness: criteria are sufficiently complete when relevant factors that could affect the conclusions in the context of the performance engagement circumstances are not omitted. Complete criteria include, where relevant, benchmarks for presentation and disclosure;
• reliability: reliable criteria allow reasonably consistent evaluation of measurement of the activity, including when used in similar circumstances by similarly qualified assurance practitioners;
• neutrality: neutral criteria contribute to conclusions that are free from bias; and
• understandability: understandable criteria contribute to conclusions that are clear, comprehensive, and not subject to significantly different interpretations.

207.33 Sources of suitable criteria and sub-criteria may include:

• policy decisions or policy statements;
• legislation and regulations;
• published performance measures and internal measures;
• policies and guidance developed by central entities, regulators or government;
• standards of good practice, relevant benchmarks and relevant practice guides developed by professions, associations or other recognised authorities;
• statistics, practices, benchmarks, performance standards or procedures developed within the entity; and
• subject matter or general literature⁶.

207.34 Criteria may require interpretation and modification to ensure their relevance to the audit. Criteria may need to be modified or refined as the audit proceeds and more information becomes available. Should the criteria require substantial amendment and impact on the scope of the audit, the proposed changes should be discussed, in the first instance, with the Engagement Executive. If agreed, the proposed changes can then be considered by the ANAO executive at the formal review stages (see the PASG Workflow).

207.35 For further details refer to ANAO Audit Manual – PASG Specific Chapter 208 Generic audit criteria.

Audit scope

207.36 The audit scope defines the boundary of the audit. The audit scope may identify:

• the part of the entity, management control system or organisational unit to be examined;
• the matters subject to audit;
• particular entity locations to be visited during the audit;
• the time period being examined by the audit; and/or
• any associated matters that are not within the scope of the audit and the reasons for their proposed exclusion from the audit.

207.37 The scope is usually established based on information gathered during the planning phase or obtained in previous audits.

207.38 In establishing the scope of an audit, it may be necessary to have a broad statement of scope at the outset and refine this during the course of planning the audit and in the early stages of conducting the audit.

207.39 When determining the scope of the audit, the AWP should make reference to any known relevant reports published by other external and internal scrutineers; for example, the Inspector-General of Taxation, Inspector-General of Intelligence and Security, Parliamentary committees or internal audit.

207.40 The audit team considers matters such as materiality, risks to successful program performance and/or service delivery and auditability when establishing the scope. These matters should be considered throughout the planning and conduct of the audit and, particularly, in developing the detailed audit criteria.

Audit method

207.41 The audit method sets out the extent of evidence-gathering procedures to be undertaken and the reasons for selecting them, and the means to be used to collect information relating to the audit criteria. The method explains the intended use of specific data collection tools such as sample surveys, case studies, meetings, formal interviews, document reviews, compliance and/or system control analysis and testing. The method should also consider whether testing is likely to be performed on a sample basis.

207.42 The audit method also specifies where and why particular fieldwork is to be carried out and lists the involvement of any external stakeholders.

Cost of an audit

207.43 The cost of an audit includes the estimated costs of staff resources and the engagement of contractors and experts, and the estimated costs of travel and report publication. The costs of the initial planning phase of the audit and scoping study, where undertaken, are also to be included.

207.44 The budget for the audit is derived from a consideration of:

• the estimated hours required to be spent on the audit by the audit team, Engagement Executive and the Group Executive Director;
• the cost of initial planning;
• the cost of FSASG, PSASG and SADA staff, contractors, specialists and experts;
• costs of travel, including attendance at any audit related conferences. Travel costs should include airfares, accommodation, travel allowance and taxi/car hire that will be incurred by all team members, including any contractors engaged to assist with the audit; and
• the cost of publishing the audit report.

Audit milestones and target dates

207.45 AWPs are to include the target dates for the following audit milestones:

• AWP discussion meeting;
• Audit start date (Designation letter sent);
• Progress Review 1;
• Progress Review 2;
• Report Preparation Papers to entity;
• Section 19 workshop;
• Section 19 report to entity; and
• Audit tabled.

207.46 The PASG Workflow references a Ready Reckoner tool to assist with planning the audit phases and milestone dates.

The audit team

207.47 In determining the composition of the audit team, the following factors are taken into consideration:

• the experience of the Audit Manager;
• the number, level and experience of other team members;
• the benefit of engaging FSASG, PSASG and/or SADA to assist in conducting elements of the audit;
• the benefit of engaging specialists and/or experts (including from PSG and CMB) to support the in-house team in addressing complex and/or technical issues for example, relating to methodology); and
• the complexity and expected impact of the audit.

207.48 A specialist is an audit practitioner (either from ANAO or external) who specialises in auditing a particular subject area (e.g. IT Auditor).

207.49 An expert is a person or organisation whose expertise in an area other than auditing is used by the ANAO to assist in obtaining sufficient appropriate audit evidence. An expert would normally be external to ANAO but may include internal non-audit staff such as CMG or PSG Legal Services staff.

207.50 Where gaps are identified in the skills necessary to conduct a particular audit, there are a number of options to address this, including:

• undertaking training to obtain the skills;
• obtaining advice and assistance from within PASG or other areas of the ANAO; or
• engaging a contractor, expert and/or specialist.

207.51 As part of the AWP, the audit team are encouraged to seek the input of other PASG Audit Managers and Engagement Executives who have conducted similar themed audits or audits of the same entity to assist in scoping the audit.

207.52 As part of the AWP, the audit team should outline a summary of the pre-audit consultation with FSASG, PSASG and SADA. This summary should include any potential or planned involvement of FSASG, PSASG and SADA in the performance audit, any risks identified from financial statement auditing, and other intelligence gathered from past audits that may be of relevance for the proposed audit.

Risk

207.53 A detailed risk assessment and management plan is completed and attached to the AWP that addresses each risk and its corresponding mitigation strategy (refer to ANAO Audit Manual – PASG Specific Chapter 206 Materiality and risk assessment and management).

Contributions to audit planning

External stakeholders

207.54 External stakeholders include people or organisations with an interest in the operations, activities, results or resources of an entity. The primary external stakeholders are members of the public, clients of the audited entity, and non-government organisations such as industry associations and special interest groups.

207.55 Increasingly, entities are expected to develop close links with interest groups such as consumer and industry associations, provider organisations and think-tanks. These relationships can go beyond the exchange of information and may involve more formal collaboration or negotiation about government decision-making.

207.56 The identification of external stakeholders — who and what their interests are in the audited entity or topic — should be completed during the audit planning stage.

Approval of the AWP

207.57 The approval processes for the AWP are outlined in the PASG Workflow, which contains corporate templates for staff to use during an audit.

208. Generic audit criteria

208.1 Generic audit criteria have been developed for the following four common types of performance audits conducted in the ANAO:

• Grants administration (Post PGPA Act - 1 July 2014)
• Procurement (Post PGPA Act - 1 July 2014)
• Regulation (Post PGPA Act - 1 July 2014)
• Contract Management (Post PGPA Act - 1 July 2014)

208.2 The purpose of each of the four criteria documents is to provide performance audit teams with a starting point for drafting criteria for these particular types of audits. Audit teams will still need to tailor specific criteria to their audits after identifying the time period and relevant legislative framework that applies (either prior to or post PGPA Act) and carefully considering the audit objective, the type of program or project being audited and the business risks of the entity being audited.

208.3 The generic criteria broadly describe key structures, processes and policies that would be expected to be in place within an entity. These criteria were developed using the ANAO’s Better Practice Guides (now withdrawn), relevant recent performance audits as well as Government directives. References are provided at the bottom of each document. Audit teams are encouraged to review this reference material when developing criteria for individual performance audits.

208.4 In relation to the criteria for Grants Administration, a tiered approach is warranted because the relevance of various parts of the Commonwealth Grant Rules and Guidelines often depends on the circumstances and parameters of the particular program.

209. Planning audit procedures

Guidance

209.5 Combined, the AWP and audit test program document the planned audit procedures, including the nature, timing and extent of evidence-gathering procedures and the rationale for selecting the approach.

209.6 When determining the extent of time and resources required for planning, audit teams are to consider the:

1. audit team’s experience with and understanding of the entity and the audit topic;
2. size of the team;
3. level of the audit team’s auditing experience;
4. scope of the audit; and
5. complexity of the audit criteria and proposed tests and evidence-gathering techniques.

209.7 The AWP and/or audit test program includes:

1. the types and expected sources of audit evidence;
2. the techniques planned to be used to gather evidence;
3. the planned audit procedures, including timing and extent (e.g., target testing of a specified number or items with defined characteristics, random sampling);
4. personnel and expertise requirements, including the nature and extent of the use of specialists or experts when applicable;
5. the allocation of tasks to be performed by audit team members;
6. a link between the criteria, evidence, audit procedures performed and the results and findings;
7. materiality: and
8. assessment of engagement risk.

209.8 Planning is not a discrete phase, but a continual process and the test program may need to be revised to reflect any changes in the planned approach. It is recommended that updates to the planned approach and rationale are documented to explain why the change was necessary.

209.9 In planning and conducting a performance audit, it is not expected that audit procedures will include directly assessing whether fraud or other wrongdoing is occurring in the program or activity subject to audit. It is not the auditors’ responsibility to prevent or detect fraud or other wrongdoing through the conduct of its audits. This is the responsibility of the entity itself. The ANAO is also not in a position to determine whether a fraud or other wrongdoing has actually occurred.

209.10 As part of planning and conducting a performance audit, the audit team is required to obtain an understanding of the entity and the program or activity subject to audit. This should include making an assessment of whether fraud, or related wrongdoing, may have a significant impact on the program or activity. Such an assessment could include: a review of the entity’s fraud control plan and related information and documentation, such as fraud plans of individual work areas; and a review of the entity’s systems and procedures relating to fraud prevention, investigations, prosecutions, and reporting. If the audit team concludes that the risk of fraud and related wrongdoing may have a significant impact on the program or activity subject to audit, the audit should assess the adequacy of the entity’s management of this risk.

209.11 The extent to which matters such as potential fraud and other wrongdoing are referred to in a performance audit report will depend on the individual circumstances. It is generally not necessary or appropriate to refer to individuals or specific instances in audit reports. In circumstances where such matters are systemic and have the potential to significantly impact the operations of the program or activity subject to audit, they may warrant specific audit coverage and reference in the audit report.⁷

Chapters 210 to 223

210. Designation: Communicating the terms of the audit

Background

210.1 The designation email provides notification of the Auditor-General’s decision to conduct an audit and mandate, states the section of the A-G Act under which the audit is to be conducted and advises that the audit report will be tabled in the Parliament as soon as practicable after the completion of the audit. It also provides specific details about the audit and audit team.

Guidance

210.4 The ANAO issues a designation email that communicates the terms of the performance engagement to each entity subject to audit at the commencement of each audit before commencing audit fieldwork. A standard designation template is attached to the PASG Workflow, which contains corporate templates for staff to use during an audit.

210.5 The designation email is sent to the accountable authority of a Commonwealth entity (refer to PGPA Act, section 12 – normally the Chair of the accountable authority where the authority has more than one member) and to the Chair of the Board of Directors of a Commonwealth company.

210.6 The designation message provides notification of the:

• Auditor-General’s decision to conduct and audit;
• Auditor-General’s mandate;
• section of the A-G Act under which the audit will be conducted;
• audit focus, including the objective and scope of the proposed audit;
• entity’s opportunity to provide written representations on the program or activity subject to the audit; and
• ANAO contact details.

210.7 In addition, it may be useful to provide the entity with a scope diagram that includes the audit objective, high-level criteria and sub-criteria, and a separate list of the initial information requirements for the audit.

210.8 Section 17 of the A-G Act is limited to the audit of a single Commonwealth entity, company or subsidiary. Section 18 of the A-G Act does not limit an audit to an audit of single Commonwealth entity, company or subsidiary. It is necessary for audits conducted of more than one entity, company or subsidiary to be separated from audits of a single entity, company or subsidiary. Should the audit change or be extended to include more than one entity, company or subsidiary, it is to be conducted under section 18 of the A-G Act and the change shall be communicated to the original entity, company or subsidiary. Refer to paragraph 210.3.

211. Entry meetings

Guidance

211.4 Following the designation of an audit, the entry meeting serves a number of purposes, including introducing the audit team and allowing discussion of the audit objective, process and timing with the entity.

211.5 The purpose of an entry meeting is to inform the entity and establish the basis for a successful engagement by:

• outlining the purpose of performance audits and opportunities to minimise the impact on the auditee;
• drawing attention to information access powers under the A-G Act;
• introducing to entity management the staff of the ANAO who will be involved in the audit;
• explaining the background and objectives of the audit, discussing the audit criteria and responding to any questions the entity may have about these issues;
• giving entity management the opportunity to ask questions about the audit process or any other relevant matter relating to the audit, including key milestones and expectations;
• drawing attention to subsection 36(1) of the A-G Act concerning the confidentiality of information obtained during the course of an audit, particularly when an entity has had little previous exposure to performance audit requirements and processes;
• allowing entity managers to bring to the attention of the ANAO any contextual matters that might influence the way the audit is conducted, particularly documentation that assists in explaining the program and/or relevant issues;
• informing the entity about information the audit team is expected to require — specifically, classified and/or sensitive records and where arrangements need to be made to access systems together with applicable milestones — to help ensure an efficient and timely audit process; and
• discussing the administrative arrangements surrounding the audit and establishing coordination arrangements with the entity.

211.6 It is advisable that the auditee(s) is informed at the entry meeting about the citizen contribution facility on the ANAO website and that the audit is open for stakeholders contributing information during the evidence collection stage. Also refer to the PASG Workflow.

211.7 ANAO officials are expected to approach entry meetings as an educative opportunity for audited entities and to not assume a high level of knowledge about ANAO processes.

212. Progress reviews

Guidance

212.9 Performance audit teams are required to undertake a series of progress reviews with the Service Group Executive and ANAO Executive staff at key specified intervals during the conduct of the audit.

212.10 The audit team completes an initial progress review (PR1) in order to confirm that planning for the audit remains appropriate or to suggest a variation to the conduct of the audit or to discuss any emerging issues, such as difficulties in accessing information or data.

212.11 The PR1 should explicitly consider the ongoing appropriateness of scope, budget and schedule. Where deficiencies are identified, treatments should be applied to support delivery on time and on budget. Audit Managers and Engagement Executives should consider a range of treatments to support the timely delivery of the audit.

212.12 The audit team completes a PR2 meeting to identify and distil the key issues arising from the audit and to consider the most effective structure for the presentation of the audit findings and potential recommendations in the proposed Report Preparation Papers.

212.13 Report Preparation Papers should be fully defined, findings and conclusions identified and Report Preparation Papers approximately 50% drafted when the PR2 meeting is conducted.

212.14 The audit team can request a PR3 meeting to discuss the key issues arising from the entity’s response to the Report Preparation Papers and/or the exit meeting and to consider the most effective structure for the presentation of the audit findings and recommendations in the section 19 report. The PR3 may occur either before or after the exit meeting.

212.15 Specific requirements for the briefing are included in the PASG Workflow, which contains guidance and corporate templates for staff to use during an audit.

212.16 Progress review meetings are an important opportunity to canvas and discuss issues such as:

• key messages from the audit findings;
• the test program;
• the proposed audit report structure, including headings for:
  • key findings in the report summary;
  • chapter headings in the report body; and
  • appendices;
• potential sensitivities that may arise from the audit findings and the impact that could have on reporting of audit evidence, including those arising from legal advice received or security issues;
• potential audit recommendations;
• the linkages (or otherwise) back to the original request in the case of audits undertaken following a request from a Member or Senator of the Parliament of Australia;
• outline of any significant proposed divergence from the audit objective and/or audit criteria set out in the approved AWP, with reason(s) for this divergence;
• any difficulties envisaged in meeting stakeholder expectations;
• estimated dates for the remaining milestones until the report is tabled, against approved milestone dates; and
• the preliminary overall audit conclusion.

212.17 To assist in drawing on the experience of other audit teams, the responsible Engagement Executive may invite members of other audit teams to participate in progress review meetings on a case-by-case basis. As noted at 212.6 members of the FSASG team are required to be invited to attend the PR2 meeting and may be invited to attend the PR1 and PR3 meetings.

212.18 Notwithstanding the formal requirements for briefings during key stages of the audit, the audit team would be expected to provide briefings to the responsible GED and the ANAO Executive that are commensurate with the likely impact of the audit.

212.19 Discussions with the Executive could occur:

• during or at the completion of fieldwork so that any significant issues that are identified in the course of the fieldwork are discussed; and
• after the completion of the exit meeting to discuss entity feedback, key findings and conclusions for the draft report.

212.20 These briefings are also an opportunity to bring the Executive’s attention to matters such as potential sensitivities, budget or timeframe pressures or potential difficulties in meeting stakeholder or user expectations, including for audits requested by Members and Senators of the Parliament of Australia.

213. Advising the entity of progress and significant issues

Guidance

213.4 Explaining the audit process to the entity at all key points during the audit and keeping the entity informed is crucial to maintaining a sound professional relationship with the entity being audited and ensuring the free flow of information.

213.5 Informing the entity of matters arising from the audit would include providing entity management with early advice on particularly significant issues that are identified during the course of the audit so that the entity’s perspective on the issues may be obtained at an early date.

213.6 When advising the entity on significant issues identified during the audit, the audit team must consider:

• whether there is sufficient certainty concerning the audit position to warrant the advice; and
• including an appropriate caveat for any advice provided.

213.7 This is particularly important because the ANAO’s position may change during the audit review process.

213.8 The level and frequency of contact with entity senior management is determined by the audit team but would be commensurate with the expected impact of the audit, and sensitivity to the issues arising.

213.9 Keeping the entity informed provides the opportunity for the audit team to foreshadow future steps in the audit process and flag further information and assistance that will be sought during the remainder of the audit.

213.10 Informing the entity about the conduct of the audit could include advising the entity on just completed or planned future key steps in the audit process, for example, the results of a significant meeting, the commencement/completion of fieldwork, and the development of Report Preparation Papers.

213.11 Informing the entity about the conduct of the audit could also include significant changes to the audit process and/or timing, for example, caused by the unplanned absence of key members of the audit team.

214. Entity security requirements

Guidance

214.5 When allocating resources to an audit, the responsible GED and the Engagement Executive should have regard to the likely level of security clearance required and allocate staff accordingly where possible.

214.6 The audit team needs to have regard to the reasonable security requirements of the entity being audited.

214.7 For further information, the audit team can refer to the ANAO’s Security Management Framework, available on MyANAO Protective Security page.

214.8 There are a number of factors that the audit team can consider to assess whether the entity security requirements are reasonable. These include whether the requirements:

• impinge on access to documents and systems necessary to the completion of the audit;
• introduce barriers to the efficient conduct of the audit that are contrary to the exercise of access powers under the A-G Act; and
• do not provide for the same level of clearance for the audit team as the level of clearance for relevant entity personnel.

214.9 Entity ICT security protocols may be triggered in the event of inappropriate access. Inappropriate access may result in reputational damage to the ANAO as a trusted user of entity information and action against the individuals involved.

215. Gathering audit evidence

Guidance

215.8 Audit evidence is information obtained and used to support audit findings and conclusions. The collection, analysis and use of evidence are essential elements of the effective conduct of a performance audit.

215.9 Decisions concerning the evidence gathering process begin at the planning phase of an audit. The exercise of professional judgement concerning sufficient and appropriate audit evidence then continues throughout the audit. The audit team must assess whether the evidence obtained has the sufficient quantity and the appropriate quality. This assessment will inform the decision to obtain additional or different evidence.

215.10 Performance auditors need to have discussions with the auditees about the available evidence at the planning or conducting phases. They should ascertain the nature of the evidence and how it will need to be collected and analysed and interpreted by the audit team.

215.11 It is important that the audit team obtain evidence from a variety of sources, as different perspectives and conclusions may be presented from multiple sources. It is necessary to continually identify potential sources of evidence during the conduct of the audit. This is because not all circumstances can be foreseen during planning.

215.12 The internal controls of the systems that generate the audit evidence also need to be assessed for accuracy and consistency.

215.13 It is likely that several techniques will be used to gather audit evidence in any performance audit, such as:

• analytical procedures;
• document review;
• review of email records;
• meetings;
• formal interviews;⁹
• conducting surveys (refer to ANAO Audit Manual – PASG Specific Chapter 218 Surveys);
• compliance testing;
• physical observation; and
• data analysis.

215.14 The four main types of audit evidence and their sources are set out in the table below.

215.15 Appropriateness of audit evidence is attributed to both the relevance and reliability of the evidence. While the assessment of sufficiency and appropriateness is a professional judgement of the auditor in each case, auditors may find it helpful to consider the following generalised statements concerning evidence appropriateness:

• documentary evidence is more reliable than verbal evidence, but the reliability varies depending on the source and purpose of the document;
• testimonial evidence that is corroborated in writing is more reliable than verbal evidence alone;
• evidence based on many meetings together is more reliable than evidence based on a single or a few meetings;
• testimonial evidence obtained under conditions in which people may speak freely is more reliable than evidence obtained under circumstances in which people may feel intimidated;
• evidence obtained from a knowledgeable, credible and unbiased third party is more reliable than evidence obtained from the management of the audited entity or others who have a direct interest in the audited entity;
• evidence obtained when internal control is effective is more reliable than evidence obtained when internal control is weak or non-existent;
• evidence obtained through the auditor’s direct observation, computation and inspection is more reliable than evidence obtained indirectly; and
• original documents are more reliable than copied documents.¹⁰

215.16 The performance auditor may also find the following generalised presumptions with respect to the sufficiency of audit evidence helpful in exercising their professional judgement in assessing the evidence obtained:

• the greater the audit risk, the greater the quantity and quality of evidence required;
• stronger evidence may allow less evidence to be used;
• having a large volume of audit evidence does not compensate for a lack of relevance, validity or reliability; and
• more evidence is normally necessary when the audited entity disagrees on ANAO’s conclusions over activities subject to audit.¹¹

215.17 When gathering evidence, it is important to remember that while individual pieces of data or information, such as date of birth, may not be sensitive or classified, combined with a person’s address and bank account details, the information in aggregate is likely to be more sensitive and will require appropriate handling and storage for privacy and/or security reasons.

215.18 Decisions concerning the retention in audit working papers of data and the results of analysis must balance the need to retain sufficient evidence to support the audit findings and conclusions with the need to reduce the risk of unnecessarily retaining sensitive or classified information. It may be possible, for example, to delete reference to one sensitive field, such as tax file number, without affecting the sufficiency of the audit evidence in the working papers.

215.19 Audited entities will often facilitate the ANAO’s work by giving ANAO staff privileged (researcher) level access to their document management systems. It would be inappropriate to conduct an information search with the purpose of monitoring an entity’s preparation of a response to ANAO Report Preparation Papers or a proposed Auditor-General report, or gathering additional information after an audit or review has concluded.

215.20 The information-gathering powers in the A-G Act can be used to obtain information and documents required as audit evidence. However, in practice, the information-gathering powers are used as ‘reserve’ powers and access to required information is almost always obtained through cooperation with entities. ANAO Legal Services can also assist with informing entities about the nature of the information-gathering powers.

216. Sampling and selecting items for testing

Guidance

216.13 When planning and conducting a performance audit, the auditor is required to obtain sufficient appropriate evidence on which to base audit conclusions. Audit sampling means selecting a sample for testing. Representative samples are drawn when seeking to make conclusions about the total population. It assists the auditor by obtaining evidence ‘to provide a reasonable basis for the auditor to draw conclusions about the population from which the sample is selected’. Representative sampling is one of several methods to select items for testing.

216.14 The sampling methods available to performance auditors are:

• testing all items in a population (known as census testing);
• testing a statistically representative sample of the items in a population; or
• testing a non-representative sample based on specific attributes in a population(known as target testing).

216.15 Testing all items in a given population will obtain the highest level of assurance but, especially when dealing with large populations, may not be a reasonable or efficient way to conduct audit testing.

216.16 A representative sample requires a sample size that is statistically appropriate and the sampled items to be selected randomly. This allows the testing to be representative of the total population. This can be a very efficient way to test large populations (over 800 items). For moderately sized populations (between 250 and 800 items) the relative efficiency of a representative sampling approach will depend upon the circumstances.

216.17 Targeting items for selection can be an efficient means to obtain audit evidence. This requires targeting items based on some attribute that is important to the audit. This may be based on quantitative factors (such as all items over a relevant threshold) or qualitative factors. The performance auditor chooses the relevant attributes to focus testing on the areas within the population with the highest level of audit risk. The total number of items tested will be determined according to auditor judgement. The performance auditor can only draw conclusions on the targeted items that have been tested and not on the untested portion of the population, and consequently auditor judgement on appropriate selection size would consider if the audit risk arising from the untested population is acceptable in the circumstances of the audit.

216.18 The planned approach of the audit for selecting items for testing, which may include a combination of methods, is based on professional auditor judgement. The planned method(s) for selecting items for testing needs to be clearly defined and documented. This assists in determining an audit approach that is effective, efficient and repeatable.

216.19 Selecting items for audit testing may include both targeted and representative item selection methods.

216.20 The ANAO Performance Audit testing approach is embedded in the Performance Audit Sampling template and further guidance is contained within the template and incorporated instructions.

216.21 The sample size calculator assumes that the method for selecting the sample items for testing is random sampling. Therefore, any representative sampling must be performed according to a random selection approach. This excludes a convenience¹⁴ or haphazard¹⁵ sampling approach where each item in the population does not have an equal chance of being selected.

216.22 The audit testing template also assumes that the population is homogenous. It may be appropriate to target test items in the population that are not homogenous or use stratification to split the population into separate strata with different characteristics. When stratifying populations, the sample size calculator is completed for each separate population and clearly documented in the template.

216.23 If an item selected for testing is not relevant to the test and should not have been included in the population, the item must be replaced by selecting a new randomly selected item. If an item selected is not available for testing and suitable alternative audit procedures cannot be performed, e.g. documentation relating to the item is lost, this item is treated as a deficiency or error and is not to be replaced.

Procedure following testing

216.24 At the conclusion of the audit test, it is necessary to compare the estimated proportion rate with the actual percentage of errors in the sample result. If the actual result is equal to or greater than the estimate, then the result is considered to be statistically valid and the sample size is appropriate to draw conclusions from. If the result was less than the estimate, then the audit team needs to consider a larger sample size.

216.25 For example, consider a population of 800, with a confidence level of 95%, a confidence interval of 10% and an estimated error rate of 20%. An actual test result that found 10% of the attributes being tested were incorrect would be considered statistically valid for a sample size of 58 items and no additional samples would need to be tested. For the same example, with an estimated error rate of 20% where the actual error rate was 35%, the sample size would not be statistically valid and an additional 21 items would need to be tested to obtain the statistically valid sample size of 79 items in order to draw valid conclusions at the same confidence level and interval.

216.26 If the additional sample required to achieve a statistically valid sample size is large, it may be more efficient to revise the testing strategy. If no additional sample is tested, the sampled items tested so far do not represent a statistically relevant sample and therefore cannot be relied upon as audit evidence by themselves. Consequently, the results cannot be included in the audit report in a manner that asserts or infers that the results are representative of the population. However, the audit team may consider the usefulness of the testing performed as corroboration of other audit evidence gathered and consider including the results in audit reports with suitable clear caveats about the inability to extrapolate the results of testing so far to the population.

216.27 Where the actual error rate in the original sample is significantly greater than the anticipated error rate (particularly where the actual error rate is also significantly greater than what would be considered a reasonable/acceptable error rate) this may be evidence that the process being tested has significant deficiencies related to how the process has been designed and implemented that the audit team had not detected. The audit team should re-examine these matters before determining the appropriate audit response.

Figure 1:

Sampling approaches to testing of controls

216.28 Performance audits may in some circumstances include gathering evidence that a control has been operating effectively. Audit evidence about controls effectiveness can be used by performance auditors to make relevant conclusions about the effectiveness, efficiency, economy and ethics of activities.

216.29 Where financial statement auditors have already performed audit procedures as part of their audit work, this policy allows performance auditors to rely on that work without dealing with the complexity of adjusting sample sizes to comply with the standard performance audit sampling methodology.

216.30 Alternatively, performance auditors may find it convenient to engage with financial statement auditors, or systems assurance specialists from SADA, to perform control testing procedures in accordance with the sampling methodology that they are experienced in using.

216.31 However, performance auditors may only apply the financial statement controls sampling methodology where the key assumptions underpinning this approach are appropriate to the performance audit purpose of the testing in accordance with the policy requirements specified in 216.9. These key assumptions include:

• that the testing is intended only as supporting evidence about the conclusions being drawn by the auditor and not as evidence of the audit criteria itself (that is, controls testing is supported by additional evidence as to the audit criteria);
• that the control being tested has been designed and implemented appropriately (that is, that the control, if applied consistently throughout the period relevant to the audit conclusion, would achieve the control objective on which the auditor intends to rely); and
• that the auditor identifies no or, in rare cases, very low levels of exceptions in the control effectiveness in testing.

216.32 Under the financial statement methodology, exceptions identified in the testing will cause the auditor to not be able to rely on the control, except in circumstances where a very small number of exceptions are identified and additional samples are tested in accordance with the methodology prescribed in the Audit Guide. This may not be an efficient and effective manner of obtaining audit evidence so should be carefully considered before being applied.

216.33 While not relevant to the financial statement mandate of the ANAO, the financial statement controls sampling methodology provides approaches to test controls for the purposes of concluding on the effectiveness of the control itself as part of the audit scope. This would be relevant where control effectiveness is a criterion of the performance audit. Given that this is a rarely used feature of ANAO’s financial statement methodology, the policy requires technical consultation with PSG before it is used.

217. Audit documentation: Protective marking and bulk collection of entity emails

Background

217.1 The A-G Act (section 33) provides for full and free access to any documents or other property. Documentary evidence can include material that has a ‘PROTECTED’ protective marking, including ‘PROTECTED CABINET’. The Auditor-General may also, by written notice, direct a person to provide information, produce documents and give evidence. However, most audit evidence is obtained through a cooperative approach with the audited entity.

Guidance

Application of ANAO Security Management Framework

217.8 The ANAO’s Security Management Framework, applies to all data held by the ANAO, including evidence gathered in the process of planning, execution and reporting of performance audits.

Protocol for ANAO Use of Cabinet Material in Performance Audit Reports – Obtaining and Handling Cabinet Material

217.9 Sections 32 and 33 of the A-G Act provide the Auditor-General with broad information-gathering powers that extend to any relevant document or information, regardless of their classification or status. However, in practice, the information-gathering powers are used as ‘reserve’ powers and access to required information is almost always obtained through cooperation with entities, including Cabinet materials.

• ANAO staff should seek to obtain information on a cooperative basis.
• If an entity is not cooperating, consult with the Engagement Executive and responsible GED in the first instance to escalate the request. They will consider further options, including as necessary possible use of either section 33 or requesting that the Auditor-General uses section 32 to obtain the necessary information. ANAO Legal Services can also assist with informing entities about the nature of the information-gathering powers.
• ANAO staff must at all times treat information in their custody appropriately and in accordance with the Australian Government Protective Security Policy Framework as further outlined in the ANAO Information Classification and Handling Guidelines.
• Section 36 of the A-G Act ensures protection for any information gathered by making it an offence to disclose information gained in the course of conducting an Auditor-General function for any purpose other than performing that function.
• The A-G Act also recognises that it may not be in the public interest for the Auditor-General to disclose some sensitive information.

217.10 In accordance with Cabinet material rules, CMG maintains documented processes for the requesting, handling and disposal of Cabinet materials which are documented on MyANAO.

Protocol for ANAO Use of Cabinet Material in Performance Audit Reports – Reporting on Cabinet Material

217.11 The A-G Act recognises that it may not be in the public interest for the Auditor-General to disclose some sensitive information.²⁰ When considering the publication of sensitive information, it is the Auditor-General’s role to determine whether the public interest is best served by disclosure.

217.12 As an independent officer of the Parliament, the Auditor-General has discretion in the performance or exercise of his functions or powers. In particular, the Auditor-General is not subject to direction in relation to whether or not a particular audit is to be conducted, or the way in which a particular audit is to be conducted.

217.13 The Auditor-General undertakes the following steps when considering referring to Cabinet material:

• as a matter of sound public administration, and with due regard to the long held Westminster tradition of respecting Cabinet confidentiality, the Auditor-General explicitly refers to or discusses Cabinet deliberations, processes and/or outcomes only occasionally, and when the material in question is fundamental to the conclusions of the audit;
• the proposed inclusion of any Cabinet material, whether explicitly referred to as such or not, is drawn to the attention of the auditee’s Accountable Authority during report preparation and when the Accountable Authority is provided with a draft final report for review and response; and
• the Auditor-General seeks to avoid direct quotations from deliberative documents of the Cabinet and, to the extent that it is possible, publishes reports that attribute decisions or outcomes to the responsible minister or government rather than to the Cabinet.

217.14 Any explicit discussion of Cabinet deliberations, processes or outcomes is included only when, in the considered view of the Auditor-General, the conduct of those deliberations and processes, or the associated outcomes, led directly to a conclusion expressed in the final audit report.

217.15 To ensure consistency with the protocol, audit teams must follow these steps:

• ensure at all times that any Cabinet material accessed or gathered during the course of an audit is treated and stored appropriately and in accordance with the Information Classification and Handling Guidelines. If necessary, the ANAO Security Advisor (in CMG) can be contacted for advice and assistance;
• when requesting access to Cabinet documents, which should be through the Cabinet Division of the Department of the Prime Minister and Cabinet (PM&C), please ensure the document numbers are identified (where possible) and the requested documents are relevant to the audit;
• in preparing any such documentation, due regard must be given to security and information protection obligations. Audit teams should consult the Entity Security Advisor as necessary;
• in most cases, references to Cabinet material in audit reports should use generic and minimal descriptions (i.e. ‘government’), other than when fundamental to audit conclusions; and
• if a draft audit report explicitly identifies Cabinet material, then the relevant GED or Engagement Executive should inform their counterparts in the Department of the Prime Minister and Cabinet of any intended publication of Cabinet material.

Bulk Collection of Entity Emails

217.16 Entities’ email systems are used in the administration of programs and activities. As email systems and the information and records they contain are legally the property of the entity, the ANAO is entitled to access this material in the same way as other entity records and information. Email records almost inevitably contain personal communications. As a result, the ANAO’s access will generally involve access to both official and personal communications. This can cause heightened sensitivities and may, at times, lead to disagreement between the audit team and the entity that needs to be properly managed, often at a senior level.

217.17 Without appropriate care and attention, the ANAO’s collection of bulk email records also poses the risk that audit team members do not deal with personal information in an ethical manner or in a manner that is not seen to be ethical.

217.18 When bulk email records are collected audit teams should consider:

• exploring ways of electronically searching records for relevant information in a way that limits access to personal information, and establishing internal team protocols to manage the risk that personal information is accessed and breaches to the need-to-know basis, including unreasonable access to personal information not related to the relevant audit questions;
• that relevant email records may be archived and the retrieval may involve longer timeframes and additional costs compared with the retrieval of paper files. In this context, the ANAO does not accept responsibility for entity costs arising from the conduct of audits, as section 33 of the A-G Act provides for ‘full and free access at all reasonable times to any documents or other property’. Nevertheless, it is expected that audit teams would be conscious of the costs that may be incurred by entities in meeting requests for access to records, and would seek to minimise these, to the extent possible;
• whether it may be appropriate to provide assurances to entity management that any personal information accessed will not be used in any way and will not be incorporated into the ANAO’s working papers; and
• whether it is appropriate to limit access to entity bulk email records to nominated members of the audit team and/or establishing separate security arrangements for the: storage of bulk entity emails during the audit; and destruction of emails not used as key evidence by the ANAO.

217.19 Such operational protocols may also be considered necessary to provide assurance regarding certain paper records, such as records containing highly personal information pursuant to disclosure regimes.

217.20 A balance needs to be achieved between meeting the ANAO’s responsibilities, including conducting audits in an efficient and objective manner, and recognising the legitimate concerns of entity management. This, at times, may involve the audit team explaining, in more detail than would normally be the case, the ANAO’s audit and confidentiality responsibilities and the reasons why access to certain entity information and records is necessary to meet these audit responsibilities. Potential issues in accessing data should be escalated early.

218. Surveys

Guidance

218.4 From time to time, audit teams may seek to conduct a survey to gather information from a large population to inform the scope or direction of an audit, or to provide insights into the perceived performance of a program or entity.

218.5 Approaches to conducting a survey include:

• the audit team designing, delivering and analysing the survey (subject to the audit team including members with the appropriate experience and expertise);
• contracting an expert to design a survey, with the audit team delivering and analysing the survey; or
• contracting an expert to design and deliver a survey, with the audit team analysing the survey.

218.6 Surveys can be conducted using an online tool or other means, such as a paper-based survey, telephone interviews or in-person interviews. Key considerations that audit teams should have when determining the survey approach include:

• the information security associated with proposed online survey tools;
• the information protection arrangements in place. At a minimum, any online survey tool used should offer SSL encryption to protect the privacy and integrity of collected data;
• an awareness of and being transparent about the data storage location; and
• any survey tool indemnity clauses that would require approval by a PGPA Act section 60 delegate.

218.7 Audit teams should compare and evaluate the options offered by various tools in terms of question design options, technical support and data reporting formats.

218.8 As for all procurements, audit teams should appropriately weigh up the costs and benefits of various options, ensure that they select a provider or tool that is fit for purpose and provides value for money, and clearly document the reasons for their decision. Any procurement plan should be endorsed by the PASG Practice Manager.

218.9 Audit teams should take the following steps to mitigate the risk of information leakage and manage stakeholder expectations of privacy and security:

• inform potential survey respondents about the arrangements that are in place to protect and store their responses, and should disclose the involvement of any third-party provider(s);
• minimise the collection of personal information, information about respondent organisations or contact details; and
• provide participants with alternative survey completion options (e.g. offline or by telephone).

219. Verifying audit evidence

Guidance

219.3 Verifying audit evidence is the first step in analysing audit evidence to ensure that audit findings, conclusions and recommendations are based on sound evidence.

219.4 The risk, significance and sensitivity of the matter to be reported will determine not only the nature and amount of evidence to be collected but the extent of verification.

219.5 Assessing information obtained from entity records is particularly important in circumstances where management information, such as program expenditures or key performance indicators, is to be relied on. In these cases, it is essential that the completeness and accuracy of this information is assessed through appropriate audit testing to give the ANAO confidence that the systems and processes used to produce the information can be relied on. This will involve reviewing key system controls and testing a sample of transactions.

219.6 Audit testing is required to be undertaken in these circumstances irrespective of when such information is obtained. For example, if information is provided in response to a section 19 report that is significant in settling the final audit conclusions, it needs to be assessed. When management information is contextual and is not significant in the context of the audit findings and conclusions, it is not necessary to test its completeness and accuracy. However, in these situations it is expected that the audit report would indicate that the information has been sourced from entity records or is based on entity advice, as distinct from audit analysis.

219.7 The ANAO’s policy in relation to sufficient appropriate audit evidence also applies to any work performed by an expert. When using the work of an expert to support audit findings and conclusions, it must be adequate for that purpose, recognising that the responsibility for the findings generated or conclusions drawn from the work undertaken by an expert rests solely with the ANAO.

219.8 Professional scepticism (ASAE 3500 paragraphs 16(p), A57) means an attitude that includes a questioning mind, and being alert to:

• conditions which may indicate possible deficiencies in administration;
• audit evidence that contradicts other evidence obtained;
• information that brings into question the reliability of documents and responses to enquiries;
• conditions that may indicate systemic system deficiencies; and
• circumstances that suggest the need for further analysis and enquiry.

219.9 The type of audit evidence gathered will determine the verification approaches available:

219.10 There are a number of different approaches to test completeness and accuracy of internally generated reports, and each approach provides a varying level of comfort. Professional judgement is required to determine the most appropriate method in the circumstances based on an understanding of how the report is generated and the desired level of comfort.

Standard system reports

219.11 The approach which provides the highest level of evidence is where the auditee is utilising standard system reports. When the controls for producing the reports have been tested, then there is a reasonable expectation that any errors would have been identified by the system.

219.12 Typical considerations to document include:

• auditee does not have a development environment, or access to such an environment is limited to external vendor accounts;
• review of change management logs shows no changes except vendor management;
• collaborative enquiry with different sources within the organisation confirms that the auditee does not undertake development/have access to the application source code;
• ANAO understanding of the application based on knowledge and experience is that the application is ‘off-the-shelf’ and vendor supported; and
• ANAO review of the standard user guides identifies the specific reports/controls to be part of the packaged application.

219.13 These are considerations only. It is important that the ANAO is comfortable that the standard reports are unable to be changed by the auditee and only by the vendor.

Technical re-performance

219.14 Technical re-performance involves reviewing the code (e.g. SQL query) which is used by the system to produce the report. This can only be performed when the auditee is able to access the code, and when the audit team has appropriately skilled team members who can understand the code.

Sample testing

219.15 Samples of data are tested for accuracy, completeness and validity. The procedure for this testing is detailed in the Sampling and selecting items for testing chapter of this manual.

219.16 With all approaches, there will typically be parameters applied to reports within a system. These parameters could be date ranges, product types, cost centres, etc. It is important that these parameters are reviewed as part of the process of obtaining comfort over the system reports.

220. Forming the audit conclusion

Guidance

220.7 The following diagram outlines the audit process and outcomes.

Audit findings

220.8 Audit findings are generated when the criteria (and sub-criteria) are compared with the audit evidence. Meeting or exceeding the criteria may indicate good practice leading to good performance. Failing to meet criteria would indicate that improvements are needed. It is, however, unrealistic to expect that the audited entity’s performance regarding economy, efficiency, effectiveness and ethics will always meet the criteria. This means that in addition to assessing whether the audited entity meets the criteria or not, the audit team also has to consider materiality and apply professional judgment in interpreting how this affects assessment of the entity’s performance.

220.9 The audit report should include both positive and negative points and give credit where it is due. Including positive aspects may lead to improved performance by other government organisations that read the report. It is important that the report contains all the information and arguments needed to satisfy the audit objective(s) and promote adequate and correct understanding of the matters and conditions reported.

220.10 Audits involve some type of analysis in order to understand or explain what has been observed. When analysing information collected, the auditor should focus on the criteria and objective. This will help to organise the data and also provide the focus for analysis.

220.11 While it is important to seek explanations for deviations from criteria, causes should be presented with caution. They have to be supported by sufficient and appropriate audit evidence. It is relevant to consider the audited entity’s views on reasons for performance problems or weaknesses. If such views are not supported by sufficient and appropriate audit evidence, the audit team cannot take for granted that they are relevant or correct.

220.12 The audit team should identify the possible effects of the criteria not being met. The effects could be identified either as what has already occurred or as possible future impact. The nature of the findings determines whether the audit team can present actual or potential effects.

Developing conclusions after considering the findings

220.13 Once the findings have been established and causes and effects considered, the audit team draws conclusions against the sub-criteria, criteria and objective respectively. Conclusions are statements informed by the findings. Since performance audits may point out performance relating to deficiencies in aspects of economy, efficiency, effectiveness and/or ethics, the conclusions have to specify the reasons why aspects of economy, efficiency, effectiveness or ethics may not have been fully met.

220.14 Audit conclusions clarify and add meaning to specific findings in the report. Conclusions present the opinion and go beyond merely restating the findings. Whereas the audit findings are identified by comparing ‘what should be’ according to the criteria with the audit evidence (including analytical evidence) on ‘what is’, the conclusions also reflect the auditor’s explanations and views based on these findings. Conclusions might include identifying a general theme or a certain pattern in the findings. An underlying problem that explains the findings may also be identified.

220.15 When drawing conclusions, it will often be necessary to revisit the data analysis and the audit findings to be sure that the conclusions are based on solid grounds. The analysis of data consists of combining results from different types of sources. The conclusions are based on the objective, criteria, evidence and findings.

220.16 The consideration of materiality is a matter of professional judgement. Materiality is considered in the context of qualitative factors and, when applicable, quantitative factors. The relative importance of qualitative factors and quantitative factors when considering materiality in a particular performance engagement is a matter for professional judgement. This judgement may be informed by whether the issue has had a substantive impact on the quality and/or cost efficiency of the services or policy advice provided – i.e., the primary objectives of the entity.

220.17 Concluding on the materiality of the findings identified as a result of the procedures performed requires professional judgement. For example: three criteria are identified for an audit. For one of these criteria, there are two sub-criteria. If sub-criteria one is satisfied, but the sub-criteria two is not, a finding will be reported in respect of sub-criteria two as it would be of interest to readers of the report. Professional judgement will be applied to determine the appropriate level of prominence for this finding in the audit report, given its actual or potential impact. That is, whether it will be presented within supporting findings or as an exception to the conclusion. This professional judgment will also consider the relative importance of this sub-criterion to the criterion to which it relates as well as the relative importance of that criterion to the other criteria and findings in respect of the other criteria.

220.18 In addition, professional judgement will consider whether there are factors that resulted in the satisfaction of sub-criteria one that are material to the conclusion that should be reported.

220.19 Findings are considered to be material to the overall conclusion if they, individually or in the aggregate, could reasonably be expected to change or influence the decisions of users of the report, such as the legislature or executive. Where a finding or findings is/are determined to be material to the overall conclusion, the conclusion will either be expressed in the form of the objective being met with exceptions or the objective not being met. For example, when there are findings in respect of one criterion only, it is likely that the objective would be found to be met with exceptions and the conclusion would be expressed as ‘the program was implemented effectively except for…’. Where the findings are so pervasive to the criteria and therefore objective of the audit, the conclusion would be that the objective was met and expressed as ‘the program was not implemented effectively because of …’.

221. Quality management

Background

221.1 A system of quality management is required to provide assurance that the audit team’s work has met the requirements of the ANAO Auditing Standards and complies with performance audit policies and related procedures.

Guidance

221.7 A system of quality management for performance audits is consistent with the ANAO Quality Assurance Framework and Plan.

221.8 Reviewing the audit coverage and findings provides assurance that all relevant issues and considerations have been addressed and that sufficient appropriate evidence was obtained for all issues that impact on the program or activity subject to audit and support the audit findings and conclusions.

221.9 A key element in the quality management system for performance audits is that the audit working papers should be subject to review. Working papers include all the planning documents, evidence collected and analysis undertaken during the course of an audit, and all documents relating to reporting for an audit.

221.10 The extent and nature of the review of the working papers should be sufficient for the reviewer to be satisfied that the audit findings and conclusions are supported by audit evidence, that they demonstrate an in-depth knowledge and understanding of the subject matter and that they are presented in a balanced, fair and constructive manner. Cross-referencing and filing all working papers facilitate the review of audit evidence and working papers.

221.11 The quality management system described above does not require all audit working papers to be reviewed by the Engagement Executive or more senior members of the audit team. The Engagement Executive may delegate review of some audit work to the Audit Manager or other more senior members of the team with suitable skills and experience, but the Engagement Executive retains overall responsibility for the review and supervision of the engagement. Ordinarily, the Engagement Executive reviews the higher risk aspects of the audit in addition to all those areas that specifically require Engagement Executive review in accordance with the ANAO Auditing Standards and this policy.

221.12 Reflecting that the entity subject to audit also has an important role to play in the audit process, audit working papers should indicate the extent to which the entity was consulted during the planning phase of the audit and evidence of the interaction with the entity about the audit findings, conclusions and recommendations throughout the audit.

221.13 Where the work of an expert is being used as evidence in an audit, the audit team retains full responsibility for the conclusions drawn from that evidence and presented in the audit report, recognising that it is ultimately the Auditor-General who signs each report and therefore will make the final judgement about the audit findings and conclusions.

222. Report Preparation Papers

Guidance

222.7 The audit team’s review and assessment of audit findings and conclusions, drawing on the audit evidence gathered and analysed in the delivery phase of the audit, form the basis of the Report Preparation Papers.

222.8 Specific procedures for the clearance and issuance of RPPs are included in the PASG Workflow, which contains corporate templates for staff to use during an audit.

222.9 RPPs, or equivalent working papers for audits of multiple entities, will generally be the first formal opportunity for the entity to consider the ANAO’s preliminary audit findings, conclusions and proposed recommendations. It is important that the entity understands that this is not the formal section 19 report. The audit team should explain to the entity the purpose of the RPPs, preferably in person to entity management, as well as in the correspondence that accompanies the RPPs.

222.10 RPPs can be prepared while audit evidence is being collected or towards the end of this phase when the majority of evidence has been collected. Drafting report preparation and evidence papers during the delivery phase and providing them to the entity for comment on an ongoing basis presents the opportunity to test the facts and findings at an early stage and to identify the need for more information where this is required.

222.11 RPPs, particularly when issued prior to the completion of fieldwork, can also be used to elicit further information and/or to seek further views from the entity on particular issues that are important to the audit.

222.12 RPPs provide the entity with the opportunity to correct any errors of fact or interpretation, either in writing or at the exit meeting. The entity can also provide additional information in response to this correspondence.

222.13 The iterative review of RPPs can be extensive for some audits. Audit teams are to make judgements regarding the number of reviewed versions of the RPPs that are required to be retained on the audit working papers. For example, drafts that contain minor editorial changes and comments only may not need to be retained. Any substantive edits or comments made by the GED or ANAO Executive are required to be retained.

222.14 Draft RPPs that contain substantive review comments and edits made by the responsible Engagement Executive and Group Executive Director are to be retained on the audit working papers. The comments and edits made by the ANAO Executive are required to be retained as evidence that the work performed by the audit team has been directed, supervised and reviewed, as well as to retain a documented record of the audit reporting process.

222.15 For audits that do not issue RPPs, a written summary of the key findings can be sent to the entity for discussion at the exit meeting. For efficiency, the summary of key preliminary findings can be based on the content of the Progress Review 2 briefing.

222.16 Preliminary findings, conclusions and proposed recommendations contained in the Draft RPPs should not be a surprise to the entity given sufficient and comprehensive verbal briefings throughout the audit.

222.17 In the case of cross-entity audits or audits containing information about third persons or entities, audit teams will need to consider the preparation of extracts of RPPs. The extract strategy should be agreed with the Engagement Executive and GED if necessary and should have regard to the objective of providing sufficient information to recipients to respond meaningfully to the RPP, while avoiding the release to third parties and maintaining the ANAO’s confidentiality obligations.

222.18 In determining what is a significant issue requiring communication with FSASG, PASG shall consider:

• the potential to give rise to a material mistake in the financial statements;
• any actual or suspected fraud;
• any issues revealing a weakness in key financial controls; and
• any other issues of which the audit team is aware that may be of interest to FSASG.

222.19 In determining what is a significant issue requiring communication with PSASG, PASG shall consider:

• whether there is a current performance statement audit underway or being considered for the next audit cycle;
• the potential to give rise to a material misstatement in the performance statements;
• any actual or suspected fraud;
• any issues revealing a weakness in key controls related to the setting of performance measures or the preparation, monitoring and reporting of the entity’s performance; and
• any other issues of which the audit team is aware that may be of interest to PSASG.

222.20 The ANAO Communication Unit is available to assist with any queries regarding the writing of audit reports in terms of format and style. Audit teams can also refer to the ANAO Style Guide.

222.21 RPPs should only be sent to a single officer of the entity. If that officer is not the accountable authority, director or member of the governing body, that officer would be referred to in the designation email so that the accountable authority, director or member of the governing body is aware of who the single officer will be. This will reduce the risk of accidental disclosure and give the relevant officer control over who receives the RPPs. Other officers may be sent an email informing them that the RPPs have been provided.

223. Exit meetings

Guidance

223.4 Exit meetings are an important step in the audit process and occur at the end of the period for the major collection of audit evidence, analysis of findings and identification of potential recommendations for the entity.

223.5 Where an exit meeting is to be held, it can be a forum to:

• give the entity an indication of what the final report will include;
• give the entity the opportunity to provide its perspective on the preliminary audit findings and to correct any errors of fact, interpretation or perception; and
• provide entity management with the opportunity to challenge proposed recommendations. The recommendations may then be adjusted, removed, or new recommendations developed in light of the additional information provided; and
• give the entity the opportunity to provide additional evidence, and assess whether any audit findings should be modified in light of the additional information provided.

223.6 ANAO staff attendance at the exit meeting, in addition to the audit team members, is to be determined on a case-by-case basis by the Engagement Executive.

223.7 Entities should be encouraged to invite senior representatives to exit meetings.

Chapters 224 to 227

224. Proposed section 19 report

Background

224.1 The proposed report is referred to as the section 19 report, because it is required by section 19 of the A-G Act.

Guidance

224.27 The section 19 report provides the opportunity for the entity subject to audit, and any other person that in the Auditor-General’s opinion has a special interest in the report, to provide comments that must be taken into consideration in preparing the final report.

224.28 The A-G Act provides 28 days for provision of comments on the section 19 report. Under the Acts Interpretation Act 1901 (Cth), the 28-day period is based upon calendar (and not working) days but does not include the day the recipient receives the section 19 report, meaning that where a recipient receives the section 19 report on a Monday, the last day of 28 day period will be a Tuesday. Where the last day of the 28-day period is a Saturday, Sunday or public holiday, the period is extended so that the last day is the first working day following the Sunday or public holiday.

224.29 Where one or more public holidays and/or public service annual close down days fall during the 28-day period, the Engagement Executive may consider providing the entity with more than 28 days to respond to the section 19 report.

224.30 The PASG Workflow provides instructions on preparing and issuing the section 19 report.

224.31 To be comprehensive, an audit report needs to include all the information and arguments needed to address the audit objective(s) and audit questions, while being sufficiently detailed to provide an understanding of the subject matter and the audit findings and conclusions.

224.32 To be convincing, an audit report needs to be logically structured and present a clear relationship between the audit objective(s) and/or audit questions, audit criteria, audit findings, conclusions and recommendations. It also needs to present the audit findings persuasively, address all relevant arguments to the discussion, and be accurate. Accuracy requires that the audit evidence presented and all the audit findings and conclusions are correctly portrayed. Accuracy assures readers that what is reported is credible and reliable.

224.33 Being timely requires that an audit report be issued on time in order to make the information available for use by the legislature, management, government and other interested parties.

224.34 To be reader friendly, the auditor needs to use simple language in the audit report to the extent permitted by the subject matter. Other qualities of a reader-friendly audit report include the use of clear and unambiguous language, illustrations and conciseness to ensure that the audit report is no longer than needed, which improves clarity and helps to better convey the message.

224.35 Being balanced means that the section 19 report needs to be impartial in content and tone. All audit evidence needs to be presented in an unbiased manner. The auditor needs to be aware of the risk of exaggeration and overemphasis of deficient performance. The auditor needs to explain causes and the consequences of the problems in the audit report because it will allow the reader to better understand the significance of the problem. This will in turn encourage corrective action and lead to improvements by the audited entity.

224.36 Some key questions that should be considered when drafting the section 19 report include:

• Are the sources underpinning the audit criteria / sub-criteria included in the report?
• Do the audit findings and conclusions relate to the audit objective?
• Do the recommendations flow from the findings?
• Have the questions posed in the audit been answered?
• Is the audit title appropriate?
• Is the structure of the report logical and does it flow easily?
• Does the report indicate the reliability of the audit evidence used to support audit findings and, in particular, whether evidence such as entity survey responses was corroborated by other audit evidence?
• Does the overall conclusion comprise a succinct, balanced discussion of those matters that directly support the conclusion against the audit objective, with significant findings covered in the key findings section?
• Do the recommendations focus on significant issues? Are they practical? Are they too long? Are there too many?
• Is the report internally consistent? That is, is there consistency between the body of the report, the overall conclusion and the key audit findings?

224.37 Audit criteria and their sources must be identified in the audit report because the intended users’ confidence in the audit findings and conclusions depends largely on the audit criteria. In performance audits, a wide variety of sources can be used to identify audit criteria²³.

ISO Standards

224.38 Any direct quotation of ISO standards without explicit permission from ISO would carry significant risk of breach of copyright, given ISO’s very restrictive approach. Given the potential legal implications, paraphrasing or referencing the standards is the appropriate approach.

Report summary

224.39 The general purpose, features and structure of the report summary of the section 19 report are outlined in the following paragraphs.

224.40 The report summary is an important part of the audit report. It should capture the essence of the audit, the key issues identified and the value added by the audit. The report summary is to be four to six pages in length, and aims to:

• attract the report user’s attention and interest; and
• provide a concise outline of the report and the key messages the report wishes to convey to the report user.

Background

224.41 The background is a short section that provides key introductory information about the audit.

224.42 The background section should cover:

• context, with a brief overview of the program or activity being audited;
• the service or program arrangements the audit addresses and their importance in terms of government policy, service delivery or governance;
• a summary of key outcomes, outputs and key performance indicators (where available);
• the dimensions of, or other insights into, the audit subject matter that are important to convey the significance of the audit topic;
• the rationale for undertaking the audit;
• the audit objective; and
• the high-level audit criteria.

224.43 The drafting of the section 19 report provides an opportunity to revise the audit objective, criteria and scope, to ensure they reflect the audit coverage and audit findings.

Conclusion

224.44 The overall conclusion is likely to be of most interest to the majority of users of ANAO reports.

Supporting findings

224.45 The report summary includes ‘Supporting findings’, immediately following the conclusion. This section includes the various findings that are listed throughout the report in the grey boxes attached to each chapter. These various findings should be copied directly into this section.

Recommendations

224.46 It is important that audit recommendations:

• address the cause of the issue or matter that requires improvement; for example, if the audit found that data quality was poor due to the absence of any quality management/assurance arrangements, the recommendation should be directed at the need to implement these arrangements, rather than simply recommending that data quality be improved;
• focus on significant issues. It is not necessary that every issue addressed in a report results in a recommendation;
• do not contribute to the ‘red tape’ burden and do not simply recommend an entity comply with an existing requirement;
• should stand alone; that is, they are able to be understood without reference to supporting material elsewhere in the audit report; and
• are realistic and achievable and have regard to the cost of implementation. In this context, it is important for the audit team to fully consider the cost and other implications (such as coordination and consultation requirements) of each recommendation and be satisfied that they are cost-beneficial in terms of improving performance and accountability.

224.47 Generally, recommendations should focus on what needs to be done, rather than how the entity should do it. When the means of implementing a recommendation is important, this should be addressed in the body of the report.

224.48 Even though an issue may not warrant a recommendation, a report should encourage the entity to address opportunities for improvement.

Key learnings

224.49 This section should not re-state the findings relating to the entity subject to the audit. It should include broad lessons for other government entities and inform improved business practices across the public sector which have come to light during this audit. It may relate to both areas for improvement and good practice identified in the audit. This information is also relevant to the quarterly audit insights publications and other avenues for disseminating observations from ANAO audit work.

Report length

224.50 The ANAO aims for reports of approximately 50 pages. The length may vary according to the substance of the audit and the audit conclusions reached.

Cross-entity reports

224.51 For cross-entity audits, a decision is usually made early in the audit (and advice given to the entities included in the audit) about whether the audit report will include findings, conclusions and recommendations for each entity, or will be written in generic terms.

224.52 Where the section 19 and final reports are to be written in generic terms, the audit team will draw out common audit findings, conclusions and recommendations and/or key findings from the report preparation or evidence papers that were developed for each entity included in the audit.

Procedural fairness

224.53 Procedural fairness refers to a set of rights that people derive under common law and have under the administrative law principles of natural justice.

224.54 Procedural fairness requires a decision-maker to hear a person before making a decision affecting the rights, interests and legitimate expectations of that person. In the case of performance audits, the decision is whether or not to include in the audit report certain information or opinions relating to a person or an entity.

224.55 The ANAO’s legal and procedural obligations to public sector entities and their employees will be satisfied by the normal consultation process followed during the conduct of a performance audit, including through the provision of reports under section 19 of the A-G Act. In the rare circumstances where the normal consultative process does not satisfy the ANAO’s procedural obligations, then it may be allowable for the relevant individual to be given the opportunity to comment on relevant extracts of the report.

224.56 Where third parties, such as contractors used by the entity subject to audit, could be identified in a report, they would normally be provided with an extract of the report that makes reference to them or allows them to be identified.

224.57 The decision on which party (or parties) should be provided with copies or extracts of a section 19 report rests with the Auditor-General. At the time the section 19 report is submitted to the Auditor-General for approval, details of the steps proposed to meet procedural fairness obligations should be outlined, including any proposed extracts strategy.

224.58 Specific legal advice may need to be obtained on how best to meet the ANAO’s procedural fairness obligations in particular circumstances, such as, where a number of individuals or bodies are able to be identified and it could reasonably be argued that the report could impact on their reputation.

Processing and issuing the section 19 report

224.59 The steps involved in processing the section 19 report for distribution are outlined in the PASG Workflow.

224.60 Where, following the section 19 meeting, the Auditor-General makes comment on, or requires changes to, the section 19 report, the Auditor-General will clarify whether a revised version of the report needs to be viewed and cleared or if the report can be sent to the entity on the basis that the agreed changes will be made by the Engagement Executive, or GED as necessary.

Persons or bodies provided with the section 19 report or extract

224.61 The section 19 report, or extract from the report, may be given to any person who, or entity that, the Auditor-General considers has a special interest in the report or the content of the extract.

224.62 Whether a person has a special interest and should therefore receive a full copy or extract of the section 19 report will need to be determined on a case-by- case basis. For example, where an audit of procurement has been undertaken, a full copy of the section 19 report might be provided to the Department of Finance because it has policy responsibility for procurement matters.

224.63 A person or organisation with a special interest in the report or content of an extract may also have cause to make a claim under subsection 37(1) of the A-G Act, for example under paragraph 37(2)(e) which relates to the inclusion in a public report of particular information that would unfairly prejudice the commercial interests of a person or organisation. Policies and guidance with respect to the operation of section 37 of the A-G Act are provided in Chapter 226.

224.64 The process for preparing extracts to be provided to a third party is a deliberate process of reviewing the report in its entirety and considering each part as to what to include/exclude in the extract. Audit teams should consult with GEDs and/or EDs with experience in this process.

224.65 Generally, the following arrangements for provision of the section 19 report should apply. The section 19 report of an audit of a:

• Commonwealth entity performed under paragraph 17(1)(a) of the A-G Act should be provided to a single official who is, or, in the case of a corporate entity is a member of, the accountable authority of the entity;
• Commonwealth company performed under paragraph 17(1)(b) should be provided to a director (usually the Chair) of the company; and
• corporate Commonwealth entity or a Commonwealth company, or a Commonwealth partner respectively, performed under paragraph 17(1)(c) and under section 18B should be provided to a person who is, or is a member of, the governing body of the subsidiary or the Commonwealth partner.

224.66 For Commonwealth entities, section 19 reports should only be sent to the accountable authority,²⁴ or in the case of Commonwealth companies, subsidiaries and Commonwealth partners (which do not have an accountable authority of their own), a director or member of the governing body of the entity. This will reduce the risk of accidental disclosure and give the accountable authority, director or member of the governing body control over who receives the section 19 report. Other officers (including internal audit and ANAO liaison) may be sent an email informing them that the section 19 report has been provided. Note that you may need to consult with the Auditor-General about provision of relevant papers to entities with unusual governance arrangements; for example, due to the diarchy arrangement the Auditor-General has consented to the provision of relevant papers to both the Secretary of the Department of Defence and the Chief of the Defence Force.

Comments sought on the section 19 report or extract

224.67 It is important to note that an extract from the section 19 report attracts the same protection regarding confidentiality and the associated penalties as the full section 19 report (section 36(3) of the A-G Act).

224.68 Senior members of the audit team should take an active role in communicating with the entity throughout the period that the entity has to provide comments on the section 19 report to assist in ensuring a timely response.

224.69 Discussing the audit with the entity in both formal and informal ways throughout the 28-day period can assist in identifying and resolving any issues that have the potential to delay the finalisation of the audit report. Every effort should be made to reach agreement, especially in relation to recommendations, without compromising the Auditor-General’s independence. Such efforts often include the involvement of senior management from the entity and the ANAO in further discussions during the 28-day response period. The responsible GED and ANAO Executive should be kept abreast of key developments.

224.70 Entities’ final comments on findings and recommendations should include an ‘agree’ or ‘disagree’ position on each recommendation.

Consideration of comments on the section 19 report or extract

224.71 Where substantive comments are received, it would generally be appropriate to have further discussions with senior management of the entity and to give consideration to making amendments to the final report. As a general rule, the quality of the final report will be enhanced if disagreements can be resolved and the final report amended as necessary. The responsible GED and ANAO Executive should be consulted on such proposed amendments.

224.72 Incorporating entity comments in the final report is discussed at paragraphs 226.22 to 226.25.

Retaining evidence of review of section 19 report

224.73 The iterative review of the section 19 report can be extensive for some audits. Audit teams are to make judgements regarding the number of reviewed versions of the section 19 report that are required to be retained on the audit working papers. For example, drafts that contain minor editorial changes and comments only may not need to be retained.

Liaison with FSASG

224.74 In determining what is a significant issue requiring communication with FSASG, PASG shall consider:

• the potential to give rise to a material mistake in the financial statements;
• any actual or suspected fraud;
• any issues revealing a weakness in key financial controls; and
• any other issues of which the audit team is aware that may be of interest to FSASG.

225. Key learnings

Guidance

225.3 The audit team will identify and document the key learnings from performance audits and other engagement reviews. These include lessons from other government entities that may inform improved business practices across the public sector.

225.4 PASG practice management has circulated key headings to assist in grouping findings under relevant audit activity types. These key learnings will be used to inform the ANAO Insights products.

225.5 Also refer above to paragraph 224.24 and 224.47.

226. Final report

Guidance

226.17 The final audit report, setting out the ANAO’s audit findings, overall conclusion and recommendations, is tabled in the Parliament and is available to the public.

226.18 The PASG Workflow provides instructions on preparing the final report.

Structure and content

226.19 The final audit report should be objective, balanced and constructive in the presentation of the overall conclusion and the audit findings. Recommendations should be practical and cost-effective.

226.20 The structure and content of the final audit report should be substantially the same as the section 19 report.

Section 37 of the A-G Act

226.21 In providing any advice sought by the Auditor-General in respect of the disclosure of particular information under section 37 of the A-G Act, audit teams should take into consideration that, in administering the A-G Act, specifically in respect of section 37, the Auditor‐General’s approach is in favour of disclosure to the Parliament, in the public interest, unless the Auditor‐General is of the opinion that the public interest is served by not disclosing ‘particular information’ which is otherwise prohibited from public release (for example, information with a national security classification). There is always a balance to be struck between the public interest in not disclosing information and the broader public interest in reporting transparently to the Parliament.

Incorporating entity comments in the final report

226.22 The final audit report includes the entity’s response to the section 19 report in the report summary as the final section titled Summary of entity response. It should be no more than half a page in length. If the entity’s response is longer, the audit team may discuss with the entity their willingness to prepare a shorter summary, or the full text should be included as an appendix.

226.23 Providing the entity with the opportunity for the formal comments to be amended, when a change has been made to the final report in response to an entity comment, avoids the situation where the entity’s published response relates to aspects of the report including the audit findings and conclusions that have been amended as a result of consideration of comments received on the section 19 report.

226.24 A rejoinder is the ANAO’s response to entity comments. Where the ANAO includes a rejoinder to the formal response of the entity in the final report, it should be described as an ANAO comment and placed below the entity’s comments. The ANAO comment will be a statement of fact addressing the specific entity comment and drawing attention, where appropriate, to the amended section of the report.

Approval and processing of the final report

226.25 See the PASG Workflow for detailed instructions on approving and publishing the final report.

227. Publication and closure

Guidance

227.5 The PASG Workflow provides procedures on the publication and closure processes. This includes several procedures to forward the Final Report to publications, close the audit e-hive files and complete a Continuous Improvement Report and a JCPAA Briefing. While the ANAO Communication Unit plays an important role in producing the final report, organising its presentation to Parliament, and publishing after tabling on the ANAO website, the audit team remains responsible for ensuring that the words and layout are correct and that the report is complete and accurate.

Final draft to stakeholders (‘A4s’)

227.6 Third parties that received an extract of the section 19 report do not, as a matter of course, receive a copy of the final draft of the report. If a third party requests an extract of the final report, the decision should be referred to the Group Executive Director for consideration.

Tabling of the audit report

227.7 If the final report is to be presented when the Parliament is not sitting, it will be presented under Senate standing order 166 which provides for the presentation of documents when the Senate is not sitting. Upon receipt by the President of the Senate, the document is deemed to have been presented to the Senate, is authorised for publication, parliamentary privilege is attached, and the embargo is lifted. The audit report will then be tabled at the next sittings of the House of Representatives and the Senate.

227.8 In addition to those persons and/or bodies who must receive a copy of the final report, a copy of the final report, or an extract, may also be given to a person or body that has a special interest in the report or the content of the extract.

Parliamentary review

227.9 All performance audits are required to be examined by the Joint Committee of Public Accounts and Audit in accordance with sub-section 8(1)(c) of the Public Accounts and Audit Committee Act 1951. The committee conducts public inquiries into a selection of audit reports each year, following consultation with the ANAO. The ANAO may provide a private briefing to the committee and gives evidence at these inquiries.

227.10 The PASG Workflow states that the audit team prepares a JCPAA Briefing and saves a PDF version to the JCPAA Summary folder on the ANAO shared library.

227.11 From time to time, other parliamentary committees also conduct inquiries into performance audit reports or aspects of public administration that have been the subject of performance audit coverage.

Attachment 1: Generic audit criteria for Grants administration