The Performance Audit Services Group (PASG) volume of the ANAO Audit Manual applies to the performance audit activity performed by PASG in collaboration with the Systems Assurance and Data Analytics (SADA) group. Relevant policies and guidance from the PASG volume are also applied to assurance reviews performed by PASG. Policies and guidance in the PASG volume address the planning, execution and reporting stages of the performance audit process.

Engagement performance — general

Chapters 201 to 205

201. Project managed audits

Background

201.1 Under section 27 of the Auditor-General Act 1997 (A-G Act), the Auditor-General, on behalf of the Commonwealth, may engage any person under contract to assist in the performance of any Auditor-General function1. For various reasons, the ANAO engages firms to undertake some performance audits or discrete parts of performance audits on his behalf.

Policy

201.2 Tenders and contracts for a performance audit or part of a performance audit shall require contractors to comply with the requirements of:

  1. the ANAO Auditing Standards;
  2. legislation or regulations relevant to the audit, including minimum security clearance requirements;#

  3. APES 110 Code of Ethics for Professional Accountants, as supplemented by ANAO policy in relation to the provision of other services to ANAO auditees; and
  4. the ANAO Audit Manual.

    201.3 The ANAO Engagement Executive is the ‘lead assurance practitioner’ for the purposes of the Standards on Assurance Engagements (ASAEs) and retains responsibility for the overall quality on the engagement. This includes responsibility for:

    1. The engagement being planned and performed (including appropriate direction and supervision) to comply with professional standards and applicable legal and regulatory requirements;
    2. Review of audit work being performed in accordance with the ANAO Audit Manual and reviewing the engagement documentation on or before the date of the performance audit report;
    3. Appropriate engagement documentation being maintained to provide evidence of achievement of the audit objectives, and that the engagement was performed in accordance with relevant ASAEs and relevant legal and regulatory requirements;
    4. Appropriate consultation being undertaken by the engagement team on difficult or contentious matters;
    5. Remaining alert for evidence of non-compliance with relevant ethical requirements by members of the engagement team; and
    6. Considering the results of the ANAO’s monitoring procedures and whether deficiencies noted and communicated by PSRG may affect the engagement.

      201.4 Where an Engagement Quality Control Reviewer (EQCR) is assigned to the engagement in accordance with ANAO Audit Manual – Shared Content, paragraph 8.40, the Engagement Executive shall take responsibility for discussing significant matters arising during the engagement with the EQCR, and not submit the final draft report to the ANAO Executive for clearance until completion of that review.

      201.5 The contractor is required to plan and conduct the engagement in accordance with the ANAO’s methodology for performance audits. This requires compliance with the ANAO Audit Manual and use of ANAO templates including for the Audit Work Plan, Progress Review briefings, Report Preparation Papers, draft proposed report under section 19 of the Auditor General Act 1997 and final draft report.

      201.6 The Engagement Executive shall make such enquiries and document their review of such work papers as is necessary so as to be satisfied that the quality control procedures applied to the audit are in accordance with the requirements of the contract, including:

      1. meeting ANAO Auditing Standards and ANAO Audit Manual;
      2. the use of contractor staff with appropriate competence, qualifications and experience.

        201.7 For high risk audits, the Engagement Executive’s involvement shall be extended with a greater involvement in audit planning and execution, more regular meetings with firm and auditee, and review of significant matters arising during the audit.

        201.8 Prior to submitting the final draft report to the ANAO Executive for clearance, the Engagement Executive shall be satisfied that the contractor’s work provides sufficient appropriate audit evidence to support the audit findings and conclusions and the subsequent release of the performance audit report. The contract engagement partner (or equivalent) shall also provide written sign off on the audit in accordance with the approved form Contractor’s Representation Letter.

        201.9 Audit documentation, including contractor firm files, shall be complete and ready for finalisation in accordance with ANAO Audit Manual – Shared content, paragraph 9.11. In order to facilitate this, the ANAO shall provide the contractor with access to the ANAO premises and secure access to the ANAO’s document management system (E-Hive).

        201.10 The contractor shall be required to support the ANAO’s internal and external quality assurance processes by providing audit documentation and any additional information requested relating to both in progress and completed engagements in a timely manner.

        Guidance

        201.11 The mandatory requirements governing ANAO audits need to be made known to contractors via tender and contract documentation.

        201.12 ANAO policy is to comply with the requirements of APES 110 Code of Ethics for Professional Accountants. Those requirements apply equally to contractor firms and their staff. ANAO policy also adds to these professional requirements, for example; ‘Provision of other services by ANAO Contractors to ANAO auditees’ includes additional prohibited services to the requirements of APES 110.

        201.13 To help ensure contractors comply with ANAO Policy, paragraph 201.2 of this manual requires tenders and contracts for audit services to make provision for the ANAO Engagement Executive to make the contractor aware of policies and procedures the ANAO requires be followed on the audit. The procedural steps relevant to performance audits are included in the PASG Workflow, which contains corporate templates to be used during an audit.

        201.14 The ANAO Engagement Executive is formally the ‘lead assurance practitioner’ under the ASAEs. In practice, when the ANAO contracts out an audit, or part of an audit, some of the duties of the lead assurance practitioner (e.g. supervising the conduct of fieldwork) may be fulfilled by the contactor partner (or equivalent).

        201.15 In accordance with ASQC1, the ANAO’s review policies and procedures (including the ANAO Audit Manual and PASG Workflow) are determined on the basis that the work of less experienced team members is reviewed by more experienced team members.

        201.16 The Engagement Executive shall comply with the requirement under the ANAO Auditing Standards to be satisfied that sufficient appropriate audit evidence has been obtained to support the conclusions reached and auditor’s report to be issued. The Engagement Executive’s review covers critical areas of judgement, especially those relating to difficult or contentious matters identified during the course of the engagement, significant risks and other areas the Engagement Executive has identified as important.

        201.17 The audit contractor is responsible for ensuring that audit work undertaken on behalf of the Auditor-General is performed in accordance with professional standards and to have in place quality control policies and assurance procedures to be employed throughout the audit engagement.

        201.18 Whereas financial audit work conducted by the Auditor-General is broadly similar to the assurance work performed by the external auditing profession in Australia, performance auditing is the almost exclusive domain of Auditors-General. While based upon the same assurance framework as compliance, controls and similar assurance work commonly performed by the private sector there are extensive differences. The ANAO Engagement Executive should be conscious of the level of familiarity with the ANAO performance audit methodology in determining the extent of their involvement in the performance audit.

        201.19 The expected milestone dates should be communicated to the contract partner at the commencement of the audit. Milestones usually include:

        • Preparation and approval of the Audit Work Plan;
        • Entry meeting conducted with the auditee;
        • Progress review meetings with the relevant ANAO Engagement Executive on a three weekly basis during the period of fieldwork;
        • Progress review briefing to the ANAO Executive during the finalisation of fieldwork;
        • Provisions of draft Report Preparation Papers (RPPs) to the ANAO for clearance prior to issuance of RPPs to auditee(s) for comment;
        • Exit interview conducted with the auditee;
        • Preparation of a formal draft audit report (section 19 report) and workshop / clearance of the report with the ANAO Executive prior to issuance to auditee(s) for comment;
        • Amendment of the section 19 report (as appropriate) for auditee responses, followed by provision of a final draft report to the ANAO Executive for consideration / clearance;
        • The final performance audit report to be available for tabling date; and
        • Audit wrap-up and lessons learnt meeting.

        202. Provision of other services by ANAO contractors to ANAO auditees

        Background

        202.1 This policy applies to individuals or firms contracted by the ANAO to provide performance audit services as an in-house contractor or undertake a contracted-out (project-managed) audit. It applies when a potential ANAO audit contractor tenders for an ANAO audit, and also when an existing ANAO audit contractor wishes to provide other services to an ANAO auditee.

        202.2 ASA 102 Compliance with Ethical Requirements when Performing Audits, Reviews and Other Assurance Engagements requires compliance with relevant ethical requirements when performing audits, reviews and other assurance engagements, which is defined and includes APES 110 Code of Ethics for Professional Accountants (APES 110).

        202.3 APES 110 applies to all ANAO engagements except to the extent, if any, of a conflict between APES 110 and legislative requirements.

        202.4 An important part of independence requirements involves consideration of the provision by an auditor of other services to an auditee.2 While the ANAO itself does not provide other non-assurance services to its auditees, firms or individuals contracting to the ANAO may seek to do so.

        202.5 This policy and guidance does not address other aspects of independence relevant to audit contractors. Refer to the ANAO Independence Policy and APES 110 for further requirements and guidance on other aspects of independence.

        Policy

        Application

        202.6 This policy shall apply to tenderers for ANAO performance audit work and existing contractors engaged to perform ANAO performance audit work.

        202.7 The requirements of this policy shall be made available to actual and prospective audit contractors.

        Approach

        202.8 Subject to applicable legislation and the other requirements of ANAO policy, the conceptual approach to resolving threats to independence in APES 110 shall be applied to:

        1. Identify threats to independence;
        2. Evaluate the significance of the threats identified; and
        3. Apply safeguards, when necessary, to eliminate the threats or reduce them to an acceptable level.

          202.9 In some circumstances, a threat may be so significant that no safeguard can reduce it to an acceptable level and the other service shall be refused or contract not awarded.

          Prohibited services

          202.10 In addition to the other services expressly prohibited by APES 110, where a performance audit is conducted, the contractor firm or a network firm3 is not permitted to provide consultancy services, including the provision of professional or technical advice, which relates to the criteria and recommendations of the performance audit.

          Fee parity

          202.11 The prior approval of the Group Executive Director (GED) of PSRG is required for a proposed or existing contract where the value of other services provided or to be provided by the ANAO contractor to the ANAO auditee exceeds the value of the ANAO contract to the contractor.

          Request by existing contractors to provide other services

          202.12 ANAO contracts with audit contractors shall provide that no other services may be tendered for or provided by the contractor to that ANAO auditee during the period of the audit engagement without the express agreement in writing of the ANAO.

          202.13 Requests to provide other services shall be supported by formal documentation providing detail of the other services, associated fees and a description of the procedures for monitoring conflict management. This documentation is to be prepared or endorsed by the audit contractor’s independence panel or independence partner.

          202.14 The contractor shall bring the proposed request to the attention of the Chair of the auditee’s Audit Committee for advice as to any conflict of interest perceived by the Committee. The view of the Audit Committee Chair shall be included in the proposed request to the ANAO.

          202.15 The Engagement Executive for the audit shall review the proposals for other services and provide a documented assessment with a recommendation to the GED PSRG for approval.

          202.16 The Engagement Executive for an audit shall ensure the following information is documented4 about any proposal for the provision of other services by an ANAO audit contractor to that ANAO auditee:

          1. Other services proposed;
          2. Nature of the threat posed by the proposal;
          3. Safeguards considered to reduce the threat to an acceptable level;
          4. Decision whether the threat can be reduced to an acceptable level; and
          5. Consultation undertaken in reaching decision.

            202.17 Where consultation is undertaken, this information shall be provided to those consulted.

            202.18 In respect of contracted-out audits or parts thereof, before the signing of the auditor’s report on which a contractor has been engaged, the Engagement Executive for the audit shall obtain the following from the audit contractor in the Contractor’s Representation Letter:

            1. a complete listing of other services provided, to be provided or currently being tendered for by the contractor from the beginning of the reporting period under audit;
            2. the remuneration paid or payable for those services;
            3. a declaration that the contractor has not entered into arrangements for, or provided, other services without the prior written consent of the ANAO; and
            4. a declaration that the contractor has met the independence requirements of applicable legislation, APES 110 and ANAO policy and has not otherwise been impaired.
              Reporting

              202.19 PASG is to report annually to the Executive Board of Management (EBOM):

              1. the nature and amount of each type of other services provided by audit contractors to each affected ANAO auditee;
              2. the amount of the fees paid or payable by the ANAO auditee to the contractor under each engagement; and
              3. the total fees paid or payable by the ANAO to the contractor.
                Reporting to the auditee’s audit committee

                202.20 Where other services are provided by the contractor to the ANAO auditee during the period of the audit engagement, the other services provided by the contractor to the auditee and the fees paid or payable to the contractor for those services shall be disclosed to the auditee’s Audit Committee.

                Guidance

                202.21 As part of the tender process for prospective contractors, the contractor is required to declare any conflicts of interest and any other services that they are providing to auditees.

                Approach

                202.22 APES 110 categorises threats to independence as follows:

                1. self-interest threat - the threat that a financial or other interest will inappropriately influence the audit contractors (and ANAO staff) judgement or behaviour; or
                2. self-review threat - the threat that audit contractors (and ANAO staff) will not appropriately evaluate the results of a previous judgement made or service performed by the audit contractors (and ANAO staff) or another individual within the firm (or employing organisation) on which the audit contractors (and ANAO staff) will rely when forming a judgement as part of providing a current service; or
                3. advocacy threat - the threat that audit contractors (and ANAO staff) will promote an auditee’s or employer’s position to the point that subsequent objectivity is compromised; or
                4. familiarity threat - the threat that due to a long or close relationship with an auditee or employer, audit contractors (and ANAO staff) will be too sympathetic to their interests or too accepting of their work; or
                5. intimidation threat - the threat that audit contractors (and ANAO staff) will be deterred from acting objectively because of actual or perceived pressures, including attempts to exercise undue influence over the audit contractors (and ANAO staff).
                  Non-assurance services considered in APES 110

                  202.23 APES 110 considers the provision of non-assurance services to auditees, including assuming management responsibilities. In the case of performance audits the relevant section of APES 110 is Section 291 related to assurance engagements on subject matters other than historical financial information.

                  202.24 APES 110 prohibits a firm or network firm from assuming management responsibilities of any auditee as the threats created would be so significant that no safeguards could reduce the threats to an acceptable level.

                  202.25 Threats to independence may be created when a firm or network firm provides consultancy services related to an activity which is the subject of a performance audit conducted by that firm. If the proposed consultancy services relate specifically to the recommendations of the performance audit, the firm or network firm is prohibited from providing the proposed services.

                  202.26 A self-review threat may be created if a firm or network firm provides consultancy services in relation to an activity that is subsequently the subject of a performance audit conducted by the firm.

                  202.27 In both cases, the significance of the firm’s involvement with the activity needs to be evaluated in accordance with paragraphs 202.8 - 202.11 of this policy.

                  Other services

                  202.28 For services other than those prohibited by APES 110 or ANAO policy, the threat created by the provision of another service may be able to be reduced to an acceptable level by applying safeguards including:

                  1. arranging for other services to be performed by an individual who is not a member of the audit team;
                  2. if such services are performed by a member of the audit team, using a partner or senior staff member with appropriate expertise who is not a member of the audit team to review the work performed.

                    202.29 APES 110 paragraphs 200.12 to 200.15 provide examples of ‘Firm-wide safeguards’; ‘engagement –specific safeguards’; and ‘safeguards within the client’s systems and procedures’ that may eliminate or reduce threats to an acceptable level, such as:

                    1. using different partners and engagement teams with separate reporting lines for the provision of non-assurance services to the ANAO client;
                    2. discussing ethical issues with those charged with governance, such as the audit committee;
                    3. disclosing to those charged with governance, such as the audit committee, the nature of services provided and extent of fees charged;
                    4. rotating senior assurance team personnel;
                    5. the ANAO auditee implementing internal procedures that ensure objective choices in commissioning non-assurance engagements;
                    6. the ANAO auditee’s corporate governance structure that provides appropriate oversight and communications regarding the contractor’s services.

                      202.30 Dialogue regarding the application of legislation, government policies and other relevant guidance/audit criteria that does not constitute advice is considered to be a normal part of the audit process and does not generally create threats to independence.

                      Fee considerations

                      202.31 The ANAO needs to consider the perception of independence where the proposed value of other services performed and requested to be performed by the contractor exceeds the value of the ANAO contract. The objectivity of the contractor may be impaired if they are dependent on the auditee for other services income as concern about losing the other services can create a self-interest or intimidation threat, particularly when that income is greater than the value of the ANAO audit contract.

                      202.32 A significant threat to contractor independence may arise, where, over the period of a proposed or existing ANAO contract, the value of other services provided or to be provided by the ANAO contractor to the ANAO auditee would exceed the value of the ANAO contract to the contractor.

                      202.33 For a proposed contract, the tender evaluation team and, for an existing contract, the Engagement Executive are required to seek approval from the GED PSRG where the fees for proposed or existing other services exceeds the audit fee. The fee parity policy requires consideration of the fees over the life of the audit contract. This may allow some flexibility to approve ‘one-off’ other services that exceed the audit fee in a single year, provided that the value of the total contract is not exceeded.

                      202.34 If the fee for other services is a contingent fee, this may create a self-interest threat that APES 110 requires to be evaluated and safeguards applied to eliminate the threat or reduce them to an acceptable level (refer to APES 110 paragraphs 290.221 to 290.224 for further guidance on contingent fees).

                      Network firms

                      202.35 Network firms are required to be independent of the contractor’s auditee. A Network is defined in APES 110 as ‘a larger structure that is aimed at co-operation; and that is clearly aimed at profit or cost sharing or shares common ownership, control or management, common quality control policies and procedures, common business strategy, the use of a common brand-name, or a significant part of professional resources.’

                      202.36 Requests to provide other services by ANAO contractors should include relevant information in respect of their network firms.

                      203. Role and responsibilities of Group Executive Directors

                      Policy

                      203.1 The GED shall participate in key stages of the planning and delivery of performance audits including in the initial selection of audit topics and in progress reviews held during the conduct of an audit.

                      Guidance

                      203.2 The GEDs manage the ANAO’s Performance Audit Services Group (PASG) and engage in key elements of the conduct of a performance audit.

                      203.3 The strategic priorities of the GED are set out in the Service Group plan.

                      203.4 The procedural steps relevant to GEDs are included in the PASG Workflow, which contains corporate templates for staff to use during an audit.

                      203.5 For high risk audits, the GED is expected to have greater involvement in audit planning and execution, and review significant matters arising during the audit.

                      204. Roles and responsibilities of the Engagement Executive

                      Background

                      204.1 The Engagement Executive is the ‘lead assurance practitioner’ for the purposes of the Standards on Assurance Engagements. The Engagement Executive is the head of a single administrative Branch in the Service Group that is responsible for delivering multiple performance audits in a financial year that support the delivery of the ANAO’s Annual Audit Work Program.

                      Policy

                      204.2 The Engagement Executive shall take responsibility for the overall quality on the engagement, including:

                      • The engagement being planned, performed and documented in accordance with the ANAO auditing standards, this Manual any other relevant ANAO policy, legal and regulatory requirements;
                      • Appropriate review of the engagement documentation before the date of the assurance report; and
                      • That appropriate consultation has been undertaken on difficult and contentious matters.

                      Guidance

                      204.3 The responsibilities in the auditing standards which this policy places on Engagement Executives include the following:

                      • the overall quality on each audit engagement to which the Engagement Executive is assigned;
                        1. Through their actions and appropriate messages to the engagement team, the Engagement Executive should emphasise the importance of compliance with ANAO Auditing Standards and quality control policies and procedures. In addition the engagement team should have the ability to raise concerns without fear of reprisals. Quality is essential in performing audit engagements and the overall quality of the audit will be helped by ensuring that the audit is performed in a manner consistent with ANAO standard methodology.
                      • the engagement team’s compliance with relevant ethical requirements including APES 110;
                        1. These include the principles of integrity, objectivity, professional competence and due care, confidentiality and professional behaviour. The Engagement Executive should remain alert for evidence of non-compliance and if it should occur, in consultation with others, determine the appropriate action.
                      • the engagement team’s compliance with the ANAO’s Independence Policy;
                      • the appropriate planning of the engagement consistent with ASAE 3500 Performance Engagements;
                      • the application of engagement team members, specialists and auditor’s experts which collectively have the appropriate levels of competencies and capabilities;
                        1. The Engagement Executive is required by the ANAO Auditing Standards to be satisfied that the competence and capabilities of the audit team will ensure performance of the audit engagement consistent with professional standards and regulatory requirements and enable an auditor’s report that is appropriate in the circumstances.5 When considering the competence and capability of the engagement team as a whole the Engagement Executive may take into account such matters as the team’s understanding and practical experience of audit engagements of a similar nature, their understanding of professional standards and regulatory and legal requirements, technical expertise, ability to apply professional judgement and understanding of the ANAO’s quality control policies and procedures.
                      • the direction, supervision and performance of the engagement consistent with professional and auditing standards and regulatory and legal requirements. The Engagement Executive should document the extent and timing of their reviews. Refer to the policy Direction, Supervision and Review (ANAO Audit Manual - Shared Content, paragraphs 8.2-8.4) for further guidance;
                      • that sufficient and appropriate audit evidence exists and is documented to support the conclusions reached and for the auditor’s report to be issued;
                        1. In line with the ANAO policy Audit Documentation (ANAO Audit Manual - Shared Content, paragraphs 9.2-9.14) the only work that should be done after the issuing of the final report is that of an administrative nature.
                      • following appropriate procedures for consultations and differences of opinion and in particular ensure compliance with the ANAO policy on Differences of Opinion (ANAO Audit Manual - Shared Content, paragraphs 8.64-8.68);
                      • determine that an EQCR has been appointed, as required by the ANAO Auditing Standards and ANAO policy (Refer to the Engagement Quality Control Review (ANAO Audit Manual - Shared Content, paragraph 8.42); and
                      • enough involvement in the audit engagement at appropriate stages throughout the engagement including attendance at key meetings, discussions with the engagement team, EQCR and ANAO Executive and the completion of review of the planning and completion procedures at the appropriate stages of the audit.

                      204.4 Engagement Executives are assisted by the Audit Manager and team allocated to the engagement in fulfilling these responsibilities, including assisting in:

                      • consideration of whether the audit should be delivered through in-house or externally contracted resources;
                      • ensuring all audit related documentation is filed;
                      • planning the scope of work to produce the audit deliverables in the agreed timeframe;
                      • planning the time taken to prepare for, conduct and close the engagement;
                      • delivering in accordance with agreed timeframes;
                      • monitoring the costs associated with the audit, including recommending a variation to the budget if required;
                      • ensuring the quality of the audit deliverables;
                      • resourcing the audit to ensure that the audit team has the requisite skills to undertake the audit;
                      • the direction, review and supervision of audit team members;
                      • communication within the ANAO, among audit team members, and the entity being audited, and more broadly other interested parties;
                      • assessing and managing the operational and engagement risks associated with the audit;
                      • the procurement of any specialist resources and any associated contract management;
                      • documenting the agreed scope, timeliness and quality assurance arrangements in respect of any services required from SADA or AASG to contribute to a performance audit; and
                      • ensuring all persons engaged in the audit complete the required independence documentation and action is taken to manage any declared conflicts as required.

                      204.5 Audit Managers are expected to regularly monitor progress against established audit milestones and complete the actual date that audit milestones are achieved in the ANAO’s Changepoint system in a timely manner. The data held in Changepoint forms the basis of reports to ANAO senior executives.

                      204.6 When engaging with the entity being audited, especially on difficult or contentious matters, the Engagement Executive and audit team should ensure that a professional and productive approach is taken. This includes, for example, trying to understand the audited entity’s circumstances, operating environment and point of view.

                      204.7 The Engagement Executive should be aware of any risks to audit timeliness and budget, and escalate these as soon as practicable.

                      204.8 The procedural steps involving the Engagement Executive are included in the PASG Workflow. As outlined in the PASG Workflow, the level of responsibility differs for audits with different risk ratings.

                      205. Role and responsibilities of the SADA Executive

                      Background

                      205.1 This policy sets out the responsibilities of the SADA Executive in a performance audit or any other PASG assurance engagement that engages the specialist skills of the SADA Group.

                      205.2 A SADA Executive is allocated to a performance audit or other assurance engagement consistent with ANAO Audit Manual - Shared Content, paragraph 6.4.

                      Policy

                      205.3 The role and responsibilities of the SADA Executive shall include:

                      • the direction, supervision and performance of the SADA component of the engagement;
                      • reviewing key documents and working papers on the audit file, including:
                        1. the Audit Work Plan and documentation of the agreed scope of SADA involvement;
                        2. those relating to significant IT risks, judgements and difficult or contentious matters; and
                        3. components of the section 19 report and Report Preparation Papers relevant to SADA work undertaken during the audit;
                      • having sufficient involvement in the engagement at appropriate stages including attendance at key audit meetings, discussions with the engagement team, and Engagement Executive (where appropriate); and
                      • attendance at Audit Committees, where appropriate.

                      Guidance

                      205.4 The responsibility for the direction, supervision and performance of the SADA component of the engagement includes:

                      • emphasising the importance of audit quality on each engagement;
                      • tracking the progress and quality of the SADA component of the engagement;
                      • considering the competence and capabilities of the SADA audit team assigned to the engagement;
                      • ensuring that appropriate SADA procedures are planned and performed;
                      • addressing significant SADA matters arising during the engagement and the impact on the planned approach;
                      • ensuring the SADA work performed supports the conclusions reached and is appropriately documented; and
                      • agreeing with the performance Engagement Executive any SADA services to be provided in a performance audit, the scope, timeliness and quality assurance arrangements for those services, as well as ensuring those services are appropriately resourced.

                      205.5 IT can pose specific risks to an entity’s internal control. Some examples of IT risks which may be relevant to performance audits are:

                      • Reliance on systems or programs that are inaccurately processing data, processing inaccurate data, or both.
                      • Unauthorised access to data that may result in destruction of data or improper changes to data, including the recording of unauthorised or non-existent transactions, or inaccurate recording of transactions. Particular risks may arise where multiple users access a common database.
                      • The possibility of IT personnel gaining access privileges beyond those necessary to perform their assigned duties thereby breaking down segregation of duties.
                      • Unauthorised changes to data in master files.
                      • Unauthorised changes to systems or programs.
                      • Failure to make necessary changes to systems or programs.
                      • Inappropriate manual intervention.
                      • Potential loss of data or inability to access data as required.

                      205.6 Attendance at key meetings should be determined by the nature of the audit and the level of SADA involvement. If SADA involvement has been minimal and/or there are no material findings arising from SADA work performed, it may not be necessary for the SADA Executive to attend entry / exit meetings with the auditee, progress reviews or the section 19 workshop. However attendance at key meetings would be expected if there was extensive SADA involvement and/or material findings arising from SADA work performed.

                      Engagement performance — planning

                      Chapters 206 to 212

                      206. Understanding the entity and activity subject to audit

                      Policy

                      206.1 The audit team shall obtain sufficient understanding of the entity and activity to be audited. This will assist the audit team to identify and assess the risks that the activity is not economic, efficient, effective or ethical, and design and undertake evidence-gathering procedures.

                      Guidance

                      206.2 The collection of information about the entity and activity to be audited is a key element of initial planning for an audit.

                      206.3 Obtaining an understanding of the activity and its context is an essential part of planning and conducting a performance audit. It includes gaining knowledge of the entity that is responsible for the activity, and where relevant, the broader program of which the activity is part.

                      206.4 This provides the auditor with a framework to:

                      • distinguish between the activity and its control systems;
                      • develop and assess the suitability of criteria;
                      • target sources of relevant evidence;
                      • identify performance audit operational risks;
                      • identify engagement risks and assess materiality;
                      • identify whether there is the need for specialist skills or the work of an expert; and
                      • estimate resource requirements.
                      Types of information to be collected

                      206.5 The types of information that it may be appropriate to collect about the entity subject to audit includes:

                      • objectives of the entity;
                      • external accountability relationships—who the stakeholders and clients are and what their interests are in the entity;
                      • internal accountability relationships—such as organisational arrangements, delegations and committee structures;
                      • resources—the physical, financial, human and information resources available to the entity;
                      • applicable legal and policy frameworks / requirements for the activity
                      • management processes including:
                        • governance arrangements;
                        • performance criteria used by management;
                        • assessment of performance by management;
                        • the nature and frequency of reporting on performance;
                        • the use of performance results to assess entity operations and performance;
                        • the systems and controls in place for controlling the entity’s resources and ensuring appropriate client service;
                        • risk assessments used by management; and
                        • the role of internal audit in performance auditing;
                      • performance goals—their consistency with the entity’s legislation and governance framework;
                      • methods of program delivery—intended outputs of programs and outcomes, delivery methods and constraints on effective delivery;
                      • the external environment—factors that influence the entity’s operations, over which the entity may have little control, such as economic, social and political influences, with a particular focus on changes to that environment; and
                      • other publicly available information on the program.

                      206.6 The types of information to be collected for a cross-entity audit may focus more directly on the subject matter of the activity to be examined in the audit rather than entity-wide information. For example, the subject matter could include fraud control arrangements, the corporate planning framework, project and contract management, internal audit operations, recordkeeping or human resource management.

                      Sources of information

                      206.7 As a starting point, information already available within the ANAO should be reviewed and discussions undertaken, as necessary, with other ANAO staff who have knowledge of the entity or activity to be audited. Background information may have already been collected as part of the ANAO’s annual planning process, as part of another performance audit, or by AASG as part of the audit of an entity’s financial statements.

                      206.8 Background information can also be derived from public or other sources external to the entity such as:

                      • the entity’s website;
                      • enabling and program-specific legislation;
                      • Cabinet submissions and decisions;
                      • the entity’s Corporate Plan, Annual Report and Portfolio Budget Statements;
                      • audit committee papers;
                      • Senate Estimates and other Parliamentary hearings, parliamentary committee reports, and Second Reading Speeches;
                      • media reports, Ministerial and entity press releases, newspaper and journal articles, and television and radio reports;
                      • central entity policies, standards, directives and guidelines;
                      • Government and review tribunal hearings and reports, such as those by the Commonwealth Ombudsman;
                      • reports of other external scrutineers, such as the Inspector-General of Taxation;
                      • the entity’s planning documents and organisation charts; and
                      • Australian or overseas material from entities that have similar programs or experiences.

                      206.9 In addition, the audit team should hold discussions with the entity, including Internal Audit, and obtain and review documentation, including:

                      • relevant policies, plans and procedures; and
                      • reports on any evaluations or reviews.

                      206.10 Sources of information for cross-entity audits, in relation to the subject matter rather than specific entity information, include:

                      • policies or guidance promulgated by central agencies;
                      • audits conducted by the ANAO and other audit offices within Australia or overseas;
                      • internal audits and reviews conducted by the entities included within the scope of the audit; and
                      • professional and community organisations and standard-setting bodies such as Standards Australia.

                      206.11 The audit team may also hold discussions with one or more of the entities to be included in a cross-entity audit to gain a practical understanding of the subject matter of the audit. The decision to undertake such discussions and collect documents from entities will need to be decided on a case-by-case basis. Some factors to consider when making the decision include:

                      • whether it is likely that the particular policy or practice is widely understood and is being implemented; and
                      • the need for information to assist in developing a suitable audit objective, criteria and approach.

                      206.12 The information-gathering powers in the A-G Act can be used to obtain information and documents required for planning an audit. However, in practice, the information-gathering powers are used as ‘reserve’ powers and access to required information is almost always obtained through cooperation with entities. If an entity is not cooperating, consult with the Engagement Executive and responsible GED in the first instance to escalate the request. If necessary, they will consider possible use of either section 33 or requesting that the Auditor-General uses section 32 to obtain the necessary information. ANAO Legal Services can also assist with informing entities about the nature of the information-gathering powers.

                      207. Materiality and risk assessment and management

                      Policy

                      207.1 Materiality shall be considered when determining the nature, timing and extent of procedures.

                      207.2 Materiality shall be assessed in planning and reassessed if there is any indication that the basis on which the materiality was determined has changed.

                      207.3 Materiality shall also be considered when evaluating the effect of any identified findings, taken individually and in combination.

                      207.4 A matter shall be considered material if it is significant to the performance of the activity in relation to economy, efficiency and/or effectiveness evaluated against the criteria. During the performance engagement the assurance practitioner shall reassess the materiality of any matter if there is any indication that the basis on which the materiality was determined has changed.

                      207.5 The determination of materiality is a matter of professional judgement and the basis for the professional judgements made shall be documented. Materiality assessments are documented throughout the audit in the Audit Work Plan and working papers related to the audit steps at the planning (paragraphs 208.9, 208.24, 208.38) and execution (paragraph 212.7(g)) stages.

                      207.6 Audit risk shall be assessed at planning and considered and addressed throughout the audit in order to reduce it to an acceptably low level. Audit risks shall be documented in the Audit Work Plan and the audit risk assessment plan, including the risk ratings and any mitigation actions put in place.

                      207.7 At each progress review point (see paragraph 215.1 in Engagement Performance-Execution), the audit team shall review the risk assessment and management plan. This review includes identifying any new risks, assessing if the planned mitigation actions have been and continue to be effective and if any changes in treatments are required.

                      207.8 Major changes to the risk levels or mitigation actions and new identified risks shall be discussed with the Engagement Executive.

                      Guidance

                      Materiality

                      207.9 ASAE 3500 defines materiality as variations in performance of an activity evaluated against the identified criteria which have the potential to affect the economy, efficiency and/or effectiveness of the activity and be reasonably expected to influence relevant decisions of the intended users or the discharge of accountability by the responsible party or governing body of the entity.

                      207.10 As such, findings are considered to be material if they, individually or in the aggregate, could reasonably be expected to influence relevant decisions taken by intended users on the basis of the auditor’s report. The Parliament is considered to be the primary user of the ANAO’s performance audit reports. The audit team’s consideration of materiality is a matter of professional judgement. The audit team needs to also consider the aggregate effect of individually insignificant findings.

                      207.11 Materiality can also be understood as the relative importance of a matter to change or influence the decisions of users of the report, such as legislatures or the executive. Materiality is considered in the context of quantitative and qualitative factors, such as relative magnitude, the nature and effect on the subject matter and the interests expressed by intended users or recipients. In addition to monetary value, materiality includes issues of social and political significance, compliance, transparency, governance and accountability. Materiality can vary over time, and can depend on the perspective of the intended users and responsible parties (ISSAI 3100).

                      207.12 Guidance in respect of applying materiality to findings to form a conclusion is at Forming the Audit Conclusion.

                      Materiality and audit risk

                      207.13 Materiality and audit risk need to be considered together as considerations of materiality will consequently impact performance engagement risk.

                      207.14 Audit risk includes performance engagement risk, which is the risk of expressing an inappropriate conclusion based on evidence that is not soundly based. This may include evidence that is improper or incomplete as a result of inadequacies in the evidence gathering process, misrepresentation or fraud. Audit risk also includes operational engagement risk, which is the risk that an audit will not be completed in accordance with the approved budget and timeframe and to the required quality (ISSAI 3100).

                      207.15 Assessing and applying the concept of materiality throughout the audit addresses performance engagement risk by driving examination of:

                      • material areas where the performance engagement risk is high; and
                      • material areas where the performance engagement risk is low, but any significant variations or deficiencies could have a material effect on the economy, efficiency or effectiveness of the activity/subject matter.

                      207.16 Determining materiality and performance engagement risk as a team, with Engagement Executive involvement, is vital to avoiding under, or over-auditing. This can result where team members have different views on materiality and performance engagement risk.

                      207.17 Audit risk is assessed in planning and throughout the conduct of a performance audit. The assessment of audit risk requires the audit team to:

                      • understand the entity and its environment;
                      • assess risks to the audit conclusion; and
                      • design and conduct audit procedures to reduce engagement risk to an acceptably low level.
                      Materiality considerations

                      207.18 The consideration of materiality is relevant to all aspects of performance audits. Therefore, the auditor needs to consider materiality when selecting the audit topics, determining the audit objective(s), questions and scope, defining the criteria, evaluating the evidence, documenting the findings and developing the conclusions and recommendations.

                      207.19 Professional judgement about materiality is made in light of surrounding circumstances, but is not affected by the level of assurance. Materiality for a reasonable assurance engagement is the same as for a limited assurance engagement because materiality is based on the information needs of intended users.

                      207.20 Quantitative materiality factors relate to audit findings that are expressed or evaluated numerically. Generally, audit findings are more material where they relate to relatively larger numbers or values in the context of the audit.. There is no standard threshold for quantitative materiality in performance audits and in each case the auditor exercises their professional judgement in determining what numerical value represents a finding that is important in the measurement of the activity’s performance. For example, an audit finding that $1 million out of $2 million in spending activity lacked documentation is likely to be material, given the relative value of the finding to the subject matter under audit. On the other hand, a finding that a high proportion of low value transactions contained an error might not be considered quantitatively material if those errors related only to rounding errors that were within the audited entity’s accepted error rates. The auditor needs to consider the aggregate effect of individually insignificant findings.

                      207.21 Qualitative factors affecting materiality may include such things as:

                      • the number of persons or entities affected by the matter being audited;
                      • the interaction between, and relative importance of, various components of the activity when it is made up of multiple components, such as a report that includes numerous performance indicators;
                      • the wording chosen with respect to the activity that is expressed in narrative form;
                      • the nature of a finding, for example, the nature of findings in respect of a control when the assurance report includes a statement that the control is effective;
                      • whether a finding affects compliance with law or regulation;
                      • in the case of periodic reporting on an activity, the effect of an adjustment that affects past or current activities or is likely to affect future activities;
                      • whether a finding is the result of an intentional act or is unintentional;
                      • whether a finding is significant having regard to known previous communications to users, for example, in relation to the expected outcome of the audit;
                      • whether a finding relates to the relationship between the ANAO and the auditee, or their relationship with other parties;
                      • when a threshold or benchmark value has been identified, whether the result of the procedure deviates from that value;
                      • whether a particular aspect of the program or entity is significant with regard to the nature, visibility and sensitivity of the program or audited entity;
                      • whether the health or safety of citizens is affected; and
                      • whether a finding relates to transparency or accountability.
                      Performance engagement risk considerations

                      207.22 In the context of a performance audit, risk assessment is the identification and analysis of the key risks to the achievement of objectives concerning economy, efficiency and effectiveness, thus forming a basis for developing potential audit questions and determining the potential audit scope.

                      207.23 Performance engagement risks have the potential to adversely affect the auditee in some way. As well as failure to deliver a policy or program economically, efficiently or effectively, other performance engagement risks could include exposure to financial loss, loss of reputation of the entity or its Minister(s), or concerns regarding national security or commercial confidence.

                      207.24 Indicators that there is a high level of engagement risk may include:

                      • highly complex entities with multiple programs or functions;
                      • deficiencies in corporate governance;
                      • significant business risks which impact on the economy, efficiency or effectiveness of a program or the entity as a whole;
                      • poorly controlled, or changing, processes and systems;
                      • frequent changes in key personnel, systems or programs which are not well managed;
                      • previous performance engagements may have reported on significant findings.

                      207.25 Relevant factors for consideration when determining performance engagement risk include:

                      Factors that may impact the assessment of performance engagement risk

                      Subject matter characteristics

                      The nature of transactions, for example, high volumes, large dollar values and complex transactions.

                      The nature, size and complexity of the activity/subject matter.

                      External environment

                      The economic, social, political and environmental impact of the activity/subject matter.

                      Internal factors

                      The extent of management’s actions regarding issues raised in previous performance engagements.

                      The complexity and quality of management information and external reporting.

                      The effectiveness of internal control.

                      The nature and degree of change in the environment or within the entity that impact on the activity or subject matter.

                         

                      207.26 The risk assessment should be informed by the auditor’s understanding of the entity and activity subject to audit. Based on this understanding, risks could be identified and analysed by answering the following questions:

                      • What can go wrong?
                      • What assets are at risk and from what sources?
                      • With whom does the risk lie?
                      • What factors are / can be constraining performance (economy, efficiency, effectiveness)?
                      • What could be the cause (including weaknesses in controls?
                      • What could be the consequences or the impact, including on the entity’s reputation?
                      • How could this risk be managed?

                      207.27 The risks identified should be closely examined in order to decide on the ones that are key (significant and relevant). The risk level of the key risks should be determined by assessing the likelihood and potential impact of each risk using the risk matrix in the Risk Assessment Management template.

                      207.28 Audit responses to address the key risks identified should then be documented in the template, which is available from a link on the PASG Audit Manual web page.

                      207.29 The Risk Assessment Management template provides a repository for all established information on the risks and serves as a resource when communicating risk information to stakeholders. Each risk is allocated to a member of the audit team or other ANAO personnel who has the responsibility to regularly review the risks and assess the effectiveness of any mitigation treatments.

                      208. The Audit Work Plan

                      Background

                      208.1 The AWP documents the planning activities for the audit which are required by the Auditing Standards (ASAE 3500).

                      Policy

                      208.2 An AWP shall be prepared for each audit.

                      208.3 Audit teams shall determine whether an examination of the performance of the entity or subject matter is a suitable focus for a performance audit.

                      208.4 Prior to preparing an AWP, the Group Executive Director shall seek the approval of the Auditor-General to ascertain whether the proposed audit topic is suitable for planning. This will usually occur quarterly and for multiple audits at one time via a batch approval.

                      208.5 The audit objective shall be rational, clearly defined and relate to the principles of compliance, economy, efficiency, effectiveness and/or ethics. It shall be expressed in terms that can be concluded against.

                      208.6 Suitable criteria, corresponding to the audit objective, shall be identified for each audit. They shall be reasonable quantitative or qualitative measures of performance against which the activity’s performance may be assessed.

                      208.7 Criteria shall be relevant, complete, reliable, neutral and understandable.

                      208.8 The AWP shall document materiality and engagement risk level.

                      208.9 The AWP shall include the estimated cost of the audit, milestones and target dates. Performance audit teams shall discuss proposed audits with SADA in the planning and conduct of individual audits.

                      208.10 The AWP shall be provided to Assurance Audit Services Group. Assurance Audit Services Group shall provide the Audit Strategy Document, interim and final management letters (where issued) and the closing letter, related to the financial statements audit.

                      208.11 Performance audit teams shall discuss proposed audits with Assurance Audit Services Group and consider financial statement audit findings in the planning and conduct of individual audits.

                      208.12 Any major variation from the details about the rationale and background to the audit outlined in the Annual Audit Work Program shall be clearly communicated to the Executive in the AWP, with a detailed explanation of the reasons underpinning the variation. In those cases where the audit topic was not included in the Annual Audit Work Program, the reasons for undertaking the audit shall be set out in detail.

                      208.13 All AWPs shall be provided to the Auditor-General for approval, with a copy provided at the same time to the Deputy Auditor-General.

                      208.14 Where the audit is not included in the Annual Audit Work Program or where there is a significant variation in the nature and intent of the audit compared with that set out in the Annual Audit Work Program, the AWP must be endorsed by the responsible Group Executive Director before submission to the Deputy Auditor-General, and Auditor-General.

                      208.15 Following approval of the AWP by the Auditor-General, any subsequent changes to audit criteria shall be reviewed by the responsible Engagement Executive, and approved by the responsible GED and the Auditor-General.

                      208.16 The AWP for each audit shall briefly identify any significant engagement and operational risks confronting the audit, including key risks identified in the portfolio overview of the most recent Annual Audit Work Program, where relevant.

                      208.17 Where audit teams are likely to have intermittent contact during the course of the audit with vulnerable people such as Indigenous communities, people with disabilities, people from non-English speaking backgrounds or children under the age of 18 years of age, they shall indicate this in the audit work plan. Advice shall be sought from the Senior Director, Human Resources to ensure that the appropriate support arrangements can be put in place, including whether audit team members are required to apply for a working with vulnerable people registration or a working with children check.

                      Guidance

                      208.18 Audit planning varies according to the size, nature and complexity of the audit. The resources to be used in planning an audit should be commensurate with the nature and complexity of each audit and the assessment of the risks to the audit.

                      208.19 The preparation of an AWP necessitates understanding the entity and activity subject to audit. Per section 206.12 of this manual, the information-gathering powers in the A-G Act can be used to obtain information and documents required for planning an audit.

                      208.20 An AWP template and instructions on planning are available as part of the PASG Workflow, which contains corporate templates for staff to use during an audit.

                      208.21 Each AWP should contain sufficient information to allow the GED and the Executive to make a fully informed decision on the conduct of the proposed performance audit. Once approved, the AWP provides the authority to conduct the audit.

                      208.22 In preparing the AWP, the audit team should refer to the criteria, approach, budget and timeframes for similar audits considered previously and as outlined in their AWPs.

                      208.23 In a small number of cases, it may not be possible to finalise elements of the AWP until the delivery phase of the audit has commenced. In these circumstances, the audit team should clearly outline in the AWP those aspects of the audit plan that may be subject to change and set a timeframe to confirm any changes. Approval will be needed to proceed with the audit while the details of the AWP are still being finalised. The audit team is responsible for progressing the audit in parallel with the refinement of the audit plan during the delivery phase.

                      208.24 The following paragraphs outline considerations for preparing specific elements of the AWP.

                      Rationale for undertaking the audit

                      208.25 The AWP outlines the rationale for conducting the audit. Usually, the rationale for undertaking a particular audit has been identified in the course of preparing the Annual Audit Work Program. This rationale will be included in the final audit report in accordance with paragraph 227.4 of this manual, and hence should be clear and understandable to a broad audience, with reference to the relative importance of the subject matter from a stakeholder perspective. The following table illustrates examples that should be incorporated in a rationale.

                      Element

                      Description

                      Materiality

                      Qualitative: High public visibility of the program; importance of the program to particular client groups; strong Parliamentary or community interest in the performance of the program.

                      Quantitative: high value of assets, annual expenditure or annual revenue of the entity or the program, activity or function. Scope may be limited to items above a certain dollar threshold, e.g. procurement contracts greater than $xx.

                      Impact

                      Significant impact of the activity, even when it is undertaken by a small unit within an entity with low materiality.

                      Key area/issue presenting risks or challenges to Commonwealth administration

                      The program or activity being a government initiative that is directly related to a key area/issue presenting risks or challenges to Commonwealth public administration.

                      Potential benefits from the audit

                      More efficient business processes; greater accuracy in claims processing; better management of contracts; improved adherence to Commonwealth policies; greater accountability through accurate performance reporting; earlier detection of risks to good management or prevention of fraud.

                      Previous coverage

                      No previous ANAO performance audit coverage; very limited internal review of a significant program; possibility of a follow-up audit foreshadowed in a previous ANAO audit; a follow-up audit requested by a parliamentary committee.

                      Value for money

                      Multiple factors need to be taken into account when determining value for money. These include:

                      • reducing costs/managing budget cuts;
                      • contract management; and
                      • digital transformation.

                      Refer to Developing a Value for Money Perspective for Performance Audits for details on applying a value for money perspective.

                      Auditability

                      Relates to the ability to carry out the audit according to the ANAO Auditing Standards. Although some areas may be significant, they may not be auditable for one or more of the following reasons:

                      • the area is outside our mandate;
                      • the audit team does not have or cannot acquire the required expertise;
                      • the area is undergoing significant and fundamental change;
                      • suitable criteria or approaches are not available to assess performance; or
                      • the information or evidence required is not available or cannot be obtained efficiently.
                         

                      Background to the audit

                      208.26 Each AWP should include background information regarding the entity, program or function to be audited. This background information reflects and generally builds on the material for the particular audit that was included in the Annual Audit Work Program.

                      Audit objective

                      208.27 The audit objective, outlined in the AWP, is a key statement that is intended to define the intention of the audit. The objective of a performance audit is to provide an assessment of specified elements of an entity’s operations. The assessment should address one or more of the following terms: effectiveness; efficiency; economy; or compliance.

                      These terms are defined in ASAE 3100 and ASAE 3500 as follows:

                      Term

                      Definition

                      Effectiveness

                      The performance principle relating to the extent to which the intended objectives at a program or entity level are achieved.

                      Efficiency

                      The performance principle relating to the minimisation of inputs employed to deliver the intended outputs in terms of quality, quantity and timing.

                      1. See Special Considerations for Auditing Efficiency: Methodology and Guidance for further guidance about conducting audits focussed on efficiency.

                      Economy

                      The performance principle relating to the minimisation of the costs of resources, within the operational requirements of timeliness and availability of required quantity or quality.

                      Compliance

                      The assessment of adherence to the requirements, as measured by suitable criteria.

                         

                      208.28 The audit objective and the audit scope (see paragraph 208.34) are interrelated and should be considered together. The audit objective needs to be realistic and achievable and give sufficient understanding to the entity and other relevant parties about the focus of the audit. The audit objective also provides the basis for developing the audit criteria and the audit approach.

                      Audit criteria

                      208.29 Audit criteria are the specific measures used to assess the performance of the activity. In accordance with ASAE 3500 paragraph 16(d), the criteria are benchmarks used to evaluate the underlying subject matter. Audit criteria are reasonable and attainable standards of performance against which the extent of effectiveness, efficiency, economy or compliance aspects of an entity’s programs or activities can be assessed.

                      208.30 Audit criteria are important because they provide:

                      • a common understanding between the audit team, the ANAO Executive and the entity regarding the standards against which the entity is to be assessed; and
                      • a structure for the evidence-gathering phase of the audit.

                      208.31 Suitable criteria are those that are relevant to the subject matters being audited and appropriate to the circumstances. As outlined in ASAE 3500 the characteristics of suitable criteria include:

                      • Relevance: relevant criteria contribute to conclusions that assist decision-making by the intended users;
                      • Completeness: criteria are sufficiently complete when relevant factors that could affect the conclusions in the context of the performance engagement circumstances are not omitted. Complete criteria include, where relevant, benchmarks for presentation and disclosure;
                      • Reliability: reliable criteria allow reasonably consistent evaluation of measurement of the activity, including when used in similar circumstances by similarly qualified assurance practitioners;
                      • Neutrality: neutral criteria contribute to conclusions that are free from bias; and
                      • Understandability: understandable criteria contribute to conclusions that are clear, comprehensive, and not subject to significantly different interpretations.

                      208.32 Suitable criteria and sub-criteria may be developed from:

                      • policy decisions or policy statements;
                      • legislation and regulations;
                      • published performance measures and internal measures;
                      • policies and guidance developed by central entities, regulators or government;
                      • standards of good practice, relevant benchmarks and relevant practice guides developed by professions, associations or other recognised authorities;
                      • statistics, practices, benchmarks, performance standards or procedures developed within the entity; and
                      • criteria used in previous audits, including those conducted by the ANAO, and other Australian or overseas audit offices;
                      • subject matter or general literature.

                      208.33 Criteria may require interpretation and modification to ensure their relevance to the audit. Criteria may need to be modified or refined as the audit proceeds and more information becomes available. Should the criteria require substantial amendment and impact on the scope of the audit, the proposed changes should be discussed, in the first instance, with the Executive Director. If agreed, the proposed changes can then be considered by the ANAO executive at the formal review stages (see the PASG Workflow).

                      208.34 For further details refer to Chapter 11 Generic Audit Criteria.

                      Audit scope

                      208.35 The audit scope defines the boundary of the audit. The audit scope may identify:

                      • the part of the entity, management control system or organisational unit to be examined;
                      • the matters subject to audit;
                      • particular entity locations to be visited during the audit;
                      • the time period being examined by the audit; and/or
                      • any associated matters that are not within the scope of the audit and the reasons for their proposed exclusion from the audit.

                      208.36 The scope is usually established based on information gathered during the planning phase or obtained in previous audits.

                      208.37 In establishing the scope of an audit, it may be necessary to have a broad statement of scope at the outset and refine this during the course of planning the audit and in the early stages of conducting the audit.

                      208.38 When determining the scope of the audit, the AWP should make reference to any known relevant reports published by other external and internal scrutineers, for example, the Inspector-General of Taxation, Inspector-General of Intelligence and Security, Parliamentary committees or internal audit.

                      208.39 The audit team considers matters such as materiality, risks to successful program performance and/or service delivery and auditability when establishing the scope. These matters should be considered throughout the planning and conduct of the audit and, particularly, in developing the detailed audit criteria.

                      Audit method

                      208.40 The audit method sets out the extent of evidence-gathering procedures to be undertaken and the reasons for selecting them, and the means to be used to collect information relating to the audit criteria. The method explains the intended use of specific data collection tools such as sample surveys, case studies, interviews, document reviews, compliance and/or system control analysis and testing. The method should also consider whether testing is likely to be performed on a sample basis.

                      208.41 The audit method also specifies where and why particular fieldwork is to be carried out and lists the involvement of any external stakeholders.

                      Cost of an audit

                      208.42 The cost of an audit includes the estimated costs of staff resources and the engagement of contractors and experts, and the estimated costs of travel and report publication. The costs of the initial planning phase of the audit and scoping study, where undertaken, are also to be included.

                      208.43 The budget for the audit is derived from a consideration of:

                      • the estimated hours required to be spent on the audit by the audit team, Engagement Executive and the Group Executive Director;
                      • the cost of initial planning;
                      • the cost of Assurance Audit Services Group and SADA staff, contractors, specialists and experts;
                      • costs of travel, including attendance at any audit related conferences. Travel costs should include airfares, accommodation, travel allowance and taxi/car hire that will be incurred by all team members, including any contractors engaged to assist with the audit; and
                      • the cost of publishing the audit report.
                      Audit milestones and target dates

                      208.44 Audit work plans are to include the target dates for the following audit milestones:

                      • AWP discussion meeting
                      • Audit start date (Designation letter sent)
                      • Progress Review 1
                      • Progress Review 2
                      • Report Preparation Papers to entity
                      • Section 19 workshop
                      • Section 19 report to entity
                      • Audit tabled

                      208.45 The PASG Workflow references a Ready Reckoner tool to assist with planning the audit phases and milestone dates.

                      The audit team

                      208.46 In determining the composition of the audit team, the following factors are taken into consideration:

                      • the experience of the Audit Manager;
                      • the number, level and experience of other team members;
                      • the benefit of engaging the Assurance Audit Services Group and/or the SADA to assist in conducting elements of the audit;
                      • the benefit of engaging specialists and/or experts (including from PSRG and CMB) to support the in-house team in addressing complex and/or technical issues for example, relating to methodology); and
                      • the complexity and expected impact of the audit.

                      208.47 A specialist is an audit practitioner (either from ANAO or external) who specialises in auditing a particular subject area (e.g. IT Auditor).

                      208.48 An expert is a person or organisation whose expertise in an area other than auditing is used by the ANAO to assist in obtaining sufficient appropriate audit evidence. An expert would normally be external to ANAO but may include internal non-audit staff such as CMB or PSRG Legal Services staff.

                      208.49 Where gaps are identified in the skills necessary to conduct a particular audit, there are a number of options to address this, including:

                      • undertaking training to obtain the skills;
                      • obtaining advice and assistance from within PASG or other areas of the ANAO; or
                      • engaging a contractor, expert and/or specialist.

                      208.50 As part of the AWP, the audit team are encouraged to seek the input of other PASG Audit Managers and Executive Directors who have conducted similar themed audits or audits of the same entity to assist in scoping the audit.

                      208.51 As part of the AWP, the audit team should outline a summary of the pre-audit consultation with Assurance Audit Services Group and SADA. This summary should include any potential or planned involvement of Assurance Audit Services Group and SADA in the performance audit, any risks identified from financial statement auditing, and other intelligence gathered from past audits that may be of relevance for the proposed audit.

                      Risk

                      208.52 A detailed risk assessment and management plan is completed and attached to the AWP that addresses each risk and its corresponding mitigation strategy (discussed in Risk Assessment and Management Plan section).

                      Contributions to audit planning
                      External stakeholders

                      208.53 External stakeholders include people or organisations with an interest in the operations, activities, results or resources of an entity. The primary external stakeholders are members of the public, clients of the audited entity, and non-government organisations such as industry associations and special interest groups.

                      208.54 Increasingly, entities are expected to develop close links with interest groups such as consumer and industry associations, provider organisations and think-tanks. These relationships can go beyond the exchange of information and may involve more formal collaboration or negotiation about government decision-making.

                      208.55 The identification of external stakeholders—who and what their interests are in the audited entity or topic—should be completed during the audit planning stage.

                      Approval of the AWP

                      208.56 The approval processes for the AWP are outlined in the PASG Workflow, which contains corporate templates for staff to use during an audit.

                      209. Generic audit criteria

                      209.1 Generic audit criteria have been developed for the following four common types of performance audits conducted in the ANAO:

                      Grants administration (Post PGPA Act - 1 July 2014)

                      Procurement (Post PGPA Act - 1 July 2014)

                      Regulation (Post PGPA Act - 1 July 2014)

                      Contract Management (Post PGPA Act - 1 July 2014)

                      209.2 The purpose of each of the four criteria documents is to provide performance audit teams with a starting point for drafting criteria for these particular types of audits. Audit teams will still need to tailor specific criteria to their audits after identifying the time period and relevant legislative framework that applies (either prior to or post PGPA Act) and carefully considering the audit objective, the type of program or project being audited and the business risks of the entity being audited.

                      209.3 The generic criteria broadly describe key structures, processes and policies that would be expected to be in place within an entity. These criteria were developed using the ANAO’s Better Practice Guides (now withdrawn), relevant recent performance audits as well as Government directives. References are provided at the bottom of each document. Audit teams are encouraged to review this reference material when developing criteria for individual performance audits.

                      209.4 In relation to the criteria for Grants Administration, a tiered approach is warranted because the relevance of various parts of the Commonwealth Grant Rules and Guidelines often depends on the circumstances and parameters of the particular program.

                      210. Planning audit procedures

                      Policy

                      210.1 The audit team shall document the planned audit procedures including the nature, timing, extent and rationale for the planned procedures for each criterion.

                      210.2 The planned audit procedures shall be updated as necessary throughout the audit and all changes in the planned procedures shall be clearly documented and explained.

                      210.3 At a minimum, an audit test program shall provide a link between the criteria, evidence, audit procedures performed and the results and findings.

                      210.4 The planning of an audit shall include an assessment of whether the audit team has adequate skills, competence and knowledge to undertake the particular audit.

                      Guidance

                      210.5 Combined, the Audit Work Plan (AWP) and audit test program document the planned audit procedures, including the nature, timing and extent of evidence-gathering procedures and the rationale for selecting the approach.

                      210.6 When determining the extent of time and resources required for planning, audit teams are to consider the:

                      1. audit team’s experience with and understanding of the entity and the audit topic;
                      2. size of the team;
                      3. level of the audit team’s auditing experience;
                      4. scope of the audit; and
                      5. complexity of the audit criteria and proposed tests and evidence-gathering techniques.

                        210.7 The AWP and/or audit test program includes:

                        1. the types and expected sources of audit evidence;
                        2. the techniques planned to be used to gather evidence;
                        3. the planned audit procedures, including timing and extent (e.g., target testing of a specified number or items with defined characteristics, random sampling);
                        4. personnel and expertise requirements, including the nature and extent of the use of specialists or experts when applicable;
                        5. the allocation of tasks to be performed by audit team members;
                        6. a link between the criteria, evidence, audit procedures performed and the results and findings;
                        7. materiality: and
                        8. assessment of engagement risk.

                          210.8 Planning is not a discrete phase, but a continual process and the test program may need to be revised to reflect any changes in the planned approach. It is recommended that updates to the planned approach and rationale are documented to explain why the change was necessary.

                          210.9 In planning and conducting a performance audit, it is not expected that audit procedures will include directly assessing whether fraud or other wrongdoing is occurring in the program or activity subject to audit. It is not the auditors’ responsibility to prevent or detect fraud or other wrongdoing through the conduct of its audits. This is the responsibility of the entity itself. The ANAO is also not in a position to determine whether a fraud or other wrongdoing has actually occurred.

                          210.10 As part of planning and conducting a performance audit, the audit team is required to obtain an understanding of the entity and the program or activity subject to audit. This should include making an assessment of whether fraud, or related wrongdoing, may have a significant impact on the program or activity. Such an assessment could include: a review of the entity’s fraud control plan and related information and documentation, such as fraud plans of individual work areas; and a review of the entity’s systems and procedures relating to fraud prevention, investigations, prosecutions, and reporting. If the audit team concludes that the risk of fraud and related wrongdoing may have a significant impact on the program or activity subject to audit, the audit should assess the adequacy of the entity’s management of this risk.

                          210.11 The extent to which matters such as potential fraud and other wrongdoing are referred to in a performance audit report will depend on the individual circumstances. It is generally not necessary or appropriate to refer to individuals or specific instances in audit reports. In circumstances where such matters are systemic and have the potential to significantly impact the operations of the program or activity subject to audit, they may warrant specific audit coverage and reference in the audit report6.

                          Engagement performance — execution

                          Chapters 211 to 224

                          211. Designation: Communicating the terms of the audit

                          Background

                          211.1 The designation email provides notification of the Auditor-General’s decision to conduct an audit and mandate, states the section of the Auditor-General Act 1997 (Cth) under which the audit is to be conducted and advises that the audit report will be tabled in the Parliament as soon as practicable after the completion of the audit. It also provides specific details about the audit and audit team.

                          Policy

                          211.2 The ANAO shall issue a written designation message that communicates the terms of the performance engagement (audit or review) to each entity subject to the performance engagement. (ASAE 3500)

                          211.3 Where the scope of an audit is subsequently extended to include more than one entity, the section under which the audit is conducted will change from section 17 to section 18 of the Auditor-General Act 1997 (Cth). In these circumstances a revised designation message shall be sent to the original entity and a section 18 designation message provided to the other entity(s).

                          Guidance

                          211.4 The ANAO issues a designation email that communicates the terms of the performance engagement to each entity subject to audit at the commencement of each audit before commencing audit fieldwork. A standard designation template is attached to the PASG Workflow, which contains corporate templates for staff to use during an audit.

                          211.5 The designation email is sent to the accountable authority of a Commonwealth entity (refer to PGPA Act, section 12 – normally the Chair of the accountable authority where the authority has more than one member) and to the Chair of the Board of Directors of a Commonwealth company.

                          211.6 The designation message provides notification of the:

                          • Auditor-General’s decision to conduct and audit;
                          • Auditor-General’s mandate;
                          • section of the A-G Act under which the audit will be conducted;
                          • audit focus, including the objective and scope of the proposed audit;
                          • entity’s opportunity to provide written representations on the program or activity subject to the audit; and
                          • ANAO contact details.

                          211.7 In addition, it may be useful to provide the entity with a scope diagram that includes the audit objective, high-level criteria and sub-criteria, and a separate list of the initial information requirements for the audit.

                          212. Entry interviews

                          Policy

                          212.1 The ANAO shall conduct an Entry Interview with each entity, unless the entity declines an Entry Interview or it is impractical to do so, for example, where the audit involves a survey of a large number of entities.

                          212.2 Approval to not proceed with an Entry Interview shall be obtained from the Engagement Executive and documented in the Audit File.

                          212.3 A record of the Entry Interview meeting, including the outcomes and any significant decisions made, shall be documented in the audit file.

                          Guidance

                          212.4 Following the designation of an audit, the Entry Interview meeting serves a number of purposes including introducing the audit team and allowing discussion of the audit objective, process and timing with the entity.

                          212.5 The purpose of an Entry Interview is to inform the entity and establish the basis for a successful engagement by:

                          • outlining the purpose of performance audits and opportunities to minimise the impact on the auditee;
                          • drawing attention to information access powers under the Auditor-General Act 1997 (Cth);
                          • introducing to entity management the staff of the ANAO who will be involved in the audit;
                          • explaining the background and objectives of the audit, discussing the audit criteria and responding to any questions the entity may have about these issues;
                          • giving entity management the opportunity to ask questions about the audit process or any other relevant matter relating to the audit, including key milestones and expectations;
                          • drawing attention to subsection 36(1) of the Auditor-General Act 1997 (Cth) concerning the confidentiality of information obtained during the course of an audit, particularly when an entity has had little previous exposure to performance audit requirements and processes;
                          • allowing entity managers to bring to the attention of the ANAO any contextual matters that might influence the way the audit is conducted, particularly documentation that assists in explaining the program and/or relevant issues;
                          • informing the entity about information the audit team is expected to require—specifically, classified and/or sensitive records and where arrangements need to be made to access systems together with applicable milestones—to help ensure an efficient and timely audit process; and
                          • discussing the administrative arrangements surrounding the audit and establish coordination arrangements with the entity.

                          212.6 It is advisable that the auditee(s) is informed at the Entry Interview about the citizen contribution facility on the ANAO website and that the audit is open for stakeholders contributing information during the evidence collection stage. Also refer to the PASG Workflow.

                          212.7 ANAO officials are expected to approach Entry Interviews as an educative opportunity for audited entities and to not assume a high level of knowledge about ANAO processes.

                          213. Progress reviews

                          Policy

                          213.1 Progress reviews shall occur at:

                          • 20 per cent into the allocated audit hours; and
                          • 50 per cent into the allocated audit hours;
                          • Before or after the Exit Interview (optional).

                          213.2 The third Progress Review (before or after Exit interview) shall occur when:

                          • the ANAO Executive requests a meeting;
                          • there is a significant audit issue arising from previous progress reviews remaining unresolved;
                          • the responsible Executive Director or Group Executive Director considers a meeting necessary.

                          213.3 The audit team shall prepare briefs on the audit’s progress in accordance with the appropriate templates provided by the service group, or as otherwise specified by practice management.

                          213.4 The Engagement Executive shall seek approval from the ANAO Executive in those cases where there has been a change since the Audit Work Plan was approved that affects the audit objective, scope, proposed response, budget or tabling dates.

                          213.5 The outcomes and all significant decisions made in key meetings and briefings with the ANAO Executive, responsible GED and/or Executive Director shall be documented in the audit working papers.

                          213.6 The relevant AASG Executive Director or Signing Officer, as appropriate, shall be invited to attend the 2nd Progress Review meeting.

                          213.7 In addition to the minimum progress review meetings outlined in paragraph 215.1, the audit team shall engage with the GED and the ANAO Executive at any other critical stages of the audit, and in relation to any significant issues.

                          213.8 The outcomes and all significant decisions made in key meetings and briefings with the ANAO Executive shall be documented in the audit working papers. Where notes or comments are made by the ANAO Executive on Progress Review briefing papers prepared by audit teams, these records shall be retained on the audit working papers.

                          Guidance

                          213.9 Performance audit teams are required to undertake a series of progress reviews with the Service Group Executive and ANAO Executive staff at key specified intervals during the conduct of the audit.

                          213.10 The audit team completes an initial progress review (PR1) in order to confirm that planning for the audit remains appropriate or to suggest a variation to the conduct of the audit or to discuss any emerging issues, such as difficulties in accessing information or data.

                          213.11 The initial progress review should explicitly consider the ongoing appropriateness of scope, budget and schedule. Where deficiencies are identified, treatments should be applied to support delivery on time and on budget. Audit Managers and Executive Directors should consider a range of treatments to support the timely delivery of the audit. If changes to budget, timeframe or scope are required, then a variation should be sought from the GED.

                          213.12 The audit team completes a second progress review (PR2) meeting to identify and distil the key issues arising from the audit and to consider the most effective structure for the presentation of the audit findings and potential recommendations in the proposed Report Preparation Papers.

                          213.13 Report Preparation Papers should be fully defined, findings and conclusions identified and Report Preparation Papers approximately 50% drafted when the second progress review meeting is conducted.

                          213.14 Prior to the commencement of the drafting of the section 19 report, the audit team can request a third progress review meeting, to discuss the key issues arising from the entity’s response to the Report Preparation Papers and/or the Exit Interview and to consider the most effective structure for the presentation of the audit findings and recommendations in the section 19 report.

                          213.15 Specific requirements for the briefing are included in the PASG Workflow, which contains guidance and corporate templates for staff to use during an audit.

                          213.16 Progress review meetings are an important opportunity to canvas and discuss issues such as:

                          • key messages from the audit findings;
                          • the proposed audit report structure, including headings for :
                            • key findings in the report summary;
                            • chapter headings in the report body; and
                            • appendices;
                          • potential sensitivities that may arise from the audit findings, including those arising from legal advice received or security issues;
                          • potential audit recommendations;
                          • the linkages (or otherwise) back to the original request in the case of audits undertaken following a request from a Member or Senator of the Parliament of Australia;
                          • outline of any significant proposed divergence from the audit objective and/or audit criteria set out in the approved Audit Work Plan, with reason(s) for this divergence;
                          • any difficulties envisaged in meeting stakeholder expectations;
                          • estimated dates for the remaining milestones until the report is tabled, against approved milestone dates; and
                          • the preliminary overall audit conclusion.

                          213.17 To assist in drawing on the experience of other audit teams, the responsible Executive Director may invite members of other audit teams to participate in progress review meetings on a case-by-case basis. As noted at 215.6 members of the AASG team are required to be invited to attend the 2nd Progress Review meeting, and may be invited to attend the 1st and 3rd Progress Review meetings.

                          213.18 Notwithstanding the formal requirements for briefings during key stages of the audit, the audit team would be expected to provide briefings to the responsible GED and the ANAO Executive that are commensurate with the likely impact of the audit.

                          213.19 Discussions with the Executive could occur:

                          • during or at the completion of fieldwork so that any significant issues that are identified in the course of the fieldwork are discussed; and
                          • after the completion of the Exit Interview to discuss entity feedback, key findings and conclusions for the draft report.

                          213.20 These briefings are also an opportunity to bring the Executive’s attention to matters such as potential sensitivities, budget or timeframe pressures or potential difficulties in meeting stakeholder or user expectations, including for audits requested by Members and Senators of the Parliament of Australia.

                          214. Advising the entity of progress and significant issues

                          Policy

                          214.1 The audit team shall keep the management of the entity subject to audit informed in a timely manner of the conduct of, and significant issues arising from, the audit.

                          214.2 The audit team shall inform the entity of any deficiencies in controls and systems, and material findings on a timely basis to allow the entity sufficient time to investigate and respond to the findings.

                          214.3 The audit team shall consider, within the terms of the engagement or regarding relevant legislative requirements, whether any other matter has come to their attention that needs to be communicated with the entity.

                          Guidance

                          214.4 Explaining the audit process to the entity at all key points during the audit and keeping the entity informed is crucial to maintaining a sound professional relationship with the entity being audited and ensuring the free flow of information.

                          214.5 Informing the entity of matters arising from the audit would include providing entity management with early advice on particularly significant issues that are identified during the course of the audit so that the entity’s perspective on the issues may be obtained at an early date.

                          214.6 When advising the entity on significant issues identified during the audit, the audit team must consider:

                          • whether there is sufficient certainty concerning the audit position to warrant the advice; and
                          • including an appropriate caveat for any advice provided.

                          214.7 This is particularly important because the ANAO’s position may change during the audit review process.

                          214.8 The level and frequency of contact with entity senior management is determined by the audit team, but would be commensurate with the expected impact of the audit, and sensitivity to the issues arising.

                          214.9 Keeping the entity informed provides the opportunity for the audit team to foreshadow future steps in the audit process and flag further information and assistance that will be sought during the remainder of the audit.

                          214.10 Informing the entity about the conduct of the audit could include advising the entity on just completed or planned future key steps in the audit process, for example, the results of a significant meeting, the commencement/completion of fieldwork, and the development of Report Preparation Papers.

                          214.11 Informing the entity about the conduct of the audit could also include significant changes to the audit process and/or timing, for example, caused by the unplanned absence of key members of the audit team.

                          215. Entity security requirements

                          Policy

                          215.1 When planning an audit, the audit team shall consider the level of security clearance that will be required to conduct the audit in an entity, and to take appropriate action to obtain the necessary clearance (if not already in place for all audit team members).

                          215.2 All staff shall comply with the relevant requirements of the Australian Government’s Protective, as implemented by the entity subject to audit.

                          215.3 Where the audit team considers that entity security requirements are not reasonable, this shall be promptly brought to the attention of the responsible Executive Director and, as necessary, the responsible GED, and resolved through professional communication with the entity as soon as practicable.

                          215.4 Entity information about individuals shall only be accessed for a clearly defined audit purpose, and ANAO staff shall not attempt to access information from entity systems or records relating to themselves, relatives or acquaintances. If there is a risk of accessing such information in the course of an audit or review, staff shall bring the risk to the attention of their audit executive and consult with the relevant entity as necessary.

                          Guidance

                          215.5 When allocating resources to an audit, PASG Practice Management should have regard to the likely level of security clearance required and allocate staff accordingly where possible.

                          215.6 The audit team needs to have regard to the reasonable security requirements of the entity being audited.

                          215.7 For further information, the audit team can refer to the ANAO’s Personnel Security Policy, available on the Audit Central Protective Security page.

                          215.8 There are a number of factors that the audit team can consider to assess whether the entity security requirements are reasonable. These include whether the requirements:

                          • impinge on access to documents and systems necessary to the completion of the audit;
                          • introduce barriers to the efficient conduct of the audit that are contrary to the exercise of access powers under the Auditor-General Act 1997 (Cth); and
                          • do not provide for the same level of clearance for the audit team as the level of clearance for relevant entity personnel.

                          215.9 Entity ICT security protocols may be triggered in the event of inappropriate access. Inappropriate access may result in reputational damage to the ANAO as a trusted user of entity information and action against the individuals involved.

                          216. Gathering audit evidence

                          Policy

                          216.1 The audit team shall gather evidence that is sufficient and appropriate to address the audit’s objectives, and support the audit’s findings and conclusions.

                          216.2 The audit team shall exercise professional judgement to obtain the sufficient quantity and appropriate quality of audit evidence.

                          216.3 This evidence shall be documented and stored appropriately to provide the basis for concluding that the audit was conducted in accordance with the ANAO Auditing Standards, and other legal requirements.

                          216.4 All evidence gathered shall be stored appropriately in accordance with the ANAO’s Protective Security Governance Policy and ANAO Recordkeeping Framework.

                          216.5 ANAO staff granted privileged (researcher) access to entity document management systems must ensure that all information searches relate directly to an active audit or review and are conducted for the purpose of gathering evidence.

                          Guidance

                          216.6 Audit evidence is information obtained and used to support audit findings and conclusions. The collection, analysis and use of evidence are essential elements of the effective conduct of a performance audit.

                          216.7 Decisions concerning the evidence gathering process begin at the planning phase of an audit. The exercise of professional judgement concerning sufficient and appropriate audit evidence then continues throughout the audit. The audit team must assess whether the evidence obtained has the sufficient quantity and the appropriate quality. This assessment will inform the decision to obtain additional or different evidence.

                          216.8 Performance auditors need to have discussions with the auditees about the available evidence at the planning or conducting phases. They should ascertain the nature of the evidence and how it will need to be collected and analysed and interpreted by the audit team.

                          216.9 It is important that the audit team obtain evidence from a variety to sources, as different perspectives and conclusions may be presented from multiple sources. It is necessary to continually identify potential sources of evidence during the conduct of the audit. This is because not all circumstances can be foreseen during planning.

                          216.10 The internal controls of the systems that generate the audit evidence also need to be assessed for accuracy and consistency.

                          216.11 It is likely that several techniques will be used to gather audit evidence in any performance audit, such as:

                          • analytical procedures;
                          • document review;
                          • review of email records;
                          • interviews;
                          • conducting surveys (refer to Surveys);
                          • compliance testing;
                          • physical observation; and
                          • data analysis.

                          216.12 The four main types of audit evidence and their sources are set out in the table below.

                          Types of evidence

                          Sources of evidence

                          Physical

                          Physical evidence is obtained by observing people and events or examining property. It can take the form of photographs, charts or maps and detailed written descriptions of observations made.

                          Oral

                          Oral or testimonial evidence is obtained in the form of statements in response to inquiries or interviews. Oral evidence can be obtained from entity staff and clients, other stakeholders or experts. Interviews conducted to obtain oral evidence can be structured or unstructured and may involve the use of formal questionnaires and sampling techniques.

                          Documentary

                          Documentary evidence in physical or electronic form is the most common form of evidence. It may be obtained from within or outside the entity and includes such things as: policy statements and legislation; reports; reviews and evaluations; letters and minutes, emails and phone records; procedures and guidelines; risk assessment plans; planning and budget documents; contracts and leases; performance results; client feedback; computer system records; personnel documents; and organisation charts.

                          Analysis of information and data

                          Analyses of data and other information can be obtained from the entity (and may need verification) or can be generated by the audit team. It includes the analysis of ratios and trends, comparisons of procedures and operations with standards or specified requirements, and analysis of substantive testing of transactions.

                           

                             

                          216.13 Appropriateness of audit evidence is attributed to both the relevance and reliability of the evidence. While the assessment of sufficiency and appropriateness is a professional judgement of the auditor in each case, auditors may find it helpful to consider the following generalised statements concerning evidence appropriateness:

                          • Documentary evidence is more reliable than oral evidence, but the reliability varies depending on the source and purpose of the document;
                          • Testimonial evidence that is corroborated in writing is more reliable than oral evidence alone;
                          • Evidence based on many interviews together is more reliable than evidence based on a single or a few interviews;
                          • Testimonial evidence obtained under conditions in which people may speak freely is more reliable than evidence obtained under circumstances in which people may feel intimidated;
                          • Evidence obtained from a knowledgeable, credible and unbiased third party is more reliable than evidence obtained from the management of the audited entity or others who have a direct interest in the audited entity;
                          • Evidence obtained when internal control is effective is more reliable than evidence obtained when internal control is weak or non-existent;
                          • Evidence obtained through the auditor’s direct observation, computation and inspection is more reliable than evidence obtained indirectly; and
                          • Original documents are more reliable than copied documents7.

                          216.14 The performance auditor may also find the following generalised presumptions with respect to the sufficiency of audit evidence helpful in exercising their professional judgement in assessing the evidence obtained:

                          • The greater the audit risk, the greater the quantity and quality of evidence required;
                          • Stronger evidence may allow less evidence to be used;
                          • Having a large volume of audit evidence does not compensate for a lack of relevance, validity or reliability; and
                          • More evidence is normally necessary when the audited entity disagrees on ANAO’s conclusions over activities subject to audit8.

                          216.15 When gathering evidence, it is important to remember that while individual pieces of data or information, such as date of birth, may not be sensitive or classified, combined with a person’s address and bank account details, the information in aggregate is likely to be more sensitive and will require appropriate handling and storage for privacy and / or security reasons.

                          216.16 Decisions concerning the retention in audit working papers of data and the results of analysis must balance the need to retain sufficient evidence to support the audit findings and conclusions with the need to reduce the risk of unnecessarily retaining sensitive or classified information. It may be possible, for example, to delete reference to one sensitive field, such as tax file number, without affecting the sufficiency of the audit evidence in the working papers.

                          216.17 Audited entities will often facilitate the ANAO’s work by giving ANAO staff privileged (researcher) level access to their document management systems. It would be inappropriate to conduct an information search with the purpose of: monitoring an entity’s preparation of a response to ANAO Report Preparation Papers or a proposed Auditor-General report; or gathering additional information after an audit or review has concluded.

                          217. Sampling and selecting items for testing

                          Policy

                          217.1 When selecting a representative sample to obtain audit evidence, the auditor shall use a statistically relevant sample size, and the items shall be selected randomly.

                          217.2 When representative sampling is the primary method used for selecting items for testing the standard range of 90-95% confidence level and 5% confidence interval shall be applied to obtain reasonable assurance. A lower confidence level of 80-90% may be considered and applied only when the procedures performed to which the sampling approach applies are in addition to other audit evidence obtained through separate procedures that address the same criteria.

                          217.3 The sample size for a representative sample shall be based upon a reasonable estimate of the proportion of the population estimated to have/not have the attribute being tested. If the test results differ from the estimated proportion, then the auditor shall assess whether an increased sample size is required.

                          217.4 If the population being tested is relatively small (under 250 items), or is estimated to have a high expected error rate (greater than 40%), then a sampling approach other than representative sampling shall be applied unless otherwise approved by the Auditor-General.

                          217.5 In considering the alternative approaches to sampling, the auditor shall:

                          • evaluate the relative efficiency of each approach;
                          • evaluate whether the evidence expected to be obtained will be sufficient and appropriate for the audit; and
                          • document the rationale for the approach chosen.

                          217.6 When choosing a method for selecting items for audit testing:

                          The planned approach shall be outlined in the Audit Work Plan and updated in the audit’s Progress Reviews.

                          Where a non-representative testing strategy is employed, the audit report shall provide a description of the characteristics, or attributes, of the population being tested and not assert or imply that the items tested were representative or that the sample size was statistically valid.

                          217.7 Audit teams shall apply the sampling approach outlined in this document, unless the circumstances require the assistance of a statistical expert and approval is obtained from the responsible Engagement Executive. A statistical expert shall be used when the attributes of the population being tested are of a more specialised nature or are outside the scope of the sample size calculator. This includes, but is not limited to, measurement of relationships between variables and the conduct of surveys.

                          217.8 The ANAO performance audit sampling template shall be completed when determining and documenting the approach to selecting items to test in a performance audit.

                          217.9 All judgements made in determining the planned audit approach for selecting items for testing, and any subsequent revisions to the approach, shall be documented in the audit file.

                          Guidance

                          217.10 When planning and conducting a performance audit, the auditor is required to obtain sufficient appropriate evidence on which to base audit conclusions. Audit sampling means selecting a representative sample for testing with a view to making statistically appropriate conclusions about the total population. It assists the auditor by obtaining evidence ‘to provide a reasonable basis for the auditor to draw conclusions about the population from which the sample is selected’. Representative sampling is one of several methods to select items for testing.

                          217.11 The test item selection methods available to performance auditors are:

                          • testing all items in a population (also known as census testing);
                          • targeted testing based upon items with specific attributes in a population (also known as target testing); or
                          • testing a representative sample of the items in a population.

                          217.12 Testing all items in a given population will obtain the highest level of assurance. But, especially when dealing with large populations, this may not be a reasonable or efficient way to conduct audit testing.

                          217.13 Targeting items for selection can be an efficient means to obtain audit evidence. This requires targeting items based on some attribute that is important to the audit. This may be based on quantitative factors (such as all items over a relevant threshold) or qualitative factors. The performance auditor chooses the relevant attributes to focus testing on the areas within the population with the highest level of audit risk. The total number of items tested will be determined according to auditor judgement. The performance auditor can only draw conclusions on the targeted items that have been tested and not on the untested portion of the population and consequently auditor judgement on appropriate selection size would consider if the audit risk arising from the untested population is acceptable in the circumstances of the audit.

                          217.14 A representative sample requires a sample size that is statistically appropriate and the sampled items to be selected randomly. This allows the testing to be representative of the total population. This can be a very efficient way to test large populations (over 800 items). For moderately sized populations (between 250 and 800 items) the relative efficiency of a representative sampling approach will depend upon the circumstances.

                          217.15 The planned approach of the audit for selecting items for testing, which may include a combination of methods, is based on professional auditor judgement. The planned method(s) for selecting items for testing needs to be clearly defined and documented. This assists in determining an audit approach that is effective, efficient and repeatable.

                          217.16 Selecting items for audit testing may include both targeted and representative item selection methods.

                          217.17 The ANAO Performance Audit testing approach is embedded in the Performance Audit Sampling template and further guidance is contained within the template and incorporated instructions.

                          217.18 The sample size calculator assumes that the method for selecting the sample items for testing is random sampling. Therefore, any representative sampling must be performed according to a random selection approach. This excludes a convenient or haphazard sampling approach where each item in the population does not have an equal chance of being selected.

                          217.19 The audit testing template also assumes that the population is homogenous. It may be appropriate to target test items in the population that are not homogenous or use stratification to split the population into separate strata with different characteristics. When stratifying populations, the sample size calculator is completed for each separate population and clearly documented in the template.

                          217.20 If an item selected for testing is not relevant to the test and should not have been included in the population, the item must be replaced by selecting a new randomly selected item. If an item selected is not available for testing and suitable alternative audit procedures cannot be performed, e.g. documentation relating to the item is lost, this item is treated as a deficiency or error and is not to be replaced.

                          Procedure following testing

                          217.21 At the conclusion of the audit test, it is necessary to compare the estimated proportion rate with the actual percentage of errors in the sample result. If the actual result is equal to or greater than the estimate, then the result is considered to be statistically valid and the sample size is appropriate to draw conclusions from. But if the result was less than the estimate, then the audit team need to consider a larger sample size.

                          217.22 For example, consider a population of 800, with a confidence level of 95%, a confidence interval of 10% and an estimated error rate of 20%. An actual test result that found 10% of the attributes being tested were incorrect would be considered statistically valid for a sample size of 58 items and no additional samples would need to be tested. For the same example, with an estimated error rate of 20% where the actual error rate was 35%, the sample size would not be statistically valid and an additional 21 items would need to be tested to obtain the statistically valid sample size of 79 items in order to draw valid conclusions at the same confidence level and interval.

                          217.23 If the additional sample required to achieve a statistically valid sample size is large, it may be more efficient to revise the testing strategy. If no additional sample is tested, the sampled items tested so far do not represent a statistically relevant sample and therefore cannot be relied upon as audit evidence by themselves. Consequently, the results cannot be included in the audit report in a manner that asserts or infers that the results are representative of the population. However, the audit team may consider the usefulness of the testing performed as corroboration of other audit evidence gathered and consider including the results in audit reports with suitable clear caveats about the inability to extrapolate the results of testing so far to the population.

                          217.24 Where the actual error rate in the original sample is significantly greater than the anticipated error rate (particularly where the actual error rate is also significantly greater than what would be considered a reasonable/acceptable error rate) this may be evidence that the process being tested has significant deficiencies related to how the process has been designed and implemented that the audit team had not detected. The audit team should re-examine these matters before determining the appropriate audit response.

                          Figure 1: 

                          A decision tree to see if a sample is statistically valid

                          218. Audit documentation: Protective marking and bulk collection of entity emails

                          Background

                          218.1 The Auditor-General Act 1997 (Cth) (section 33) provides for full and free access to any documents or other property. Documentary evidence can include material that has a ‘PROTECTED’ protective marking, including ‘PROTECTED CABINET’. The Auditor-General may also, by written notice, direct a person to provide information, produce documents and give evidence. However, most audit evidence is obtained through a cooperative approach with the audited entity.

                          Policy

                          Application of ANAO Protective Security framework

                          218.2 All staff shall comply with the ANAO’s Protective Security Governance Policy in the conduct of their audits..

                          218.3 When collecting, analysing, and documenting evidence (including the bulk collection and analysis of entity emails) and drafting the audit report, all audit staff shall comply with the ANAO’s Information Security Policy, ICT Security Policy and Information, Classification and Handling Guidelines.

                          Dealing with Cabinet Material

                          218.4 When dealing with Cabinet documents9, audit teams shall have regard to:

                          • ANAO Protocol on use of Cabinet-in-Confidence Information in Audit Reports
                          • Practice Note: Use of Cabinet Material in Performance Audit Reports; and
                          • Access to Cabinet Documents Policy and Procedures.

                          218.5 In applying the requirements of 218.4, audit teams shall:

                          1. it will be appropriate to refer to the material in the final report without explicitly identifying it as Cabinet information; or
                          2. explicit discussion of Cabinet deliberations, processes and/or outcomes is likely to be required; and
                          • ensure that Cabinet documents are requested, handled and disposed in accordance with Cabinet requirements as they apply to the ANAO;
                          • through the Engagement Executive and responsible GED, advise the Auditor-General when the audit team expects that the performance audit report will refer to Cabinet material. This will commonly occur through the section 19 workshop process but may occur separately where deemed appropriate by the Engagement Executive. However this advice is communicated, the audit file shall document the Auditor-General’s involvement10, including the audit team’s advice as to whether:
                          • ensure that the approval by the Auditor-General for the inclusion of Cabinet material in an audit file is documented in the audit file.
                          Bulk Collection of Data, including Emails

                          218.6 When a performance audit collects the emails of audited entities on a bulk basis11 the audit team shall ensure that auditee information is held, accessed, analysed and reported by the ANAO in a manner that meets the audit objective with the minimum reasonable risk posed to the confidentiality and privacy12 of individual auditee staff, contractors and other stakeholders.

                          218.7 This policy also applies to any other circumstance where bulk data is obtained for a performance audit where personal data is likely to be included, including other forms of electronic correspondence being collected on a bulk basis, such as instant messaging or telephone records.

                          Guidance

                          Application of ANAO Protective Security framework

                          218.8 The ANAO’s Protective Security framework, as specified in the Protective Security Governance Policy, applies to all data held by the ANAO, including evidence gathered in the process of planning, execution and reporting of performance audits.

                          Protocol for ANAO Use of Cabinet Material in Performance Audit Reports – Obtaining and Handling Cabinet Material

                          218.9 Sections 32 and 33 of the A-G Act provide the Auditor-General with broad information-gathering powers that extend to any relevant document or information, regardless of their classification or status. However, in practice, the information-gathering powers are used as ‘reserve’ powers and access to required information is almost always obtained through cooperation with entities, including Cabinet materials:

                          • ANAO staff should seek to obtain information on a cooperative basis.
                          • If an entity is not cooperating, consult with the Engagement Executive and responsible GED in the first instance to escalate the request. They will consider further options, including as necessary possible use of either section 33 or requesting that the Auditor-General uses section 32 to obtain the necessary information. ANAO Legal Services can also assist with informing entities about the nature of the information-gathering powers.
                          • ANAO staff must at all times treat information in their custody appropriately and in accordance with the Protective Security Policy Framework as specified in the Information, Classification and Handling Guidelines.
                          • Section 36 of the A-G Act ensures protection for any information gathered by making it an offence to disclose information gained in the course of conducting an Auditor-General function for any purpose other than performing that function.
                          • The A-G Act also recognises that it may not be in the public interest for the Auditor-General to disclose some sensitive information.

                          218.10 In accordance with Cabinet material rules, Corporate Management Group (CMG) maintain documented processes for the requesting, handling and disposal of Cabinet materials which are documented on Audit Central.

                          Protocol for ANAO Use of Cabinet Material in Performance Audit Reports – Reporting on Cabinet Material

                          218.11 The ANAO has developed a protocol and practice note that summarises the circumstances in which the Auditor-General may decide to publish Cabinet material in performance audit reports or other audit products prepared under the Auditor-General Act 1997 (Cth) (the A-G Act)13 for tabling in the Parliament.

                          218.12 The A-G Act also recognises that it may not be in the public interest for the Auditor-General to disclose some sensitive information14. When considering the publication of sensitive information, it is the Auditor-General’s role to determine whether the public interest is best served by disclosure.

                          218.13 As an independent officer of the Parliament, the Auditor-General has discretion in the performance or exercise of his functions or powers. In particular, the Auditor-General is not subject to direction in relation to whether or not a particular audit is to be conducted, or the way in which a particular audit is to be conducted.

                          218.14 The Auditor-General undertakes the following steps when considering the publication of Cabinet material:

                          • As a matter of sound public administration, and with due regard to the long held Westminster tradition of respecting Cabinet confidentiality, the Auditor-General only publishes Cabinet material occasionally, and only when the material in question is fundamental to the conclusions of the audit.
                          • ANAO officers provide written advice and consult with the Auditor-General about the potential need to include any such material.
                          • The proposed inclusion of any such material is drawn to the attention of the auditee’s Accountable Authority during report preparation and when the Accountable Authority is provided with a draft final report for review and response.
                          • The Auditor-General seeks to avoid direct quotations from deliberative documents of the Cabinet and, to the extent that it is possible, publishes reports that attribute decisions or outcomes to the responsible minister or government rather than to the Cabinet.

                          218.15 Any explicit discussion of Cabinet deliberations, processes or outcomes is included only when, in the considered view of the Auditor-General, the conduct of those deliberations and processes, or the associated outcomes, led directly to a conclusion expressed in the final audit report.

                          218.16 To ensure consistency with the protocol, audit teams must follow these steps:

                          • Ensure at all times that any Cabinet material accessed or gathered during the course of an audit is treated and stored appropriately and in accordance with the Information, Classification and Handling Guidelines. If necessary, the ANAO Security Adviser (in CMG) can be contacted for advice and assistance.
                          • When requesting access to Cabinet documents – which should be through the Cabinet Division of the Department of the Prime Minister and Cabinet (PM&C), please ensure the document numbers are identified (where possible) and the requested documents are relevant to the audit, as determined by the Auditor General.
                          • In preparing any such documentation, due regard must be given to security and information protection obligations. Audit teams should consult the Entity Security Adviser as necessary.
                          • In most cases, references to Cabinet material in audit reports should use generic and minimal descriptions (i.e. ‘government’), other than when fundamental to audit conclusions.
                          • if a draft audit report explicitly identifies Cabinet material, then the relevant GED or Executive Director should inform their counterparts in the Department of the Prime Minister and Cabinet of any intended publication of Cabinet material.
                          Bulk Collection of Entity Emails

                          218.17 Entities’ email systems are used in the administration of programs and activities. As email systems and the information and records they contain are legally the property of the entity, the ANAO is entitled to access this material in the same way as other entity records and information. Email records almost inevitably contain personal communications. As a result, the ANAO’s access will generally involve access to both official and personal communications. This can cause heightened sensitivities and may, at times, lead to disagreement between the audit team and the entity that need to be properly managed, often at a senior level.

                          218.18 Without appropriate care and attention, the ANAO’s collection of bulk email records also poses the risk that audit team members do not deal with personal information in an ethical manner or in a manner that is not seen to be ethical.

                          218.19 When bulk email records are collected audit teams should consider:

                          • exploring ways of electronically searching records for relevant information in a way that limits access to personal information and establishing internal team protocols to manage the risk that personal information is accessed and breaches to the need-to-know basis, including unreasonable access to personal information not related to the relevant audit questions;
                          • that relevant email records may be archived and the retrieval may involve longer timeframes and additional costs compared with the retrieval of paper files. In this context, the ANAO does not accept responsibility for entity costs arising from the conduct of audits, as section 33 of the Auditor-General Act 1997 (Cth) provides for ‘full and free access at all reasonable times to any documents or property’. Nevertheless, it is expected that audit teams would be conscious of the costs that may be incurred by entities in meeting requests for access to records, and would seek to minimise these, to the extent possible;
                          • whether it may be appropriate to provide assurances to entity management that any personal information accessed will not be used in any way and will not be incorporated into the ANAO’s working papers; and
                          • whether it is appropriate to limit access to entity bulk email records to nominated members of the audit team and/or establishing separate security arrangements for the: storage of bulk entity emails during the audit; and destruction of emails not used as key evidence by the ANAO.

                          218.20 Such operational protocols may also be considered necessary to provide assurance regarding certain paper records, such as records containing highly personal information pursuant to disclosure regimes.

                          218.21 A balance needs to be achieved between meeting the ANAO’s responsibilities, including conducting audits in an efficient and objective manner, and recognising the legitimate concerns of entity management. This, at times, may involve the audit team explaining, in more detail than would normally be the case, the ANAO’s audit and confidentiality responsibilities and the reasons why access to certain entity information and records is necessary to meet these audit responsibilities. Potential issues in accessing data should be escalated early.

                          219. Surveys

                          Policy

                          219.1 Audit teams shall document their intention to conduct a survey as part of their initial Audit Work Plan.

                          219.2 The Audit Work Plan shall include details on the design and delivery of the survey, and the estimated costs of the survey in the proposed budget.

                          219.3 The audit team shall also discuss the planned survey with the auditee entity at the entry interview.

                          Guidance

                          219.4 From time to time, audit teams may seek to conduct a survey to gather information from a large population to inform the scope or direction of an audit, or to provide insights into the perceived performance of a program or entity.

                          219.5 An audit team may design and deliver an audit survey internally, subject to the audit team including members with the appropriate experience and expertise. Alternative approaches include:

                          • contracting an expert to design and deliver the survey, or
                          • making use of an online survey tool.

                          219.6 Key considerations that audit teams should have when determining the survey approach include:

                          • the information security associated with proposed online survey tools;
                          • the information protection arrangements in place. At a minimum, any online survey tool used should offer SSL encryption to protect the privacy and integrity of collected data;
                          • an awareness of and being transparent about the data storage location; and
                          • any survey tool indemnity clauses that would require approval by a PGPA Act section 60 delegate.

                          219.7 Audit teams should compare and evaluate the options offered by various tools in terms of branding, question design options, technical support and data reporting formats.

                          219.8 As for all procurements, audit teams should appropriately weigh up the costs and benefits of various options; ensure that they select a provider or tool that is fit for purpose and provides value for money; and clearly document the reasons for their decision. Any procurement plan should be endorsed by the PASG Practice Manager.

                          219.9 Audit teams should take the following steps to mitigate the risk of information leakage and manage stakeholder expectations of privacy and security:

                          • inform potential survey respondents about the arrangements that are in place to protect and store their responses, and should disclose the involvement of any third-party provider(s);
                          • minimise the collection of personal information, information about respondent organisations or contact details; and
                          • provide participants with alternative survey completion options (e.g. offline or by telephone).

                          220. Verifying audit evidence

                          Policy

                          220.1 Audit teams shall adopt an attitude of professional scepticism in making independent judgements about audit evidence during an audit.

                          220.2 When using information produced by the entity to perform audit procedures and gather audit evidence, the audit team shall evaluate whether the information is sufficiently reliable for the purpose of the audit, including obtaining audit evidence about the accuracy and completeness of the information.

                          Guidance

                          220.3 Verifying audit evidence is the first step in analysing audit evidence to ensure that audit findings, conclusions and recommendations are based on sound evidence.

                          220.4 The risk, significance and sensitivity of the matter to be reported will determine not only the nature and amount of evidence to be collected but the extent of verification.

                          220.5 Assessing information obtained from entity records is particularly important in circumstances where management information, such as program expenditures or key performance indicators, is to be relied on. In these cases it is essential that the completeness and accuracy of this information is assessed through appropriate audit testing to give the ANAO confidence that the systems and processes used to produce the information can be relied on. This will involve reviewing key system controls and testing a sample of transactions.

                          220.6 Audit testing is required to be undertaken in these circumstances irrespective of when such information is obtained. For example, if information is provided in response to a section 19 report that is significant in settling the final audit conclusions, it needs to be assessed. When management information is contextual and is not significant in the context of the audit findings and conclusions, it is not necessary to test its completeness and accuracy. However, in these situations it is expected that the audit report would indicate that the information has been sourced from entity records or is based on entity advice, as distinct from audit analysis.

                          220.7 The ANAO’s policy in relation to sufficient appropriate audit evidence also applies to any work performed by an expert. When using the work of an expert to support audit findings and conclusions, it must be adequate for that purpose, recognising that the responsibility for the findings generated or conclusions drawn from the work undertaken by an expert rests solely with the ANAO.

                          220.8 Professional scepticism (ASAE 3500 paras 16(p), A57) means an attitude that includes a questioning mind, and being alert to:

                          • conditions which may indicate possible deficiencies in administration;
                          • audit evidence that contradicts other evidence obtained;
                          • information that brings into question the reliability of documents and responses to enquiries;
                          • conditions that may indicate systemic system deficiencies; and
                          • circumstances that suggest the need for further analysis and enquiry.

                          220.9 The type of audit evidence gathered will determine the available verification approaches available:

                          Type of evidence

                          Verification technique

                          Physical

                          The observation is recorded using, for example, image capture technologies.

                          Oral/testimonial

                          The interviewee confirms in writing the facts of the discussion.

                          A number of independent sources confirm the evidence, for example, interviews with staff, documentation and expert advice.

                          A survey provides valid results.

                          The facts of a discussion are confirmed from other sources. The oral evidence is recorded.

                          Documentary

                          Documentary evidence is obtained from more than one source.

                          Documentary evidence is the final version and has the appropriate approvals and sign-offs.

                          The matters set out in the document, for example, performance results, are tested for accuracy.

                          Internal system controls are tested or electronic records are validated.

                          Adherence to procedures or plans is tested to determine that they are followed in practice.

                          Internally generated reports

                          Ensure that the report is a standard system report, unable to be changed by the auditee.

                          Technical re-performance using IT specialists.

                          Samples of data are tested for accuracy, completeness and validity.

                          Further guidance on these approaches is provided below.

                             

                          220.10 There are a number of different approaches to test completeness and accuracy of internally generated reports and each approach provides a varying level of comfort. Professional judgement is required to determine the most appropriate method in the circumstances based on an understanding of how the report is generated and the desired level of comfort.

                          Standard system reports

                          220.11 The approach which provides the highest level of evidence is where the auditee is utilising standard system reports. When the controls for producing the reports have been tested, then there is a reasonable expectation that any errors would have been identified by the system.

                          220.12 Typical considerations to document include:

                          • auditee does not have a development environment, or access to such an environment is limited to external vendor accounts;
                          • review of change management logs shows no changes except vendor management;
                          • collaborative enquiry with different sources within the organisation confirms that the auditee does not undertake development / have access to the application source code;
                          • ANAO understanding of the application based on knowledge and experience is that the application is ‘off-the-shelf’ and vendor supported; and
                          • ANAO review of the standard user guides identifies the specific reports / controls to be part of the packaged application.

                          220.13 These are considerations only. It is important that the ANAO is comfortable that the standard reports are unable to be changed by the auditee and only by the vendor.

                          Technical re-performance

                          220.14 Technical re-performance involves reviewing the code (e.g. SQL query) which is used by the system to produce the report. This can only be performed when the auditee is able to access the code, and when the audit team have appropriately skilled team members who can understand the code.

                          Sample testing

                          220.15 Samples of data are tested for accuracy, completeness and validity. The procedure for this testing is detailed in the Sampling and selecting items for testing chapter of this manual.

                          220.16 With all approaches there will typically be parameters applied to reports within a system. These parameters could be date ranges, product types, cost centres, etc. It is important that these parameters are reviewed as part of the process of obtaining comfort over the system reports.

                          221. Forming the audit conclusion

                          Policy

                          221.1 The audit team shall form the overall audit conclusion with regard to the objective of the audit, as evaluated against the criteria.

                          221.2 In accordance with the policy at paragraph 227.5 – 227.6, the overall conclusion is the first sentence under the Conclusion heading of the Summary and Recommendations chapter. It shall directly address the question of whether or not the audit objective has been met and, if not, be specific about the findings that resulted in exceptions to the conclusion.

                          221.3 The paragraphs following the overall conclusion shall be identical to the ‘conclusion’ section of the blue summary boxes in subsequent chapters. These conclusions shall directly address the question of whether or not the individual criteria have been met and, if not, be specific about the findings that resulted in exceptions to the conclusion.

                          221.4 Materiality shall be considered when evaluating the effect on the conclusion(s) of any identified findings, taken individually and in combination.

                          221.5 Only findings determined to be material shall be presented as exceptions to the conclusion(s) and included in the Conclusion section of the report. Other findings shall be presented in the supporting findings section of the report.

                          221.6 In accordance with the requirement at 227.2 for the report to be balanced, the conclusion section shall also include reference to evidence of good performance that has had a material impact on the conclusion. That is, the conclusion shall specify the material matters that resulted in the objective and / or criteria being met.

                          Guidance

                          221.7 The following diagram outlines the audit process and outcomes.

                          Stages are audit objective; audit criteria; sub-criteria; evidence and findings; conclusions (against sub-criteria); conclusions (against criteria); and audit conclusion (against objective)

                          Audit findings

                          221.8 Audit findings are generated when the criteria (and sub-criteria) are compared with the audit evidence. Meeting or exceeding the criteria may indicate good practice leading to good performance. Failing to meet criteria would indicate that improvements are needed. It is, however, unrealistic to expect that the audited entity’s performance regarding economy, efficiency, and effectiveness will always meet the criteria. This means that in addition to assessing whether the audited entity meets the criteria or not, the audit team also has to consider materiality and apply professional judgment in interpreting how this affects assessment of the entity’s performance.

                          221.9 The audit report should include both positive and negative points and give credit where it is due. Including positive aspects may lead to improved performance by other government organisations that read the report. It is important that the report contains all the information and arguments needed to satisfy the audit objective(s), and promote adequate and correct understanding of the matters and conditions reported.

                          221.10 Audits involve some type of analysis in order to understand or explain what has been observed. When analysing information collected, the auditor should focus on the criteria and objective. This will help to organise the data and also provide the focus for analysis.

                          221.11 While it is important to seek explanations for deviations from criteria, causes should be presented with caution. They have to be supported by sufficient and appropriate audit evidence. It is relevant to consider the audited entity’s views on reasons for performance problems or weaknesses. If such views are not supported by sufficient and appropriate audit evidence, the audit team cannot take for granted that they are relevant or correct.

                          221.12 The audit team should identify the possible effects of the criteria not being met. The effects could be identified either as what has already occurred or as possible future impact. The nature of the findings determines whether the audit team can present actual or potential effects.

                          Developing conclusions after considering the findings

                          221.13 Once the findings have been established and causes and effects considered, the audit team draws conclusions against the sub-criteria, criteria and objective respectively. Conclusions are statements informed by the findings. Since performance audits may point out performance relating to deficiencies in aspects of economy, efficiency and/or effectiveness, the conclusions have to specify the reasons why aspects of economy, efficiency or effectiveness may not have been fully met.

                          221.14 Audit conclusions clarify and add meaning to specific findings in the report. Conclusions present the opinion and go beyond merely restating the findings. Whereas the audit findings are identified by comparing ‘what should be’ according to the criteria with the audit evidence (including analytical evidence) on ‘what is’, the conclusions also reflect the auditor’s explanations and views based on these findings. Conclusions might include identifying a general theme or a certain pattern in the findings. An underlying problem that explains the findings may also be identified.

                          221.15 When drawing conclusions it will often be necessary to revisit the data analysis and the audit findings to be sure that the conclusions are based on solid grounds. The analysis of data consists of combining results from different types of sources. The conclusions are based on the objective, criteria, evidence and findings.

                          221.16 The consideration of materiality is a matter of professional judgement. Materiality is considered in the context of qualitative factors and, when applicable, quantitative factors. The relative importance of qualitative factors and quantitative factors when considering materiality in a particular performance engagement is a matter for professional judgement. This judgement may be informed by whether the issue has had a substantive impact on the quality and/or cost efficiency of the services or policy advice provided – i.e. the primary objectives of the entity.

                          221.17 Concluding on the materiality of the findings identified as a result of the procedures performed requires professional judgement. For example: three criteria are identified for an audit. For one of these criteria there are two sub-criteria. If sub-criteria one is satisfied, but the sub-criteria two is not, a finding will be reported in respect of sub-criteria two as it would be of interest to readers of the report. Professional judgement will be applied to determine the appropriate level of prominence for this finding in the audit report, given its actual or potential impact. That is, whether it will be presented within supporting findings or as an exception to the conclusion. This professional judgment will also consider the relative importance of this sub-criterion to the criterion to which it relates as well as the relative importance of that criterion to the other criteria and findings in respect of the other criteria.

                          221.18 In addition, professional judgement will consider whether there are factors that resulted in the satisfaction of sub-criteria one that are material to the conclusion that should be reported.

                          221.19 Findings are considered to be material to the overall conclusion if they, individually or in the aggregate, could reasonably be expected to change or influence the decisions of users of the report, such as the legislature or executive. Where a finding or findings is / are determined to be material to the overall conclusion, the conclusion will either be expressed in the form of the objective being met with exceptions or the objective not being met. For example, when there are findings in respect of one criterion only, it is likely that the objective would be found to be met with exceptions and the conclusion would be expressed as ‘the program was implemented effectively except for…’. Where the findings are so pervasive to the criteria and therefore objective of the audit, the conclusion would be that the objective was met and expressed as ‘the program was not implemented effectively because of …’.

                          222. Quality control

                          Background

                          222.1 A system of quality control is required to provide assurance that the audit team’s work has met the requirements of the ANAO Auditing Standards and complies with performance audit policies and related procedures.

                          Policy

                          222.2 There shall be a system of quality control to provide assurance that all audits have been conducted in accordance with performance audit policies and related procedures. Evidence that key elements of quality control have been undertaken shall be documented by relevant staff using the Performance Audit Report Policy Compliance Certification. The Performance Audit Report Policy Compliance Certification must be completed prior to the auditee’s response to section 19 report and draft final report being provided to the ANAO Executive for clearance.

                          222.3 Where work is delegated to members of an audit team, the Engagement Executive and/or the Audit Manager shall carefully direct, supervise and review the work undertaken to assist with the effective implementation of the audit. The Engagement Executive shall conduct a sufficient review of the audit team’s work to provide assurance that the audit has been properly performed and appropriate evidence-based conclusions reached.

                          222.4 Supervision and review of the work of experts shall also be undertaken to assess compliance with performance audit policies and related procedures. The audit team shall make enquiries and test the views expressed by the expert to the extent necessary to be satisfied that the findings and conclusions are supportable and appropriate.

                          222.5 The Audit Work Plan, Report Preparation Papers and draft reports shall be subject to review, including that appropriate approvals were obtained and all significant issues were discussed with the ANAO Executive.

                          222.6 The evidence contained in the working papers to support each of the key issues/findings shall be subject to review by the Audit Manager and the Executive Director. Where audit working papers have been subject to review, evidence of the review shall be documented.

                          Guidance

                          222.7 A system of quality control for performance audits is consistent with the ANAO Quality Assurance Framework.

                          222.8 Reviewing the audit coverage and findings provides assurance that all relevant issues and considerations have been addressed and that sufficient appropriate evidence was obtained for all issues that impact on the program or activity subject to audit and support the audit findings and conclusions.

                          222.9 A key element in the quality control system for performance audits is that the audit working papers should be subject to review. Working papers include all the planning documents, evidence collected and analysis undertaken during the course of an audit, and all documents relating to reporting for an audit.

                          222.10 The extent and nature of the review of the working papers should be sufficient for the reviewer to be satisfied that the audit findings and conclusions are supported by audit evidence, that they demonstrate an in-depth knowledge and understanding of the subject matter and that they are presented in a balanced, fair and constructive manner. Cross-referencing and filing all working papers facilitates the review of audit evidence and working papers.

                          222.11 The quality control system described above does not require all audit working papers to be reviewed by the Engagement Executive or more senior members of the audit team. The Engagement Executive may delegate review of some audit work to the Audit Manager or other more senior members of the team with suitable skills and experience but the Engagement Executive retains overall responsibility for the review and supervision of the engagement. Ordinarily the Engagement Executive reviews the higher risk aspects of the audit in addition to all those areas that specifically require Engagement Executive review in accordance with the ANAO Auditing Standards and this policy.

                          222.12 Reflecting that the entity subject to audit also has an important role to play in the audit process, audit working papers should indicate the extent to which the entity was consulted during the planning phase of the audit and evidence of the interaction with the entity about the audit findings, conclusions and recommendations throughout the audit.

                          222.13 Where the work of an expert is being used as evidence in an audit, the audit team retains full responsibility for the conclusions drawn from that evidence and presented in the audit report, recognising that it is ultimately the Auditor-General who signs each report and therefore will make the final judgement about the audit findings and conclusions.

                          223. Report Preparation Papers

                          Policy

                          223.1 Report Preparation Papers (RPPs) shall address the audit objective(s) and audit criteria. The RPPs shall be sufficiently detailed to allow the entity to understand the basis for, and facilitate meaningful understanding and/or discussion of, the preliminary audit findings, conclusions and any potential recommendations.

                          223.2 All RPPs be prepared in accordance with the Service Group’s current ‘PASG Report’ template style and format.

                          223.3 RPPS shall be provided to the entity in accordance with the PASG Workflow.

                          223.4 In circumstances where RPPs are not provided to the entity, for example, a short duration audit where the entity was formally briefed about the finding, preliminary conclusions and possible recommendations before issuing a section 19 report, the reasons for not providing the Report Preparation Papers shall be documented.

                          223.5 RPPs shall be provided to Assurance Audit Services Group. Briefings shall also be provided to AASG on significant issues as they arise in the course of a performance audit and as Report Preparation Papers are prepared.

                          223.6 Draft RPPs shall be cross-referenced to the audit evidence and documented in the audit working papers.

                          Guidance

                          223.7 The audit team’s review and assessment of audit findings and conclusions, drawing on the audit evidence gathered and analysed in the delivery phase of the audit, form the basis of the Report Preparation Papers.

                          223.8 Specific procedures for the clearance and issuance of RPPs are included in the PASG Workflow, which contains corporate templates for staff to use during an audit.

                          223.9 RPPs, or equivalent working papers for audits of multiple entities, will generally be the first formal opportunity for the entity to consider the ANAO’s preliminary audit findings, conclusions and proposed recommendations. It is important that the entity understands that this is not the formal section 19 report. The audit team should explain to the entity the purpose of the RPPs, preferably in person to entity management, as well as in the correspondence that accompanies the RPPs.

                          223.10 RPPs can be prepared while audit evidence is being collected or towards the end of this phase when the majority of evidence has been collected. Drafting report preparation and evidence papers during the delivery phase and providing them to the entity for comment on an ongoing basis presents the opportunity to test the facts and findings at an early stage and to identify the need for more information where this is required.

                          223.11 RPPs, particularly when issued prior to the completion of fieldwork, can also be used to elicit further information and/or to seek further views from the entity, on particular issues that are important to the audit.

                          223.12 RPPs provide the entity with the opportunity to correct any errors of fact or interpretation, either in writing or at the Exit Interview. The entity can also provide additional information in response to this correspondence.

                          223.13 The iterative review of RPPs can be extensive for some audits. Audit teams are to make judgements regarding the number of reviewed versions of the RPPs that are required to be retained on the audit working papers. For example, drafts that contain minor editorial changes and comments only may not need to be retained. Any substantive edits or comments made by the GED or ANAO Executive are required to be retained.

                          223.14 Draft RPPs that contain substantive review comments and edits made by the responsible Executive Director and Group Executive Director are to be retained on the audit working papers. The comments and edits made by the ANAO Executive are required to be retained to evidence that the work performed by the audit team has been directed, supervised and reviewed, as well as to retain a documented record of the audit reporting process.

                          223.15 For audits that do not issue RPPs, a written summary of the key findings can be sent to the entity for discussion at the Exit Interview. For efficiency, the summary of key preliminary findings can be based on the content of the Progress Review 2 briefing.

                          223.16 Preliminary findings, conclusions and proposed recommendations contained in the Draft RPPs should not be a surprise to the entity given sufficient and comprehensive verbal briefings throughout the audit.

                          223.17 In the case of cross-entity audits or audits containing information about third persons or entities, audit teams will need to consider the preparation of extracts of RPPs. The extract strategy should be agreed with the Engagement Executive and GED if necessary and should have regard to the objective of providing sufficient information to recipients to respond meaningfully to the RPP, while avoiding the release to third parties and maintaining the ANAO’s confidentiality obligations.

                          223.18 In determining what is a significant issue requiring communication with AASG, PASG shall consider:

                          • the potential to give rise to a material mistake in the financial statements;
                          • any actual or suspected fraud;
                          • any issues revealing a weakness in key financial controls; and
                          • any other issues of which the audit team is aware that may be of interest to AASG.

                          223.19 The ANAO Communication Unit is available to assist with any queries regarding the writing of audit reports in terms of format and style. Audit teams can also refer to the ANAO Style Guide.

                          223.20 RPPs should only be sent to a single officer of the entity that is not an accountable authority, director or member of the governing body of the entity. That officer would be referred to in the designation email so that the accountable authority, director or member of the governing body is aware of who the single officer will be. This will reduce the risk of accidental disclosure and give the relevant officer control over who receives the RPPs. Other officers may be sent an email informing them that the RPPs have been provided.

                          224. Exit interviews

                          Policy

                          224.1 An Exit interview shall be held with each entity subject to audit unless the entity declines the invitation or it is impractical to do so. For example, in the case where the audit approach involved a large number of entities participating in a survey, it may not be practical to conduct an Exit Interview with each entity.

                          224.2 RPPs, or their equivalent for audits of multiple entities, shall be provided prior to the Exit interview to allow the entity sufficient time to review and formulate a considered response to the preliminary audit findings, tentative conclusions and proposed recommendations.

                          224.3 A record of the Exit interview, including the outcomes, any sensitivities and all significant decisions made, shall be documented in the audit working papers prior to the finalisation of the proposed section 19 report.

                          Guidance

                          224.4 Exit interviews are an important step in the audit process and occur at the end of the period for the major collection of audit evidence, analysis of findings and identification of potential recommendations for the entity.

                          224.5 Where an Exit Interview is to be held, it can be a forum to:

                          • give the entity an indication of what the final report will include;
                          • give the entity the opportunity to provide its perspective on the preliminary audit findings and to correct any errors of fact, interpretation or perception;
                          • provide entity management with the opportunity to challenge proposed recommendations. The recommendations may then be adjusted, removed, or new recommendations developed in light of the additional information provided; and
                          • give the entity the opportunity to provide additional evidence, and assess whether any audit findings should be modified in light of the additional information provided.

                          224.6 ANAO staff attendance at the Exit Interview, in addition to the audit team members, is to be determined on a case-by-case basis by the Engagement Executive.

                          224.7 Entities should be encouraged to invite senior representatives to exit interviews.

                          Engagement performance — reporting

                          Chapters 225 to 228

                          225. Proposed section 19 report

                          Background

                          225.1 The proposed report is referred to as the section 19 report, because it is required by section 19 of the A-G Act.

                          Policy

                          225.2 The audit team shall draft a section 19 report that addresses issues that are relevant to the audit objective and the audit criteria. Audit reports shall be comprehensive, convincing, timely, reader friendly, and balanced.

                          225.3 All audit reports shall contain a report summary that consists of: background (including key information about the audit, audit objectives and criteria); conclusion; supporting findings; recommendations (where appropriate); summary of the entity response; and key learnings for Australian Government entities.

                          225.4 The background section shall outline the rationale for undertaking the audit, the audit objective and high-level audit criteria, including the source of those criteria, to provide the report user with information about the basis on which the audit findings and conclusions have been made. The high-level criteria included in the report summary shall be fleshed out in the main body of the report.

                          225.5 The overall conclusion shall be framed against the audit objective, and directly address the question of whether or not the audit objective has been met and, if not, be specific about aspects or matters that require improvement.

                          225.6 The overall conclusion shall:

                          • be expressed in the first sentence under the Conclusion heading of the Summary and Recommendations chapter;
                          • contain paragraphs identical to the ‘conclusion’ section of the blue summary boxes in subsequent chapters;
                          • summarise areas where the entity could improve its performance; and
                          • present the significant matters and issues in a concise and balanced way and link them in a logical manner—this avoids the appearance of presenting a list of unrelated matters.

                          225.7 In the draft section 19 report prepared for review by the Engagement Executive, Group Executive Director and the ANAO Executive, the findings in the section 19 report shall be cross-referenced to the audit evidence.

                          225.8 The section 19 report shall also include the following:

                          • methods of data gathering and analysis applied;
                          • time period covered;
                          • sources of data; and
                          • limitations to the data used.

                          225.9 Audit teams shall retain evidence of review by the Engagement Executive, Group Executive Director or the ANAO Executive of the draft section 19 report as part of audit working papers.

                          225.10 The section 19 report shall be forwarded through the Engagement Executive to the Group Executive Director, the Deputy Auditor-General and the Auditor-General for discussion at the section 19 meeting, agreement and approval.

                          225.11 All section 19 reports shall be reviewed by the Engagement Executive and provided to the responsible Group Executive Director, Deputy Auditor-General and Auditor-General concurrently for clearance. Following the incorporation of any comments, the Engagement Executive and, as necessary, the GED, reviews the section 19 report for final approval and formal sign-off by the Auditor-General.

                          225.12 In circumstances where they are considered to require amendment, approval to amend the audit criteria shall be sought from the ANAO Executive.

                          225.13 The section 19 report shall be sent from the Office of the Auditor-General - Performance Audit email address, using the Auditor-General’s signature block.

                          225.14 For performance audits of individual Commonwealth entities, Commonwealth companies, subsidiaries or Commonwealth partners under sections 17, 18A or 18B of the A-G Act, the complete section 19 report shall be sent to the auditee.

                          225.15 For cross-entity performance audits conducted under section 18 of the A-G Act, arrangements for providing the section 19 report (either a complete version of the report or relevant extract(s) only) shall be determined in consultation with the Auditor-General..

                          225.16 Legal and procedural fairness obligations shall be fulfilled for all audits. The ANAO shall consider written comments provided within 28 days by the entity, or persons or bodies with a special interest in the section 19 report or extract, before finalising the audit report (subsection 19(7) of the A-G Act).

                          225.17 Where a person or organisation has a special interest in all or part of the section 19 report, the Auditor-General shall determine whether to provide that person or organisation with a copy of the section 19 report, or relevant extract, for comment.

                          225.18 Where an extract is to be provided to a person or organisation with a special interest, the Auditor-General shall approve the approach taken by the team to determine the content of the extract to be provided. Written notice shall be provided to the Auditor-General, through the GED, describing the approach proposed, with a copy of the extract attached.

                          225.19 Extracts from section 19 reports shall include sufficient material to allow the person or organisation receiving it to understand and comment on the accuracy of the facts that are relevant to them, and the context in which they are presented, in an informed way. Unless approved by the Auditor-General, the extract shall not disclose:

                          • Cabinet material;
                          • security classified material;
                          • briefing notes, advice or other material provided in confidence to Government;
                          • confidential material of other parties; or
                          • the audit conclusion.

                          225.20 To meet the requirements of the A-G Act, any written comments received on the section 19 report within 28 calendar days of the section 19 report being received by the recipient shall be given full consideration by the audit team in preparing the final audit report15.

                          225.21 Key learnings for other government entities shall, unless the contrary is approved by the relevant GED, be incorporated into the summary section.

                          225.22 Any substantive edits or comments made by the GED and ANAO Executive shall be retained. The comments and edits shall be retained to evidence that the work performed by the audit team has been directed, supervised and reviewed, as well as to retain a documented record of the audit reporting process.

                          225.23 The section 19 report shall be provided to Assurance Audit Services Group. Briefings shall also be provided to AASG on significant issues as they arise in the course of a performance audit and as section 19 reports are prepared.

                          Guidance

                          225.24 The section 19 report provides the opportunity for the entity subject to audit, and any other person that in the Auditor-General’s opinion has a special interest in the report, to provide comments that must be taken into consideration in preparing the final report.

                          225.25 The PASG Workflow provides instructions on preparing and issuing the section 19 report.

                          225.26 To be comprehensive, an audit report needs to include all the information and arguments needed to address the audit objective(s) and audit questions, while being sufficiently detailed to provide an understanding of the subject matter and the audit findings and conclusions.

                          225.27 To be convincing, an audit report needs to be logically structured and present a clear relationship between the audit objective(s) and/or audit questions, audit criteria, audit findings, conclusions and recommendations. It also needs to present the audit findings persuasively, address all relevant arguments to the discussion, and be accurate. Accuracy requires that the audit evidence presented and all the audit findings and conclusions are correctly portrayed. Accuracy assures readers that what is reported is credible and reliable.

                          225.28 Being timely requires that an audit report needs to be issued on time in order to make the information available for use by the legislature, management, government and other interested parties.

                          225.29 To be reader friendly, the auditor needs to use simple language in the audit report to the extent permitted by the subject matter. Other qualities of a reader-friendly audit report include the use of clear and unambiguous language, illustrations and conciseness to ensure that the audit report is no longer than needed, which improves clarity and helps to better convey the message.

                          225.30 Being balanced means that the section 19 report needs to be impartial in content and tone. All audit evidence needs to be presented in an unbiased manner. The auditor needs to be aware of the risk of exaggeration and overemphasis of deficient performance. The auditor needs to explain causes and the consequences of the problems in the audit report because it will allow the reader to better understand the significance of the problem. This will in turn encourage corrective action and lead to improvements by the audited entity.

                          225.31 Some key questions that should be considered when drafting the section 19 report include:

                          • Do the audit findings and conclusions relate to the audit objective?
                          • Do the recommendations flow from the findings?
                          • Have the questions posed in the audit been answered?
                          • Is the audit title appropriate?
                          • Is the structure of the report logical and does it flow easily?
                          • Does the report indicate the reliability of the audit evidence used to support audit findings and, in particular, whether evidence such as entity survey responses was corroborated by other audit evidence?
                          • Does the overall conclusion comprise a succinct, balanced discussion of those matters that directly support the conclusion against the audit objective, with significant findings covered in the key findings section?
                          • Do the recommendations focus on significant issues? Are they practical? Are they too long? Are there too many?
                          • Is the report internally consistent? That is, is there consistency between the body of the report, the overall conclusion and the key audit findings?
                          Report summary

                          225.32 The general purpose, features and structure of the report summary of the section 19 report are outlined in the following paragraphs.

                          225.33 The report summary is an important part of the audit report. It should capture the essence of the audit, the key issues identified and the value added by the audit. The report summary aims to:

                          • attract the report user’s attention and interest; and
                          • provide a concise outline of the report and the key messages the report wishes to convey to the report user.
                          • The report summary is to be four to six pages in length.
                          Background

                          225.34 The background is a short section that provides key introductory information about the audit.

                          225.35 The background section should cover:

                          • context, with a brief overview of the program or activity being audited;
                          • the service or program arrangements the audit addresses and their importance in terms of government policy, service delivery or governance;
                          • a summary of key outcomes, outputs and key performance indicators (where available);
                          • the dimensions of, or other insights into, the audit subject matter that are important to convey the significance of the audit topic;
                          • the rationale for undertaking the audit;
                          • the audit objective; and
                          • the high-level audit criteria, including the source of those criteria.

                          225.36 The drafting of the section 19 report provides an opportunity to revise the audit objective, criteria and scope, to ensure they reflect the audit coverage and audit findings.

                          Conclusion

                          225.37 The overall conclusion is likely to be of most interest to the majority of users of ANAO reports.

                          Supporting findings

                          225.38 The report summary includes ‘Supporting findings’, immediately following the conclusion. This section includes the various findings that are listed throughout the report in the grey boxes attached to each chapter. These various findings should be copied directly into this section.

                          Recommendations

                          225.39 It is important that audit recommendations:

                          • address the cause of the issue or matter that requires improvement, for example, if the audit found that data quality was poor due to the absence of any quality control/assurance arrangements, the recommendation should be directed at the need to implement these arrangements, rather than simply recommending that data quality be improved;
                          • focus on significant issues. It is not necessary that every issue addressed in a report results in a recommendation;
                          • do not contribute to the ‘red tape’ burden and do not simply recommend an entity comply with an existing requirement;
                          • should stand alone, that is, they are able to be understood without reference to supporting material elsewhere in the audit report; and
                          • are realistic and achievable and have regard to the cost of implementation. In this context, it is important for the audit team to fully consider the cost and other implications (such as coordination and consultation requirements) of each recommendation and be satisfied that they are cost-beneficial in terms of improving performance and accountability.

                          225.40 Generally, recommendations should focus on what needs to be done, rather than how the entity should do it. When the means of implementing a recommendation is important, this should be addressed in the body of the report.

                          225.41 Even though an issue may not warrant a recommendation, a report should encourage the entity to address opportunities for improvement.

                          Key learnings

                          225.42 This section should not re-state the findings relating to the entity subject to the audit. It should include broad lessons for other government entities, and inform improved business practices across the public sector, which have come to light during this audit. It may relate to both areas for improvement or good practice identified in the audit. This information is also relevant to the quarterly audit insights publications and other avenues for disseminating observations from ANAO audit work.

                          Report length

                          225.43 The ANAO aims for reports of approximately 50 pages. The length may vary according to the substance of the audit and the audit conclusions reached.

                          Cross-entity reports

                          225.44 For cross-entity audits, a decision is usually made early in the audit (and advice given to the entities included in the audit) about whether the audit report will include findings, conclusions and recommendations for each entity, or will be written in generic terms.

                          225.45 Where the section 19 and final reports are to be written in generic terms, the audit team will draw out common audit findings, conclusions and recommendations and/or key findings from the report preparation or evidence papers that were developed for each entity included in the audit.

                          Procedural fairness

                          225.46 Procedural fairness refers to a set of rights that people derive under common law and have under the administrative law principles of natural justice.

                          225.47 Procedural fairness requires a decision-maker to hear a person before making a decision affecting the rights, interests and legitimate expectations of that person. In the case of performance audits, the decision is whether or not to include in the audit report certain information or opinions relating to a person or an entity.

                          225.48 The ANAO’s legal and procedural obligations to public sector entities, and their employees, will be satisfied by the normal consultation process followed during the conduct of a performance audit, including through the provision of reports under section 19 of the A-G Act. In the rare circumstances where the normal consultative process does not satisfy the ANAO’s procedural obligations, then it may be allowable for the relevant individual to be given the opportunity to comment on relevant extracts of the report.

                          225.49 Where third parties, such as contractors used by the entity subject to audit, could be identified in a report, they would normally be provided with an extract of the report that makes reference to them or allows them to be identified.

                          225.50 The decision on which party (or parties) should be provided with copies or extracts of a section 19 report rests with the Auditor-General. At the time the section 19 report is submitted to the ANAO Executive for approval, details of the steps proposed to meet procedural fairness obligations should be outlined, including any proposed extracts strategy.

                          225.51 Specific legal advice may need to be obtained on how best to meet the ANAO’s procedural fairness obligations in particular circumstances, such as, where a number of individuals or bodies are able to be identified and it could reasonably be argued that the report could impact on their reputation.

                          Processing and issuing the section 19 report

                          225.52 The steps involved in processing the section 19 report for distribution are outlined in the PASG Workflow.

                          225.53 Where, following the section 19 meeting, the ANAO Executive makes comment on, or requires changes to, the section 19 report, the Executive will clarify whether it needs to view and clear a revised version of the report or if the report can be sent to the entity on the basis that the agreed changes will be made by the Engagement Executive, or GED as necessary.

                          Persons or bodies provided with the section 19 report or extract

                          225.54 The section 19 report, or extract from the report, may be given to any person who, or entity that, the Auditor-General considers has a special interest in the report or the content of the extract.

                          225.55 Whether a person has a special interest and should therefore receive a full copy or extract of the section 19 report will need to be determined on a case-by- case basis. For example, where an audit of procurement has been undertaken, a full copy of the section 19 report might be provided to the Department of Finance because it has policy responsibility for procurement matters.

                          225.56 A person or organisation with a special interest in the report or content of an extract may also have cause to make a claim under subsection 37(1) of the A-G Act, for example under paragraph 37(2)(e) which relates to the inclusion in a public report of particular information that would unfairly prejudice the commercial interests of a person or organisation. Policies and guidance with respect to the operation of section 37 of the A-G Act is provided in Chapter 229.

                          225.57 The process for preparing extracts to be provided to a third party is a deliberate process of reviewing the report in its entirety and considering each part as to what to include / exclude in the extract. Audit teams should consult with GEDs and / or EDs with experience in this process.

                          225.58 Generally, the following arrangements for provision of the section 19 report should apply. The section 19 report of an audit of a:

                          • Commonwealth entity performed under paragraph 17(1)(a) of the A-G Act should be provided to a single official who is, or, in the case of a corporate entity is a member of, the accountable authority of the entity;
                          • Commonwealth company performed under paragraph 17(1)(b) should be provided to a director (usually the Chair) of the company; and
                          • corporate Commonwealth entity or a Commonwealth company, or a Commonwealth partner respectively, performed under paragraph 17(1)(c) and under section 18B should be provided to a person who is, or is a member of, the governing body of the subsidiary or the Commonwealth partner.

                          225.59 For Commonwealth entities, section 19 reports should only be sent to the accountable authority16, or in the case of Commonwealth companies, subsidiaries and Commonwealth partners (which do not have an accountable authority of their own), a director or member of the governing body of the entity. This will reduce the risk of accidental disclosure and give the accountable authority, director or member of the governing body, control over who receives the section 19 report. Other officers (including internal audit and ANAO liaison) may be sent an email informing them that the section 19 report has been provided. Note that you may need to consult with the Auditor-General about provision of relevant papers to entities with unusual governance arrangements, for example, due to the diarchy arrangement the Auditor-General has consented to the provision of relevant papers to both the Secretary of the Department of Defence and the Chief of the Defence Force.

                          Comments sought on the section 19 report or extract

                          225.60 It is important to note that an extract from the section 19 report attracts the same protection regarding confidentiality and the associated penalties as the full section 19 report (subsection 36(3) of the A-G Act).

                          225.61 Senior members of the audit team should take an active role in communicating with the entity throughout the period that the entity has to provide comments on the section 19 report, to assist in ensuring a timely response.

                          225.62 Discussing the audit with the entity in both formal and informal ways throughout the 28-day period can assist in identifying and resolving any issues that have the potential to delay the finalisation of the audit report. Every effort should be made to reach agreement, especially in relation to recommendations, without compromising the Auditor-General’s independence. Such efforts often include the involvement of senior management from the entity and the ANAO in further discussions during the 28-day response period. The responsible GED and ANAO Executive should be kept abreast of key developments.

                          225.63 Entities’ final comments on findings and recommendations should include an ‘agree’ or ‘disagree’ position on each recommendation.

                          Consideration of comments on the section 19 report or extract

                          225.64 Where substantive comments are received, it would generally be appropriate to have further discussions with senior management of the entity and to give consideration to making amendments to the final report. As a general rule, the quality of the final report will be enhanced if disagreements can be resolved and the final report amended as necessary. The responsible GED and ANAO Executive should be consulted on such proposed amendments.

                          225.65 Incorporating entity comments in the final report is discussed at paragraphs 229.24 to 229.25.

                          Retaining evidence of review of section 19 report

                          225.66 The iterative review of the section 19 report can be extensive for some audits. Audit teams are to make judgements regarding the number of reviewed versions of the section 19 report that are required to be retained on the audit working papers. For example, drafts that contain minor editorial changes and comments only may not need to be retained.

                          Liaison with AASG

                          225.67 In determining what is a significant issue requiring communication with AASG, PASG shall consider:

                          • the potential to give rise to a material mistake in the financial statements;
                          • any actual or suspected fraud;
                          • any issues revealing a weakness in key financial controls; and
                          • any other issues of which the audit team is aware that may be of interest to AASG.

                          226. Key learnings

                          Policy

                          226.1 At the conclusion of an audit, the audit team shall document any key learnings and/or instances of good practice that were identified during the audit.

                          226.2 Key learnings, and instances of good practice, from other government entities shall, unless the contrary is approved by the relevant GED, be incorporated into the summary section of the section 19 and final reports.

                          Guidance

                          226.3 The audit team will identify and document the key learnings from performance audits and other engagement reviews. These include lessons from other government entities that may inform improved business practices across the public sector.

                          226.4 PASG practice management has circulated key headings to assist in grouping findings under relevant audit activity types. These key learnings will be used to inform the ANAO Insights products.

                          226.5 Also refer above to paras 227.19 and 227.40.

                          227. Final report

                          Policy

                          227.1 The final audit report shall convey a sufficient understanding of the entity and issues related to the audit objective, and be based on sound evidence and well-targeted analysis.

                          227.2 The audit team shall consider all material events relevant to the audit up to the date of issue of the final report. Following fieldwork, the effect of any subsequent events on the audit findings shall be considered up to the date of the final report.

                          227.3 Audit reports shall not include particular information that is sensitive, if its disclosure would be contrary to the public interest in the opinion of the Auditor-General (section 37 of the A-G Act).

                          227.4 If, during the course of an audit, an entity indicates, either verbally or in writing, that they believe that including specific content in an audit report would not be in the public interest pursuant to section 37 of the A-G Act, the Engagement Executive shall inform the Auditor-General immediately. The Auditor-General may seek further advice as to whether the relevant content may be information of the type listed in subsection 37(2) of the A-G Act. The purpose of such advice would be to assist the Auditor-General to form an opinion under paragraph 37(1)(a) of the A-G Act.

                          227.5 If the Auditor-General decides that information that is material to the conclusion is to be excluded from a section 19 or final report because it is considered by the Auditor-General to be particular information for the purpose of section 37 of the A-G Act, the Engagement Executive shall document, in writing through the GED, the content that is proposed to be excluded and the Auditor-General’s reasons for its exclusion.

                          227.6 If the entity’s response includes comments on a third party, the third party shall be provided the opportunity to confirm the facts and to reply to any adverse comments, unless otherwise approved by the Group Executive Director.

                          227.7 Any changes of substance made to the section 19 report in the final report shall be discussed with the entity.

                          227.8 As per 227.18, to meet the requirements of the A-G Act, any comments received on the section 19 report shall be given full consideration, in preparing the final audit report. This consideration and the response to the comments shall be documented on the audit file.

                          227.9 Where the ANAO does not agree with entity comments (and consequently has not changed the report), the reasons for not agreeing with the comments shall be documented and reported to the ANAO Executive as evidence that the entity’s views were fully considered.

                          227.10 Formal comments on the section 19 report, or extract of the report, shall be included in full as an appendix to the final report in accordance with subsection 19(8) of the A-G Act.

                          227.11 Where the Auditor-General or Deputy Auditor-General request amendments to the draft final report, the amended report shall be provided to the Auditor-General, Deputy Auditor-General and the responsible Group Executive Director for final comments and clearance.

                          227.12 All final audit reports shall be formally signed-off by the Engagement Executive and Group Executive Director, and approved for tabling in Parliament by the Auditor-General.

                          227.13 Any rejoinders made by the ANAO in the final report—to the entity’s response—shall be approved by the Auditor-General

                          227.14 Evidence of review of the final report shall be retained as part of the audit working papers.

                          227.15 The final audit report shall state that the audit was carried out in compliance with the ANAO Auditing Standards, and shall also include the full expected cost of the audit to the ANAO.

                          227.16 The audit working papers shall, if applicable:

                          • include formal comments from the entity provided with the section 19 report or extracts of the report;
                          • clearly outline changes made to the report as a result of entity comments;
                          • include a record of the Auditor-General’s, Deputy Auditor-General’s and Group Executive Director’s review and/or request to amend the report.

                          Guidance

                          227.17 The final audit report, setting out the ANAO’s audit findings, overall conclusion and recommendations, is tabled in the Parliament and is available to the public.

                          227.18 The PASG Workflow provides instructions on preparing the final report.

                          Structure and content

                          227.19 The final audit report should be objective, balanced and constructive in the presentation of the overall conclusion and the audit findings. Recommendations should be practical and cost-effective.

                          227.20 The structure and content of the final audit report should be substantially the same as the section 19 report.

                          Section 37 of the A-G Act

                          227.21 In providing any advice sought by the Auditor-General in respect of the disclosure of particular information under section 37 of the A-G Act, audit teams should take into consideration that, in administering the A-G Act, specifically in respect of section 37, the Auditor‐General’s approach is in favour of disclosure to the Parliament, in the public interest, unless the Auditor‐General is of the opinion that the public interest is served by not disclosing ‘particular information’ which is otherwise prohibited from public release (for example, information with a national security classification).

                          227.22 There is always a balance to be struck between the public interest in not disclosing information and the broader public interest in reporting transparently to the Parliament.

                          Incorporating entity comments in the final report

                          227.23 The final audit report includes the entity’s response to the section 19 report in the report summary as the final section titled Summary of entity response. It should be no more than half a page in length. If the entity’s response is longer, the audit team may discuss with the entity their willingness to prepare a shorter summary, or the full text should be included as an appendix.

                          227.24 Providing the entity with the opportunity for the formal comments to be amended, when a change has been made to the final report in response to an entity comment, avoids the situation where the entity’s published response relates to aspects of the report including the audit findings and conclusions that have been amended as a result of consideration of comments received on the section 19 report.

                          227.25 A rejoinder is the ANAO’s response to entity comments. Where the ANAO includes a rejoinder to the formal response of the entity in the final report, it should be described as an ANAO comment and placed below the entity’s comments. The ANAO comment will be a statement of fact addressing the specific entity comment and drawing attention, where appropriate, to the amended section of the report.

                          Approval and processing of the final report

                          227.26 See the PASG Workflow for detailed instructions on approving and publishing the final report.

                          228. Publication and closure

                          Policy

                          228.1 Prior to the tabling of the final report, an embargoed copy shall be made available to the following stakeholders two days prior to tabling:

                          • the Prime Minister and responsible Minister (for all audits);
                          • the Finance Minister, the Prime Minister’s Office and the Department of the Prime Minister and Cabinet (for cross-entity audits conducted under section 18);
                          • an official who is, or is a member of, the accountable authority of the entity (if the audit is of a Commonwealth entity);
                          • a director of the company (if the auditee is of a Commonwealth company);
                          • a person who is, or is a member of, the governing body of the subsidiary (if the auditee is of a subsidiary of a corporate Commonwealth entity or Commonwealth company);
                          • to the extent that the report relates to the operations of a Commonwealth partner—give a copy of the report to a person who is, or is a member of, the governing body of the Commonwealth partner; and
                          • to the extent that the report relates to the operations of a person who is not a Commonwealth partner—give a copy of the report to the person.

                          228.2 The audit report shall be tabled as soon as practicable after the audit is completed (as required by subsections 17(4) and 18(2) of the A-G Act).

                          228.3 The final audit report shall be transmitted to the Presiding Officers to enable tabling in the Parliament as soon as practicable after the completion of each audit.

                          228.4 A Continuous Improvement Summary Report and Joint Committee of Public Accounts and Audit Briefing as part of the final stage of the audit.

                          228.5 For the purposes of the Final Report, the relevant stakeholders include:

                          • the Prime Minister;
                          • the responsible Minister;
                          • the relevant Parliamentary Secretary, where they have direct responsibility for the matters subject to audit;
                          • the Secretary of the Department of the Prime Minister and Cabinet; and
                          • the responsible accountable authority of the entity subject to audit.

                          Guidance

                          228.6 The PASG Workflow provides procedures on the publication and closure processes. This includes several procedures to forward the Final Report to publications, and close the audit e-hive files, complete a Continuous Improvement Report and a JCPAA Briefing. While the ANAO Communication Unit plays an important role in producing the final report, organising its presentation to Parliament, and published after tabling on the ANAO website, the audit team remains responsible for ensuring that the words and layout are correct and that the report is complete and accurate.

                          Final draft to stakeholders (‘A4s’)

                          228.7 Third parties that received an extract of the section 19 report do not, as a matter of course, receive a copy of the final draft of the report. If a third party requests an extract of the final report, the decision should be referred to the Group Executive Director for consideration.

                          Tabling of the audit report

                          228.8 If the final report is to be presented when the Parliament is not sitting, it will be presented under Senate standing order 166 which provides for the presentation of documents when the Senate is not sitting. Upon receipt by the President of the Senate, the document is deemed to have been presented to the Senate, is authorised for publication, parliamentary privilege is attached and the embargo is lifted. The audit report will then be tabled at the next sittings of the House of Representatives and the Senate.

                          228.9 In addition to those persons and/or bodies who must receive a copy of the final report, a copy of the final report, or an extract, may also be given to a person or body that has a special interest in the report or the content of the extract.

                          Continuous improvement

                          228.10 The ANAO’s investment in the performance audit program is significant and it is incumbent on all performance audit teams to contribute insights and observations arising from performance audits that assist PASG to periodically review and enhance its policies and practices as part of the Service Group’s objective of achieving continuous improvement.

                          228.11 The audit team submits a Continuous Improvement Report to the PASG Business Unit within one month of the tabling of the audit report. The Engagement Executive approves the report and emails to PASGpracticemanagement@anao.gov.au.

                          228.12 The formal capture of learnings from individual audits, in the Continuous Improvement Summary Report (see the PASG Workflow), provides a useful base from which to assess the effectiveness and efficiency of existing audit management practices, and the processes followed in conducting each phase of a performance audit.

                          Parliamentary review

                          228.13 All performance audits are required to be examined by the Joint Committee of Public Accounts and Audit in accordance with sub-section 8(1)(c) of the Public Accounts and Audit Committee Act 1951. The committee conducts public inquiries into a selection of audit reports each year, following consultation with the ANAO. The ANAO provides a private briefing to the committee and gives evidence at these inquiries.

                          228.14 The PASG Workflow states that the audit team prepares a JCPAA Briefing and saves a PDF version to the JCPAA Summary folder on the ANAO shared library.

                          228.15 From time to time, other parliamentary committees also conduct inquiries into performance audit reports or aspects of public administration that have been the subject of performance audit coverage.

                          Footnotes

                          1 The ANAO for the purposes of the PGPA Act is a listed entity. An official of a Commonwealth entity that is a listed entity is a person who is prescribed by an Act or the rules to be an official of the entity. Under s38 of the A-G Act, the persons listed as officials of the Audit Office include persons engaged under contract as referred to in section 27.

                          2APES 110 addresses the provision of non-assurance services to assurance clients at paragraphs 291.140 to 291.157.

                          3 Network firm is defined in APES110.

                          4 Non-Audit Services Approval Form is located on the PSRG Audit Support Page – Current Audit Cycle Template, should be used to document the requirements of paragraph 202.16.

                          5ASA 220, paragraph 15.

                          6 Refer to ANAO Audit Manual – Shared Content 10.31 for more information about the ANAO’s responsibilities for documenting and communicating about potential fraud or other wrongdoing.

                          7 INTOSAI GUIDEGUID 3920 The Performance Auditing Process paragraph 75.

                          8 INTOSAI GUIDEGUID 3920 The Performance Auditing Process paragraph 76.

                          9 Cabinet documents are defined by the Cabinet Handbook and the Freedom of Information Act 1982. The Cabinet Handbook 13th edition defines Cabinet documents at paragraph 132. Paragraph 133 notes that Cabinet documents are to be marked ‘PROTECTED Cabinet’.

                          10 This may be achieved through the documented briefing papers or minutes of the section 19 workshop.

                          11 Bulk email collection involves the gathering of email records on a non-specific basis where, for any reason, ANAO is not able to identify and obtain from the auditee the specific email correspondence required to provide the evidence sought.

                          12 In accordance with Chapter 2, while the Australian Privacy Principles under the Privacy Act 1988 do not apply to the ANAO, it is ANAO’s position that the spirit and intent of Act will be followed to the extent it is appropriate.

                          14 Section 37 of the A-G Act

                          15 Under the Acts Interpretation Act 1901 (Cth), the 28-day period is based upon calendar (and not working) days but does not include the day the recipient receives the section 19 report, meaning that where a recipient receives the section 19 report on a Monday, the last day of 28 day period will be a Tuesday. Where the last day of the 28-day period is a Saturday, Sunday or public holiday, the period is extended so that the last day is the first day following the end of the period that is not a weekend or public holiday.

                          16 For corporate Commonwealth entities where the accountable authority is normally a board of directors or similar, it is normally sent to the chairperson or equivalent of that board.