The objective of the audit was to examine the implementation of the annual performance statements requirements under the Public Governance, Performance and Accountability Act 2013 and the enhanced Commonwealth performance framework.

Summary

Background

1. Performance reporting frameworks have been in place in the Australian public sector for several decades, to enable the measurement and assessment of the impact of government programs. The current performance measurement and reporting requirements for Commonwealth entities are established under the Public Governance, Performance and Accountability Act 2013 (PGPA Act) and the accompanying Public Governance, Performance and Accountability Rule 2014 (PGPA Rule). These are supported by the enhanced Commonwealth performance framework, which took effect on 1 July 2015.

2. The framework aims to provide financial and non-financial information to the Parliament and the public, and improve the line of sight between the use of public resources and the results achieved by government entities. It differs from previous frameworks in that it broadens the performance information collected and reported by entities, and introduced the requirement for entities to publish corporate plans at the beginning of each reporting period. The framework also introduced the requirement for entities to publish annual performance statements in their annual report. Annual performance statements provide an assessment of entities’ progress in achieving their purpose/s, as set out in their corporate plans and aligned to the Portfolio Budget Statements.

3. Portfolio Budget Statements, which are updated throughout the year1, are required to describe at a strategic level, the outcomes intended to be achieved with the funding appropriated by the Parliament. The performance criteria presented in the Portfolio Budget Statements are required to be a strategically focused subset of the performance information reported in an entity’s corporate plan.2 This positions corporate plans as the primary document for setting out an entity’s planned non-financial performance and provides the reader with an understanding of how this will be measured and assessed.

4. The Department of Finance (Finance) is responsible for the whole-of-government administration of the enhanced Commonwealth performance framework, and related legislation. Accountable authorities3 are responsible for the implementation of the framework within their entities. To assist entities in implementing the enhanced Commonwealth performance framework, Finance has provided various forms of guidance and support.

5. The ANAO plays a role in advising the Parliament, and the Joint Committee of Public Accounts and Audit (JCPAA), on the implementation of the reforms to the enhanced Commonwealth performance framework. One aspect of this role includes undertaking audits to provide feedback on, and influence the development, implementation and operation of, an effective performance framework.

6. The Auditor-General’s responsibilities, as set out in the Auditor-General Act 1997, include auditing the annual performance statements of Commonwealth entities in accordance with the PGPA Act. The PGPA Act does not require the Auditor-General to conduct annual audits of performance statements unless requested by either the Minister for Finance or the responsible minister. This means that the Parliament does not receive assurance, as a matter of course, on performance statements included in annual reports, as it does over financial statements, where an independent audit is mandatory.

7. The ongoing implementation of the Commonwealth’s resource management framework, including the enhanced performance framework, will continue to be a focus in future ANAO audit work programs.

Audit objective and criteria

8. The objective of the audit was to examine the implementation of the annual performance statements requirements under the PGPA Act and the enhanced Commonwealth performance framework.

9. To form a conclusion against the audit objective, the following high level criteria were adopted:

  • the selected entities met their obligations to publish annual performance statements;
  • the performance criteria were appropriate and were reported against in the selected entities’ performance statements for 2015–16;
  • the selected entities had effective supporting frameworks and processes to gather, assess, assure and report information included in their annual performance statements; and
  • sufficient records were retained to support the results reported by the entities against their non-financial performance measurement frameworks.

10. The ANAO reviewed one purpose in the 2015–16 performance statements of the Australian Federal Police and the Department of Agriculture and Water Resources (the selected entities), including the supporting systems and processes.

11. This performance audit is one of three audits in the ANAO’s current work program that address key aspects of the implementation of the PGPA Act. These audits have been identified by the JCPAA as priorities of the Parliament. This will assist in keeping the Parliament, government and the community informed on implementation of the resource, risk and performance management frameworks introduced by the PGPA Act.

Conclusion

12. The Australian Federal Police (AFP) and the Department of Agriculture and Water Resources (Agriculture) met the minimum requirements for the preparation and publication of the first annual performance statements under the PGPA Act and the PGPA Rule. For both entities, the performance statements included reporting against the entities’ purposes, activities, and performance criteria reviewed as part of the audit.

13. The performance criteria were mostly relevant to the activities undertaken by the selected entities. Alignment of entity activities to performance criteria and measurement of the attribution of specific activities to the achievement of the entities’ purposes could be enhanced.

14. Both entities’ performance criteria mostly provided a reliable method of assessing the entities’ progress in fulfilling their purposes. Addressing any potential bias in the reported results should also be considered. In addition, describing the methodology for measurement and basis for assessment, including through a target or baseline, needs to be addressed to improve the reliability of the entities’ performance criteria.

15. As a whole, the performance criteria for both entities were substantially complete, collectively providing a balanced basis for assessing the entities’ progress in fulfilling their purposes. The selection of performance criteria will require ongoing effort by both entities to identify opportunities to clarify or increase the overall alignment of performance criteria to the purpose. The entities’ balance of performance criteria—for example qualitative, quantitative, efficiency-focused and short, medium and long term timeframes—should be reviewed.

16. Both entities established or adapted existing systems and processes to meet the requirements of the PGPA Act and the PGPA Rule. These remained in development during the audit, with further work being undertaken in 2016–17, to support the quality of information reported in future performance statements.

17. The selected entities established assurance processes to certify that the reported performance information accurately reflected entity performance. Planning and assurance processes for the entities should mature over time. As part of this process entities should give further consideration to the role and function of their respective audit committees, to ensure that the intent and requirements of the framework are met, as neither audit committee could fully demonstrate compliance with the PGPA Rule.

18. The majority of results presented in the selected entities’ annual performance statements were supported by complete and accurate records as required by the PGPA Act and PGPA Rule. Both entities could improve record-keeping to better demonstrate the calculations and analysis applied to raw data to produce results, and to support the analysis in the annual performance statements.

Supporting findings

Measurement and reporting of performance

19. Both of the 2015–16 performance statements reviewed as part of the audit complied with the PGPA Act and PGPA Rule, in relation to their publication. The performance statements were published as part of the entities’ 2015–16 Annual Reports. They included the required statements, results and analysis against the performance criteria outlined in the corporate plan and Portfolio Budget Statements reviewed as part of the audit.

20. The analysis section of both entities’ annual performance statements included some consideration of the entities’ operating environment and were supported in some cases by case studies and trend information. However the quality of the analysis could be improved, in particular, by providing further discussion of how the entities’ activities, through the results of the performance criteria, had contributed towards the achievement of their purpose/s and the external factors which impacted performance.

21. Both entities built on their existing external performance measurement and reporting framework, to meet the requirements of the enhanced Commonwealth performance framework under the PGPA Act. The information published by the entities in their corporate plan and Portfolio Budget Statements provides a foundation for reporting in the annual performance statements, although there was scope for both entities to improve how material was presented to achieve a clearer ‘line of sight’.

22. The performance criteria reviewed as part of the audit enabled the reporting of, and accountability for, the progress of each entity towards fulfilling their purposes.

23. The performance criteria for both entities were mostly relevant, providing a basis to make decisions on the entities’ progress in fulfilling their purposes. One of the AFP’s performance criteria required improvements to assist the reader to identify the benefit or beneficiary measured by the performance criterion and its link to the AFP’s activities. Agriculture’s performance criteria were mostly relevant, however the benefit or beneficiary was often not clear, or the focus of the measure was not clearly attributable to the entity.

24. The performance criteria for both entities were mostly reliable, providing a basis for reasonably consistent assessment of the entities’ progress in fulfilling their purposes. Improvements to two of the AFP’s performance criteria are required to limit the level of potential bias in the reported results. The majority of Agriculture’s performance criteria did not describe the method or basis for measurement, or provide a target or baseline, impacting the reliability of the performance criteria.

25. As a whole, the performance criteria for both entities were substantially complete, collectively providing a balanced basis for assessing the entities’ progress in fulfilling their purposes. To improve the completeness of the performance criteria, the selection of performance criteria will require ongoing effort by both entities to identify opportunities to clarify or increase the overall alignment of performance criteria to the purpose, and present a greater balance of performance criteria across the different forms of performance information and their timeframes.

Systems and processes to support performance measurement and reporting

26. Each entity adapted pre-existing processes for the preparation of the Annual Report, to facilitate the coordination and collation of information for the annual performance statements. In addition, both entities developed a project plan outlining the roles and responsibilities, risks and mitigating controls, milestones for delivery, and assurance mechanisms to guide the preparation of the performance statements. Neither entity completed a comprehensive pre-assessment of the processes of producing performance statements as part of their planning. The incremental development of performance reporting by both entities was focused on identifying lessons learnt.

27. Both entities had established, or leveraged from existing systems and methodologies to collect and report performance information for the purposes of the annual performance statements. The AFP would benefit from considering the frequency and extent of reviews of the methodology supporting one performance criterion, and strengthening processes for the validation of information sourced outside of management systems. Further consideration of data availability is required by Agriculture to establish a system to support the consistent collection, analysis and reporting of non-financial performance information.

28. Processes were established by both entities to provide assurance that the results reported in the annual performance statements were an accurate representation of performance. Further refinement of these arrangements is required by the entities. This would include documenting: guidance on the assurance process; and the review and endorsement of the annual performance statements to ensure that evidence of management and audit committee assurance is recorded and retained.

29. Each entity relied on management certifications over the selected performance criteria and the completeness and accuracy of underlying records. The entities’ audit committees also received regular briefings on the preparation of the annual performance statements, including details on the management certification processes. Additionally, Agriculture’s audit committee commissioned an internal audit on the Key Performance Indicators (performance criteria) and Performance Reporting, to inform the committee’s review responsibilities. The audit committee also monitored the implementation of key recommendations. However, the audit committee’s sign off to Agriculture’s accountable authority was limited, and did not meet the requirements of the department’s audit committee charter or the PGPA Rule and its intent. The AFP was unable to locate the final certification by its audit committee to the accountable authority over the performance statements, limiting an assessment against the audit committee charter, or the PGPA Rule and its intent.

30. There would be benefit in both entities further considering the role of the audit committee as a source of independent assurance to the accountable authority and how their audit committee charters and processes establish a basis to provide this assurance. The PGPA Rule provides that an audit committee’s functions must include reviewing the appropriateness of an accountable authority’s performance reporting. This function would necessarily involve the committee forming a view on how the entity should measure its performance. As a result, an audit committee’s charter, and any certification by the audit committee to an accountable authority discharging their performance reporting function, should reflect this requirement.

31. Records were largely available and supported the results and analysis sections reported in the annual performance statements for both entities. The AFP maintained complete and accurate records for all but one measure. Agriculture was unable to provide complete records for one performance criterion, and relied on an absence of advice as confirmation of compliance for another. Both entities could further improve record-keeping to demonstrate the calculations and analysis applied to raw data to produce results, and to support analysis in the annual performance statements.

Opportunities for improvement and key learnings

32. The ANAO recognises that this is the first year of published performance statements under the PGPA Act and has taken this into account in the conduct of this audit. It is expected that entity processes will take some time to mature. On this basis the ANAO has not made any recommendations in this audit, but has highlighted a range of matters which warrant further attention by the AFP and Agriculture.

33. In addition, guidance from Finance is being incrementally updated as lessons are learnt, through its role as policy owner in the Commonwealth. The need for further clarity in guidance, in particular for audit committees, has been acknowledged by the Department of Finance in recent discussions with audit committee chairs.4 Finance has previously advised the JCPAA that section 112 of the PGPA Act provides for an independent review of the framework in 2017.5 This review of the framework would include the PGPA Act and the rules, and presents an opportunity for Finance to further consider the accompanying guidance.

34. Below is a summary of key learnings identified in this audit report that may be considered by other Commonwealth entities in preparing their annual performance statements.

Box 1: Key learnings for all entities

Presentation of results and analysis

  • Presenting results alongside established targets, and providing comparisons to results from previous years or references to related indicators and results, can assist the reader in assessing performance.
  • The analysis section of the performance statements provides entities the opportunity to supplement the reported results with contextual information. This can enhance the reader’s understanding of the environment within which the entity operates and the contributions that it makes.

Purposes and activities

  • Clearly identifying and grouping activities can assist the reader to assess the alignment of the performance criteria with the entity’s purpose. This provides the basis for a clear read between the corporate plan and the performance statements.

Relevance, reliability and completeness of performance criteria

  • Relevant performance criteria should clearly align to an entity’s purpose and activities, indicate who will benefit from the related activity and how, and be understandable to readers.
  • Reliable performance criteria should be measurable, disclose the method or basis for assessment such as a target or benchmark, and not lead to biased results.
  • Complete performance criteria should collectively address the entity’s purpose, and provide a balanced examination of the entity’s effectiveness and efficiency across the different forms of performance information and their timeframes.

Systems, processes and methodologies

  • The enhanced Commonwealth performance framework provides entities the opportunity to apply a fit for purpose approach to their performance measurement. This principle should be kept in mind by entities in designing or reviewing their own processes to inform the preparation of the annual performance statements.
  • Entity processes should support the complete cycle of performance measurement and reporting. This would reflect the preparation and publication of the Portfolio Budget Statements, corporate plans, performance statements and annual reports, evaluating lessons learnt and incorporating these into the following cycle to contribute to continuous improvement.
  • Where appropriate, entities should consider how existing IT systems and controls can be leveraged from to support performance reporting. This would include clearly documenting any calculations or analysis applied to data drawn from systems to enable recalculation.

Assurance processes

  • A check list is a useful tool to assist entity management representatives to understand the detailed considerations expected, including the level of evidence required, to support a certification of performance criteria and accompanying results.
  • An audit committee’s charter, and any certification by the audit committee to an accountable authority discharging their performance reporting function, should clearly reflect the PGPA Rule and its intent.

Record-keeping

  • To ensure appropriate access to supporting records for future years, entities may consider establishing a centralised repository.
  • Clearly document and retain records detailing the methods used for calculations and analysis to reach the result reported in the annual performance statements to enable recalculation.

Summary of entities’ responses

35. Summary responses from the selected entities are provided below, while the full responses are provided at Appendix 1.

Department of Finance

The Department of Finance supports the findings of the report.

Australian Federal Police

The AFP welcomes the ANAO’s findings and acknowledges the assessment provided on the performance statement and on processes and systems related to performance measurement and reporting. The consolidated list of key learnings in the report will be useful for continued improvement.

This audit has assisted the AFP to focus efforts in ongoing performance measurement reform across the entire cycle of planning, monitoring, analysis and reporting.

The AFP will continue active participation in the Department of Finance performance community of practice and also maintain a localised law enforcement performance group to promote best practice.

Department of Agriculture and Water Resources

The Department of Agriculture and Water Resources 2015–16 annual performance statements were the first prepared under the enhanced Commonwealth Performance Framework. The department’s work under the framework is an evolving process and, as acknowledged in this report, the department has already made a range of changes to its performance framework. This audit has been an opportunity to identify further improvements.

The department agrees with most of the ANAO’s findings indicating areas in which its performance measurement and reporting can be improved. These findings will inform its continued efforts.

The department does not accept the finding that its Audit Committee did not meet the requirements of its charter, or the requirements and intent of the PGPA Rule, in providing assurance of the certification process.1 The committee undertook a range of work to meet its charter, and the department considers the assurance provided was consistent with advice from the Department of Finance on the role of audit committees in the certification process.2 The department is committed to establishing processes to ensure the Audit Committee meets the requirements of the PGPA Rule.

ANAO comments on the Department of Agriculture and Water Resources’ summary response

1. The Audit Committee’s Charter required the committee to review and provide independent advice and assurance about the appropriateness of the department’s performance reporting (see paragraph 3.49). The ANAO concluded that the advice provided to the Accountable Authority by the Audit Committee did not provide assurance about the appropriateness of the department’s performance reporting (see paragraph 3.51).

2. See paragraph 33. The Department of Finance has acknowledged the need for further clarity in guidance to audit committees.

1. Background

Introduction

1.1 Since the mid-1980s, public sector management frameworks have emphasised the importance of measuring program performance. While the focus of the frameworks has changed over the years, the fundamental goals have remained largely consistent—to be able to measure and assess the impact of government programs and not just measure the activities, deliverables and associated costs. Some frameworks have also included a focus on program efficiency.

1.2 The current performance measurement and reporting requirements for Commonwealth entities (corporate and non-corporate) are established under the Public Governance, Performance and Accountability Act 2013 (PGPA Act) and the accompanying Public Governance, Performance and Accountability Rule 2014 (PGPA Rule).

1.3 The Auditor-General’s responsibilities, as set out in the Auditor-General Act 1997, include auditing annual performance statements of Commonwealth entities in accordance with the PGPA Act. The PGPA Act does not require the Auditor-General to conduct annual audits of performance statements unless requested by either the Minister for Finance or the responsible minister. This means that the Parliament does not receive assurance, as a matter of course, on performance statements included in annual reports as it does over financial statements where an independent audit report is mandatory.

1.4 The ongoing implementation of the Commonwealth’s resource management framework, including the enhanced performance framework, will continue to be a focus in future ANAO audit work programs.

Enhanced Commonwealth performance framework

1.5 The enhanced Commonwealth performance framework (the framework) took effect on 1 July 2015, to provide both financial and non-financial information to the Parliament and the public, under the PGPA Act.6 The aim is to improve the line of sight between the use of public resources and the results achieved by entities. A key change from the previous framework was to broaden the performance information collected and published by entities, in demonstrating progress against their purposes and outcomes.

1.6 The framework introduced the requirement for entities to publish corporate plans at the beginning of each reporting cycle from 2015–16. Plans must set out the entities’ strategies for achieving their purpose/s and determining how progress will be measured. In addition, the framework introduced the requirement for entities to prepare annual performance statements (performance statements) at the end of the reporting period. These are to be included in entities’ annual reports. Performance statements provide an assessment of the extent to which the entity has progressed in achieving its purpose/s, as set out in the corporate plan and aligned to the Portfolio Budget Statements (PBS).

1.7 PBSs, which are updated throughout the year7, are to describe at a strategic level, the outcomes intended to be achieved with the funding appropriated by the Parliament. The performance criteria presented in the PBSs are a strategically focused subset of the performance information included in an entity’s corporate plan. Entities are expected to clearly map the performance information from the PBSs to the entities’ purpose/s published in their corporate plans.8 This positions the corporate plan as the primary document for setting out planned non-financial performance and provides the reader with an understanding of how an entity intends to measure and assess its performance.9 Key elements of the enhanced Commonwealth performance framework, and the broader Commonwealth Resource Management Framework, are set out in Figure 1.1.

Figure 1.1: Elements of the Commonwealth Resource Management Framework

Source: ANAO analysis of guidance from the Department of Finance, incorporating relevant legislation.

Roles and responsibilities

Minister for Finance and Department of Finance

1.8 The Minister for Finance is the Minister responsible for administering the PGPA Act. The Finance Minister may, by legislative instrument, make rules prescribing matters required or permitted by the PGPA Act to be prescribed by the rules, including those relating to performance.10 On 25 April 2015, the Finance Minister amended the PGPA Rule to reflect the requirements for entities to publish corporate plans and annual performance statements from 2015–16.

1.9 As noted previously, under section 40 of the PGPA Act, the Minister for Finance or a responsible Minister also has powers to request the Auditor-General to examine and report on the annual performance statements of an entity.

1.10 The Minister for Finance is supported by the Department of Finance (Finance). Finance is responsible for the whole-of-government administration of the enhanced Commonwealth performance framework and related legislation. Finance has previously advised the Joint Committee of Public Accounts and Audit that section 112 of the PGPA Act provides for an independent review of the framework in 2017.11 This review of the framework would include the PGPA Act and the rules.

1.11 To assist entities in implementing the enhanced Commonwealth performance framework, Finance has:

  • published written guidance in the form of Resource Management Guides (RMGs)12;
  • reviewed draft corporate plans and performance statements on behalf of entities, when requested;
  • established the Performance Community of Practice (Performance CoP) to provide a forum to share expertise and examples of better practice and establish a feedback loop between framework design, implementation and results. Four Performance CoP performance statements workshops were conducted in 201713;
  • released regular Public Management Reform Agenda newsletters and emails to keep entities informed of emerging better practice and other developments;
  • published lessons learned papers in February 2016 and January 2017, based on Finance’s assessment of a selection of 2015–19 and 2016–20 corporate plans, including examples of better practice; and
  • released a draft lessons learned paper in March 2017 and published the final paper in May 2017, based on Finance’s assessment of a selection of 2015–16 performance statements, including examples of better practice.

1.12 Finance also completed a review of entities’ compliance with the PGPA Act and PGPA Rule in regard to the preparation and publication of annual performance statements for 2015–16. Finance’s review concluded that all Commonwealth entities required to produce annual performance statements for 2015–16 did so per section 39(1) of the PGPA Act. In addition, 82 per cent of entities were fully compliant with Section 16F of the PGPA Rule. The primary compliance matters identified through Finance’s review were minor issues in entities’ expression of the accountable authority’s introductory statement.

Joint Committee of Public Accounts and Audit

1.13 The Joint Committee of Public Accounts and Audit (JCPAA) resolved to inquire into and report on the development of the enhanced Commonwealth performance framework on 26 March 2015. Through the inquiry, the Committee sought to ensure that the rules, directions and guidance that underpin the framework provided clarity to entities regarding performance monitoring and reporting expectations, and also facilitated scrutiny.

1.14 The JCPAA released Report 453 Development of the Commonwealth Performance Framework in December 2015 and made the following observations relevant to this audit:

  • The importance of a ‘clear read’ of performance information, ensuring it is presented clearly and consistently across reports produced by an entity within an annual reporting period, and across several cycles. This would also include the comparability of performance information across entities.14
  • The need for strong and sustained leadership, from all levels, including all senior leadership teams within entities.15
  • Active planning, monitoring, reporting and evaluation initiatives could allow programs and policies to be tracked and adjusted in real-time to improve results.16
  • Central monitoring, reporting and evaluation of performance across an entity may allow specific and systemic issues to be identified and addressed.17

Portfolio Ministers

1.15 Under the PGPA Act, a Portfolio Minister has powers to:

  • access the records kept about the performance of the entities within their portfolio (section 37); and
  • request the Auditor-General to examine and report on the annual performance statements of entities within their portfolio (section 40).

1.16 A Portfolio Minister who makes a request of the Auditor-General, is required to table the report in each House of Parliament with the accompanying annual performance statements, as soon as practicable after the report is received.

Accountable Authorities

1.17 Accountable authorities are responsible for the implementation of the requirements of the enhanced Commonwealth Performance framework in their entities. Part 2–3 of the PGPA Act—relating to planning, performance and accountability under the Act—sets out the detailed requirements. Key requirements relevant to this audit include:

  • prepare a corporate plan each reporting period that complies, is published, and is provided to the responsible Minister and Finance Minister in accordance with any requirements prescribed by the PGPA Rules;
  • cause records to be kept about the entity’s performance in accordance with any requirements prescribed by the Rules;
  • measure and assess the entity’s performance and comply with any requirements prescribed by the Rules; and
  • prepare annual performance statements about the entity’s performance that comply with any requirements prescribed by the Rules and include it in the annual report.

Audit Committees

1.18 Audit committees are appointed by the accountable authority of an entity. The functions of an audit committee are prescribed by section 17 of the PGPA Rule, and must be set out by the accountable authority in a written charter. The required functions of an audit committee are detailed in Box 2.

Box 2: Functions of the audit committee

PGPA Rule subsection 17(2) outlines the functions of the audit committee:

The functions must include reviewing the appropriateness of the accountable authority’s:

  1. financial reporting; and
  2. performance reporting; and
  3. system of risk oversight and management; and
  4. system of internal control for the entity.

Source: PGPA Rule.

Requirements for annual performance statements

1.19 The first performance statements under the PGPA Act were required to be published in entities’ annual reports in late 2016.18 Specific requirements are outlined below:

Box 3: Matters to be included in a Commonwealth entity’s annual performance statements

Under the PGPA Rule section 16F, entities’ annual performance statements must:

  • measure and assess the entity’s performance in achieving the entity’s purposes in the reporting period in accordance with the method of measuring and assessing the entity’s performance in the reporting period that was set out in the entity’s corporate plan, and in any Portfolio Budget Statement, Portfolio Additional Estimates Statement or other portfolio estimates statement, that were prepared for the reporting period; and
  • include the following information in the annual performance statements:

Statements

  • a statement that the performance statements are prepared for paragraph 39(1)(a) of the Act;
  • a statement specifying the reporting period for which the performance statements are prepared;
  • a statement that, in the opinion of the accountable authority of the entity, the performance statements:
    • (i) accurately present the entity's performance in the reporting period; and
    • (ii) comply with subsection 39(2) of the Act.

Results

  • The results of the measurement and assessment referred to in subsection (1) of this section of the entity's performance in the reporting period in achieving its purposes.

Analysis

  • An analysis of the factors that may have contributed to the entity's performance in achieving its purposes in the reporting period, including any changes to:
    • (a) the entity's purposes, activities or organisational capability; or
    • (b) the environment in which the entity operated; that may have had a significant impact on the entity's performance in the reporting period.

Source: PGPA Rule.

1.20 Section 37 of the PGPA Act sets out the requirement for Commonwealth entities to keep records that properly record and explain the entity’s non-financial performance as outlined below.

Box 4: Records about performance of Commonwealth entities

Under PGPA Act section 37:

(1) The accountable authority of a Commonwealth entity must cause records to be kept that properly record and explain the entity’s performance in achieving its purposes.

(2) The accountable authority must ensure that the records are kept in a way that:

a) complies with any requirements prescribed by the rules; and

b) enables the preparation of the annual performance statements required by section 39.

(3) The responsible Minister and the Finance Minister are entitled to full and free access to the records kept under this section. However, those Ministers’ access is subject to any Commonwealth law that prohibits disclosure of particular information.

Source: PGPA Act.

1.21 As noted above, the accountable authority of a Commonwealth entity must also ensure that the entity has an audit committee. The committee must perform functions in accordance with the requirements prescribed by the Rules (Box 2).

Audit coverage

1.22 This performance audit is one of three audits in the ANAO’s current work program that address key aspects of the implementation of the PGPA Act. The other two audits are:

  • Corporate Planning in the Australian Public Sector. This performance audit is the second in a series of audits that is assessing progress in implementing the corporate planning requirement under the PGPA Act. The first in the series was ANAO Audit Report No.6 2016–17 Corporate Planning in the Australian Public Sector; and
  • The Management of Risk by Public Sector Entities. This audit is assessing how effectively selected public sector entities manage risk, including compliance with the Commonwealth Risk Management Policy.

1.23 These audits have been identified by the JCPAA as priorities of the Parliament19 and will assist in keeping the Parliament, government and the community informed on implementation of the resource, risk and performance management frameworks introduced by the PGPA Act.

Audit approach

1.24 The objective of the audit was to examine the implementation of the annual performance statements requirement under the PGPA Act and the enhanced Commonwealth performance framework. To form a conclusion against the audit objective, the following high level criteria were adopted:

  • the selected entities met their obligations to publish annual performance statements;
  • the performance criteria were appropriate and were reported against in the selected entities’ performance statements for 2015–16;
  • the selected entities had effective supporting frameworks and processes to gather, assess, assure and report information included in their annual performance statements; and
  • sufficient records were retained to support the results reported by the entities against their non-financial performance measurement frameworks.

1.25 The audit involved:

  • reviewing one purpose/s statement from each entity’s performance statements, and all performance criteria established to demonstrate progress against the following strategic objectives20:
    • Australian Federal Police – Federal Policing and National Security (see Table 1.1); and
    • Department of Agriculture and Water Resources – Building successful primary industries (see Table 1.2);
  • reviewing internal systems, processes and procedures, including the governance and oversight put in place by entities, to support their development of the annual performance statements; and
  • reviewing records and interviewing staff of the two selected entities.

1.26 The audit scope also included reviewing Finance’s role in whole-of-government administration of the annual performance statements requirements.

1.27 The ANAO recognises that this is the first year of published performance statements under the PGPA Act and has taken this into account. It is expected that entity processes will take some time to mature. Finance is incrementally updating guidance as lessons are learned. On this basis the audit seeks to share key learnings and point to areas for improvement.

1.28 The audit was conducted in accordance with the ANAO Auditing Standards at a cost to the ANAO of approximately $390,700.

1.29 The team members for this audit were Corinne Horton, Jennifer Hutchinson, Kara Ball, Alicia Vaughan and Michael White.

Table 1.1: Audit scope: performance framework for the Australian Federal Police21

Australian Federal Policea

Purpose

Our purpose is to enforce Commonwealth criminal law, contribute to combating organised crime and protecting Commonwealth interests from criminal activity in Australia and overseas. This purpose is derived from section 8 of the Australian Federal Police Act 1979 and informed by an associated Ministerial Direction articulated under section 37 (2) of the Act.

Strategic objective

Federal policing and national security—Promoting the safety and security of the Australian community and infrastructure; preventing, deterring, disrupting and investigating serious and organised crime and crimes of Commonwealth significance; and ensuring effective collaboration with international, Commonwealth, state and territory partners.b

Activities

  • Federal policing.
  • National security.
  • Specialist and support operations.

Performance criteria

  • Level of external client/stakeholder satisfaction (percentage of clients satisfied or very satisfied).
  • Percentage of cases before court that result in conviction.
  • Percentage of counter-terrorism investigations that result in a prosecution, disruption or intelligence referral outcome.
  • Level of community confidence in the contribution of the AFP to aviation law enforcement and security (percentage of aviation network users satisfied or very satisfied).
  • Response to aviation law enforcement and /or security incidents within priority response times.
  • Number of avoidable incidents per 5,000 Protection hours.
  • Return on investment for investigation of transnational crime.
  • Assets restrained.
  • Increased or reinforced cyber safety and security awareness (percentage of surveyed sample indicating increased awareness or reinforced awareness after delivery of presentations).

Note a: Supplementary notes from the original publications have been removed.

Note b: This reflects the AFP’s PBS Programme 1.1.

Source: Australian Federal Police 2015–19 Corporate Plan.

Table 1.2: Audit scope: performance framework for the Department of Agriculture and Water Resources22

Department of Agriculture and Water Resourcesa

Purpose

  • Provides advice to the Australian Government on how to help our primary industries remain competitive, productive and sustainable into the future.
  • Provides advice to the government on how best to achieve social, economic and environmental benefits from the use of water resources in the national interest.
  • Administers government programmes and legislation that support these objectives, including the collection of levies for research, development and marketing.
  • Regulates the import of food and other goods to ensure that Australia is safeguarded against exotic animal and plant pests and diseases.
  • Regulates the provision of export certification of agriculture, fish and forest products to meet importing country requirements.

Strategic objective

Building successful primary industries—Improve the farm-gate returns for agriculture, fisheries, food and fibre industries.b

Activities

  • Deliver Agricultural Competitiveness White Paper initiatives including measures that seek a smarter approach to farming based on a strong research and development system.
  • Provide analysis and advice to the government and external decision-makers on support for portfolio industries to make them more productive, globally competitive and profitable.
  • Support current government priorities including country-of origin labelling and the relocation of rural research and development corporations.
  • Provide levies collection and distribution services to industry and government.
  • Support industry bodies and research and development corporations to meet statutory and contractual requirements and objectives.
  • Administer, improve and implement reforms in industry regulations in accordance with the government’s agenda.

Performance criteria

  • Portfolio industries record an increase in productivity.
  • Rate of return on capital invested across portfolio industries is maintained or increased.
  • Rate of profit for producers and businesses is maintained or increased.
  • Access to water, land, forest and marine resources for primary production is maintained or improved.
  • Improved availability of safe, efficient and effective agricultural and veterinary chemicals.
  • Investment in rural research and development corporation programmes demonstrates positive returns.
  • 100 per cent of allocated funding under the Research and Development for Profit programme is expended in accordance with the agreed timetable.
  • 100 per cent of rural research and development corporations are compliant with statutory and contractual requirements.
  • High level of efficiency in collecting and distributing levies to fund research and development in research and development corporations.
  • Inspections of levy agent records cover at least 30 per cent of levy revenue.
  • Less than 5 per cent of quota allocations are rejected because of quota certification failures.

Note a: Supplementary notes from the original publications have been removed.

Note b: This reflects Agriculture’s PBS Programme 1.5: Horticulture Industry, Programme 1.6: Wool Industry, Programme 1.7: Grains Industry, Programme 1.8: Diary Industry, Programme 1.9: Meat and Livestock Industry and Programme 1.10: Agricultural Resources.

Source: Department of Agriculture and Water Resources 2015–19 Corporate Plan.

2. Measurement and reporting of performance

Areas examined

This chapter considers whether the selected entities met their obligations to publish annual performance statements in their 2015–16 Annual Report. It also examines a selection of performance criteria and the reporting against those criteria in the selected entities’ 2015–16 performance statements.

Conclusion

The Australian Federal Police (AFP) and the Department of Agriculture and Water Resources (Agriculture) met the minimum requirements for the preparation and publication of the first annual performance statements under the Public Governance, Performance and Accountability Act 2013 (PGPA Act) and the accompanying Public Governance, Performance and Accountability Rule 2014 (the Rule). For both entities, the performance statements included reporting against the entities’ purposes, activities, and performance criteria reviewed as part of the audit.

The performance criteria were largely relevant to the activities undertaken by the entities. Alignment of activities to criterion and measurement of the attribution of specific activities to the achievement of the entities’ purposes could be enhanced.

Both entities’ performance criteria generally provided a reliable method of assessing the entities’ progress in fulfilling their purposes. Addressing any potential bias in the reported results should be considered. In addition, describing the methodology for measurement and basis for assessment, including through a target or baseline, needs to be addressed to improve the reliability of the entities’ performance criteria.

As a whole, the performance criteria for both entities were complete, collectively providing a balanced basis for assessing the entities’ progress in fulfilling their purposes. The performance criteria selection will require ongoing effort by both entities to identify opportunities to clarify or increase the overall alignment of performance criteria to the purpose. The entities’ balance of performance criteria, for example qualitative, quantitative, and efficiency and short, medium and long term timeframes, should be reviewed.

Opportunities for improvement

The ANAO identified opportunities for improvement relating to:

  • providing meaningful analysis of external factors affecting performance;
  • the presentation and expression of the activities to assist a reader to identify and assess the alignment of the performance criteria to the entities’ purposes;
  • a greater balance of the number of effectiveness and efficiency criteria, including: describing, where necessary, why output measures are a reasonable basis for assessing effectiveness; and how performance criteria might better reflect a mixture of short, medium and longer-term strategic objectives; and
  • presenting performance criteria that clearly state who will benefit and how, and are accompanied by a target, timeframe or baseline for assessing the results.

Have entities met the requirements of the Public Governance, Performance and Accountability Act 2013 and accompanying Rule?

Both of the 2015–16 performance statements reviewed as part of the audit complied with the PGPA Act and the accompanying Rule, in relation to their publication. The performance statements were published as part of the entities’ 2015–16 Annual Reports. They included the required statements, results and analysis against the performance criteria outlined in the corporate plan and Portfolio Budget Statements reviewed as part of the audit.

The analysis section of both entities’ annual performance statements included some consideration of the entities’ operating environment and were supported in some cases by case studies and trend information. However the quality of the analysis could be improved, in particular, by providing further discussion of how the entities’ activities, through the results of the performance criteria, had contributed towards the achievement of their purposes and the external factors which impacted performance.

2.1 Table 2.1 outlines the ANAO’s summary assessment of the selected entities’ compliance with the PGPA Act and Rule for the presentation of performance statements. Both entities complied with section 39 of the PGPA Act, however improvements could be made to address the requirements of section 16F of the PGPA Rule more comprehensively.

Table 2.1: Compliance with PGPA Act and PGPA Rule requirements

Requirement

AFP Compliant?

Agriculture Compliant?

Section 39 of the PGPA Act

Subsection (1)

Prepare annual performance statements for the entity as soon as practicable after the end of each reporting period for the entity. Include a copy of the annual performance statements in the entity’s annual report that is tabled in the Parliament.

Yes

Yes

Subsection (2)

 

 

The annual performance statements must:

(a) provide information about the entity’s performance in achieving its purposes; and

(b) comply with any requirements prescribed by the rules.

Yes – refer to compliance with Section 16F of the PGPA Rule below.

Yes – refer to compliance with Section 16F of the PGPA Rule below.

Section 16F of the PGPA Rule

Subsection (1) – Measuring and assessing entity’s performance

The accountable authority of the entity must measure and assess the entity’s performance in achieving the entity’s purposes in the reporting period in accordance with the method of measuring and assessing the entity’s performance in the reporting period that was set out in the entity’s corporate plan, and in any Portfolio Budget Statement, Portfolio Additional Estimates Statement or other portfolio estimates statement, that were prepared for the reporting period.a

Yes

Partly

Subsection (2) – Matters that must be included in annual performance statements

Item 1: Statements

  1. A statement specifying the performance statements were prepared for subsection 39(1)(a) of the PGPA Act.

Yes

Yes

  1. A statement specifying the reporting period for which the performance statements are prepared.

Yes

Yes

  1. A statement that, in the opinion of the accountable authority of the entity, the performance statements:

    (i) accurately present the entity’s performance in the reporting period; and

    (ii) comply with subsection 39(2) of the Act.

Yes

Yes

Item 2: Results

The results of the measurement and assessment referred to in subsection 16F(1) of PGPA Rule of the entity’s performance in the reporting period in achieving its purposes.

Yes

Partly

Item 3: Analysis

An analysis of the factors that may have contributed to the entity’s performance in achieving its purposes in the reporting period, including any changes to:

(a) the entity’s purposes, activities or organisational capability; or

(b) the environment in which the entity operated;

that may have had a significant impact on the entity’s performance in the reporting period.

Partly

Partly

     

Note a: Compliance with subsection 16F(1) of the PGPA Rule is linked to compliance with subsection (2), Item 2: Results.

Source: ANAO analysis.

Results

2.2 The selected entities addressed the requirement, under section 16F of the PGPA Rule, to provide the results of the measurement and assessment of their performance in achieving their purposes. The performance statements included results against each performance criterion and references to the location of each criterion within the entities’ PBSs and corporate plans.

2.3 Subsection 16F(1) of the PGPA Rule also requires that results are presented in accordance with the method set out in the corporate plan and PBS. Finance guidance indicates that a corporate plan should include a description of performance measures, when they will be reported on, the data collection techniques to be used and any targets the performance measures will be assessed against. Agriculture’s 2015–19 Corporate Plan did not include information on the data collection techniques that would be used to measure results. Data sources were identified in the performance statements for six of the 11 performance criteria examined.

Analysis

2.4 The analysis sections of the selected entities’ performance statements contained background information to enhance the reader’s understanding of the results, including graphical representations of historical trends for some criteria. Both entities included some analysis of the factors contributing to the reported performance, in accordance with the requirements of sub-section 16F(2) of the PGPA Rule. The analysis could have been improved by further identifying and clarifying links between the reported results, and internal or external factors that influenced these results.

Australian Federal Police

2.5 The AFP’s analysis section consisted of high level statements regarding the results against each performance criterion, and how these reflected achievement of the AFP’s purpose. Where results did not meet the expected targets, the AFP identified factors contributing to the outcome and noted actions it had taken to address these factors. There was limited discussion of changes to the environment during the reporting period which may have contributed to the AFP’s performance.

2.6 The AFP provided more detailed analysis against the individual measures in the results section. A page was dedicated to each measure, detailing historical quantitative results and comparative analysis of the previous reporting period. For some criteria, relevant case studies were also listed in the performance statements and detailed later in the annual report.

Department of Agriculture and Water Resources

2.7 Agriculture discussed its results under six sub-headings, and provided case studies and references to other publications from which information was sourced. The analysis section included consideration of the broader environment, comprising the Australian economy, the overall market sector, and seasonal conditions, which may have affected the reported results.

2.8 There was limited analysis to demonstrate how the department’s activities had contributed towards the reported results. For example, the sections on ‘Managing industry levies’ and ‘Rural research and development’ advised how much revenue was collected and distributed to research and development corporations, but not how the levies or work undertaken by the corporations had contributed to improved farm-gate returns. The inclusion of case studies of specific activities that the department had undertaken throughout the year that contributed towards the reported results would have improved the quality of reporting.

Key Learnings 1. Presentation and analysis of results

  • In line with section 16F of the PGPA Rule, performance statements should include an analysis of the factors that may have contributed to the entity’s performance in achieving its purposes in the reporting period.
  • A clear presentation of results enables the reader to draw conclusions about the entity’s overall performance against its purpose. Presenting results alongside the established targets, comparing results to previous years, and including references to related indicators and results, can assist the reader in this process.
  • The analysis sections of the performance statements provide the opportunity to supplement the reported results with contextual information. This can enhance the reader’s understanding of the environment within which the entity operates and the contributions that it makes.
  • Performance information is strengthened by a discussion of how the entity’s activities are expected to have contributed towards the reported results.

Did the entities’ corporate plans support performance measurement and reporting in the annual performance statements?

Both entities built on their existing external performance measurement and reporting framework, to meet the requirements of the enhanced Commonwealth performance framework under the Public Governance, Performance and Accountability Act 2013. The information published by the entities in their corporate plan and Portfolio Budget Statements provides a foundation for reporting in the annual performance statements, although there was scope for both entities to improve how material was presented to achieve a clearer ‘line of sight’.

2.9 For its 2015–19 Corporate Plan, the AFP built on its existing performance measurement and reporting framework as published in previous Portfolio Budget Statements (PBS). The AFP has since commenced a project to review and redevelop its performance measurement and reporting framework. The redeveloped framework will be subject to a phased implementation commencing in 2017–18.

2.10 Agriculture commenced re-development of its performance measurement and reporting framework in January 2015. This process combined the 15 programs reported in the PBSs into five objectives, identifying effectiveness, efficiency and output performance measures and deliverables. In February 2015, a Performance Reporting Project Board was established to oversee the re-development of the department’s non-financial performance reporting framework. The department reviewed its performance framework for 2016–17 and 2017–18, and advised that it intends to review the framework annually to drive continuous improvement. Performance measures from the 2015–19 Corporate Plan were revised and an amended set of measures and targets were included in the 2016–1723 Corporate Plan. The department advised that changes included:

  • clarification of language;
  • inclusion of relevant evaluation processes; and
  • inclusion of new measures and initiatives.

Purpose statements and activities

2.11 Section 16E of the PGPA Rule requires that an entity’s corporate plan state the entity’s purposes over the next four years. The PGPA Act defines purpose/s as including the objectives, functions or role of an entity. The aim of the purpose/s statement is to give context to the significant activities that the entity will pursue over the period covered by the plan.24 Finance guidance notes that the purpose/s of an entity should be stated in a relevant and concise manner.25

2.12 Finance has defined an activity as a distinct effort of an entity undertaken to achieve a specific result (fulfilling purpose/s). Activities should be aligned with the entity’s purpose/s, and be a focus for performance measurement and reporting.

2.13 As noted in paragraph 1.7, the PBS and corporate plan are intended to be complementary and provide the foundations for the performance story presented in entity performance statements.

Australian Federal Police

2.14 The 2015–19 Corporate Plan met the requirements of the PGPA Rule and provided a foundation for performance reporting.26

2.15 The AFP’s 2015–19 Corporate Plan identified the following purpose:

to enforce Commonwealth criminal law and contribute to combating organised crime and protecting Commonwealth interests from criminal activity in Australia and overseas.

2.16 The purpose reflects the strategic objectives of the entity and is derived from the Australian Federal Police Act 1979 and associated Ministerial Direction issued in May 2014. The purpose also reflects Outcome 1 as set out in the AFP’s 2015–16 PBS. Outcome 2 is not covered by the AFP’s 2015–19 Corporate Plan, as it is subject to a separate purchasing agreement with the ACT Government. Readers are directed to separate reporting on this arrangement. The AFP’s outcomes and related programs for 2015–16 are outlined in Appendix 2.

2.17 The presentation and expression of the AFP’s activities could be improved to assist a reader to identify and assess the alignment of the performance criteria to the purpose. The AFP’s program objectives set out in the 2015–16 PBS are used as the basis for describing the AFP’s activities in the corporate plan. The activities described under Program 1.1 and 1.2 collectively address the AFP’s purpose. The activities for Program 1.1 are dispersed through narrative across the sub-sections on ‘Federal Policing’, ‘National Security’, and ‘Specialist and supporting capabilities’. Each sub-section also includes information on the AFP’s organisational structures, functions, and aspirations. As a result, it can be difficult for the reader to easily identify specific activities, and obtain a clear read of how the activities contribute towards the AFP’s purpose and objectives.

2.18 The AFP identified the presentation and expression of its activities as an area for improvement as part of its ongoing review of the performance measurement and reporting framework. As a result, the AFP’s 2016–17 Corporate Plan has been structured so that activities intended to be undertaken by the AFP to fulfil its purpose are clearly articulated. This will provide the basis for a clearer read between the 2016–17 Corporate Plan and the performance statements to be included in the 2016–17 Annual Report.

Department of Agriculture and Water Resources

2.19 The 2015–19 Corporate Plan met the requirements of the PGPA Rule and provided a foundation for the reporting of Agriculture’s annual performance. The key priorities and objectives published in the 2015–16 PBS were reflected in the 2015–19 Corporate Plan purpose statement, meeting the requirements of the PGPA Act. The plan’s purpose statement comprised five purposes, three outcomes and eight objectives. Each objective included performance measures and the key activities the department planned to undertake to achieve intended results. There was scope to better focus the purpose statement, to allow the intended results to be more readily identifiable. The structure of the purpose statement was improved in the 2016–17 Corporate Plan. The three outcomes of the department have been consolidated into a clear and concise purpose statement, including intended activities and results.

2.20 In re-developing the performance framework for the 2015–19 Corporate Plan, Agriculture also considered the outcomes, programs and performance measures already published in its PBS. The 2015–19 Corporate Plan has clear links to the outcomes and programs within the PBS. The PBS performance measures and targets are a subset of the performance measures included in the 2015–19 Corporate Plan, consistent with Finance guidance. Agriculture’s outcomes and related programs for 2015–16 are outlined in Appendix 2.

2.21 This audit focused on the following strategic objective included in the 2015–19 Corporate Plan:

Building successful primary industries – Improve the farm-gate returns for agriculture, fisheries, food and fibre industries.

2.22 The key activities from the 2015–19 Corporate Plan for the ‘Building successful primary industries’ objective are reproduced in Table 1.2.

2.23 The activities were clearly identified and reflect the government priorities listed in publications such as the Agricultural Competitiveness White Paper and on the department’s website. Improvements could be made to the identification of key activities by including the specific activities that the department will undertake, rather than broad references such as ‘in accordance with government agenda’. Broad references can hinder the development of meaningful performance criteria and reduce the reader’s ability to judge the extent to which intended results were achieved, and the factors that affected the delivery of those results.

Key Learnings 2. Expression and identification of activities

  • Clearly identifying and grouping activities can assist the reader to assess the alignment of the performance criteria with the entity’s purpose. This provides the basis for a clear read between the corporate plan and the performance statements.
  • When identifying key activities, broad references such as ‘in accordance with government agenda’ should be avoided. Clearly specifying activities enables the development of meaningful performance criteria and enhances stakeholders’ ability to judge the extent to which intended results were achieved.

Did the entities’ performance criteria support the reporting of progress against their purposes?

The performance criteria reviewed as part of the audit enabled the reporting of, and accountability for, the progress of each entity towards fulfilling their purposes.

The performance criteria for both entities were mostly relevant, providing a basis to make decisions on the entities’ progress in fulfilling their purposes. One of the AFP’s performance criteria required improvements to assist the reader to identify the benefit or beneficiary measured by the performance criterion and its link to the AFP’s activities. Agriculture’s performance criteria were mostly relevant, however the benefit or beneficiary was often not clear, or the focus of the measure was not clearly attributable to the entity.

The performance criteria for both entities were mostly reliable, providing a basis for reasonably consistent assessment of the entities’ progress in fulfilling their purposes. Improvements to two of the AFP’s performance criteria are required to limit the level of potential bias in the reported results. The majority of Agriculture’s performance criteria did not describe the method or basis for measurement, or provide a target or baseline, impacting the reliability of the performance criteria.

As a whole, the performance criteria for both entities were substantially complete, collectively providing a balanced basis for assessing the entities’ progress in fulfilling their purposes. To improve the completeness of the performance criteria, the selection of performance criteria will require ongoing effort by both entities to identify opportunities to clarify or increase the overall alignment of performance criteria to the purpose, and present a greater balance of performance criteria across the different forms of performance information and their timeframes.

2.24 The ANAO assessed the selected entities’ performance criteria for relevance, reliability and completeness. The basis for assessment has been adapted from the criteria for the evaluation of the appropriateness of key performance indicators as set out in ANAO Report No.21 2013–14 Pilot Project to Audit Key Performance Indicators. The criteria have been updated to reflect Finance’s guidance to support the enhanced Commonwealth performance framework and can be found at Appendix 3.

Relevance and reliability of entities’ performance criteria

2.25 In applying the ‘relevant’ criterion, the ANAO assessed whether each performance measure adopted by the selected entities for the strategic objectives under review:

  • clearly indicated who benefited and how they benefited from the entity’s activities27;
  • was focused on a specific aspect of the entity’s purpose/s and activity/ies, and the attribution of the result to the entity is clear 28; and
  • was easily understandable.29

2.26 In applying the ‘reliable’ criterion the ANAO assessed whether each relevant performance measure:

  • was measurable, that is, it used and disclosed information sources and methodologies (including a basis or baseline for measurement or assessment, for example a target or benchmark) that were fit-for-purpose and verifiable30;and
  • was free from bias, allowing for clear interpretation and an objective basis for assessment of the results.31

2.27 The characteristics of the ‘relevant’ and ‘reliable’ criteria set out above are linked and observations made during an assessment against those characteristics may contribute to reporting for more than one criterion. Tables 2.2 and 2.3 summarise the ANAO’s overall assessment for the selected entities’ performance criteria. The scale used to rate each entity’s performance criteria was:

  • did not display the characteristics of the criterion (No—Not relevant/reliable);
  • displayed in part the characteristics of the criterion (Partly relevant/reliable);
  • displayed most of the characteristics of the criterion (Mostly relevant/reliable); and
  • displayed all of the characteristics of the criterion (Yes—Relevant/reliable).
Australian Federal Police

2.28 The majority of the AFP’s performance criteria were assessed to meet the relevant and reliable criteria. A summary of the ANAO’s analysis is outlined in the following paragraphs and Table 2.2. Further detail of the analysis can be found at Appendix 4.

2.29 Eight of the nine performance criteria assessed by the ANAO met the ‘relevant’ criterion. One criterion, the ‘level of external client/stakeholder satisfaction’, partly met the ‘relevant’ criterion. The satisfaction measure does not define who stakeholders or clients are, and in what context they have engaged with the AFP to be able to assess satisfaction. This limits the understanding of the performance criterion and its focus, limiting an assessment of what aspect of the purpose or activities it relates to.

2.30 The corporate plan defines and/or directs readers to further information in relation to the methodologies the measures are based on. The methodologies range from accessing results from internal databases, to participant and stakeholder surveys, and calculations that rely on inputs from internal and external sources. All of the performance criteria were capable of measurement by these methodologies, and targets have been assigned across the four year corporate plan.

2.31 Of the AFP’s performance criteria presented in Table 2.2, ‘assets restrained’, mostly met the reliable criterion, and ‘counter-terrorism investigation outcomes’ did not meet the criterion at all. This assessment was made due to the variability of the measurement outcomes of ‘assets restrained’ and the inbuilt bias of ‘counter-terrorism investigation outcomes’.

2.32 The reliability of the ‘assets restrained’ measure is impacted by the variability in the size, value and number of restraints that may occur in a given year. The ANAO notes that an explanation of contributing factors to the movement in the results of this measure is included in the AFP’s performance statements. It is important that accompanying analysis acknowledges this variability and includes evaluation of the factors contributing to the level restrained in a particular year, to allow the reader to determine the reliability of the measure as an indicator of past and future performance by the AFP.

2.33 The ‘counter-terrorism investigation outcomes’ measure compares the number of finalised (closed) counter-terrorism investigations to those that resulted in a prosecution, disruption or an intelligence referral. These three variables represent the breadth of potential outcomes for a finalised counter-terrorism investigation and invariably return a result of 100 per cent. This limits the reliability of the measure as it is biased to a positive outcome. Entities should assess performance criteria for inbuilt bias, before inclusion in their corporate plan

2.34 The remaining performance criteria were assessed to be reliable. Further discussion in relation to the suitability of the information sources supporting the performance measurement methodologies is provided in Chapter 3.

Table 2.2: Assessment of AFP performance criteria relevance and reliability

Strategic objective: Federal policing and national security—Promoting the safety and security of the Australian community and infrastructure; preventing, deterring, disrupting and investigating serious and organised crime and crimes of Commonwealth significance; and ensuring effective collaboration with international, Commonwealth, state and territory partners.

AFP performance criteria

ANAO assessment

Relevant

Reliable

Level of external client/stakeholder satisfaction (percentage of clients satisfied or very satisfied).

Partly

Yes

Percentage of cases before court that result in conviction.

Yes

Yes

Percentage of counter-terrorism investigations that result in a prosecution, disruption or intelligence referral outcome.

Yes

No

Level of community confidence in the contribution of the AFP to aviation law enforcement and security (percentage of aviation network users satisfied or very satisfied).

Yes

Yes

Response to aviation law enforcement and/or security incidents within priority response times.

Yes

Yes

Number of avoidable incidents per 5 000 Protection hours.

Yes

Yes

Positive return on investment for investigation of transnational crime.

Yes

Yes

Assets restrained.

Yes

Mostly

Increased or reinforced cyber safety and security awareness (percentage of surveyed sample indicating increased awareness or reinforced awareness after delivery of presentations).

Yes

Yes

     

Source: ANAO analysis.

Department of Agriculture and Water Resources

2.35 Agriculture’s performance criteria mostly met the ‘relevant’ and ‘reliable’ criteria. A summary of the ANAO’s analysis is outlined in the following paragraphs and Table 2.3 below. Further detail of the analysis is provided at Appendix 4.

2.36 For five of the performance criteria, the performance statements did not specify how the department’s activities contributed to the results reported, impacting their focus and relevance. The performance measures focus on the changes in the broader Australian economy and the primary industries sector, which can lead to improved farm-gate returns (meeting the department’s purposes). The outcomes are not always within the direct control or influence of the department (for example, Portfolio industries may record an increase in productivity), or are shared with another entity and do not clearly describe Agriculture’s contribution.32 This makes it difficult to attribute the results of a particular departmental activity to changes observed in the primary industries sector.

2.37 Where it is difficult to attribute the results of a particular departmental activity to changes observed in the primary industries sector and broader environment, there would be value in acknowledging this limitation in the corporate plan. The limitation could be further addressed by including specific case studies or details of activities Agriculture will undertake throughout the reporting period that influence the results being reported. For example, Agriculture could highlight the key projects or programs that are expected to contribute towards the results reported through the performance criteria and include an analysis of the progress, or otherwise, of the specific projects or programs.

2.38 To improve the relevance of the performance criteria, Agriculture should also ensure that it is clear to readers who is the beneficiary of the activity measured by the criterion, and how they will benefit. This was not clear for six of the performance criteria.

2.39 None of Agriculture’s performance criteria in 2015–16 disclosed the method and parameters (targets, timeframes or baselines) that would be used to collect data and measure performance to enable assessment by the reader. This affects the reliability of the performance criteria presented in Agriculture’s annual performance statements. For some measures, results have been informed by old data or trend analysis across a decade or more, which does not reflect progress in 2015–16. Other measures are not able to be easily measured and rely on assumptions or estimates. The performance measures included in Agriculture’s 2016–17 Corporate Plan have addressed some of these issues with the inclusion of targets, baselines or information on the basis for how results will be measured and assessed for future performance statements.

Table 2.3: Assessment of Agriculture performance criteria relevance and reliability

Strategic objective: Building successful primary industries—Improve the farm-gate returns for agriculture, fisheries, food and fibre industries.

Agriculture performance criteria

ANAO assessment

Relevant

Reliable

Portfolio industries record an increase in productivity.

Mostly

Mostly

Rate of return on capital invested across portfolio industries is maintained or increased.

Mostly

Mostly

Rate of profit for producers and businesses is maintained or increased.

Mostly

Mostly

Access to water, land, forest and marine resources for primary production is maintained or improved.

Mostly

Mostly

Improved availability of safe, efficient and effective agricultural and veterinary chemicals.

Mostly

Mostly

Investment in rural research and development corporation programs demonstrates positive returns.

Mostly

Mostly

100% of allocated funding under the Research and Development for Profit program expended in accordance with the agreed timetable.

Partly

Mostly

100% of rural research and development corporations are compliant with statutory and contractual requirements.

Partly

Mostly

High level of efficiency in collecting and distributing levies to fund research and development in research and development corporations.

Mostly

Mostly

Inspections of levy agent records cover at least 30% of levy revenue.

Partly

Mostly

Less than 5% of quota allocations are rejected because of quota certification failures.

Partly

Mostly

     

Source: ANAO analysis.

Key Learnings 3. Relevance and reliability of performance criteria

Relevance of performance criteria

  • Performance criteria should clearly align to an entity’s purpose and activities, indicate who will benefit from the related activity and how, and be understandable to readers.

Reliability of performance criteria

  • Performance criteria should be measurable and disclose a methodology for measurement and basis for assessment, such as a target or benchmark. This allows for consistent interpretation and measurement of results.
  • To be reliable, performance criteria should not result in biased reporting of results.

Completeness of the performance criteria in assessing the entities’ activities and purpose

2.40 Finance guidance notes that it is rare for a single measure to be able to adequately determine the effectiveness of an activity. It advises that good performance information will draw on multiple sources. The quality of performance information should be emphasised over quantity. The guidance recommends a small set of measures that is sufficiently comprehensive to cover those factors that affect an activity’s performance.

2.41 Finance guidance indicates that performance measures should be a mix of quantitative and qualitative data or information. It is stated that efficiency, output and input measures will typically be used to complement effectiveness measures. The guidance also notes that in some cases, effectiveness may not be measurable, due to cost or a lack of complete information. In these cases, input, output and efficiency measures can be used as proxies for effectiveness. Entities are advised that in these circumstances, they should be clear on why effectiveness cannot be measured and how the proxy measures are suitable.

2.42 The guidance also recognises that activities often work on different timeframes, and information on one or more timeframes may not be available at the time of reporting. It is therefore recommended that the performance story told at a particular time reflect the outcome that can be reasonably expected from the relevant activities at that time.

2.43 In assessing the selected entities’ performance criteria for completeness, the ANAO considered whether the performance criteria present a basis for a collective and balanced assessment of the entity against its purpose. In particular, the ANAO considered whether the selected entities’ performance criteria:

  • clearly align to the entity’s purpose through the activities identified in the corporate plan (collective);
  • provide a basis for assessment of the effectiveness and efficiency of the entity in fulfilling its purpose (balanced);
  • relied on a mixture of quantitative and qualitative data (balanced); and
  • assess a mixture of short, medium and long-term objectives (balanced).
Australian Federal Police

2.44 The performance criteria assessed as part of the audit are substantially complete, providing a collective and balanced basis for assessing the AFP’s progress in fulfilling its purpose. Areas for improvement include clearer alignment of performance criteria to the activities, program objectives and purposes, and presenting a mixture of performance criteria that focus on short, medium and long-term objectives. In the 2016–17 Corporate Plan, the AFP has described its approach to enhancing performance measurement and reporting over the next three years. The proposed approach is to include more qualitative discussion and evaluation of overall performance.

2.45 As discussed earlier, the presentation of the activities of the AFP could be improved in the corporate plan, to assist readers to align the performance criteria to the AFP’s purpose. The ANAO identified that ‘Specialist and supporting capabilities’ could not be directly linked to the performance criteria presented in the corporate plan or performance statements. It was also not clear whether ‘External client/stakeholder satisfaction’ linked directly to a single activity, multiple activities or all AFP activities. The AFP also relies on a high proportion of output measures, rather than measures of effectiveness to assess progress in fulfilling its purpose. While this approach can be appropriate in certain circumstances, the AFP should include an explanation of why measuring outputs provides a reasonable basis for assessing its effectiveness.

2.46 Further assessment of the quantitative results against the AFP’s performance criteria may assist the AFP to present a more meaningful performance story. For example, against ‘percentage of cases before court that result in conviction’, the AFP may consider cases where a conviction was not recorded, and include a discussion of the contributing factor/s and what measures the AFP has implemented to increase the likelihood of a conviction in future cases. This type of analysis would provide users with additional information to assist in evaluating the performance of the AFP.

2.47 The AFP’s performance criteria include a mixture of quantitative and qualitative data, providing a basis for a more comprehensive performance story. Each performance criterion is reported against annually, and static targets have been set over the four years of the corporate plan for eight of the nine measures reviewed by the ANAO. ‘Assets restrained’ reports against a target of an ‘increase’ against a rolling average of the previous five years’ results.

2.48 There would be benefit in the AFP including in the Corporate Plan a description of the rationale for setting static targets, particularly where the target has been historically met, and how incremental improvement is expected to be demonstrated over time. The ANAO notes that the AFP provided historical results in the annual report appendices for performance criteria with static targets. This enables a reader to assess the AFP’s performance against targets over the longer-term, however including more detailed analysis of longer-term performance in the performance statements would be beneficial.

2.49 Table 2.4 below details the AFP’s performance criteria against the ‘complete’ criterion characteristics set out in paragraph 2.43.

Table 2.4: ANAO assessment of AFP performance criteria—completeness

Strategic objective: Federal policing and national security—Promoting the safety and security of the Australian community and infrastructure; preventing, deterring, disrupting and investigating serious and organised crime and crimes of Commonwealth significance; and ensuring effective collaboration with international, Commonwealth, state and territory partners.

Activitiesa

Performance criteria

Type

Basis

Timeframe

Federal policing

Percentage of cases before court that result in conviction.

Output

Quantitative

Short-term

Positive return on investment for investigation of transnational crime.

Efficiency

Quantitative

Short-term

Assets restrained.

Output

Quantitative

Medium-term

Increased or reinforced cyber safety and security awareness (percentage of survey sample indicating increased awareness or reinforced awareness after delivery of presentations).

Effectiveness

Qualitative

Short-term

National security

Percentage of counter-terrorism investigations that result in a prosecution, disruption or intelligence referral outcome.

Output

Quantitative

Short-term

Level of community confidence in the contribution of the AFP to aviation law enforcement and security.

Effectiveness

Qualitative

Short-term

Response to aviation law enforcement and/or security incidents within priority response times.

Output

Quantitative

Short-term

Number of avoidable incidents per 5 000 Protection hours.

Output

Quantitative

Short-term

Specialist and supporting capabilities

The ANAO could not:

  • identify a directly related criterion; or
  • determine how the other criteria might contribute to measuring this activity.

The ANAO could not determine whether this criterion related to a single activity, multiple activities, or all.

Level of external client/stakeholder satisfaction (percentage of clients satisfied or very satisfied).

Effectiveness

Qualitative

Short-term

         

Note a: Performance criteria have been mapped to specific activities by the ANAO. Note that some performance criteria may relate to more than one activity.

Source: ANAO analysis.

Department of Agriculture and Water Resources

2.50 Agriculture’s performance criteria assessed as part of the audit are substantially complete, providing a collective and balanced basis for assessing the department’s progress in fulfilling its purpose. For future reporting, consideration could be given to ensuring that there are performance criteria that cover all identified key activities, to provide a more complete overview of the department’s performance in fulfilling its purposes.

2.51 As noted in Table 2.5, of the 11 performance criteria reviewed as part of the audit:

  • four can be linked to measuring the activity of supporting advice to government on how to assist primary industries to remain competitive, productive and sustainable into the future;
  • three provide a basis to assess the performance of the department’s administered programs supporting Research and Development Corporations;
  • two relate to the collection of levies for research, development and marketing;
  • one reflects the administration of reforms in industry regulations; and
  • one could not be linked to the activities listed for the strategic objective or purpose.

2.52 The performance criteria ‘Improved availability of safe, efficient and effective agricultural and veterinary chemicals’ and ‘Access to water, land, forest and marine resources for primary production is maintained or improved’ were matched to specific activities by the ANAO. This required further investigation into the details of the relevant activities, and was not immediately clear from information provided in the corporate plan. The activity of ‘support current government priorities including country-of-origin labelling and the relocation of rural research and development corporations’ could not be clearly linked to the performance criteria.

2.53 The performance criteria were balanced, providing a mixture of qualitative and quantitative measures, and covering the four year term of the 2015–19 Corporate Plan (though not specific to any one year) along with specific measures for 2015–16. The majority of the performance criteria focussed on effectiveness (six of the 11 reviewed), one had a focus on efficiency, two were output measures, with the remaining two being input measures. The ANAO notes that in the 2016–17 Corporate Plan, the input measures have become input targets. The performance measures had a mixture of short, medium and long-term timeframes.33

2.54 Table 2.5 details Agriculture’s performance criteria against the ‘complete’ criterion characteristics set out in paragraph 2.43.

Table 2.5: ANAO assessment of Agriculture performance criteria—completeness

Strategic objective: Building successful primary industries—Improve the farm-gate returns for agriculture, fisheries, food and fibre industries

Activitiesa

Performance criteria

Type

Basis

Timeframe

Provide analysis and advice to the government and external decision-makers on support for portfolio industries to make them more productive, globally competitive and profitable.

Portfolio industries record an increase in productivity.

Effectiveness

Quantitative

Long-term

Rate of return on capital invested across portfolio industries is maintained or increased.

Effectiveness

Quantitative

Long-term

Rate of profit for producers and businesses is maintained or increased.

Effectiveness

Quantitative

Long-term

Access to water, land, forest and marine resources for primary production is maintained or improved.

Effectiveness

Quantitative

Short and medium-term

Administer, improve and implement reforms in industry regulations in accordance with the government’s agenda.

Improved availability of safe, efficient and effective agricultural and veterinary chemicals.

Effectiveness

Qualitative

Short and medium-term

Deliver Agricultural Competitiveness White Paper initiatives including measures that seek a smarter approach to farming based on a strong research and development system.

Support industry bodies and research and development corporations to meet statutory and contractual requirements and objectives.

Investment in rural research and development corporation programs demonstrates positive returns.

Effectiveness

Qualitative

Long-term

100% of allocated funding under the Research and Development for Profit program expended in accordance with the agreed timetable.

Input

Quantitative

Short and medium-term

100% of rural research and development corporations are compliant with statutory and contractual requirements.

Input

Quantitative

Short-term

Provide levies collection and distribution services to industry and government.

High level of efficiency in collecting and distributing levies to fund research and development in research and development corporations

Efficiency

Qualitative

Short-term

Inspections of levy agent records cover at least 30% of levy revenue.

Output

Quantitative

Short-term

Support current government priorities including country-of origin labelling and the relocation of rural research and development corporations.

The ANAO could not:

  • identify a directly related criterion; or
  • determine how the other criteria might contribute to measuring this activity.

The ANAO could not determine how this criterion linked to activities listed for the strategic objective or purpose.

Less than 5% of quota allocations are rejected because of quota certification failures.

Output

Quantitative

Short-term

         

Note a: Performance criteria have been mapped to specific activities by the ANAO. Note that some performance criteria may relate to more than one activity under the strategic objective of ‘Building successful primary industries’. In addition, some performance criteria reflect the aggregate of activities by government/s and industries, and include factors not directly attributable to Agriculture.

Source: ANAO analysis.

Key Learnings 4. Completeness of performance criteria

  • An entity’s performance statement should contain a variety of meaningful performance criteria to demonstrate how each activity contributes towards progress against the entity’s purpose/s, across the different timeframes.
  • Achievement against an entity’s purpose/s is best demonstrated by criteria that focus on effectiveness. Where this is not possible, entities should explain why the substituted input, output and/or efficiency criteria are suitable proxy measures for effectiveness.
  • The insights provided by performance information can be enhanced by drawing on a variety of different data sources and evaluation tools. These can include stakeholder surveys, benchmarking, peer review, and program or project evaluations.a

Note a: Department of Finance, Resource Management Guide No. 130: Overview of the enhanced Commonwealth Performance Framework, April 2015, p. 33.

3. Systems and processes to support performance measurement and reporting

Areas examined

This chapter examines whether the selected entities implemented processes to collect, assess, assure and report performance information in the annual performance statements. It also examines whether sufficient records were retained to support the results reported.

Conclusion

Both entities established or adapted existing systems and processes to meet the requirements of the Public Governance, Performance and Accountability Act 2013 (PGPA Act) and the Public Governance, Performance and Accountability Rule 2014 (the Rule). These remained in development during the audit, with further work being undertaken in 2016–17, to support the quality of information reported in future performance statements.

The entities established assurance processes to certify that the reported performance information accurately reflected the entities’ performance. Planning and assurance processes for the entities should mature over time. This should include further consideration of the role and function of the respective audit committees, to ensure the intent and requirements of the framework are met, as neither committee could fully demonstrate compliance with the PGPA Rule.

The majority of results presented in the Australian Federal Police and Department of Agriculture and Water Resources annual performance statements were supported by complete and accurate records as required by the PGPA Act. Both entities could improve record-keeping to better demonstrate the calculations and analysis applied to raw data to produce results, and to support the analysis in the annual performance statements.

Opportunities for improvement

The ANAO identified opportunities for improvement relating to:

  • strengthening systems and processes for collecting, consolidating, validating and reporting performance information;
  • establishing clearer policies and guidance for assurance of performance information, including review of the audit committee role and function; and
  • documenting and maintaining consistent and comprehensive records to support reported performance in the annual performance statements.

Did the entities’ processes inform the coordination and collation of the annual performance statements?

Each entity adapted pre-existing processes for the preparation of the Annual Report, to facilitate the coordination and collation of information for the annual performance statements. In addition, both entities developed a project plan outlining the roles and responsibilities, risks and mitigating controls, milestones for delivery, and assurance mechanisms to guide the preparation of the performance statements. Neither entity completed a comprehensive pre-assessment of the processes of producing performance statements as part of their planning. The incremental development of performance reporting by both entities was focused on identifying lessons learnt.

3.1 In reviewing the processes implemented by the entities in developing their first annual performance statements, the ANAO considered whether the entities:

  • established, or leveraged on, structured approaches to support the development of their performance statements;
  • clearly defined roles, responsibilities and accountabilities;
  • consulted internal and external stakeholders; and
  • fully engaged their management.

3.2 Prior to the introduction of the requirements for all Commonwealth entities and companies to prepare performance statements, entities were required to publish the results of non-financial performance measures set out in their Portfolio Budget Statements, in the Annual Report. As a result, there were pre-existing processes for the preparation of the Annual Report in both entities. These were built upon to facilitate the coordination and collation of information for the preparation of the annual performance statements.

3.3 The Department of Agriculture and Water Resources (Agriculture) established a Planning and Reporting Working Group, comprising representatives of the divisions involved in performance reporting, and their divisional business areas. The Working Group provided oversight and monitoring of the department’s business planning and performance reporting arrangements, including:

  • reviewing the department’s annual business planning and performance reporting cycle;
  • identifying gaps in performance information and the ability to report on planned reporting of measures;
  • approving arrangements to prepare each year’s corporate plan, annual performance statements and annual report, and performance information required for the Portfolio Budget Statements;
  • reviewing arrangements for the divisional business planning process, including the preparation of divisional business plans, mid-year and end-of-year divisional performance reviews;
  • approving arrangements to report against the Regulator Performance Framework34; and
  • reporting to the department’s Executive Management Committee on these matters.

3.4 The Australian Federal Police (AFP) has an equivalent group, the Organisational Performance team, within the then Internal Audit and Business Analysis branch.35 The team has been established for a number of years and is responsible for the collation of data and monitoring of organisational performance for the AFP, including performance criteria set out in the PBS, corporate plan and annual performance statements. This team was also responsible for:

  • engagement with the AFP’s National Managers, Executive Leadership Committee and Commissioner, on the form and content of the performance statements;
  • consultation with the audit committee;
  • participation in the Department of Finance’s Communities of Practice; and
  • participation in a peer group of other law-enforcement and regulatory entities to discuss learnings and future collaboration regarding performance criteria.

3.5 Both entities developed a project plan outlining the roles and responsibilities, risks and mitigating controls, milestones for delivery, and assurance mechanisms to support the preparation of their performance statements. The plans provided a basis for monitoring the progress of the performance statements and to assist relevant stakeholders understand the assurance framework underpinning the preparation processes.

3.6 Agriculture’s plan was finalised in March 2016 and the AFP received endorsement from the Audit Committee of its plan in May 2016. To be able to identify or address issues in the design or underlying processes supporting the annual performance statements, the timing of both entities’ project plans could be moved forward. A comprehensive pre-assessment of the processes of producing performance statements as part of their planning would also assist.

3.7 In addition, the entities should consider how current and planned future processes can support their annual cycle of planning and performance reporting. Corporate plans and annual performance statements are the bookends of the performance framework for Commonwealth entities36, and should be viewed by entities as a continuous reporting process, rather than annual and distinct publications. Implementing processes that support objective planning and performance reporting, including the preparation of corporate plans and annual performance statements throughout the year and into the future, would also assist in standardising the assurance mechanisms that underpin entity planning and reporting.

Key learnings 5. Implementing processes to support continuous improvement

  • Entities should design processes that support the complete cycle of planning, performance measurement and reporting. This would reflect the preparation and publication of the Portfolio Budget Statements, corporate plans, performance statements and annual reports, evaluating lessons learnt and incorporating these into the following cycle to contribute to continuous improvement.
  • The enhanced Commonwealth performance framework provides entities the opportunity to apply a fit for purpose approach to their performance measurement.a This principle should be kept in mind by entities in designing or reviewing their own processes to inform the preparation of the annual performance statements.

Note a: Department of Finance, Resource Management Guide No. 130: Overview of the enhanced Commonwealth Performance Framework, reissued in July 2016, p. 9.

Were systems and methodologies established to collect and report performance information?

Both entities leveraged from existing systems and methodologies to collect and report performance information for the purposes of the annual performance statements. The AFP would benefit from considering the frequency and extent of reviews of the methodology supporting one performance criterion, and strengthening processes for the validation of information sourced outside of management systems. Further consideration of data availability is required by Agriculture to establish a system to support the consistent collection, analysis and reporting of non-financial performance information.

3.8 In designing appropriate performance criteria, an important consideration for entities is the information that the entity will need to collect to report against those measures. Resource Management Guide No. 131: Developing good performance information, issued by the Department of Finance, sets out that entities should consider the following for information used to report performance:

  • availability—is information readily accessible or feasible to access;
  • suitability—is the information accurate, auditable and reliable;
  • timeliness—the information should be collected on a timescale relevant to the measure; and
  • cost—the effort required to collect and analyse information should be commensurate with the benefit it provides.

3.9 One focus of this audit was whether the selected entities considered the above factors and established systems and methodologies to support the collection of performance information that could be relied upon as a complete and accurate reflection of the results reported in the performance statements. In assessing the approaches adopted by the selected entities, the ANAO considered the following:

  • the method and frequency used to collect the data for each criterion and whether it was suitable;
  • the known limitations of the data;
  • how data is recorded and stored and what controls exist to ensure it is not inadvertently or purposely manipulated;
  • how data is extracted from systems and what processes are applied to reach the reported result; and
  • whether adequate documentation of policies and procedures to support consistent data outcomes exist.

3.10 As part of its review the ANAO also considered the extent of reliance placed on data extracted from IT systems for performance reporting, and the controls established by the entities to ensure the integrity of that data. Details of the ANAO’s assessment of the selected entities’ systems and methodologies is provided in Appendices 5 and 6. The following section provides a summary for each entity.

Australian Federal Police

3.11 The AFP maintains a KPI Business Rulebook (updated each reporting cycle) which summarises for each internal and external criterion reported:

  • the data sources and collection arrangements;
  • the frequency of their internal measurement or reporting; and
  • any identified limitations of the data source or collection methods.

3.12 Frequent measurement, monitoring and interrogation of performance results increase the likelihood that issues affecting the completeness and accuracy of data supporting the performance criteria are identified. The frequency of the AFP’s reporting of results is high for all performance criteria, except those based on annual surveys, and is presented to senior management groups within the AFP each month. The emphasis placed on regular reporting of results by the AFP, and the engagement of its senior management groups in reviewing and considering the AFP’s performance internally, provides users with confidence in the consistency and accuracy of information presented externally by the AFP in the annual performance statements.

3.13 Six of the AFP’s performance criteria use data sourced from the AFP’s management systems, extracted through a corporate reporting module. Further discussion on the reliability of these systems can be found at paragraphs 3.22 to 3.26 of this audit report.

3.14 Only one of the criteria (return on investment) requires further analysis of the systems data to reach the reported result. The ANAO noted that the methodology and supporting inputs underpinning the performance criterion were developed over a number of years by the AFP and augmented with research work conducted jointly with the University of Queensland Institute for Social Science Research. The methodology was finalised in 2008 and 2009 and inputs to the methodology were updated in August 2013 following a review by the AFP.37 No further updates to the methodology have been made since then. There would be benefit in the AFP considering the frequency and extent of these reviews given the complexity of the underlying methodology.

3.15 Two criteria are based on the results of annual surveys conducted through external research companies and the information used to report the results against these measures is drawn directly from the external reports. Engaging reputable research companies with expertise in the survey field provides a level of confidence in the quality and objectivity of the survey construction and analysis of results. The risk of potential misstatement of the results in the annual performance statements is also limited, as there is no need to perform further analysis or manipulation of the result as presented in the third party report.

3.16 The final criterion is based on a collation of responses to hardcopy questionnaires posted to the AFP from presentation participants. The ANAO noted that the calculated response rate, based on the number of participants and responses received, was 8.1 per cent38, and there was no evidence of analysis of the impact of partial or incomplete responses. Without consideration of these factors, the analysis of the effectiveness of the AFP’s activity is limited.

3.17 There would be benefit in the AFP assessing the frequency and extent of reviews of the inputs for the return on investment methodology, and giving further consideration to the impact of response rates on the meaningfulness of the analysis of questionnaire responses.

Department of Agriculture and Water Resources

3.18 Documentation of the processes for collection, consolidation, validation and reporting of performance measures was not consistent, and remained at a high-level. Performance criteria data and information was drawn from operating systems across the department and a number of external sources, such as the Australian Bureau of Statistics. The performance criteria reflected a variety of relevant data collection techniques and methodologies. Data collection tools included surveys, internal business systems (including ICT data management systems), satellite imaging and advice from line areas. Agriculture has considered opportunities going forward to improve the quality and diversity of data sources and make greater use of tools such as benchmarking, stakeholder surveys, peer review and evaluation.

3.19 Agriculture would have benefited from further consideration of the availability of data to support the performance criteria in developing its methodologies. Results for two of the 11 performance criteria relied on alternative data sources to produce the result presented in the annual performance statements compared to sources identified earlier in the year. In addition, the results reported against five criteria did not fully reflect the original criterion due to the absence of available data or analysis that was expected to be available. This indicates a need for further development by Agriculture of the system for identifying and collecting performance information, including a revision of reporting frequency. Agriculture, in its review of the performance reporting framework in 2017, identified this need to reconsider the frequency of reporting of some performance criteria.

Key learnings 6. Evaluating data sources

When considering the suitability of information and data sources for performance criteria, entities should identify and document:

  • where the data to support the result is to be sourced;
  • how data is going to be extracted and whether further calculations and analysis will be required to reach the result;
  • any limitations to the reliability of the data; and
  • the level of confidence that the data will be available when required.

Reliance on IT systems

3.20 The use of IT systems or applications, when not adequately controlled or monitored, can increase the risks posed to the accuracy and completeness of information being relied upon to produce reports. The existence of effective controls mitigates these risks.

3.21 The ANAO focused its testing on the selected entities’ key controls for data capture, quality assurance and reporting integrity. The controls tested included:

  • user access restriction to systems and any data warehouse systems;
  • data input validation and quality assurance;
  • monitoring and reporting of changes and/or deletions of records; and
  • extraction and transformation of data.39
Australian Federal Police

3.22 The AFP places reliance on its management systems (PROMIS40 and CAD41) and supporting corporate reporting modules to record and extract data used to report results against a number of performance criteria.

3.23 The ANAO noted the maintenance of user manuals and delivery of user training to reinforce behaviours and assist in minimising errors and maintain data quality. The AFP has established an overarching National Guideline on the Security of ICT System Access (the Guideline). The Guideline defines system user obligations to securely manage access to all AFP ICT systems, including key controls for how access is granted, maintained and audited. The ANAO’s assessment of user access controls in both PROMIS and the data warehouse systems did not identify any practice that deviated from the Guideline.

3.24 The regular performance measurement, reporting and internal monitoring referred to in paragraph 3.12 above, also includes quality assurance reviews of the data being drawn from the AFP’s corporate reporting systems. These reviews are undertaken by the Organisational Performance team prior to analysis. This process includes:

  • confirming the dataset has been extracted using the fields required in the KPI Business Rulebook;
  • reviewing the consistency of related fields within PROMIS, for example that status reasons align to case notes;
  • investigating any incompatible fields through follow up with individual case officers; and
  • reviewing fields to confirm only relevant data is included in the report.

3.25 Where the Organisational Performance team identifies that data supporting performance criteria has been input incorrectly, the relevant case officer is notified and a correction is made to PROMIS. Changes cannot be made retrospectively by an operator to the CAD system,and all errors are manually confirmed and updated in the performance reporting data. These changes are recorded by the Organisational Performance team throughout the year, providing a snapshot of potential error rates arising from inaccurate data input. No formal analysis of the potential error rate and impact on the data is performed by the team.

3.26 The ANAO did not identify any IT control weaknesses that would affect the integrity of data extracted from AFP systems to support performance reporting in the performance statements.42

Department of Agriculture and Water Resources

3.27 Agriculture leveraged on its existing data and management systems (Phoenix43, EXDOC44, SARA and DAISY45, and E546) to generate the results presented in the performance statements, against the following performance criteria:

  • high level of efficiency in collecting and distributing levies to fund research and development in research and development corporations;
  • inspections of levy agent records cover at least 30 per cent of levy revenue; and
  • less than 5 per cent of quota allocations are rejected because of quota certification failures.

3.28 Access to these systems is managed by Agriculture through an established control framework. This framework includes controls restricting access to management systems and to the supporting ICT environment, including monitoring any changes to system records. The ANAO’s assessment of key user access controls for the relevant management systems did not identify any control weaknesses that would affect the integrity of data extracted to support performance reporting in the performance statements.

3.29 Existing systems’ controls were relied on by Agriculture to reduce the risk of inaccurate and incomplete information being produced and used for performance reporting. The systems’ controls comprised data input validation, automated calculations and data cleansing47 programs. For example, where a system control identified an incomplete record, it would prompt the user to seek and input additional information. These changes are then subject to automated revalidation by the system. The ANAO did not identify any IT control weaknesses related to these systems’ controls that would affect the integrity of the results presented in the performance statements.

3.30 The ANAO noted that, in some instances, data was required to be extracted and then transformed by Agriculture staff to generate the results used for performance reporting. This was the result of the systems not being originally designed to present data in the required manner for performance reporting. Clearly documenting the methodology for performing additional analysis over data extracted from systems enables a party, either internal or external to Agriculture, to re-perform the analysis supporting the annual performance statements.

Key learnings 7. Reliance on IT systems

  • Where appropriate, entities should consider how existing IT systems and controls can be leveraged from to support performance reporting.
  • Entities should clearly document the methodology for performing additional analysis over data extracted from systems to enable recalculation.

Were processes established to obtain assurance over the annual performance statements?

Processes were established by both entities to provide assurance that the results reported in the annual performance statements were an accurate representation of performance. Further refinement of these arrangements is required by the entities. This would include documenting: guidance on the assurance process; and the review and endorsement of the annual performance statements to ensure that evidence of management and audit committee assurance is recorded and retained.

Each entity relied on management certifications over the selected performance criteria and the completeness and accuracy of underlying records. The entities’ audit committees also received regular briefings on the preparation of the annual performance statements, including details on the management certification processes. Additionally, Agriculture’s audit committee commissioned an internal audit on the Key Performance Indicators (performance criteria) and Performance Reporting, to inform the committee’s review responsibilities. The audit committee also monitored the implementation of key recommendations. However, the audit committee’s sign off to Agriculture’s accountable authority was limited, and did not meet the requirements of the department’s audit committee charter or the PGPA Rule and its intent. The AFP was unable to locate the final certification by its audit committee to the accountable authority over the performance statements, limiting an assessment against the audit committee charter, or the PGPA Rule and its intent.

There would be benefit in both entities considering the role of the audit committee as a source of independent assurance to the accountable authority and how the audit committees’ charters and processes establish a basis to provide this assurance. The PGPA Rule provides that an audit committee’s functions must include reviewing the appropriateness of an accountable authority’s performance reporting. This function would necessarily involve the committee forming a view on how the entity should measure its performance. As a result, an audit committee’s charter, and any certification by the audit committee to an accountable authority discharging their performance reporting function, should reflect this requirement.

3.31 Section 16F of the PGPA Rule requires an accountable authority to certify that their annual performance statements accurately present the entity’s performance for the reporting period. An equivalent certification by the accountable authority is required by section 42 of the PGPA Act for the annual financial statements. It is important that an accountable authority has confidence that the entity has adequate processes in place to provide sufficient assurance to support these certifications.

3.32 The requirement for entities to prepare and publish their financial statements has been in place for a number of decades. As a result, the financial statements preparation processes of Commonwealth entities are mature, and supported by established assurance frameworks including mandatory annual audits by the ANAO. Entities may leverage from these frameworks and implement similar processes to provide an accountable authority with sufficient assurance to certify the annual performance statements. Relevant processes may include:

  • documented procedures and processes supporting consistent reporting;
  • assessment and documentation of risk and risk mitigation strategies;
  • management reporting of progressive results to internal stakeholders such as the accountable authority and audit committee;
  • management certifications; and
  • audit committee involvement and endorsement.

3.33 Earlier sections of this chapter considered the selected entities’ processes and procedures, including risk identification and management, and the frequency of management reporting to support the annual performance statements. The following sections focus on the AFP and Agriculture’s implementation of management certifications and audit committee involvement and endorsement, as assurance mechanisms for their accountable authority’s certification of the annual performance statements.

Management certifications

Australian Federal Police

3.34 The key management certification relied on by the AFP was provided by the National Manager’s Forum on 12 September 2016. National Managers were assigned responsibility for the AFP’s performance criteria included in the corporate plan in May 2016. This responsibility included reviewing and endorsing the performance criteria relevant to their business area, as set out in the KPI Business Rulebook referred to in paragraph 3.11 above, that would be reported against in the annual performance statements.

3.35 National Managers received regular updates of the progress of their performance criteria as part of the reporting referred to in paragraph 3.12 of this audit report. This was in addition to the quality assurance reviews of the data being drawn from the corporate reporting systems by the Organisational Performance team, referred to in paragraphs 3.24 and 3.25 above. On this basis, the National Managers:

endorsed that the annual performance statements are based on properly maintained records, accurately reflect the AFP’s performance in 2015–16 and comply with subsection 39(2) of the PGPA Act.

3.36 In accordance with the Performance Statements Preparation Plan, the performance statements were then circulated to the Executive Leadership Committee by the Deputy Commissioner Capability. The Deputy Commissioner Capability advised the Manager Capability Development of the endorsement of the performance statements by the Executive Leadership Committee on 21 September 2016. The AFP was unable to provide formal minutes recording this endorsement. The performance statements were subsequently recommended to the Commissioner for signing on 23 September 2016.

3.37 The performance statements were signed by the Commissioner and presented in the AFP’s 2015–16 Annual Report on 10 October 2016.

Department of Agriculture and Water Resources

3.38 Advice to Agriculture’s Secretary centred on a risk based approach and the specific assurance process followed prior to signing the necessary certifications was at the discretion of each First Assistant Secretary. The assurance processes varied, from meeting with senior managers to gain assurance, to email correspondence and reliance on results published in internal and external reports.

3.39 The ANAO reviewed the certification statements for the 11 performance criteria examined as part of this audit. Five First Assistant Secretaries (FASs) provided signed certifications across the 11 measures. A summary of the signed certifications was provided to the Secretary prior to the signing of the annual performance statements. The certifications were limited to the performance results tables in the performance statements and did not include the supporting analysis sections presented in the performance statements.

3.40 Two of the five certifications included caveats. The caveats were footnoted to the tables in the summary provided to the Secretary, but the summary table itself did not draw attention to them. The effect of the caveats on the results was not disclosed in the certifications or the summary, however, Agriculture did disclose the caveats in the annual performance statements. Agriculture also provided additional information in the form of graphs and analysis in the performance statements to further support the reported result. This additional information assists users to make an assessment of the reliability of the result. In future, the certification provided by the FASs should state the impact of any caveats and these should be clearly brought to the attention of the Secretary. Agriculture should also, where possible, limit the use of information which may require caveats in the future.

3.41 In its 2016–17 Annual Report implementation plan, the department noted plans to ensure certification of the entire document by relevant FASs, to provide additional assurance of compliance with the PGPA Rule. While the precise process was yet to be determined, it was proposed that FASs certify reporting in both the results tables and analysis sections for their divisions, and that the department’s Regulatory Reform Unit certify industry reporting for which it is responsible.

3.42 The proposed process could be refined by developing a check list for each FAS to complete prior to signing the certification. This could cover aspects such as: the availability of records to support the results reported in the performance statements; documentation of any known limitations to the data or information reported; and when and how to adequately and meaningfully caveat the certifications. As discussed above, all caveats should be clearly brought to the attention of the Secretary, with sufficient documentation to validate the effect of the caveat on the results.

Key learning 8. Management certifications

A check list is a useful tool to assist entity management representatives to understand the detailed considerations expected, including the level of evidence required, to support a certification of performance criteria and accompanying results.

The role and function of audit committees and entity performance reporting48

3.43 The accountable authority of a Commonwealth entity is required to ensure that the entity has an audit committee49 and to determine by written charter the functions of this committee.50 The minimum functions of an audit committee are set out in the PGPA Rule and include reviewing the appropriateness of the accountable authority’s performance reporting (Box 5, emphasis added).

Box 5: Section 17 of the PGPA Rule

Functions of the audit committee:

(1) The accountable authority of a Commonwealth entity must, by written charter, determine the functions of the audit committee for the entity.

(2) The functions must include reviewing the appropriateness of the accountable authority’s:

  1. financial reporting; and
  2. performance reporting; and
  3. system of risk oversight and management; and
  4. system of internal control;

for the entity.

3.44 An information sheet issued by the Department of Finance in August 2015, Audit Committees and performance reporting, provided further guidance, including:

  • This function51 would necessarily involve the committee forming some view on how the entity should measure its performance and how these measures can be weaved into an effective performance story.
  • Audit committees might also choose to look at the alignment of corporate plans with annual performance statements, compliance with rule minimum requirements, and the types of performance methodologies able to be used or integrated into forward plans.
  • Another potential role for audit committees could be to give assurance on support systems that sit behind performance reporting. This could involve testing the robustness of results and providing assurance that claimed results are valid and supportable.
Australian Federal Police

3.45 The AFP’s audit committee charter, endorsed by the Commissioner on 2 July 2015, sets out the committee’s functions in regard to performance reporting in accordance with the PGPA Rule as follows:

Review the appropriateness of the AFP’s performance reporting: including the framework for developing, measuring and reporting key performance indicators and the AFP’s annual performance statements.

3.46 The ANAO’s review of AFP’s Audit Committee meeting minutes from 2015–16 noted the following actions by the Audit Committee on the AFP’s performance measurement and reporting:

  • discussion of the potential benefit of Internal Audit involvement to provide assurance to the Commissioner around the preparation of the AFP’s Performance Statement. (No internal audit on this topic was completed, although it has been included as a potential topic in the 2016–2019 Internal Audit Work Program);
  • requested and received advice from the Internal Audit and Business Analysis section on how the performance criteria were derived; what risks there were to the accuracy of underlying data; and what assurance AFP management had that those risks were acceptable;
  • requested the development and presentation of a Performance Statements Preparation Plan, including a timeline, to demonstrate the assurance process supporting the preparation of the performance statements; and
  • enquired as to the knowledge and understanding of management representatives of their role and responsibility for assuring information contained within the performance statements.

3.47 The audit committee was scheduled to review and recommend the signing of the annual performance statements at the 6 September 2016 meeting through a formal minute to the Commissioner. At the time of that meeting the management certification of the annual performance statements by the National Managers and Executive Leadership Committee had not occurred.

3.48 The draft annual performance statements were provided to the audit committee for review as part of the papers distributed in advance of the 6 September 2016 meeting. The Chair advised members that any significant change to the content or process of the annual performance statements provided to the Committee would be distributed to members out of session for information. The minuted outcome from the 6 September 2016 meeting was that the Audit Committee endorsed the process without specific reference to the appropriateness of the performance reporting, which would not meet the requirements of the audit committee’s charter discussed at paragraph 3.45 above. The signed recommendation to the Commissioner from the Chair of the Audit Committee could not be located for ANAO review. This prevents assessment against the PGPA Rule and its intent.

Department of Agriculture and Water Resources

3.49 Agriculture’s audit committee charter sets out the committee’s functions in regard to performance reporting in accordance with the PGPA Rule as follows:

review and give independent advice and assurance about the appropriateness of the department’s:

b. performance reporting – including the framework for developing, measuring and reporting key performance indicators and the department’s annual performance statement.

3.50 The Audit Committee received quarterly briefings during 2015–16 on the process for the preparation of the 2015–16 Annual Performance Statements. In addition, to address its responsibility under the PGPA Rule and its charter, the audit committee:

  • reviewed the framework for the selection of performance measures;
  • commissioned an internal audit (which was completed in March 2016) on the Key Performance Indicators (performance criteria) and Performance Reporting, to inform the committee’s review responsibilities, and monitored the implementation of key recommendations of the audit;
  • reviewed the proposed process for completing the performance statement, including the assurance mechanisms (sign-offs); and
  • noted in March 2017 the department’s review of the performance reporting framework which included the department’s proposed next steps for the 2017–18 PBS and corporate plan, and 2016–17 Annual Performance Statements.

3.51 The committee provided a report to the Secretary on 8 September 2016, advising ‘that the performance framework and assurances adequately supported management’s recommendation’ that the Secretary sign the 2015–16 Performance Statements Certificate. This limited advice does not meet the requirements of the charter discussed at paragraph 3.49 above, which requires the committee to provide assurance about the appropriateness of the department’s performance reporting. As the department’s charter reflects the requirements and intent of the PGPA Rule, these framework requirements have not been met.

Summary

3.52 There would be benefit in the selected entities further considering the role of the audit committee as a source of independent assurance to the accountable authority and how their audit committee charters and processes establish a basis to provide this assurance. This function would necessarily involve the committee forming a view on how the entity should measure its performance. An audit committee’s charter, and any certification by the audit committee to an accountable authority discharging their performance reporting function, should reflect this requirement.

Key learning 9. Audit committee’s roles and responsibilities

The PGPA Rule sets out that an audit committee’s function must include reviewing the appropriateness of an accountable authority’s performance reporting. This function would necessarily involve the committee forming a view on how the entity should measure its performance. As a result, an audit committee’s charter, and any certification by the audit committee to an accountable authority discharging their performance reporting function, should reflect this requirement.

Comparison to the financial statements assurance process

3.53 As noted in paragraph 3.31 of this audit, the certification required by an accountable authority of their performance statements is equivalent to that of the financial statements. Both entities’ audit committees noted that ‘reporting of performance information is not necessarily subject to the same established framework of internal controls or accounting standards which apply in respect of the annual financial statements’. As the entities’ non-financial performance reporting frameworks mature, management structures and systems of internal control for performance statements can be expected to mirror those of the financial statements. In addition, audit committee charters and certifications should reflect the requirements of the PGPA Rule and clearly enunciate the level of assurance provided to accountable authorities.

3.54 Chief Financial Officers have responsibility for the preparation, and joint certification with the accountable authority, of the financial statements. To meet this responsibility, there is an expectation that the Chief Financial Officer has the relevant knowledge, skills and experience of accounting to prepare the financial statements. Similar expectations were not evident within the performance statements certification processes. In addition, audit committee charters did not highlight that there must be a mix of relevant skills and experience, including performance measurement and reporting, within the committee.52

3.55 Agriculture’s Audit Committee charter requires all members to be conversant with financial management reporting, and that at least one member of the committee should have accounting or related financial management experience and/or qualifications, and a comprehensive understanding of accounting and auditing standards. The AFP’s Audit Committee charter also includes a requirement that at least one member possess these skills. There are no similar requirements for performance reporting in either entity’s charter for the audit committee.

3.56 Agriculture has no single internal unit dedicated to continuous development and monitoring of organisation-wide performance, but has established a working group (as noted in paragraph 3.3). The AFP has a section dedicated to monitoring organisational performance and is staffed by personnel with relevant skills and experience in performance measurement.

Have entities maintained records to support the results and analysis included in the annual performance statements?

Records were largely available and supported the results and analysis sections reported in the annual performance statements for both entities. The AFP maintained complete and accurate records for all but one measure. Agriculture was unable to provide complete records for one performance criterion, and relied on an absence of advice as confirmation of compliance for another. Both entities could further improve record-keeping to demonstrate the calculations and analysis applied to raw data to produce results, and to support analysis in the annual performance statements.

3.57 Section 37 of the PGPA Act sets out the requirement for Commonwealth entities to keep records that properly record and explain the entity’s non-financial performance. Box 6 provides the specific requirements.

Box 6: Section 37(1) of the PGPA Act

37 Records about performance of Commonwealth entities

(1) The accountable authority of a Commonwealth entity must cause records to be kept that properly record and explain the entity’s performance in achieving its purposes.

(2) The accountable authority must ensure that the records are kept in a way that:

a) complies with any requirements prescribed by the rules; and

b) enables the preparation of the annual performance statements required by section 39.

(3) The responsible Minister and the Finance Minister are entitled to full and free access to the records kept under this section. However, those Ministers’ access is subject to any Commonwealth law that prohibits disclosure of particular information.

3.58 The ANAO reviewed documentation made available by both entities to record and explain the selected performance criteria results and analysis presented in the annual performance statements.

3.59 Records were available to support the majority of results reported in the performance statements of both entities. The ANAO’s review, discussed below, indicates that detailing the methods used for calculations and analysis based on datasets, would assist to properly explain the entities’ performance, maintain records integrity and ensure consistent methodologies are applied in future reporting periods.

Australian Federal Police

3.60 The AFP collated and maintained all results centrally by a team responsible for organisational performance.

3.61 The ANAO identified inconsistencies in the underlying dataset for the results of only one of the nine performance criteria reported in the annual performance statements. The calculation and workings applied to the raw data to achieve the reported result were also not retained. These inconsistencies did not lead to a material difference in the overall result reported by the AFP.

3.62 Excluding the performance criterion noted above, and noting matters raised in paragraph 3.48 in regard to maintaining records of endorsements, the AFP maintained adequate records to support results reported in the annual performance statements.

3.63 The ANAO’s observations of the AFP’s record-keeping supporting the annual performance statements are summarised in Table 3.1 below.

Table 3.1: ANAO assessment of AFP record-keeping

Performance criteria

Result

Are there records to support the reported result?

Level of external client/stakeholder satisfaction (% of clients satisfied or very satisfied).

92 per cent

Yes. The final survey report was kept on file and able to be provided for audit.

Percentage of cases before the court that result in a conviction.

95 per cent

Yes. The final dataset was maintained on an internal system and was able to be provided for audit.

Percentage of counter-terrorism investigations that result in a prosecution, disruption or intelligence referral outcome.

100 per cent

Yes. The final dataset was maintained on an internal system and was able to be provided for audit.

Level of community confidence in the contribution of the AFP to aviation law enforcement and security (percentage of aviation network users satisfied or very satisfied).

78 per cent

Yes. The final survey report was kept on file and able to be provided for audit.

Response to aviation law enforcement and/or security incidents within priority response times.

Range depending on Priority Rating

Yes. The final dataset was maintained on an internal system and was able to be provided for audit.

Number of avoidable incidents per 5,000 Protection hours.

Nil

Yes. The final dataset was maintained on an internal system and was able to be provided for audit.

Return on investment for investigation of transnational crime.

5.0

Yes. The final dataset was maintained on an internal system and was able to be provided for audit.

Assets restrained.

$96.5 million

Yes. The final dataset was maintained on an internal system and was able to be provided for audit.

Increased or reinforced cyber safety and security awareness (percentage of surveyed sample indicating increased awareness or reinforced awareness after delivery of presentations).

94 per cent

No. Original dataset retained by the program area did not align to the advice provided to the Organisational Performance team or the result reported in the AFP’s 2015–16 Annual Performance Statements.

Analysis also does not include impact of incomplete responses or assess the potential bias presented by low response rates.a

     

Note a: The total questionnaire responses reported in the 2015–16 Annual Performance Statement was 814, compared to participant numbers of more than 10 000 parents, carers and teachers. This reflects an approximate response rate of 8 per cent.

Source: ANAO analysis of AFP records.

Department of Agriculture and Water Resources

3.64 Agriculture’s record-keeping was dispersed, with relevant records retained by the responsible areas within the department.

3.65 Agriculture records were available to support the majority of results reported in the performance statements. This ranged from regular internal reports, external reporting and analysis of existing data and a selection of case studies. Most performance criteria reviewed as part of the audit were based on information that was publicly available and therefore subject to public scrutiny. This information was often provided by professional sources, such as the Australian Bureau of Statistics (ABS) and Australian Bureau of Agricultural and Resource Economics and Sciences (ABARES).

3.66 As noted in paragraph 3.19, Agriculture relied on different data sources to those identified in internal documentation to produce the results presented in the annual performance statements for two performance criteria. In addition, the results of three criteria did not fully reflect the original criterion due to the absence of available data or analysis.53 In these cases, the ANAO has assessed the completeness and accuracy of records used to support the result as reported in the annual performance statements.

3.67 Agriculture was unable to provide complete records to support the result reported for marine resources against the performance criterion ‘Access to water, land, forest and marine resources for primary production is maintained or improved’. Records for the reported result of access to land and forest resources were publicly available.54

3.68 For the performance criterion ‘100% of allocated funding under the Research and Development for Profit program expended in accordance with the agreed timetable’, the funding details of the overall program are reported on the department’s website. This includes the total approved spending per project for each round. It does not detail the agreed timetables or expenditure breakdown against these timeframes. The department was unable to provide further evidence to support the reported result of ‘projects progressing in accordance with agreed timeframes’.

3.69 The result reported against the performance criteria ‘100% of rural research and development corporations are compliant with statutory and contractual requirements’, noted that ‘Pending formal notification for 2015–16 of material compliance, no research and development corporation has advised of any events [of non-compliance]’. As the result is based on an absence of advice, the ANAO has concluded only on the basis that no evidence that this statement is not accurate came to our attention.

3.70 For the remaining performance criteria, the relevant records were able to be obtained from the responsible areas within the department. To enable consistent record-keeping across responsible areas, guidance outlining procedures for producing and storing records should be considered. The establishment of a dedicated repository may assist in this.

3.71 The ANAO’s observations of Agriculture’s record-keeping supporting the annual performance statements are summarised in Table 3.2 below.

Table 3.2: ANAO assessment of Agriculture record-keeping

Performance criteria

Result

Are there records to support the reported result?

Portfolio industries record an increase in productivity.

Over the period 1995–96 to 2015–16, the average annual productivity growth for the agriculture, fishing and forestry sector was 0.98%. This exceeds average annual market sector productivity growth of 0.36% over the same period.

Yes. The raw data is available on the ABS website. The rationale and calculations were documented by ABARES.

Rate of return on capital invested across portfolio industries is maintained or increased.

Over the period 1995–96 to 2015–16, the annual trend growth for rate of return on capital was 0.01% for broadacre and dairy farms.

Yes. The ABARES farm survey results are available on the department’s website. The rationale and calculations were documented by ABARES.

Rate of profit for producers and businesses is maintained or increased.

Over the period 1995–96 to 2015–16, the annual trend growth in profit was 2.69 per cent for broadacre and dairy farms.

Yes. The ABARES farm survey results are available on the department’s website. The rationale and calculations were documented by ABARES.

Access to water, land, forest and marine resources for primary production is maintained or improved.

Land and forest resource

The total area reported as agriculture and forestry land use declined marginally by 1.5% between 2005–06 and 2010–11.

There was a decline in access to the native public forest resources from 1995–96 to 2010–11. After a sustained period of growth in plantations over several decades, the area of plantations has declined marginally in recent years.

Fisheries resources

The area of the marine environment available for fisheries production remained the same, however some areas for specific fisheries were open or closed to achieve fisheries management outcomes.

In 2015–16 all fisheries managed solely by the Commonwealth were operating with approval under Part 13A of the Environment Protection and Biodiversity Conservation Act 1999.

Partial.

The raw land use data, satellite imagery and forestry data are publicly available.

The records supporting the ‘fisheries resources’ result could not be located for ANAO review.

Improved availability of safe, efficient and effective agricultural and veterinary chemicals.

The establishment of a Collaborative Forum for determining priority chemical needs has been completed.

The first round of a grants program to provide funds supporting new chemical uses was fully subscribed. A second round has been approved and is ready to start.

Projects to develop official crop groupings and move uses from temporary permits onto permanent labels, are ongoing.

Yes. The information is available on the department’s website.

Investment in rural research and development corporation programs demonstrates positive returns.

Past public investments are estimated to have generated average rates of return that could be as high as 28% for research and development, and 47% for extension.

Yes. A publicly available ABARES study was used.

100% of allocated funding under the Research and Development for Profit program expended in accordance with the agreed timetable.

The program is a $190.5 million competitive grants program from 2014–15 to 2021–22 to fund the rural research and development corporations for collaborative projects to improve farm-gate profitability and productivity.

Nearly $79 million of funding has been approved, with $26.7 million for 12 projects under round 1 and $52.2 million for 17 projects under round 2.

Funding agreements for all projects are in place, with projects progressing in accordance with agreed timeframes.

Partial.

The funding details are reported on the department’s website. This includes the total approved spending. It does not detail the timetable, agreed expenditure breakdown per project or dates. The department was unable to provide further evidence to support the reported result.

100% of rural research and development corporations are compliant with statutory and contractual requirements.

Under their funding agreements research and development corporations provide annual compliance audit reports and certification reports.

All research and development corporations advised that they were materially compliant with their obligations for 2014–15.

All research and development corporations are obliged to notify the department of any significant events that may materially impact their ability to comply with their funding agreements. Pending formal notification for 2015–16 of material compliance, no research and development corporation has advised of any events.

Partial.

The reported result relies on notifications by research and development corporations, facilitated by twice yearly meetings with the department, and annual declarations.

The ANAO can only conclude that nothing came to our attention during the course of the audit that would lead us to believe that an instance of non-compliance was reported by a research and development corporation and not disclosed by Agriculture.

High level of efficiency in collecting and distributing levies to fund research and development in research and development corporations.

Cost-recovery charges in 2015–16 represented less than 1.2% of levies disbursed.

Yes. The information is available in the department’s biannual reports to levies stakeholders.

Inspections of levy agent records cover at least 30% of levy revenue.

Met.

Yes. The reported results are based on data drawn from internal operating systems (SARA and DAISY referred to in paragraph 3.27). The results are monitored through monthly reports.

The information is available in the department’s biannual reports to levies stakeholders.

Less than 5% of quota allocations are rejected because of quota certification failures.

All quotas were allocated in accordance with relevant legislation.

No consignments were rejected because of quota certification failures.

Yes. The information is available on the department’s website and is updated regularly.

     

Source: ANAO analysis of Agriculture records.

Key learnings 10. Record-keeping

When compiling the data to be used for reporting on results against performance criteria in the annual performance statements, entities should consider:

  • establishing a dedicated repository to ensure appropriate access to supporting records for future years; and
  • documenting and retaining records detailing the methods used for calculations and analysis of raw data to reach a reported result.

Appendices

Appendix 1 Entity responses

Department of Finance

Australian Federal Police

Department of Agriculture and Water Resources

ANAO Comments

1. The Audit Committee’s Charter required the committee to review and provide independent advice and assurance about the appropriateness of the department’s performance reporting (see paragraph 3.49). The ANAO concluded that the advice provided to the Accountable Authority by the Audit Committee did not provide assurance about the appropriateness of the department’s performance reporting (see paragraph 3.51).

2. The Department of Finance has acknowledged the need for further clarity in guidance to audit committees (see paragraph 33).

Appendix 2 Summary of the selected entities’ outcomes and programmes

1. The following tables summarise the 2015–16 outcomes and programmes of the AFP and Agriculture.

Table A.1: AFP’s 2015–16 outcomes and related programmes

Outcomes

Related programmes

Outcome 1

Reduced criminal and security threats to Australia’s collective economic and societal interests through co-operative policing services.

Programme 1.1: Federal Policing and National Security

Programme 1.2: International Police Assistance

Outcome 2

A safe and secure environment through policing activities on behalf of the Australian Capital Territory Government.

Programme 2.2: ACT Community Policing

   

Source: Australian Federal Police Portfolio Budget Statements 2015–16.

Table A.2: Agriculture’s 2015–16 outcomes and related programs

Outcomes

Related programmes

Outcome 1

More sustainable, productive, internationally competitive and profitable Australian agricultural, food and fibre industries through policies and initiatives that promote better resource management practices, innovation, self-reliance and improved access to international markets.

Programme 1.1: Agriculture adaptation

Programme 1.2: Sustainable Management – Natural Resources

Programme 1.3: Forestry Industry

Programme 1.4: Fishing Industry

Programme 1.5: Horticulture Industry

Programme 1.6: Wool Industry

Programme 1.7: Grains Industry

Programme 1.8: Dairy Industry

Programme 1.9: Meat and Livestock Industry

Programme 1.10: Agricultural Resources

Programme 1.11: Drought Programmes

Programme 1.12: Rural Programmes

Programme 1.13: International Market Access

Outcome 2

Safeguard Australia’s animal and plant health status to maintain overseas markets and protect the economy and environment from the impact of exotic pests and diseases, through risk assessment, inspection and certification and the implementation of emergency response arrangements for Australian agricultural, food and fibre industries.

Programme 2.1: Biosecurity and Export Services

Programme 2.2: Plant and Animal Health

Outcome 3

Improve the health of rivers and freshwater ecosystems and water use efficiency through implementing water reforms, and ensuring enhanced sustainability, efficiency and productivity in the management and use of water resources.

Programme 3.1: Water Reform

   

Source: Department of Agriculture and Water Resources Portfolio Additional Estimates Statements 2015–16.

Appendix 3 Criteria for the assessment of the relevance, reliability and completeness of performance information

2. The following criteria have been adapted from the evaluation of the appropriateness of key performance indicators as depicted in ANAO Report No.21 2013–14 Pilot Project to Audit Key Performance Indicators. The criteria have been updated to reflect Finance’s guidance to support the enhanced Commonwealth performance framework.

Table A.3: Criteria for the assessment of the relevance, reliability and completeness of performance information

 

Criteria

Characteristics

Explanation

Individual assessment

Relevant

A relevant performance criterion assists users’ decision making in regard to an entity’s progress in fulfilling its purpose.

Benefit

The performance criterion clearly indicates who will benefit and how they will benefit from the entity’s activities.

The performance criterion should explain who will benefit from the activity and how the recipient will benefit.

Focus

The performance criterion should address a significant aspect/s of the purpose, via the activities.

The performance criterion should assist significantly in informing whether the purpose is being achieved.

Understandable

The performance criterion should provide sufficient information in a clear and concise manner.

The performance criterion should be stated in plain English and signal the impacts of activities to inform users.

Reliable

A reliable performance criterion allows for reasonably consistent assessment of an entity’s progress in fulfilling its purpose.

Measurable

The performance criterion should use information sources and methodologies that are fit for purpose.

The performance criterion should be capable of being measured to demonstrate the progress of fulfilling the purpose. This includes documenting a basis or baseline for measurement or assessment, for example a target or benchmark.

Free from Bias

The performance criterion should be free from bias and where possible, benchmarked against similar activities.

The performance criterion should allow for clear interpretation of results and provide an unbiased basis for assessment.

Overall assessment

Complete

Performance criteria allow for the overall assessment of an entity’s progress in fulfilling its purpose to inform users’ decision making.

Balanced

The performance criteria should provide a balanced examination of the overall performance story.

The performance criteria should reflect a balance of measurement types (effectiveness and efficiency), bases (quantitative and qualitative) and timeframes (short, medium and long-term).

Collective

The performance criteria should collectively address the purpose.

The performance criteria should demonstrate the extent of achievement against the purpose.

       

Source: Adapted from ANAO Report No.21 2013–14 Pilot Project to Audit Key Performance Indicators, incorporating Department of Finance Resource Management Guides No: 130; 131; 132; and 134.

Appendix 4 Assessment of entities’ performance criteria relevance and reliability

1. The following tables details the results of the ANAO’s assessment of the relevance and reliability of the Australian Federal Police, and the Department of Agriculture and Water Resource’s performance criteria reviewed as part of this audit.

Table A.4: Assessment of AFP’s performance criteria relevance and reliability

Activities

AFP performance criteria

Relevant

Reliable

ANAO Comment

The ANAO could not determine whether this criterion related to a single activity, multiple activities, or all.

Level of external client/stakeholder satisfaction (percentage of clients satisfied or very satisfied).

Partly

Yes

Relevant

Client/stakeholder is not defined to determine the basis for assessing satisfaction and the link to the activity.

The performance criterion does not make clear who will benefit and how.

Reliable

The criterion met the characteristics of measurable and free from bias.

Federal policing

Percentage of cases before court that result in conviction.

Yes

Yes

Relevant

The criterion met the characteristics of benefit, focus and understandable.

Reliable

The criterion met the characteristics of measurable and free from bias.

Positive return on investment for investigation of transnational crime.

Yes

Yes

Relevant

The criterion met the characteristics of benefit, focus and understandable.

Reliable

The criterion met the characteristics of measurable and free from bias.

Assets restrained.

Yes

Mostly

Relevant

The criterion met the characteristics of benefit, focus and understandable.

Reliable

The variability of the measurement outcomes leads to a potential for biased decision making.

Increased or reinforced cyber safety and security awareness (percentage of surveyed sample indicating increased awareness or reinforced awareness after delivery of presentations).

Yes

Yes

Relevant

The criterion met the characteristics of benefit, focus and understandable.

Reliable

The criterion met the characteristics of measurable and free from bias.

National security

Percentage of counter-terrorism investigations that result in a prosecution, disruption or intelligence referral outcome.

Yes

No

Relevant

The criterion met the characteristics of benefit, focus and understandable.

Reliable

The criterion contains an inherent bias. The definition is too broad, capturing all outcomes, and leads to biased reporting.

Level of community confidence in the contribution of the AFP to aviation law enforcement and security (percentage of aviation network users satisfied or very satisfied).

Yes

Yes

Relevant

The criterion met the characteristics of benefit, focus and understandable.

Reliable

The criterion met the characteristics of measurable and free from bias.

Response to aviation law enforcement and/or security incidents within priority response times.

Yes

Yes

Relevant

The criterion met the characteristics of benefit, focus and understandable.

Reliable

The criterion met the characteristics of measurable and free from bias.

Number of avoidable incidents per 5 000 Protection hours.

Yes

Yes

Relevant

The criterion met the characteristics of benefit, focus and understandable.

Reliable

The criterion met the characteristics of measurable and free from bias.

         

Source: ANAO analysis.

Table A.5: Assessment of Agriculture’s performance criteria relevance and reliability

Activities

Performance Measure

Relevant

Reliable

ANAO Comment

Provide analysis and advice to the government and external decision-makers on support for portfolio industries to make them more productive, globally competitive and profitable.

 

Portfolio industries record an increase in productivity.

Mostly

Mostly

Relevant

The impact measured by the performance criterion is not directly attributable to the activity as described– the provision of analysis and advice would be expected to lead to further activities that would influence the outcome more directly.

Reliable

The method and parameters that will be used to collect data and measure performance are not disclosed to enable assessment by the reader.

Rate of return on capital invested across portfolio industries is maintained or increased.

Mostly

Mostly

Relevant

The impact measured by the performance criterion is not directly attributable to the activity as described– the provision of analysis and advice would be expected to lead to further activities that would influence the outcome more directly.

Reliable

The method and parameters that will be used to collect data and measure performance are not disclosed to enable assessment by the reader.

Rate of profit for producers and businesses is maintained or increased.

Mostly

Mostly

Relevant

The impact measured by the performance criterion is not directly attributable to the activity as described– the provision of analysis and advice would be expected to lead to further activities that would influence the outcome more directly.

Reliable

The method and parameters used to collect data and measure performance are not disclosed to enable assessment by the reader.

Access to water, land, forest and marine resources for primary production is maintained or improved.

Mostly

Mostly

Relevant

The impact measured by the performance criterion is not directly attributable to the activity as described– the provision of analysis and advice would be expected to lead to further activities that would influence the outcome more directly.

Reliable

The method and parameters that will be used to collect data and measure performance are not disclosed to enable assessment by the reader.

Administer, improve and implement reforms in industry regulations in accordance with the government’s agenda.

Improved availability of safe, efficient and effective agricultural and veterinary chemicals.

Mostly

Mostly

Relevant

The corporate plan does not state that this is a shared responsibility. There is scope to improve the measure by distinguishing between the role of the-department and that of the regulator (APVMA).

While the ANAO linked the measure to a key activity, it was not easily identifiable to the reader without background knowledge.

Reliable

The method and parameters that will be used to collect data and measure performance are not disclosed to enable assessment by the reader.

Deliver Agricultural Competitiveness White Paper initiatives including measures that seek a smarter approach to farming based on a strong research and development system.

Support industry bodies and research and development corporations to meet statutory and contractual requirements and objectives.

Investment in rural research and development corporation programs demonstrates positive returns.

Mostly

Mostly

Relevant

The performance criterion does not make clear who will benefit and how – what are the positive returns?

Reliable

The method and parameters that will be used to collect data and measure performance are not disclosed to enable assessment by the reader.

100% of allocated funding under the Research and Development for Profit program expended in accordance with the agreed timetable.

Partly

Mostly

Relevant

The performance criterion does not make clear who will benefit and how.

The performance criterion is not easily understandable to the reader. Further explanation of the Research and Development for Profit program is required.

Reliable

The method and parameters that will be used to collect data and measure performance are not disclosed to enable assessment by the reader.

100% of rural research and development corporations are compliant with statutory and contractual requirements.

Partly

Mostly

Relevant

While linked to a key activity, this is an indicator of the necessary contract management rather than the performance of R&D activities. The outcome is beyond the department’s control and it is not obvious how this will increase farm-gate returns.

The performance criterion does not make clear who will benefit and how.

Reliable

The method and parameters that will be used to collect data and measure performance are not disclosed to enable assessment by the reader.

Provide levies collection and distribution services to industry and government.

High level of efficiency in collecting and distributing levies to fund research and development in research and development corporations.

Mostly

Mostly

Relevant

The performance criterion does not make clear who will benefit and how.

Reliable

The method and parameters that will be used to collect data and measure performance are not disclosed to enable assessment by the reader.

Inspections of levy agent records cover at least 30% of levy revenue.

Partly

Mostly

Relevant

The performance criterion does not make clear who will benefit and how.

The performance criterion is not easily understandable.

Reliable

The method and parameters that will be used to collect data and measure performance are not disclosed to enable assessment by the reader.

The ANAO could not determine how this criterion linked to activities listed for the strategic objective or purpose.

Less than 5% of quota allocations are rejected because of quota certification failures.

Partly

Mostly

Relevant

The performance criterion does not make clear who will benefit and how.

The performance criterion is not easily understandable.

Reliable

The method and parameters that will be used to collect data and measure performance are not disclosed to enable assessment by the reader.

         

Source: ANAO analysis.

Appendix 5 Summary of entity’s sources, collection arrangements, measurement frequency and limitations of data for performance reporting

1. The following tables summarise the sources, collection arrangements, measurement frequency and limitations of data used by the Australian Federal Police, and the Department of Agriculture and Water Resources for reporting against selected performance criteria in the annual performance statements.

Table A.6: AFP performance criteria data sources, collection arrangements, measurement frequency and limitations

Performance criteria

Data sources, collection arrangements, measurement frequency and limitations identified

Level of external client/stakeholder satisfaction (% of clients satisfied or very satisfied).

Source and collection: AFP Business Satisfaction Survey through an online survey of AFP nominated clients/stakeholders.

Frequency: Annual (internal and external)

Limitations: Composition of stakeholder population, focus of survey questions and response rate.

Percentage of cases before the court that result in a conviction.

Source and collection: National Cases cubea (PROMISb) through the PROMIS data entry.

Frequency: Bi-monthly (internal)

Limitations: Reliance on the completeness and accuracy of PROMIS data

Percentage of counter-terrorism investigations that result in a prosecution, disruption or intelligence referral outcome.

Source and collection: National Cases cube (PROMIS) and Counter Terrorism area manual categorisation through PROMIS data entry and direct liaison with Counter Terrorism area

Frequency: Bi-monthly (internal)

Limitations: Reliance on the completeness and accuracy of PROMIS data and categorisation of Counter Terrorism outcomes by Counter Terrorism area

Level of community confidence in the contribution of the AFP to aviation law enforcement and security (percentage of aviation network users satisfied or very satisfied).

Source and collection: Airport Consumer Confidence Survey through a face to face survey (external party).

Frequency: Annual

Limitations: Sample size and quotas per airport

Response to aviation law enforcement and/or security incidents within priority response times.

Source and collection: PROMIS through the Computer aided dispatch (CAD)c system entry (uploaded daily to PROMIS)

Frequency: Monthly (internal)

Limitations: Reliance on the completeness and accuracy of CAD data

Number of avoidable incidents per 5,000 Protection hours.

Source and collection: Case Notes National Cube (PROMIS) through the PROMIS data entry.

Frequency: Bi-monthly (internal)

Limitations: Reliance on the completeness and accuracy of PROMIS data

Return on investment for investigation of transnational crime.

Source and collection: Drug Harm Index and Estimated Financial Return methodologies and time recording based on PROMIS data entry.

Frequency: Bi-monthly (internal)

Limitations: Reliance on the completeness and accuracy of PROMIS data and up to date supporting methodology

Assets restrained.

Source and collection: Proceeds of Crime Cube (PROMIS) through the PROMIS data entry.

Frequency: Monthly (internal)

Limitations: Reliance on the completeness and accuracy of PROMIS data

Increased or reinforced cyber safety and security awareness (percentage of surveyed sample indicating increased awareness or reinforced awareness after delivery of presentations).

Source and collection: External questionnaire respondents to a hard copy questionnaire.

Frequency: Annual (internal and external)

Limitations: Response rate

Note a: ‘Cube’ refers to reports generated by the AFP, built via a Statistical Analysis System (SAS) tool, for the purpose of extracting specific data fields from management systems to support internal and external reporting.

Note b: PROMIS (Police Realtime Online Management Information System) is the key case management information system used by the AFP.

Note c: Computer Aided Dispatch (CAD) is a system developed to record the AFP’s response to reported incidents. This includes recording the time the incident was reported, who was dispatched in response and when, and how the incident was resolved.

Source: ANAO analysis.

2. To examine Agriculture’s assessment approach for 2015–16, the ANAO determined the data sources, measurement frequency and limitations identified by the department. Information provided in the 2015–16 Annual Performance Statements, the department’s data collection summaries and advice from the department is summarised in Table A.7 below.

Table A.7: Agriculture performance criteria data sources, collection arrangements, measurement frequency and limitations

Performance criteria

Data sources, collection arrangements, measurement frequency and limitations identified

Portfolio industries record an increase in productivity.

Source and collection: ABS data cube produced from statistical surveys and government administrative records

Frequency: Annual

Limitations: Initially excluded forestry and fisheries however these were later included

Rate of return on capital invested across portfolio industries is maintained or increased.

Source and collection: ABARES broadacre and dairy surveys

Frequency: Annual

Limitations: Excludes some portfolio industries

Rate of profit for producers and businesses is maintained or increased.

Source and collection: ABARES broadacre and dairy surveys

Frequency: Annual

Limitations: Excludes some portfolio industries

Access to water, land, forest and marine resources for primary production is maintained or improved.

Source and collection: National scale land use data, obtained through the ABS Agricultural census and satellite imagery; advice from state/territory agencies reported in Australia’s State of the Forests; and advice from the fisheries line area

Frequency: 5 yearly (annual for fisheries)

Limitations: No limitations- well established database

Improved availability of safe, efficient and effective agricultural and veterinary chemicals.

Source and collection: Advice from line areas

Frequency: Variable

Limitations: This is a shared performance measure with the APVMA. Care needs to be taken in identifying metrics that focus more on the department’s contribution rather than on APVMA performance (which will be reported in their own performance statement/annual report).

Investment in rural research and development corporation programs demonstrates positive returns.

Source and collection: Publicly available ABARES study using data from various sources including ABS, ABARES farm surveys and state/territory departments

Frequency: Single publication based on longitudinal data from 1952–53 to 2006–07

Limitations: Initial concerns about variable and possibly inconsistent methodologies for assessing return on investment by the different RDCs, however these were addressed by using the ABARES study instead of individual RDC evaluations

100% of allocated funding under the Research and Development for Profit program expended in accordance with the agreed timetable.

Source and collection: Advice from program managers

Frequency: Regular, public updates on program implementation

Limitations: No limitations- information verifiable against published sources

100% of rural research and development corporations are compliant with statutory and contractual requirements.

Source and collection: Records of meetings with RDCs, compliance audits and certification reports

Frequency: Twice yearly meetings and annual reports

Limitations: No limitations- obligations are specified in contractual and regulatory obligations on RDCs

High level of efficiency in collecting and distributing levies to fund research and development in research and development corporations.

Source and collection: Annual departmental Report to Levy Stakeholders, based on data from levy management system (Phoenix)

Frequency: Entered continuously, reported annually

Limitations: No limitations- well established methodology for calculating the metric

Inspections of levy agent records cover at least 30% of levy revenue.

Source and collection: Business data from levy management system (Phoenix), provided by the responsible division

Frequency: Annual as part of a 3 year rolling program

Limitations: No limitations

Less than 5% of quota allocations are rejected because of quota certification failures.

Source and collection: Quota administration statistics and reporting systems (SARA and Daisy) and information from the Quota Unit (exporter reporting)

Frequency: Annual

Limitations: No limitations

Source: ANAO summary of the 2015–16 Annual Performance Statements, the department’s data collection summaries and advice from the department.

Appendix 6 Assessment of entities’ methods for data collection and collation of results

1. The following tables summarise the ANAO’s observations of the data collection methodologies used by the Australian Federal Police, and the Department of Agriculture and Water Resources, including collation of results, for reporting against selected performance criteria in the annual performance statements.

Table A.8: Review of AFP’s methods for data collection and collation of results

Performance criteria

Methodology/approach

ANAO Comments

Level of external client/stakeholder satisfaction (% of clients satisfied or very satisfied).

Annual Business Satisfaction Survey, administered online by specialised research company.

Stakeholders to be surveyed are nominated by the AFP based on engagements with the AFP in the previous 6 months.

Survey achieved a response rate of 48 per cent (2014–15: 48 per cent) based on AFP nominated stakeholders.

The survey results were accurately collated for presentation in the performance statements.

Percentage of cases before the court that result in a conviction.

Based on dataset comparing cases resulting in a conviction or proven without conviction, to all cases reaching court.

Extracted from PROMIS via corporate reporting by Organisational Performance team.

Dataset used met the definitions set out in KPI Rulebook for presentation in the performance statements.

Percentage of counter-terrorism investigations that result in a prosecution, disruption or intelligence referral outcome.

Based on dataset of finalised counter-terrorism investigations which are then classified to the relevant outcome by the Counter-Terrorism area.

Extracted from PROMIS via corporate reporting.

Outcomes are broad, leading to all investigations classified as meeting the definition and 100 per cent result.

Dataset used met the definitions set out in KPI Rulebook and evidence of Counter Terrorism area assignment of outcomes included.

Level of community confidence in the contribution of the AFP to aviation law enforcement and security (percentage of aviation network users satisfied or very satisfied).

Airport Consumer Confidence Survey administered face to face by specialised research company.

Sample proportioned amongst capital city airports based on allocation of passenger numbers.

Representativeness of result within +/- 2.5 per cent.

The survey results were accurately collated for presentation in the performance statement.

Response to aviation law enforcement and/or security incidents within priority response times.

Reports responses to reported priority incidents within CAD system and uploaded to PROMIS daily.

Extracted from PROMIS via corporate reporting by Organisational Performance team.

Dataset used met the definitions set out in KPI Rulebook.

Required amendments to data resulting from review were made and recorded appropriately.

Number of avoidable incidents per 5,000 Protection hours.

Number of incidents reported via case notes, which are reviewed and then compared against time attributed to Protection operations.

Extracted from PROMIS and time recording system via corporate reporting by Organisational Performance team.

There were no avoidable incidents recorded in 2015–16. Data fields used to confirm result met the definitions set out in KPI Rulebook.

Return on investment for investigation of transnational crime.

Methodology considers the benefit of AFP drug and financial crime investigations versus the cost of completing those investigations.

The benefit is determined by two measures – the Drug Harm Index and Estimated Financial Return.

Both were developed by the AFP and augmented by the research of the University of Queensland Institute for Social Science Research in 2008 and 2009 respectively.

The cost is an estimate of a number of factors including: staff costs attributable to cases, legal proceedings and prison costs.

Draws on PROMIS datasets extracted via corporate reporting and collated via macros in a spreadsheet by Organisational Performance team.

The Drug Harm Index and Estimated Financial Return measures have not been externally reviewed since 2008 and 2009 respectively.

In 2013, an internal review of selected inputs, including purity and cost data was completed.

The AFP should consider the continuing appropriateness of inputs and underlying assumptions supporting the methodology.

Datasets used from PROMIS met the KPI Rulebook definitions and the methodology was consistently applied by the macros in the spreadsheet.

Assets restrained.

Based on dataset compiling the number of seizures and confirmed value of assets restrained during 2015–16.

Extracted from PROMIS via corporate reporting by Organisational Performance team.

Year to year variability in this measure is expected, acknowledged and disclosed by the AFP.

Dataset used met the definitions set out in KPI Rulebook.

Increased or reinforced cyber safety and security awareness (percentage of surveyed sample indicating increased awareness or reinforced awareness after delivery of presentations).

Based on questionnaire responses submitted via post from presentation participants which are then manually entered into a spreadsheet.

Responses to two questions are combined to determine an overall satisfaction percentage.

Responses were collected and input into spreadsheet in accordance with Standard Operating Procedures.

No evidence of analysis to determine the impact of incomplete responses on the reliability of the result.

Response rate equivalent to 8.1 per cent (814 responses from more than 10,000a participants) is too low to allow for meaningful analysis.

Note a: Participants consisted of “more than 10,000 parents, carers and teachers” (AFP 2015–16 Annual Performance Statements, p. 33).

Source: ANAO analysis.

Table A.9: Review of Agriculture’s methods for data collection and collation of results

Performance criteria

Methodology/approach

ANAO Comments

Portfolio industries record an increase in productivity.

Data analytics method developed by ABARES (using a projection for 2015–16 and final estimates for the preceding years) to calculate trend growth in productivity over a 20 year period.

The timeframe analysed was extended from 10 (as per initial planning) to 20 years.

A comparison with competitor countries was planned but not conducted. The rationale for this was not documented.

Fisheries and forestry data was also used, while original planning included only agricultural data.

Rate of return on capital invested across portfolio industries is maintained or increased.

Data analytics method developed by ABARES (using final estimates for the years to 2013–14, a preliminary estimate for 2014–15 and a projection for 2015–16). Rate of return calculated based on broadacre and dairy data over a 20 year period.

The timeframe analysed was extended from 10 (as per initial planning) to 20 years, using data for broadacrea and dairy.

Fisheries data was not included.

Rate of profit for producers and businesses is maintained or increased.

Data analytics method developed by ABARES (using final estimates for the years to 2013–14, a preliminary estimate for 2014–15 and a projection for 2015–16). Rate of profit calculated based on broadacre and dairy data over a 20 year period.

The timeframe analysed was extended from 10 (as per initial planning) to 20 years, using data for broadacre and dairy.

Fisheries data was not included.

Access to water, land, forest and marine resources for primary production is maintained or improved.

Method developed by ABARES to simplify and derive results from land use data via a three-year process. Forestry results sourced from a public report; fisheries data based on approvals for fisheries.

Results were estimated using established sources and methodologies identified in the planning.

Access to water (agricultural water use) was not reported.

Improved availability of safe, efficient and effective agricultural and veterinary chemicals.

Attribution of three types of activities conducted by the department intended to increase market access by improving chemical availability.

Original planning considered the number of new chemicals listed during the year. The data was not available at the time of reporting.

Reporting was based on information available on the department’s website and focussed on the:

  • establishment of a Collaborative Forum; and
  • progress of a grants program and two projects to streamline chemical registration requirements.

Investment in rural research and development corporation programs demonstrates positive returns.

Longitudinal study conducted by ABARES, based on established economic models, supported by advice from program managers.

Analysis of end project evaluations was originally intended to inform the reported result.

When concerns were raised about possibly inconsistent methodologies used by different RDCs to assess return on investment, a long-term economic study was used instead.

100% of allocated funding under the Research and Development for Profit program expended in accordance with the agreed timetable.

Advice from program managers based on existing externally available information

The approach to tracking this information was not provided to the ANAO. The department has advised that expenditure is closely monitored by the relevant First Assistant Secretary.

100% of rural research and development corporations are compliant with statutory and contractual requirements.

Meetings with RDCs, and review of compliance audits and certification reports for 2014–15.

Result was based on the absence of notifications of non-compliance by the 15 RDCs for 2015–16.

Opportunities for self-reporting are provided at biannual meetings with the department and mandatory annual certification reports.

Certification reports and compliance audit reports are not required to be provided until five months after the end of each RDC’s financial year. As a result, these reports are not necessarily available before the department’s mandatory annual report publication deadline.

High level of efficiency in collecting and distributing levies to fund research and development in research and development corporations.

A costing methodology has been developed. Activity and costing data is extracted from several internal systems, including the levies management information system, Phoenix. This data is used to support the derivation of results.

Data is manually extracted from several internal systems and calculations are performed using spreadsheets that support the costing methodology.

There is potential to automate this process to reduce the risk of errors and improve efficiency.

Inspections of levy agent records cover at least 30% of levy revenue.

Phoenix stores data relating to levy returns, receipts and disbursements and compliance management. This data is extracted from Phoenix and processed through an established method.

Data is manually extracted and calculations are performed using spreadsheets.

There is potential to automate this process to reduce the risk of errors and improve efficiency.

Less than 5% of quota allocations are rejected because of quota certification failures.

Quota allocations and certifications are managed through several internal systems. The two key systems are SARA and Daisy.

Data is extracted from SARA and Daisy and a processed through a developed method. The process is similar to the department’s approach to regular quota management reporting.

Data is manually extracted and calculations are performed using spreadsheets. This is to ensure that data is presented in the required format.

Although data on quota allocations and certifications is available, the department has identified a misalignment with the measure and its business operations. This misalignment may lead to an unintended view on performance.

The department has made adjustments to the measure.

     

Note a: The Australian broadacre sector includes industries for wheat and other crops, and livestock such as sheep and cattle. Department of Agriculture and Water Resources, Australian farm survey results 2013-14 to 2015-16, April 2016, p. 8.

Source: ANAO analysis.

Footnotes

1 A Portfolio Budget Statement is produced for every appropriation bill where a Commonwealth entity within a portfolio is appropriated an amount by the Parliament. Department of Finance, Resource Management Guide No. 134: Annual Performance Statements, July 2016, p. 3.

2 The Finance Secretary Direction, issued on 24 February 2016, sets out that entities’ 2016–17 Portfolio Budget Statements must include at least one high level performance criterion for existing programs, and all performance criteria for new, or materially changed existing programs.

3 An accountable authority for a Commonwealth entity is generally the person or group of persons that has responsibility for, and control over, the entity’s operations. Sub-section 12(2) of the PGPA Act sets out the person(s) or body that is the accountably authority of a Commonwealth entity.

4 Department of Finance presentation and discussion, Audit Committee Chairs Forum, 7 June 2017.

5 Joint Committee of Public Accounts and Audit, Report 457 Development of the Commonwealth Performance Framework— Second Report, p. 4.

6 Department of Finance, Resource Management Guide No. 130: Overview of the enhanced Commonwealth performance framework, July 2016, p. 2.

7 A Portfolio Budget Statement is produced for every appropriation bill where a Commonwealth entity within a portfolio is appropriated an amount by the Parliament. Department of Finance, Resource Management Guide No.134: Annual Performance Statements, July 2016, p. 3.

8 Department of Finance, Resource Management Guide No. 134: Annual performance statements for Commonwealth entities, July 2016, p. 4.

9 Department of Finance, Resource Management Guide No. 130: Overview of the Enhanced Commonwealth Performance Framework, April 2015, p. 8.

10 PGPA Act 2013, Sections 101–104.

11 Joint Committee of Public Accounts and Audit, Report 457 Development of the Commonwealth Performance Framework— Second Report, p. 4.

12 The ANAO previously noted in Audit Report No.21 2013–14 Pilot Project to Audit Key Performance Indicators that Finance guidance supporting entity performance measurement and reporting was maintained in multiple sources, rather than consolidated into a single reference document. For more information refer to p. 51 of ANAO Audit Report No.21 2013–14.

13 27 & 28 March 2017 in Canberra, 3 April 2017 in Melbourne and 5 April 2017 in Sydney.

14 JCPAA, Report 453 Development of the Commonwealth Performance Framework, December 2015, pp. 54–55.

15 ibid., p. 56.

16 ibid., p. 57.

17 ibid., p. 57.

18 Finance guidance states that the timeframe for the tabling of an entity’s annual report may vary according to the legislation the entity is subject to. For example, Commonwealth entities that report on a financial year basis are required to present their annual report to the responsible Minister by 15 October in each reporting period. Other entities may have annual report tabling requirements in their enabling legislation.

19 The JCPAA identified the following cross entity audit priorities of the 44th Parliament for 2016–17: Procurement; Performance Statement Implementation by Commonwealth Entities; Corporate Plans of Commonwealth Entities; and the Management of Risks by Commonwealth Entities. See Joint Committee of Public Accounts and Audit, Report 459: Annual Report 2015–16, October 2016, p. 12.

20 Both entities presented strategic objectives as a subset of their purpose in the performance statements as outlined in Tables 1.1 and 1.2.

21 For an overview of the Australian Federal Police, and proposed future audit coverage, refer to the ANAO’s Annual Audit Work Program.

22 For an overview of the Department of Agriculture and Water Resources, and proposed future audit coverage, refer to the ANAO’s Annual Audit Work Program.

23 Agriculture’s 2016–17 Corporate Plan covers the required period 2016–20.

24 Department of Finance, Resource Management Guide No. 132: Corporate plans for Commonwealth entities, April 2015, p. 10.

25 ANAO Report No.6 2016–17 Corporate Planning in the Australian Public Sector (p. 44) noted that future Finance guidance should clarify the requirements relating to the presentation of the purposes of an entity and reporting on each reporting period covered by the corporate plan.

26 ANAO Report No.6 2016–17 Corporate Planning in the Australian Public Sector, p. 33.

27 Department of Finance, Quick Reference Guide No. 131: Developing Good Performance Information, April 2015.

28 ANAO Audit Report No.21 2013–14 Pilot Project to Audit Key Performance Indicators, p. 41.

29 ibid., p. 41.

30 Department of Finance, Quick Reference Guide No. 131: Developing Good Performance Information, April 2015.

31 ANAO Audit Report No.21 2013–14 Pilot Project to Audit Key Performance Indicators, p. 41.

32 For example, improved availability of safe, efficient and effective agricultural and veterinary chemicals, is a shared responsibility with the Australian Pesticides and Veterinary Medicines Authority.

33 As noted in paragraph 2.39, none of Agriculture’s performance criteria disclosed the method and parameters (targets, time frames or baselines) that would be used to collect data and measure performance in the corporate plan. The timeframes set out above were determined by the ANAO through examination of the results reported in the annual performance statements.

34 Agriculture’s reporting under the Regulator Performance Framework was not considered as part of this audit.

35 The Organisational Performance team now sits within the Capability Development Office following an organisational restructure in 2016–17.

36 Department of Finance, Resource Management Guide No. 130: Overview of the enhanced Commonwealth Performance Framework, issued in April 2015, p. 5.

37 The AFP advised the ANAO that the review included a literature review, and a consideration of changing drug groupings, purity, consumption, social costs and precursors. The AFP also advised that the original developer of the methodology from the University of Queensland Institute for Social Science Research was consulted during this review.

38 814 responses from “more than 10,000 parents, carers and teachers” (AFP 2015–16 Annual Report, p. 33).

39 Data transformation refers to applying processes to raw data in order to present it in a way that is fit for purpose. For example, ordering fields to allow for the upload of data extracted from a system to a data warehouse.

40 Police Realtime Online Management Information System (PROMIS) is the key case management system used by the AFP.

41 Computer Aided Dispatch (CAD) is a system developed to record the AFP’s response to reported incidents. This includes recording the time the incident was reported, who was dispatched in response and when, and how the incident was resolved.

42 ANAO Report No.30 2015–16 Management of the Use of Force Regime, pp. 50–53, noted issues related to the completeness of information recorded in, and subsequently extracted from, PROMIS in support of reporting use of force incidents to internal stakeholders.

43 Phoenix supports the collection and disbursement of levies imposed by Commonwealth legislation on a range of commodities and products such as meat, grains and horticultural products.

44 EXport DOCumentation (EXDOC) supports the production of documentation for the export of prescribed goods.

45 Quota Statistics and Report Administration (SARA & DAISY) supports the management of beef and dairy product export quotas for overseas markets.

46 E5 is Agriculture’s financial management information system.

47 Data cleansing programs are used to identify inconsistencies in raw data. For example, duplicate records.

48 ANAO Report No.33 2016–17 Audits of the Financial Statements of Australian Government Entities for the Period Ended 30 June 2016 includes commentary on the governance arrangements for the audit committees of a sample of material entities, including their composition, operational arrangements and the administrative support they receive.

49 PGPA Act, Section 5.

50 PGPA Rule, Section 17.

51 ANAO comment: the function relates to sub-section 17(2)(b) of the PGPA Rule.

52 Subsection 17(3) of the PGPA Rule states ‘The audit committee must consist of at least 3 persons who have appropriate qualifications, knowledge, skills or experience to assist the committee to perform its functions.’

53 Refer to Appendix 6 for further detail on the data collection and methodologies applied by Agriculture.

54 As noted in Appendix 6, a result for access to water resources was not reported by Agriculture in the annual performance statements.

Related documents: