The objective of the audit was to examine the effectiveness of Defence’s quarterly performance report as a mechanism to inform senior stakeholders about risks and issues in the delivery of capability to the Australian Defence Force.

Summary and recommendations

Background

1. The Department of Defence’s Capability Acquisition and Sustainment Group is responsible for purchasing and maintaining military equipment and supplies in the quantities and to the service levels that are required by Defence and approved by government. Defence’s forecast expenditure for 2018–19 is $8.6 billion for acquisition projects and $6.0 billion for sustainment products.

2. The Capability Acquisition and Sustainment Group prepares a Quarterly Performance Report focusing on key major acquisition projects and key sustainment products, with the purpose of providing:

senior stakeholders within Government and the Department of Defence with a clear and timely understanding of emerging risks and issues in the delivery of capability to our Australian Defence Force end-users. These risks and issues are highlighted so that stakeholders can respond in a coordinated manner to guide the conduct of remediation actions.1

3. Internal Defence guidance on the Quarterly Performance Report further states that:

In keeping with the primary goal of the QPR, the focus remains on highlighting Projects and Products of Interest. By identifying and remediating issues early on in the Capability Life Cycle the QPR aims to prevent the capability from becoming a Project of Concern (PoC).

4. The Capability Acquisition and Sustainment Group’s Business Framework states that:

the Deputy Secretary of Capability Acquisition and Sustainment Group confirms business assurance through the Quarterly Performance Report provided to the Minister, Secretary of Defence and Chief of the Defence Force.

5. The report provides performance information for approximately 30 per cent of the total number of projects and products being managed by the Capability Acquisition and Sustainment Group — representing approximately 70 per cent of the Major Capital Equipment budget and 70 per cent of the sustainment program budget in Defence. The report is largely compiled manually using data sourced primarily from Defence’s key information technology systems for reporting on acquisition projects and sustainment products, including the Sustainment Performance Management System. Additional data is sourced from the areas responsible for the management of projects and products in Defence, and Independent Assurance Reviews.2

6. The report is approved by the Deputy Secretary, Capability Acquisition and Sustainment and provided to the Secretary and Chief of the Defence Force (CDF) for endorsement. The Secretary and CDF provide the report to the Ministers for Defence and Defence Industry. The report is also provided to the Defence Investment Committee for noting and feedback. The Defence Investment Committee includes representatives from the Department of the Prime Minister and Cabinet and the Department of Finance.

Rationale for undertaking the audit

7. The Quarterly Performance Report is an area of interest to the Parliament. Quality performance reporting supports the accountable authority and those charged with governance in their oversight of an entity.3 Regular reporting on performance to senior management also provides incentives for local line management to improve performance.

8. The Quarterly Performance Report is prepared to facilitate oversight by Defence senior leaders, help identify projects and products that require additional senior leadership attention and escalation, and support informed decision making. The report is also provided to Defence Ministers and key external stakeholders — the Department of the Prime Minister and Cabinet and Department of Finance in their capacity as members of the Defence Investment Committee — to support their understanding of Defence’s administration and their own decision-making.

Audit objective and criteria

9. The objective of the audit was to examine the effectiveness of Defence’s Quarterly Performance Report as a mechanism to inform senior stakeholders on the status, risks and issues in the delivery of capability to the Australian Defence Force.

10. The high-level audit criteria were:

  • The Quarterly Performance Report provides senior stakeholders with accurate and timely information about status and emerging risks and issues.
  • The Quarterly Performance Report provides senior stakeholders with information they value, as an aid to monitoring status, risks and issues and to inform decision-making.

Conclusion

11. The June 2018 Quarterly Performance Report reviewed by the Australian National Audit Office was largely effective in providing Defence senior stakeholders, within the agreed benchmark timeframe, with relevant and mostly accurate information on the status, risks and issues associated with Defence’s: key acquisition projects, Top 30 sustainment products, Projects of Interest or Concern, and Products of Interest or Concern.

12. Quarterly Performance Reports for December 2017 to December 2018 were provided to senior stakeholders within the two month timeframe agreed by the Minister. The timeliness of reporting improved in 2018 when compared to average delivery times achieved in 2017. The information reported was mostly accurate. Factors affecting the accuracy of the report were the quality of data in Defence information technology systems, and when the results of other sources of performance information — such as Independent Assurance Reviews — were not appropriately reflected. While Defence has made improvements to the Sustainment Performance Management System, which informs sustainment reporting in the Quarterly Performance Report, it has not yet fully realised the commitment that was made to Parliament in August 2017 that it would implement an improved reporting system which addresses data quality in the report.

13. Feedback from senior stakeholders indicates that the Quarterly Performance Report provides them with information they value. The impact of the report on decision-making is unclear, as there is limited Defence documentation indicating that information contained in the report has prompted action of some type by recipients, and other indirect impacts, such as incentives on line management to improve performance due to the need to report, are difficult to assess. Defence has sought feedback on the report from recipients and incrementally improved the report over the past two years, based on that feedback.

Supporting findings

Timeliness and accuracy of information

14. The June 2018 Quarterly Performance Report reviewed by the ANAO was provided to senior stakeholders and Ministers within agreed benchmark timeframes. Since December 2017, all Quarterly Performance Reports have been provided to Ministers within the agreed two month timeframe and the timeliness of reporting improved in 2018 (taking an average of 48 days for the report to be provided to the Minister after quarter-end) when compared to 2017 (when it took an average of 71 days). Over the last two years, information in the Quarterly Performance Report has been between 44 days (1.5 months) and 107 days (3.5 months) old at the time of reporting to Ministers.

15. The ANAO’s review of the June 2018 Quarterly Performance Report indicated that the information reported for 26 of the 27 acquisition projects reviewed and one of the eight sustainment products reviewed was accurate. For one acquisition project and seven of the eight sustainment products examined, there were one or more issues with the accuracy of the information reported in the June 2018 report. In particular, accuracy was affected where reporting on risks and issues did not reflect risks raised in contemporary Independent Assurance Reviews, the Sustainment Performance Reporting System monthly reports and other key internal documents. The ANAO also observed shortcomings in the accuracy and quality of performance information in Defence’s information technology systems. While Defence has made improvements to the Sustainment Performance Management System, it has not yet fully realised the commitment made to Parliament in August 2017 that it would implement an improved reporting system which addresses data quality in the Quarterly Performance Report.

16. There is evidence of review activity by senior line management and the central production team during the compilation of the Quarterly Performance Report. The ANAO identified inconsistences between the Quarterly Performance Report and other Defence reporting in its sample, as well as delays in reporting on emerging risks. These findings indicate that while Defence has taken steps to implement the recommendation made in Auditor-General Report No.2 2017–18 Defence’s Management of Materiel Sustainment, the cross referencing process envisaged as part of the quality assurance process is not yet functioning effectively.

Feedback from senior stakeholders

17. Defence has sought feedback on the Quarterly Performance Report from stakeholders on the format and content of the report. Feedback provided on various Quarterly Performance Reports by the Defence Investment Committee indicates there are opportunities to add to the value of the existing report. In their responses to the ANAO’s survey of report recipients, senior stakeholders indicated that the report is valued.

18. The purpose of the Quarterly Performance Report is to provide senior stakeholders with a clear and timely understanding of emerging risks and issues for projects and products to allow a coordinated approach to remediating these risks and issues. The value of the report can be improved by including in the report: trend performance data for sustainment products; and information on emerging candidates for Defence’s Projects of Concern regime and Projects and Products of Interest list. To assure senior stakeholders that emerging risks are being addressed, the report should include information on those projects and products recommended as candidates by Independent Assurance Reviews and those under active consideration by senior Defence management.

19. The impact of the report on decision-making is unclear, as there is limited Defence documentation indicating that information contained in the report has prompted action of some type by recipients. Indirect impacts, such as incentives in line management to improve performance due to the need to report, are difficult to assess. Available evidence indicates that the Defence Investment Committee has requested further information on project and product status, costs, budget and risks. Stakeholder feedback has been used to incrementally improve the report over the past two years.

Department of Defence’s response

20. The proposed report was provided to the Department of Defence. The Department’s summary response is below and its full response is at Appendix 1.

Defence agrees with the recommendation presented in the audit report on Defence’s Quarterly Performance Report on acquisition and sustainment. Defence is committed to continuously improve the readability and utility of the Quarterly Performance Report.

The Sustainment Performance Management System was implemented by the end of 2017 and all products were reporting in the system by mid-2018. Improvements have been made in the management of the sustainment activities as a result of the Sustainment Performance Management System. Defence considers that the commitment to Parliament in 2017 to implement an improved reporting system has been met and the processes to assure data quality issues in the Quarterly Performance Report will continue to be strengthened. Defence is committed to a culture of collaboration and values transparency in reporting. These positive behaviours are well recognised since the implementation of reform following the First Principles Review.

Recommendations

Recommendation no.1

Paragraph 3.18

Defence improve the Quarterly Performance Report as a tool for senior leaders by reporting on:

  1. trend performance data for sustainment products; and
  2. emerging candidates for the Projects/Products of Concern list and Products/Projects of Interest list that have been recommended by an Independent Assurance Review or which are under active consideration by senior management.

Department of Defence response: Agreed.

Key messages from this audit for all Australian Government entities

Below is a summary of key messages, including instances of good practice, which have been identified in this audit and may be relevant for the operations of other Australian Government entities.

Governance

Timeliness and quality of performance information to support decision-making

1. Background

1.1 The Department of Defence’s Capability Acquisition and Sustainment Group (CASG) is responsible for purchasing and maintaining military equipment and supplies in the quantities and to the service levels that are required by Defence and approved by government.4 Defence’s forecast expenditure for 2018–19 is $8.6 billion for acquisition projects, and $6.0 billion for sustainment products. As at 16 May 2019, CASG was managing 204 major and minor acquisition projects and 112 sustainment products.

The Quarterly Performance Report

1.2 The Capability Acquisition and Sustainment Group prepares a Quarterly Performance Report focusing on key major acquisition projects and key sustainment products, with the purpose of providing:

senior stakeholders within Government and the Department of Defence with a clear and timely understanding of emerging risks and issues in the delivery of capability to our Australian Defence Force end-users. These risks and issues are highlighted so that stakeholders can respond in a coordinated manner to guide the conduct of remediation actions.5

1.3 Internal guidance on the Quarterly Performance Report further states that:

In keeping with the primary goal of the QPR, the focus remains on highlighting Projects and Products of Interest. By identifying and remediating issues early on in the Capability Life Cycle the QPR aims to prevent the capability from becoming a Project of Concern (PoC).6

1.4 The Capability Acquisition and Sustainment Group’s Business Framework states that:

the Deputy Secretary of Capability Acquisition and Sustainment Group confirms business assurance through the Quarterly Performance Report provided to the Minister, Secretary of Defence and Chief of the Defence Force.

1.5 The Quarterly Performance Report includes information about CASG’s performance in delivering acquisition projects and sustainment products costing approximately $15 billion (2018–19 forecast expenditure).7

Process for preparing the report

1.6 Figure 1.1 outlines the Capability Acquisition and Sustainment Group’s performance reporting cycle to measure its progress against its business plan and delivery of service to its customers.

Figure 1.1: Capability Acquisition and Sustainment Group’s performance reporting cycle

Figure 1.1 outlines the Capability Acquisition and Sustainment Group’s performance reporting cycle to measure its progress against its business plan and delivery of service to its customers.

Source: Adapted from the Capability Acquisition and Sustainment Group Business Framework, 25 July 2018.

1.7 The Quarterly Performance Report is prepared by CASG’s Program Approvals and Agreements Directorate. Defence has not determined the overall cost of producing the report (each year or each quarter). Defence advised that production of the report involves:

Three staff members make up the Strategic Reporting team and include an EL1, APS 6 and APS 5 who primarily work on the compilation of QPR, including the Projects of Concern reporting. Their responsibilities are spread across other functions of the team, dependent on work peaks and troughs. Additionally, the Director of the team will spend a portion of time on the compilation of the QPR.

Outside of the team, there is the Major Program Control team of around four who review the draft report and provide input, and one staff member (on rotation) will assist with the QPR analysis (for part of one day usually).

It is harder to quantify outside of the directorate as there are the project and product management staff and CASG Division management, as well as the Defence Finance Group staff who would work on both an ‘as usual’ and ‘as required’ basis. GBM [Group Business Manager] CASG and DEPSEC [Deputy Secretary] CASG and their staff work on the ‘compilation’ during the clearance process.

1.8 The Directorate compiles the Quarterly Performance Report manually using data primarily sourced from Defence’s Monthly Reporting System8 and the Sustainment Performance Management System.9 Additional data is sourced from Capability Acquisition and Sustainment Group’s Systems Program Offices (responsible for the management of sustainment products), Project Offices (responsible for the management of acquisition projects), and Independent Assurance Reviews. Information on projects and products is cleared through Division Heads in the Capability Acquisition and Sustainment Group. The high level process and data sources for the Quarterly Performance Report are outlined in Figure 1.2 (on the following page).

1.9 The report is approved by the Deputy Secretary, Capability Acquisition and Sustainment and provided to the Secretary and Chief of the Defence Force (CDF) for endorsement.

Report distribution

1.10 The Secretary and CDF provide the report to the Ministers for Defence and Defence Industry. The report is also provided to the Defence Investment Committee for noting and feedback. The Defence Investment Committee includes representatives from the Department of the Prime Minister and Cabinet and the Department of Finance.10

1.11 The report is also provided for information to the Australian Industry Capability team, the Capability Acquisition and Sustainment Group Statutory Reporting team, and the Program Integration and Interoperability Office in Joint Systems Division.11 In addition, the following committees12 within Defence also receive a copy of the Quarterly Performance Report, or an Executive Summary for information:

  • Defence Committee;
  • Defence Audit and Risk Committee;
  • Capability Acquisition and Sustainment Group Executive Advisory Committee; and
  • Capability Acquisition and Sustainment Group Risk Advisory Committee.

Figure 1.2: High level process and data sources for the Quarterly Performance Report

flow chart showing High level process and data sources for the Quarterly Performance Report

Note: In June 2018, the Defence Investment Committee’s role in the process changed. Prior to June 2018, the Committee cleared the Quarterly Performance Report. The Committee’s role is now to note and provide feedback on the report.

Source: ANAO analysis of Defence information.

What is reported in the Quarterly Performance Report?

1.12 The structure of the report has been similar across the eight reports prepared for the quarters ending March 2017 to December 2018. Each report has included information about:

  • Projects of Concern — a status update and broad description on the progress of remediation activities.
  • Acquisition projects — there are two sets of reports:
    • For each of the key acquisition projects (that is, the ‘Top 30’ acquisition projects as set out in the department’s Portfolio Budget Statements as well as projects appearing in the annual Major Projects Report) the analysis includes comments and performance information about the project’s capability, schedule and cost and ‘traffic light’ ratings based on quantitative data.13 The report also includes a comment on relevant Australian Industry Capability.
    • For each acquisition Project of Interest, the analysis provides: a project overview; summary of risks to capability, schedule and costs; ‘traffic lights’ reflecting a qualitative assessment of these risks based on criteria; and the remediation strategy for the project.
  • Sustainment products — there are two sets of reports:
    • For each of the Top 30 sustainment products (as set out in the department’s Portfolio Budget Statements) the analysis includes comments, performance information and ‘traffic light’ ratings in terms of availability and expenditure as well as a comment on relevant Australian Industry Capability.
    • For each sustainment Product of Interest, the analysis provides: a product overview; a summary of risks to availability and cost; ‘traffic lights’ reflecting a qualitative assessment of these risks based on criteria; and the remediation strategy for the product.
Coverage of operational output of the Capability Acquisition and Sustainment Group

1.13 The Quarterly Performance Report provides performance information for approximately 30 per cent of projects and products being managed by the Capability Acquisition and Sustainment Group.

1.14 The key acquisition projects and ‘Top 30’ products represent approximately 70 per cent of the Major Capital Equipment budget and 70 per cent of the sustainment program budget as reported in the Quarterly Performance Reports for June, September and December 2018. The number and value of acquisition projects and sustainment products reported on in the June 2018 Quarterly Performance Report, which is the focus of this audit, is outlined in Table 1.1 (on the following page).

1.15 A useful feature for the reader is that there are separate sections designed to draw attention to the projects/products that have been identified as requiring additional management attention due to risks to delivery. These sections include all the projects/products that are ‘of concern’ and ‘of interest’ regardless of whether they are key acquisition projects and ‘Top 30’ sustainment products. As a consequence though, some projects and products may be discussed twice in the report.14

Table 1.1: Number and value of acquisition projects and sustainment products reported on in the June 2018 Quarterly Performance Report

Item

Total number reviewed by Defence in preparing the report

No. reported in the Quarterly Performance Report

Total budget ($billion)a

Value reported on in the Quarterly Performance Report ($billion)

Acquisition projectsb

120

40 (33 per cent of acquisition projects)

104.2

71.9 (69 per cent of acquisition project budget)

Sustainment products

111

30 (27 per cent of sustainment products)

5.6

3.9 (70 per cent of sustainment product budget)

         

Note a: Figures for acquisition projects are the total budget. Figures for sustainment products are the annual budget for 2017–18.

Note b: Post second pass acquisition projects only.

Source: ANAO analysis of Defence information. Capability Acquisition and Sustainment Group June 2018 Quarterly Performance Report.

Previous ANAO and Parliamentary review of the Quarterly Performance Report

ANAO

1.16 Auditor-General Report No.2 2017–18 Defence’s Management of Materiel Sustainment identified issues with the data quality, completeness and timeliness of the Quarterly Performance Report15 and recommended that ‘Defence institutes a risk-based quality assurance process to ensure the accuracy, completeness and relevance of the information included in the Defence Quarterly Performance Report’.16 Defence records indicate that the recommendation was closed on 7 September 2017. Chapter two of this audit report discusses implementation of the recommendation.

Joint Committee of Public Accounts and Audit Report 470: Defence Sustainment Expenditure

1.17 The Parliament’s Joint Committee of Public Accounts and Audit (JCPAA) conducted an inquiry into Auditor-General Report No.2 2017–18 Defence’s Management of Materiel Sustainment. The Deputy Secretary, Capability Acquisition and Sustainment Group advised the Committee that Defence was undertaking the changes necessary to address the ANAO findings and recommendation on the Quarterly Performance Report:

I think the fact that we are, effectively, changing the entire model as we go through … would indicate that I did have concerns about the accuracy, completeness of that report. And that’s why we are changing the model and making sure that model is data drawn directly and that the domain heads who sign that off for me are getting accurate information that’s not filtered through the system, so that we can be assured, I can be assured, the capability manager can be assured and government can be assured that they’re getting timely and accurate information.17

1.18 The Deputy Secretary further advised the Committee that Defence would have a ‘much better reporting system’ by the end of 2017.18 In particular, by:

implementing the SPMS [Sustainment Performance Management] system, so that we can actually get that consistency across, take out the individual interpretation and make sure that it is data driven — that is the whole purpose of moving from an old, effectively manual system into an automated system, which draws directly from the finance data from the Defence system, from the project schedules and from the data that’s actually being referred to.19

1.19 In its report on the inquiry, Report 470: Defence Sustainment Expenditure, the Committee stated that it hoped that:

Defence’s recent IT improvements will lead to a better reporting system, but notes that technology should not be Defence’s sole focus. Defence’s internal reporting regime should be open to the Minister and the Secretary, and report using plain language. Terms like ‘overachievement’ should not be used to describe overspending. More broadly, the culture in the Department of Defence will need to continue to change to reinforce reform across this area.

1.20 Recommendation 5 of the Committee’s report was that:

… the Auditor-General consider reviewing the Department of Defence’s new Monthly Reporting system and documentation including advising whether there have been improvements in reporting to the Minister and Secretary, and that the information provided via reporting is timely and accurate.20

1.21 In response to the recommendation, the Auditor-General wrote to the Committee in September 2018 advising that he would consider expanding the audit outlined in the ANAO’s 2018 Annual Audit Work Program on the Sustainment Performance Management System to assess the governance, use and quality of data within the performance management and reporting systems for both acquisition and sustainment that inform the preparation of the Quarterly Performance Report.

Rationale for undertaking the audit

1.22 The Quarterly Performance Report is an area of interest to the Parliament. Quality performance reporting supports the accountable authority and those charged with governance in their oversight of an entity.21 Regular reporting on performance to senior management also provides incentives for local line management to improve performance.

1.23 The Quarterly Performance Report is prepared to facilitate oversight by Defence senior leaders, help identify projects and products that require additional senior leadership attention and escalation, and to support informed decision making. The report is also provided to Defence Ministers and key external stakeholders — the Department of the Prime Minister and Cabinet and Department of Finance — to support their understanding of Defence’s administration and their own decision-making.

Audit approach

Audit objective, criteria and scope

1.24 The objective of the audit was to examine the effectiveness of Defence’s Quarterly Performance Report as a mechanism to inform senior stakeholders on the status, risks and issues in the delivery of capability to the Australian Defence Force.

1.25 The high-level audit criteria were:

  • The Quarterly Performance Report provides senior stakeholders with accurate and timely information about status and emerging risks and issues.
  • The Quarterly Performance Report provides senior stakeholders with information they value, as an aid to monitoring status, risks and issues and to inform decision-making.

1.26 This audit has focused on the June 2018 Quarterly Performance Report — the most recent report available at the time the Auditor-General decided to conduct this audit. The following reports were also reviewed during the course of fieldwork — March 2017, June 2017, September 2017, December 2017, March 2018, September 2018 and December 2018.

Audit methodology

1.27 The audit method involved:

  • examining and analysing data from Defence’s information technology systems including the Sustainment Performance Management System and the Monthly Reporting System, key reviews and documentation;
  • testing the Quarterly Performance Report process by examining information reported in the June 2018 Quarterly Performance Report for a targeted sample of eight sustainment products22, with a focus on accuracy, completeness and timeliness;
  • drawing on audit work conducted by the ANAO as part of the annual Defence Major Projects Report, to test the accuracy of reporting for the 26 major acquisition projects included in the Major Projects Report;
  • drawing on audit work conducted by the ANAO as part of recent performance audits of acquisition projects and testing the accuracy, completeness and timeliness of reporting for a further acquisition project;
  • analysing the eight responses received as part of an ANAO survey of 17 senior leaders who are routinely provided with the Quarterly Performance Report. The survey was sent to the Secretary of the Department of Defence, the Chief of the Defence Force, Defence Ministers and members of the Defence Investment Committee. The survey is included in Appendix 5; and
  • discussions with Defence personnel involved in the Quarterly Performance Report’s preparation.

1.28 This audit was conducted in accordance with the ANAO Auditing Standards at a cost to the ANAO of approximately $338,000.

1.29 The team members were Natalie Whiteley, Clyde Muthukumaraswamy, Jordan Bastoni, Cherise Reed and Sally Ramsey.

2. Timeliness and accuracy of information

Areas examined

This chapter examines whether the Quarterly Performance Report provides senior stakeholders with accurate and timely information on the status of projects and products, and emerging risks and issues. In particular, whether:

  • the Quarterly Performance Report is provided to senior stakeholders in a timely manner;
  • the information reported on acquisition projects and sustainment products in the Quarterly Performance Report is clear and accurate; and
  • Defence has established arrangements to provide assurance over the quality of information reported.
Conclusion

Quarterly Performance Reports for December 2017 to December 2018 were provided to senior stakeholders within the two month timeframe agreed by the Minister. The timeliness of reporting improved in 2018 when compared to average delivery times achieved in 2017. The information reported was mostly accurate. Factors affecting the accuracy of the report were the quality of data in Defence information technology systems, and when the results of other sources of performance information — such as Independent Assurance Reviews — were not appropriately reflected. While Defence has made improvements to the Sustainment Performance Management System, which underpins the report’s production, it has not yet fully realised the commitment that was made to Parliament in August 2017 that it would implement an improved reporting system which addresses data quality in the report.

Is the Quarterly Performance Report provided to senior stakeholders and Ministers in a timely manner?

The June 2018 Quarterly Performance Report reviewed by the ANAO was provided to senior stakeholders and Ministers within agreed benchmark timeframes. Since December 2017, all Quarterly Performance Reports have been provided to Ministers within the agreed two month timeframe and the timeliness of reporting improved in 2018 (taking an average of 48 days for the report to be provided to the Minister after quarter-end) when compared to 2017 (when it took an average of 71 days). Over the last two years, information in the Quarterly Performance Report has been between 44 days (1.5 months) and 107 days (3.5 months) old at the time of reporting to Ministers.

Preparation of the report

2.1 Each Quarterly Performance Report notes that the next report will be circulated two months after the end of the quarter. The majority of the time taken to prepare the report (approximately six weeks) is focussed on the compilation of the information in the report. This includes: extracting data on acquisition projects and sustainment products from Defence’s Information Technology systems, seeking input from Systems Program Offices and Project Offices including sign-off on this information at the division head level, and collating the information into the format required for the report.

2.2 Figure 2.1 summarises the process and approximate timeframe for compiling the Quarterly Performance Report after the end of a quarter.

Figure 2.1: Process and timeline for Preparation of the Quarterly Performance Report

Flow chart that summarises the process and approximate timeframe for compiling the Quarterly Performance Report after the end of a quarter.

Source: ANAO analysis of Defence documentation.

2.3 Defence advised the ANAO in June 2019 that:

  • the timeframe for the development and preparation of the Quarterly Performance Report is affected by the availability of data from performance reporting systems. For example, cash positions from financial systems are available on the third business day of the following month, and Division Head signoff on project and product information occurs in the middle of the following month;
  • the Monthly Reporting System and the Sustainment Performance Management System are located on different servers, which adds time to the preparation of the Quarterly Performance Report; and
  • the time required to centrally analyse data, conduct quality assurance and compile the draft Quarterly Performance Report is 11 days.

Timeliness of provision to Ministers

2.4 In 2014, the Minister requested that the report be provided within two months of the end of the reporting period. ANAO analysis (as shown in Figure 2.2) indicates that since December 2017, all reports have been provided to Ministers below the agreed benchmark of two months.

Figure 2.2: Number of calendar days taken to submit Quarterly Performance Reports to Ministers from the end of the quarter

 

Note: The dotted line represents the 16 August 2017 hearings of the Joint Committee of Public Accounts and Audit into Defence sustainment expenditure, discussed in paragraphs 1.17 and 2.6 of this audit report.

Source: ANAO analysis of Defence documentation.

2.5 The total days taken to submit the report to Ministers for each of the Quarterly Performance Reports produced over the last two years, from the end of the quarter, ranged from 44 days (1.5 months) to 107 days (3.5 months). Since December 2017 the time taken to submit reports has been between 44 and 54 days — an average of 48 days (1.6 months). This is an improvement on overall performance since 2017, when the average time taken to submit the report was 71 days.

2.6 The ANAO noted in Auditor-General Report No.2 2017–18 Defence’s Management of Materiel Sustainment that, for the Quarterly Performance Report reviewed (June 2017), the report was not timely as an aid to decision-making, as the information in it was nearly two months old by the time it was provided to Ministers. The timeliness of reporting was also noted at the Joint Committee of Public Accounts and Audit hearings in August 2017 into Defence sustainment expenditure, in the course of the Committee’s inquiry into that ANAO report. In that context, Defence committed to improving the timeliness of production.23 24 As shown in Figure 2.2, there has been an improvement in overall performance since late 2017.

2.7 Reporting is still impacted by the quality of data inputs. Since 2017, the Sponsor’s Papers for the Quarterly Performance Report provided to the Investment Committee noted that: ‘Timeliness of CASG [Capability Acquisition and Sustainment Group] reporting is currently characterised by IT [Information Technology] reporting systems that suffer from inconsistency in the quality of the data inputs from across the organisation.’ Available evidence indicates that this issue has not been discussed at the Investment Committee.

2.8 To mitigate the risk that the Secretary is unaware of escalating risks to acquisition projects and sustainment products, the Deputy Secretary, Capability Acquisition and Sustainment Group is expected to provide ‘exception’ reporting on any substantial variations or issues to the Secretary as they arise. Senior management may also access performance data to enable monitoring through separate reports, such as through fleet screenings.

Stakeholder views on the timeliness of the Quarterly Performance Report

2.9 The ANAO surveyed 17 senior stakeholders — who routinely receive the Quarterly Performance Report — on the timeliness of the report. Of the eight responses received, views were mixed as to whether the Quarterly Performance Report highlights risks and issues in a timely manner to support their decision making and whether they rely on the report to inform their decision making:

  • four senior stakeholders indicated that the Quarterly Performance Report always, mostly or sometimes highlights key risks in projects and products in a timely manner to assist decision making;
  • three senior stakeholders indicated that the report never does;
  • one senior stakeholder commented that the time taken to collate the report means that it is not used to assist in making decisions as it is after-the-fact reporting;
  • one senior stakeholder noted that the report covers events that have unfolded in the past, and that it has not been designed to support specific decision making processes; and
  • another senior stakeholder also considered the accuracy of data to be reduced by the time taken to compile the report (meaning information can be out of date).

Is the information included in the Quarterly Performance Report accurate?

The ANAO’s review of the June 2018 Quarterly Performance Report indicated that the information reported for 26 of the 27 acquisition projects reviewed and one of the eight sustainment products reviewed, was accurate. For one acquisition project and seven of the eight sustainment products examined, there were one or more issues with the accuracy of the information reported in the June 2018 report. In particular, accuracy was affected where reporting on risks and issues did not reflect risks raised in contemporary Independent Assurance Reviews, the Sustainment Performance Reporting System monthly reports and other key internal documents. The ANAO also observed shortcomings in the accuracy and quality of performance information in Defence’s information technology systems. While Defence has made improvements to the Sustainment Performance Management System, it has not yet fully realised the commitment made to Parliament in August 2017 that it would implement an improved reporting system which addresses data quality in the Quarterly Performance Report.

2.10 The following sections provide an assessment of the clarity and accuracy of information in the Quarterly Performance Report for acquisition projects and sustainment products.

Reporting on acquisition projects in the Quarterly Performance Report

2.11 To assess the accuracy of the information on acquisition projects in the June 2018 Quarterly Performance Report, the ANAO:

  • Compared the project information in the Quarterly Performance Report (including the project maturity score, capability risks ratings, budget and expenditure information, and key milestone dates) for 26 acquisition projects to information reported in the 201718 Major Projects Report (as at June 2018)25 and key data from the Monthly Reporting System (Defence’s primary performance reporting system for acquisition projects).
    • For two of the 26 projects, the ANAO also compared the results of two recent performance audits26 with the information included in the June 2018 Quarterly Performance Report. This comparison focussed on whether the issues around availability, capability or cost identified in these reports had been identified in the Quarterly Performance Report.
  • Reviewed the accuracy of performance reporting for one additional acquisition project (AIR 5428 Phase 1 Pilot Training System) to spot check reporting for a project not subject to additional scrutiny through the limited assurance review conducted by the ANAO as part of the annual Major Projects Report.

2.12 The ANAO found that the information included by Defence in the June 2018 Quarterly Performance Report was largely aligned with the information reported in the 201718 Major Projects Report (MPR)27 and the budget and schedule data from the Monthly Reporting System. Reporting for the Army’s Protected Mobility Vehicle–Light and the F-35 Joint Strike Fighter was also found to be an accurate representation when compared to recent performance audits. For both projects, the budget identified in the Quarterly Performance Report does not reflect the budget identified in the performance audit reports because the Quarterly Performance Report does not include operating and support costs. These costs can be significant. For example, the Joint Strike Fighter has an overall budget of $15.5 billion with additional operating and support costs of $4.6 billion.28

2.13 The ANAO’s review of the AIR 5428 Phase 1 Pilot Training System project indicated that a number of key risks, including risks to achieving Initial Operating Capability, were not reported in a timely manner through the Quarterly Performance Report. A summary of the results of this analysis is included in the box below.

Case study 1. AIR 5428 Phase 1 Pilot Training System

Under the acquisition project AIR 5428 Phase 1 Pilot Training System, Defence is acquiring civil fixed-wing aircraft, simulators and other training aids including courseware, to provide flight training to Australian Defence Force pilots at a cost of $1.11 billion.

The project became a Project of Interest in December 2017 and a March 2018 Independent Assurance Review recommended it as a candidate for Project of Concern status. As of March 2018, there were a number of issues with the project including schedule slippage with the flight simulator development, the visual system and flight model, and courseware. The management of contractor performance had also became a significant issue for the project with the potential to delay the achievement of Initial Operating Capability. In addition, there were emerging cost increases related to training for qualified instructors and flight simulator remediation. As of December 2018, the project was not on the Projects of Concern list.

The ANAO’s review indicated that a number of key risks for the project were not reported in a timely manner through the Quarterly Performance Report. These included delays in facilities completion, modification of the support contract, and resourcing issues in the project office. CASG’s view was that the AIR 5428 Quarterly Performance Report routinely highlighted key risks and remediation work.a

Note a: Defence advice to the ANAO, June 2019.

Source: ANAO analysis of Defence documentation.

Reporting on sustainment products in the Quarterly Performance Report

2.14 To assess the accuracy of reporting for sustainment products, the ANAO reviewed eight sustainment products from the June 2018 Quarterly Performance Report. The targeted sample covered: a range of domains29; risk ratings; and high, medium and low value products (within the Top 30 list). The list of sustainment products reviewed is included at Appendix 6.

2.15 The ANAO compared information reported for these products in four Quarterly Performance Reports (December 2017 to September 2018) with product data from the Sustainment Performance Management System, Independent Assurance Reviews, Materiel Sustainment Agreements and other reports, reviews and minutes.

2.16 A summary of the results of the ANAO’s review of the sample of eight sustainment products is included at Table 2.1.

Table 2.1: Sustainment product sample analysis summary

Analysis

CA39

CA48

CA59

CAF12

CAF19

CAF21

CN09

CN40

Accuracy and completeness issues

Emerging risks not adequately identified in the QPR

Inconsistencies between QPR and other reporting

Incorrect or missing KPIs in IT systems that populated the QPR

Clarity issues

Inconsistency of traffic light indicators within the QPR

Traffic Light indicators inaccurately represent availability and budget

Lack of clarity on how KPIs are reported in the QPR

                 

Note: The sample was CA 39 — ADF Clothing; CA48 — Multi Role Helicopters; CA59 — Explosive Ordnance Army Munitions; CAF12 — Air Traffic Management; CAF19 — Heavy Air Lift Weapons System; CAF21 — F/A18F Super Hornet Weapon System; CN09 — Armidale Class Patrol Boat; and CN40 — Hobart Class Destroyer.

Source: ANAO analysis of Defence documentation.

2.17 The following sections discuss the detailed results of the ANAO’s review of: the accuracy and completeness of information in the Quarterly Performance Report; and the clarity of information in the report.

Accuracy and completeness of information in the Quarterly Performance Report

2.18 A number of issues regarding accuracy and completeness of data relating to the sample were identified by the ANAO:

  • Emerging risks and issues for the products were not adequately identified in the Quarterly Performance Report — For five of the eight products in the sample, there were a number of emerging risks that were identified in Defence documentation such as Independent Assurance Reviews and Sustainment Performance Management System reports for the products that were not consistently reported in the Quarterly Performance Reports. These included emerging risks with the budget, deficiencies with the procurement contract, and deficiencies with existing Defence policy for managing the product.
  • Data inconsistencies between the Quarterly Performance Report and other reporting documents covering the same period There were inconsistencies between information that appeared in the Quarterly Performance Reports and that which appeared in other reporting documents in two products from the sample. These were:
    • For CA48 — Multi Role Helicopter Weapon System, there was a focus in the four Quarterly Performance Reports on problems with contractor performance affecting aircraft serviceability and sustainment and on the use of additional contracted manpower in pursuit of serviceability targets. Internal Defence advice and an Independent Assurance Review for the product, in September and October 2017 respectively, had raised concerns about a range of workforce issues affecting aircraft serviceability and sustainment, including the structure of the Systems Program Office, the loss of commercial experience and the high turnover of staff.
    • For CA39 — ADF Clothing, the Commonwealth elected in June 2018 to apply liquidated damages to a supplier. This was included in a report to the Deputy Secretary of Capability Acquisition and Sustainment Group, but was not reported on in the June 2018 Quarterly Performance Report.30
  • Incorrect or missing performance indicators in Defence’s information technology systems used to populate the Quarterly Performance Report — for two of the products in the sample, the performance indicators listed in the Materiel Sustainment Agreements (MSA) for the products did not match the performance indicators in the Sustainment Performance Management System. Defence recently reviewed the key performance indicators for sustainment products and identified discrepancies between the performance indicators listed in the MSA Product Schedules and those listed in the Sustainment Performance Management System. The review found a significant number of performance indicators reported on in the system were either not agreed by the Capability Manager or included in Defence records. The review also identified a number of misaligned baselines for the delivery of services.31
Clarity of information in the Quarterly Performance Report

2.19 The following issues regarding clarity of data in the sample were identified by the ANAO:

  • Traffic light indicators can be confusing for the reader when reviewing reports for the same product in different sections of the Quarterly Performance Report — Four of the eight products in the sample have been identified as Products of Interest. For these products, as well as the report in the Top 30 sustainment product section, there is an additional Product of Interest report included in the Quarterly Performance Report, with traffic light indicators for risk to availability and cost. The appendix at the back of each Quarterly Performance Report notes that the availability and cost ratings are determined qualitatively for the Product of Interest report and quantitatively for the Top 30 Sustainment Product report.32 This difference in methodologies means that the traffic light indicators for availability and cost can differ between the Product of Interest report and the Top 30 Sustainment Products report. While these differences in traffic lights do not represent an error, having different ratings with similar headings for the one product in different sections of the report requires the reader to reference the appendix in the report to understand the different ratings and may be confusing for the reader.33
  • Traffic lights can be an inaccurate representation of a product’s availability and budget position — The traffic lights for cost in the Quarterly Performance Report may not be a clear and accurate representation of the product’s budget position. Despite significant budget overspends in 2017–18 for CA59 — Explosive Ordnance Army Munitions, the traffic light for year-end expenditure was rated ‘green’ in the June 2018 Quarterly Performance Report. The rating of ‘green’ was achieved due to changes in the target for the budget in the Sustainment Performance Management System.34 The ANAO’s review of the Investment Committee Member Clearance Comments from June 2017 to December 2018 found instances where committee members had questioned the accuracy of the traffic light ratings for projects and products in the June 2017, December 2017 and September 2018 Quarterly Performance Reports.35 The traffic light for cost also reflects baseline funding for sustainment products only, and does not include operating and support costs.36
  • A lack of clarity for the reader around how performance indicators are reported in the Quarterly Performance Report including underlying targets and evidence — The Quarterly Performance Report includes a performance indicator for availability and two performance indicators for cost (year to date and year end expenditure) for each of the Top 30 sustainment products and Products of Interest. For the eight sustainment products in the sample, the ANAO found a lack of transparency around the data underpinning these performance indicators. The performance indicator for availability is an assessment on the overall availability for performance by the Division head responsible for the product. When the Division head overrides or changes a traffic light rating for a product in the Sustainment Performance Management System, there is a commentary field in the system to document the change. However, there is no transparency in the Quarterly Performance Report and the explanation for a change in a performance indicator would not be apparent to the reader.

2.20 As noted at paragraph 1.19, in Report 470: Defence Sustainment Expenditure, the Joint Committee of Public Accounts and Audit stated that the Quarterly Performance Report should use plain language and avoid terms like ‘overachievement’ to describe overspending. While the ANAO did not identify any instances in the sample of the use of this term, the term ‘overachievement’ is still used in the Quarterly Performance Report. For example, the term was used to describe overspending and failure to meet performance indicators for four products in the December 2018 Quarterly Performance Report.

2.21 The results of the ANAO’s review — of the accuracy and completeness of information in the Quarterly Performance Report, and the clarity of information in the report — as well as the IT issues discussed in paragraph 2.7, indicate that while Defence has continued to make improvements to the Sustainment Performance Management System since 2017, it has not yet fully realised the commitment made to Parliament in August 2017 that it would implement an improved reporting system which addresses data quality in the Quarterly Performance Report.

2.22 As discussed in paragraphs 1.17 to 1.19 of this audit report, Defence advised the Joint Committee of Public Accounts and Audit in August 2017 that Defence would have a ‘much better reporting system’ by the end of 2017, including by implementing the Sustainment Performance Management System. In June 2019, Defence advised the ANAO that it considered that the commitment to Parliament in 2017 to implement an improved reporting system had been met and the processes to assure data quality issues in the Quarterly Performance Report would continue to be strengthened — while acknowledging that a number of data quality issues remain. In particular, Defence indicated that the Sustainment Performance Management System was implemented by the end of 2017 and all products were reporting in the system by mid-2018. Defence also advised the ANAO that it has continued to make Quarterly Performance Report related improvements in the Sustainment Performance Management System, including:

  • Product Exception Report on a Page: provides three months of performance information on all red Product KPIs
  • Product of Concern and Product of Interest Page: provides efficient and effective CASG Division level reporting and signoff
  • Product Performance Trends: increased Product performance visibility can be easily accessed for the previous three months and 12 months respectively
  • Attention Request Close Loop reporting: has been incorporated to better address issues, concerns and or risks highlighted by Product managers to senior executives
  • KPI Remediation Estimate on Product Dashboards: this feature allows Product managers to estimate key performance remediation timeframes enhancing transparency of underperformance and performance remediation projections to senior executives and QPR team.37

2.23 Defence further advised the ANAO, in May 2019, that from June 2019 trend performance data for sustainment products would be automated in the Sustainment Performance Management System.38 In past reports performance indicators were presented as a point-in-time metric, and recording trend data for the products in the Quarterly Performance Report would make emerging risks and issues clearer to the reader. The lack of trend analysis was also identified as an issue in the stakeholder survey conducted for this audit. A senior Defence leader observed that: ‘a key gap for me is understanding trends. Without cross-referencing previous reports there is no ability to know if an amber or red traffic light has been a long standing issue or is a new pop up. Also, to help management know when to intervene there is no projection or prediction of future status provided.’

Has Defence established arrangements to provide assurance over the quality and consistency of information reported?

There is evidence of review activity by senior line management and the central production team during the compilation of the Quarterly Performance Report. The ANAO identified inconsistences between the Quarterly Performance Report and other Defence reporting in its sample, as well as delays in reporting on emerging risks. These findings indicate that while Defence has taken steps to implement the recommendation made in Auditor-General Report No.2 2017–18 Defence’s Management of Materiel Sustainment, the cross referencing process envisaged as part of the quality assurance process is not yet functioning effectively.

2.24 Auditor-General Report No.2 2017–18 Defence’s Management of Materiel Sustainment identified inconsistencies between data included in the Quarterly Performance Report and other Defence documentation.39 The audit report included a recommendation that: ‘Defence institutes a risk-based quality assurance process for the information included in the Defence Quarterly Performance Report.’ Defence agreed to implement the recommendation.40

2.25 Defence informed the ANAO that it implemented this recommendation in September 2017, through the implementation of quality assurance processes including:

  • cross-referring risks and remediation recommendations to Independent Assurance Review outcomes for Projects and Products;
  • independent review by the CASG [Capability Acquisition and Sustainment Group] Major Program Control team to validate the inputs into the Quarterly Performance Report;
  • initiating the shift in focus of QPR [Quarterly Performance Report] to Initial and Final Operational Capability instead of Initial and Final Materiel Release, with the aim of reflecting a holistic capability;
  • automating the data gathering process in order to minimise human errors as much as possible; and
  • consulting with CASG Executives, Defence Groups and Capability Manager Representatives in the development of [the] Quarterly Performance Report.

2.26 Defence included advice to senior stakeholders in the June 2017 Quarterly Performance Report that the recommendation had been fully implemented by 30 June 2017. Defence had developed and endorsed the Management Action Plan for the recommendation on 26 June 2017.41

2.27 Time for quality assurance checking has been allocated into the production process for the Quarterly Performance Report.42 Defence provided evidence indicating that the Strategic Performance Reporting team receives copies of Independent Assurance Reviews for projects and products. These copies are to be used to cross-reference the advice provided in the Quarterly Performance Report. The ANAO also identified instances of Division Heads in CASG being consulted and clearing relevant sections of the report as it is being produced.

2.28 Defence advised that part of the quality assurance process for the Quarterly Performance Report was the cross-referring of risks and remediation recommendations to Independent Assurance Review outcomes for projects and products. As discussed in paragraphs 2.14–2.19 of this audit report, the ANAO identified inconsistences between the Quarterly Performance Report and other Defence reporting in its sample, as well as delays in reporting on emerging risks (for example, risks identified by contemporary review activity such as Independent Assurance Reviews). These findings indicate that the cross-referencing process envisaged as part of the quality assurance program is not yet functioning effectively.

2.29 To automate the data gathering process for the report, Defence introduced a Quarterly Performance Report page in the Sustainment Performance Management System in July 2018. Prior to the development of this page, staff compiling the report would undertake an analysis of information from the Sustainment Performance Management System, manually compile the data and confirm the information with the Systems Program Offices and Division Heads. The ANAO’s review of Defence’s record management system for the December 2018 Quarterly Performance Report indicates that despite changes to the Sustainment Performance Management System, the compilation of the report is still a largely manual process, particularly for information on acquisition projects.43 As previously noted in paragraph 2.7, the Sponsor’s Papers for the Quarterly Performance Report to the Investment Committee note that the timeliness of reporting is impacted by the quality of data from the reporting systems.

3. Feedback from senior stakeholders

Areas examined

This chapter examines whether the Quarterly Performance Report provides senior stakeholders with information they value, as an aid to monitoring the status pf projects and products, risks and issues, and to inform decision-making. In particular:

  • whether Defence has sought feedback from senior stakeholders about the Quarterly Performance Report, to establish whether the information reported is valued; and
  • whether Defence has used feedback from senior stakeholders to improve reporting to better address their needs, and if the report has an impact on decision-making.
Conclusion

Feedback from senior stakeholders indicates that the Quarterly Performance Report provides them with information they value. The impact of the report on decision-making is unclear, as there is limited Defence documentation indicating that information contained in the report has prompted action of some type by recipients, and other indirect impacts, such as incentives on line management to improve performance due to the need to report, are difficult to assess. Defence has sought feedback on the report from recipients and incrementally improved the report over the past two years, based on that feedback.

Areas for improvement

The ANAO has made one recommendation aimed at improving the Quarterly Performance Report as a tool for senior leaders to identify and address emerging problems.

Has Defence sought feedback from senior stakeholders about the Quarterly Performance Report to establish whether the information reported is valued?

Defence has sought feedback on the Quarterly Performance Report from stakeholders on the format and content of the report. Feedback provided on various Quarterly Performance Reports by the Defence Investment Committee indicates there are opportunities to add to the value of the existing report. In their responses to the ANAO’s survey of report recipients, senior stakeholders indicated that the report is valued.

The purpose of the Quarterly Performance Report is to provide senior stakeholders with a clear and timely understanding of emerging risks and issues for projects and products to allow a coordinated approach to remediating these risks and issues. The value of the report can be improved by including: trend performance data for sustainment products; and information on emerging candidates for Defence’s Projects of Concern regime and Projects and Products of Interest list. To assure senior stakeholders that emerging risks are being addressed, the report should include advice about those projects and products recommended as candidates by Independent Assurance Reviews and those under active consideration by senior Defence management.

3.1 The Deputy Secretary, Capability Acquisition and Sustainment has invited recipients of the report to provide feedback on the format and content of the Quarterly Performance Report through the foreword of each report reviewed by the ANAO.

3.2 The ANAO reviewed the available signed briefs returned by Defence Ministers for the last two years. There was no feedback provided on the Quarterly Performance Report from the Ministers during this period. Defence advised the ANAO that the most recent feedback received from the Ministers on the Quarterly Performance Report was in 2014. This was when the Minister for Defence requested that the report be provided within two months of the end of the reporting period.

Survey responses

Senior stakeholders’ feedback on improving the report

3.3 In the ANAO survey of senior stakeholders undertaken as part of this audit, four of the eight respondents indicated awareness that their feedback had been sought for improvements to the Quarterly Performance Report, and these respondents further indicated that this feedback had been incorporated into the report. The remaining four respondents indicated that their feedback had not been sought for improvements to the report, with one of these respondents indicating that the question was not applicable to them. One respondent, who indicated that they had not been consulted for feedback, stated that they were aware of plans to consult them in the future.

Senior stakeholders’ feedback on value of the report

3.4 The majority of respondents to the ANAO survey indicated that it is a valued report, with five respondents advising that: the report was logically structured and easy to read; and information was mostly or always sufficient, accurate and of sufficient quality. Four respondents advised that the report mostly or always provides useful information on the status of acquisition projects and sustainment products.

3.5 Five respondents commented that timeliness affects the usefulness of the report in highlighting key risks or informing senior stakeholders’ decision-making. For example, one respondent commented that the report includes information on the status of projects and products in the past, and that it is not designed to support decisions under the capability life cycle. Another respondent noted that they did not use the document to assist in decision-making as it was ‘reporting after the fact’.

Internal feedback on the report

3.6 The ANAO reviewed the signed minutes returned by the Secretary and Chief of the Defence Force for the last two years. The Secretary and Chief of the Defence Force have not recently provided feedback on the value of the Quarterly Performance Report. There had been a request from the Secretary in 2016 for further advice about an activity in a Quarterly Performance Report.

Defence Investment Committee

3.7 In parallel with the clearance by the Secretary and Chief of the Defence Force prior to Ministerial submission, the report is provided to the Defence Investment Committee44 for noting and feedback (as noted previously in Figure 2.1). The report includes a covering brief that includes some comments45 and asks for feedback.

3.8 The ANAO reviewed the comments and feedback on clearance briefs by the Committee (for Quarterly Performance Reports for quarters ending in June 2017 to December 2018) to assess the feedback provided on the report. Committee feedback was provided on specific projects or products. Examples of the feedback provided includes:

  • corrections to delivery dates of projects;
  • questions about the accuracy of information provided such as traffic light ratings and consequences of budget information reported; and
  • requesting further advice on risks identified by Committee members.

3.9 Feedback from the Committee also indicates that there remains scope for the Capability Acquisition and Sustainment Group to improve its consultation with stakeholders in the development of the report. Issues have been raised internally regarding capability managers receiving adequate time to provide input to the report, and the Investment Committee being provided adequate time to review the report. The current practice is that Committee feedback is incorporated into the next Quarterly Performance Report rather than correcting or amending the current report in light of the Committee’s observations.

3.10 A substantive discussion of the report at a future meeting of the Investment Committee was suggested in February 2019, to assist Defence to respond to the risks identified by the Committee46 in a coordinated manner including to guide the conduct of remediation activities. Defence informed the ANAO in April 2019 that the Deputy Secretary, Capability Acquisition and Sustainment Group would provide an update to the Investment Committee on 17 April 2019. In June 2019 Defence further advised that at the 17 April 2019 meeting, the Deputy Secretary had committed to reviewing the performance reporting framework to design an improved enterprise reporting system for all capability projects.

Defence Audit and Risk Committee

3.11 The Defence Audit and Risk Committee raised an action item at its August 2018 meeting which called for useful information about specific projects covered by the report to be included in future updates to the Committee. This action item was closed at the March 2019 meeting.

Other indicators on value of the report

3.12 Other indicators of the value placed on the Quarterly Performance Report by Defence include the:

  • continuing production of the report and the effort and allocation of resources to do this (see paragraph 1.7);
  • investment in systems and quality assurance mechanisms (paragraphs 2.25 to 2.29);
  • circulation of the report to Defence Ministers and to senior committees including the Investment Committee and the Defence Audit and Risk Committee for information;
  • use of the report by several senior committees to inform discussion on performance;
  • development of other enterprise reports47 based on the Quarterly Performance Report’s approach and format; and
  • agreement to use the report to implement recommendation 2.12 from the First Principles Review. This decision is discussed further in Box 1 below.

Box 1: Implementation of Recommendation 2.12 of the First Principles Review

The First Principles Review envisaged that significant changes would need to be made to Defence’s management information systems to support the timely provision of information.

Recommendation 2.12 was that reporting was to be on ‘the operational output of each of [the Deputy Secretary, Capability Acquisition and Sustainment’s] divisions every quarter and on major contracts on a monthly basis’ as a benchmark. In April 2016 the Deputy Secretary, Capability Acquisition and Sustainment Group proposed to the First Principles Review Implementation Committee that he continue to provide the Secretary with: an improved Group Quarterly Performance Report covering both acquisition programs and sustainment product performance; and ‘exception’ reporting on any substantial variations or issues to the Secretary as they arise as fulfilment of the recommendation. The First Principles Review Implementation Committee agreed to the proposal.

Adding further value to the report

3.13 The Capability Acquisition and Sustainment Group takes steps to add further value to its reporting by providing a brief to Ministers on the key elements of the Quarterly Performance Report. The covering brief identifies key points in the report including an overview of its contents, whether the acquisition projects are being delivered to scope and within cost, and whether the sustainment products are meeting availability and cost targets.

3.14 The brief identifies key risks and issues. For example, the September 2018 Quarterly Performance Report identified the key risk for acquisition projects as schedule performance. In the case of sustainment products it identified workforce availability, availability of platforms and the ageing nature of platforms resulting in long-term sustainment funding pressures. The brief also listed the changes since the last Quarterly Performance Report, with a focus on projects and products added to, or removed from, the Projects of Concern and Projects or Products of Interest lists.

3.15 The reports (and the briefs) reviewed by the ANAO did not include trends in performance data. In addition, emerging candidates for listing as projects/products of interest/concern that have been identified through Independent Assurance Reviews or being actively considered by Capability Acquisition and Sustainment Group senior management are not reported.

3.16 The extent to which the Quarterly Performance Report addresses its purpose48 can be improved by including:

  • trend performance data for sustainment products. Defence advised the ANAO that from June 2019 trend performance data will be automated for sustainment products49, which would support the reporting of trends in sustainment product performance to senior stakeholders; and
  • reporting on emerging candidates for Defence’s Projects of Concern regime and Projects and Products of Interest list. The report should include those projects and products recommended as candidates by Independent Assurance Reviews and those under active consideration by Defence senior management.

3.17 The inclusion of this advice would be particularly useful in relation to projects and products that are not part of the group of materially significant key acquisition projects and Top 30 sustainment products that are routinely reported on in the Quarterly Performance Report.

Recommendation no.1

3.18 Defence improve the Quarterly Performance Report as a tool for senior leaders by reporting on:

  1. trend performance data for sustainment products; and
  2. emerging candidates for the Projects/Products of Concern list and Products/Projects of Interest list that have been recommended by an Independent Assurance Review or which are under active consideration by senior management.

Department of Defence’s response: Agreed.

Has Defence used feedback from stakeholders to improve reporting to better address their needs, and does the report have an impact on decision-making?

The impact of the report on decision-making is unclear, as there is limited Defence documentation indicating that information contained in the report has prompted action of some type by recipients. Indirect impacts, such as incentives in line management to improve performance due to the need to report, are difficult to assess. Available evidence indicates that the Defence Investment Committee has requested further information on project and product status, costs, budget and risks. Stakeholder feedback has been used to incrementally improve the report over the past two years.

3.19 As discussed, the Deputy Secretary, Capability Acquisition and Sustainment has invited recipients of the report to provide feedback on the format and content of the Quarterly Performance Report through the foreword of each report reviewed by the ANAO.

3.20 Defence advised the ANAO that feedback:

is responded to commensurate with the seniority, seriousness or formality of the feedback received. This includes directing feedback to the relevant area to respond directly or in consultation with the area of enquiry.

Steps taken to capture observations and improve the report

3.21 Defence advised the ANAO in December 2018 that over the last two years improvements have been made to the Quarterly Performance Report as a result of feedback from stakeholders, and that the improvements included:

  • streamlining the content of the report;
  • reducing Defence-centric jargon50;
  • incorporating Australian Industry Capability information; and
  • shifting the focus of reporting to Initial and Final Operational Capability instead of Initial and Final Materiel Release to reflect an holistic view of capability as recommended by the First Principles Review.

3.22 The ANAO’s review of Quarterly Performance Reports from March 2017 to December 2018 identified two further changes to the report, namely:

  • the addition of Project Maturity Scores to performance summaries of acquisition projects as part of the response to Recommendation 2 of the Joint Committee of Public Accounts and Audit’s Report 468: Defence Major Projects Report 20151651; and
  • incorporating a background page for both acquisition projects and sustainment products that provides information to assist the reader with the detailed summary reports on projects and products.

3.23 Stakeholder feedback has been used to incrementally improve the eight reports reviewed by the ANAO.

Impact of the report

3.24 The ANAO sought evidence that the Quarterly Performance Report had an impact and was being used actively by senior stakeholders — for example, prompting further inquiry or action, or demonstrably informing decision-making. The direct impact of the Quarterly Performance Report is unclear, as there is limited Defence documentation to inform an assessment. Similarly it is unclear whether the discipline of providing information for the Quarterly Performance Report itself provides an incentive for local line management to improve performance.

3.25 The following instances of active use of the report were identified in Defence documentation:

  • There were three occasions (two in 2016 and one in 2017) where there is evidence of the report being discussed by the Defence Committee (chaired by the Secretary). On one occasion, the Chair sought further advice regarding an activity outlined in the report.
  • The ANAO’s review of feedback from the Defence Investment Committee (discussed at paragraph 3.8–3.10) indicated that the Committee reviewed the information provided and considered the broader consequences, requesting additional advice and discussion where relevant. There was also evidence of the report prompting a number of requests for further information on project and product status, cost, budget and risks.
  • As discussed in paragraph 3.6, there had been a request from the Secretary in 2016 for further advice about an activity in a Quarterly Performance Report.

3.26 Defence advised the ANAO in January 2019 that the Capability Acquisition and Sustainment Group was also using the report as the basis for broader performance discussions with senior managers.

3.27 Defence advised the ANAO in May 2019 that Ministers and senior stakeholders also have access to other briefings, discussions and information on projects and products to inform their thinking and decision-making. As outlined in paragraph 3.5, respondents to the ANAO’s survey indicated that the impact of the report was mitigated to some extent by the timeliness of reporting.

Appendices

Appendix 1 Response from the Department of Defence

Defence response letter page 1

Defence response letter page 2

Appendix 2 Members of the Defence Investment Committee

Members of the Defence Investment Committee

Role

Position

Chair

Vice Chief of the Defence Force

Member

Associate Secretary

Member

Deputy Secretary Strategy, Policy and Intelligence

Member

Chief Finance Officer

Member

Chief of Navy

Member

Chief of Army

Member

Chief of Air Force

Member

Chief of Joint Capabilities

Member

Deputy Secretary Capability and Acquisition and Sustainment

Member

Chief Defence Scientist

Member

Director General Australian Signals Directorate

Member

Deputy Secretary from the Department of Finance

Member

Deputy Secretary from the Department of the Prime Minister and Cabinet

Permanently Invited Official

First Assistant Secretary Contestability

Permanently Invited Official

Deputy Director General Office of National Intelligence

   

Source: Defence documentation.

Appendix 3 Defence Committees that receive the Quarterly Performance Report

1. The table below outlines the committees in Defence that receive the Quarterly Performance Report, their responsibilities, how often they receive the Quarterly Performance Report, and their level of engagement with the Quarterly Performance Report.

Committee

Responsibilities

Frequency with which the committee receives the Quarterly Performance Report, and level of engagement

Defence Committee

The Defence Committee is the primary decision-making committee in the department. It is focussed on the major capability and resource trade-offs and the management of the Secretary and the Chief of the Defence Force’s shared accountabilities. The Defence Committee is supported by two subordinate committees — the Enterprise Business Committee and the Investment Committee.

The committee is provided with an overview of the Quarterly Performance Report on an ad hoc basis, for information.

Defence Audit and Risk Committee

The Defence Audit and Risk Committee operates under the authority of the Secretary and is a central element of governance in Defence. It provides independent advice on all aspects of Defence governance to the Secretary and the Chief of the Defence Force. The roles and responsibilities of the Defence Audit and Risk Committee are to review all aspects of the governance and assurance frameworks, including financial, performance and risk management, the internal controls framework, compliance, accountability and audit.

The committee receives the executive summary of the Quarterly Performance Report tri-annually for information.

Capability Acquisition and Sustainment Group Executive Advisory Committee

The Capability Acquisition and Sustainment Group Executive Advisory Committee provides the Deputy Secretary of the Capability Acquisition and Sustainment Group with direction on strategic management and performance monitoring to deliver the group’s Business Plan purpose, goals and activities on time, on budget and to the required standard.

The committee is provided with a copy of the Quarterly Performance Report and any recent changes to the report on a quarterly basis, for information.

Capability Acquisition and Sustainment Group Risk Advisory Committee

The Capability Acquisition and Sustainment Group Risk Advisory Committee provides independent advice and assurance to the Deputy Secretary of the Capability Acquisition and Sustainment Group on the group’s strategic risks including consideration of causes, consequences, adequacy of controls and planned mitigation; risks to the group’s outputs and objectives including through review of key domain, division acquisition and sustainment risks the maturity and effectiveness of the group’s risk governance framework and continuous improvement in risk management.

As of March 2018, the committee receives an executive summary of each Quarterly Performance Report for noting.

     

Source: ANAO analysis of Defence documentation

Appendix 4 Explanation of traffic lights in the Quarterly Performance Report

Projects and products of Interest

1. For the section on Projects of Interest in the report, the traffic lights for acquisition projects and sustainment products are a qualitative assessment based on the criteria identified in the table below.52

Assessment

Capability

Cost

Schedule

Green — Acceptable performance

On track to deliver scope.

On track to deliver within approved budget.

Project will deliver within approved schedule.

Amber — Early signs of underperformance

The approved scope may not be delivered in its entirety.

The project may exceed the approved budget.

Project may deliver later than the approved schedule.

Red — Underperformance realised

Major elements of the scope have not been delivered.

A Real Cost Increase is anticipated.

Project will deliver later than the approved schedule.

       

Source: June 2018 Quarterly Performance Report.

Key acquisition project traffic lights

2. Section 2a — Key Acquisition Project Dashboard and Section 2c — Performance Summaries for Key Acquisition Projects in the Quarterly Performance Report use a quantitative data-driven system of traffic lights, outlined in the table below.

Traffic light

Capability

Cost

Schedule

Green — Acceptable performance

On track to deliver approved scope.

On track to deliver within approved budget.

Delivery before, on, or up to no more than 14 days after the Baseline Date.

Amber — Early signs of underperformance

Major elements of scope are about to fail against the baseline.

Estimate at

Completion exceeds budget by up to 5 per cent.

Delivery more than 14 days after the

Baseline Date, but less than 5 per cent slippage.

Red — underperformance realised

Major elements of scope have not been achieved as baselined.

Estimate at Completion exceeds budget by more than 5 per cent = Need for a Real Cost Increase.

Delivery on or later than 5 per cent slippage.

       

Source: Annex A — Explanation of Traffic Lights and Major Milestones, June 2018 Quarterly Performance Report.

Top 30 sustainment product traffic lights

3. Section 3a — Top 30 Sustainment Products Dashboard and Section 3c — Performance Summaries for Top 30 Sustainment Products in the Quarterly Performance Report use a quantitative data-driven system of traffic lights, outlined in the following tables.

Monthly Reporting System Traffic Lights

Availability

Price

Green — Acceptable performance

No more than one indicator is amber.

If actual expenditure does not exceed more than 20 per cent of the planned budget for the month.

Amber — Early signs of underperformance

Two or more indicators are amber.

If actual expenditure exceeds the planned budget by between 20 per cent and 30 per cent for the month.

Red — underperformance realised

Any indicator is set to red.

If actual expenditure exceeds the planned budget by 30 per cent or more for the month.

     

Note: For the purpose of the Quarterly Performance Report, CN10 — Collins Class Submarine is the only product reporting in the Monthly Reporting System.

Source: Annex A — Explanation of Traffic Lights and Major Milestones, June 2018 Quarterly Performance Report.

Sustainment Performance Management System Traffic Lights

Availabilitya

Year End Costb

Year to Date Costc

Green — Acceptable Performance

The product’s availability is within the agreed green threshold.

Achievement of greater than or equal to 103 per cent against Year End Budget.

Achievement of greater than or equal to 97 per cent –to less than or equal to 103 per cent against Year to Date Budget.

Amber — Early signs of underperformance

The product’s availability is within the agreed amber threshold.

Achievement of greater than 103 per cent to less than or equal to 105 per cent against Year End Budget.

Achievement of greater than or equal to 95 per cent to less than 97 per cent; or greater than 103 per cent to less than or equal to 105 per cent against Year to Date Budget.

Red — underperformance realised

The product’s availability is within the agreed red threshold.

Achievement of greater than 105 per cent against Year End Budget.

Achievement of less than 95 per cent or greater than 105 per cent against Year to Date Budget.

       

Note a: In cases where there is more than one Key Performance Indicator (KPI) for a product, an average or sum is calculated for the purpose of showing a single traffic light. This is done by applying a weighting against each KPI traffic light. The weightings are: 1 for Green, 2 for Amber and 3 for Red. These are then added together and divided by the number of KPIs to get an aggregate traffic light.

Note b: This indicator measures the year end product price baseline forecast against the year-end budget. Data reported is the year end actual (forecast).

Note c: This indicator measures the year to date achievement against product price baseline funding. Data reported is the year to date actual up to the current reporting period measured against the year to date phasings for the financial year.

Source: Annex A — Explanation of Traffic Lights and Major Milestones, June 2018 Quarterly Performance Report.

Appendix 5 Survey provided to recipients of June 2018 Quarterly Performance Report

1. Are you aware that the Department of Defence’s Capability Acquisition and Sustainment Group provides you with a copy of the Quarterly Performance Report on acquisition and sustainment?

(Yes/No)

2. Do you read the Quarterly Performance Report?

(Yes/No)

3. If you read the Quarterly Performance Report, do you find it to be logically structured and easy to read?

(Yes/No/Not Applicable)

The following questions seek your views on the sufficiency, accuracy and quality of the information reported in the Quarterly Performance Report.

4. Is enough information included in the Quarterly Performance Report about projects and products to assist you?

(Always/Mostly/Sometimes/Rarely/Never)

5. In your experience, is the data included in the Quarterly Performance Report accurate at the point in time the report is circulated?

(Always/Mostly/Sometimes/Rarely/Never)

6. Is the information reported in the Quarterly Performance Report of sufficient quality to assist you?

(Always/Mostly/Sometimes/Rarely/Never)

7. Overall, do you find the Quarterly Performance Report to be a useful and timely report?

(Always/Mostly/Sometimes/Rarely/Never)

The following questions seek your views about the usefulness and timeliness of the Quarterly Performance Report as a tool to support decision making.

8. In your experience, is the Quarterly Performance Report a useful source of information about the key risks and issues that are affecting the capability, schedule and cost of acquisition projects?

(Always/Mostly/Sometimes/Rarely/Never)

9. In your experience, is the Quarterly Performance Report a useful source of information about the key risks and issues that are affecting the availability and costs of sustainment products?

(Always/Mostly/Sometimes/Rarely/Never)

10. Does the Quarterly Performance Report highlight key risks in projects and products in a timely manner to assist you in making decisions?

(Always/Mostly/Sometimes/Rarely/Never)

11. Do you rely on the Quarterly Performance Report to inform your decision-making in relation to projects and products under the Capability Acquisition and Sustainment Group’s management?

(Always/Mostly/Sometimes/Rarely/Never)

The following questions seek to identify whether the Capability Acquisition and Sustainment Group has engaged with recipients of the Quarterly Performance Report to obtain feedback and improve reporting.

12. Has your feedback been sought on the design, content and/or preparation of the Quarterly Performance Report to improve reporting and to better address your information needs?

(Yes/No)

13. Has your feedback on the design, content and/or preparation of the Quarterly Performance Report been incorporated?

(Yes/No/Not applicable)

Thank you for taking part in the survey.

Are there any other comments you wish to make about the Quarterly Performance Report?

Appendix 6 Sustainment product sample

Product

Description

2017–18 Budget ($m)

CAF21 Superhornet Weapon System

Air Force has 24 F/A-18F Super Hornets, which ensure that Australia’s air combat capability edge is maintained until the full introduction of the F-35A Lightning II.

243

CA59 Explosive Ordnance Army Munitions

The ADF’s inventory consists of small arms ammunition, pyrotechnics, mortar and artillery ammunition, special purpose ammunition, demolition stores and Army guided weapons.

204

CA48 Multi Role Helicopter Weapons System

Provision of Through Life Support for the Multi Role Helicopter and associated training and support Systems.

172

CAF19 C-17 Heavy Air Lift Weapons System

The C-17A weapon system comprises eight aircraft and associated training systems. Sustainment is primarily undertaken through a range of Foreign Military Sales arrangements with the United States Air Force.

103

CA39 ADF Clothing

ADF Clothing comprises approximately 21,000 line items, supplied by Australian based businesses using a wide range of onshore and/or offshore supply chain solutions.

92

CN09 Armidale Class Patrol Boats

The Armidale Class Patrol Boats provide operational capability for the Navy in support of border protection and other operational requirements.

88

CN40 Hobart Class Destroyer

The Hobart class is a ship class of three air warfare destroyers being built for the Royal Australian Navy.

49

CAF12 Air Traffic Management

The Air Traffic Management system consists of fixed and deployable radars, navigation aids, display and data processing systems, communications systems and related simulators at Defence sites throughout Australia.

46

     

Source: Capability Acquisition and Sustainment Group Quarterly Performance Report June 2018.

Appendix 7 2017–18 Major Projects Report — list of acquisition projects

1. The table below outlines the 26 acquisition projects from the 2017–18 Major Projects Report that also reported on in the June 2018 Quarterly Performance Report.

Project

Description

Budget ($m)a

AIR6000 PH2A/B

New Air Combat Capability

The AIR 6000 Project aims to introduce the F-35A Joint Strike Fighter capability that will meet Australia’s air combat needs out to 2030 and beyond. Phase 2A/2B of the project is approved to acquire 72 Conventional Take Off and Landing F-35A JSF aircraft to establish three operational squadrons, a training squadron and necessary supporting/enabling elements to replace the F/A-18A/B Hornet capability.

15,504

SEA4000 PH3

Air Warfare Destroyer Program

This project will acquire three Hobart Class Air Warfare Destroyers (AWD) and their support system for the Australian Defence Force (ADF). The capability provided by the AWDs will form a critical element of the ADF’s joint air warfare defence capability and will contribute to a number of other joint warfare outcomes.

9,089

AIR7000 PH2B

P-8A Maritime Patrol and Response Aircraft System

AIR 7000 Phase 2B seeks to acquire the materiel elements of the Maritime Patrol and Response Aircraft weapon system, including a Through Life Support system, as partial replacement of the AP-3C Orion aircraft.

5,212

AIR9000 PH 2/4/6 Multi-Role Helicopter

The MRH Program consists of three phases of AIR 9000:

  • Phase 2 (12 helicopters) is the acquisition of an additional Squadron of troop lift aircraft for the Australian Army;
  • Phase 4 (28 helicopters) will replace Army’s Black Hawk helicopters in the Air Mobile and Special Operations roles; and
  • Phase 6 (6 helicopters) will replace Royal Australian Navy Sea King helicopters in the Maritime Support Helicopter role.

3,771

AIR5349 PH3

Growler Airborne Electronic Attack Capability

The EA-18G Growler Airborne Electronic Attack Capability provides for the acquisition of 12 Boeing EA-18G Growler aircraft, ALQ-99 Tactical Jamming Systems, associated weapons, support and training systems to establish an Airborne Electronic Attack (AEA) capability for the Australian Defence Force (ADF).

3,430

AIR9000 PH8

Future Naval Aviation Combat System Seahawk Romeo

AIR 9000 Phase 8 has acquired 24 MH-60R Seahawk Romeo naval combat helicopters, associated weapons and support systems to replace the current 16 S-70B-2 Seahawk Bravo helicopters and the cancelled SH-2G(A) Seasprite helicopters.

3,430

LAND121 PH3B

Medium and Heavy Capability

LAND 121 Phase 3 was established to replace the current fleet of Australian Defence Force (ADF) Field Vehicles, Modules and Trailers and will enhance the ground mobility of the ADF. LAND 121 Phase 3B will upgrade and replace the existing medium and heavy vehicle and trailer fleet.

3,429

JP2048 PH4A/4B

Amphibious Ships

Joint Project (JP) 2048 Phase 4A/4B is providing the Australian Defence Force (ADF) with an increased amphibious deployment and sustainment capability through the acquisition of two Landing Helicopter Dock (LHD) ships and associated supplies and support.

3,092

LAND121 PH4

Protected Mobility Vehicle - Light

LAND 121 Phase 4 will acquire and deliver into service 1100 Protected Mobility Vehicles — Light (PMV-L) and 1058 companion trailers for command, liaison, reconnaissance and utility roles; and the associated training and support systems.

1,952

AIR8000 PH2

Battlefield Airlift - Caribou Replacement

This project was approved to replace the retired Caribou capability and provide the Australian Defence Force (ADF) with an enhanced intra-theatre and regional airlift capability through acquisition of a fleet of ten new Light Tactical Fixed Wing aircraft.

1,433

SEA1654 PH3

Maritime Operational Support Capability

The SEA 1654 Phase 3 Maritime Operational Support Capability Project will replace both HMA Ships Success and Sirius with a single class of two Auxiliary Oiler Replenishment Ships to sustain deployed maritime forces. The primary role of the AOR Ships is the provision of afloat-support capability to fleet units.

1,067

AIR5431 PH 3

Civil Military Air Traffic Management System

AIR 5431 Phase 3 seeks to replace the current Fixed Base Defence Air Traffic Management and Control Systems at 12 Australian Defence Force (ADF) fixed base locations with a new harmonised system, referred to as the Civil Military Air Traffic Management System (CMATS). The CMATS component of AIR 5431 Phase 3 is being conducted as a joint acquisition program with Airservices Australia.

974

JP2072 PH2B

Battlespace Communications System (Land)

LAND (JP) 2072 Phase 2B will provide the Battlespace Communications System Land (BCS-L) deployed wide-band backbone by replacing and enhancing the existing Battlefield Telecommunications Network (BTN) capability within Army and Airforce. LAND 2072 Phase 2B shall deliver the Integrated Battlefield Telecommunications Network (I-BTN) in two capability Releases. Release 1 shall provide transit case nodes, and Release 2 shall expand on Release 1 to provide vehicle mounted nodes and additional capabilities. The end state will be an I-BTN that provides greater capacity, more effective switching, wireless and wired network infrastructure supporting secure voice, data and video services.

LAND 2072 Phase 2B is required to provide end to end connectivity from the enhanced Deployable Local Area Network (eDLAN), through and within the I-BTN, and to the Defence Terrestrial Communications Network (provided by JP2047 Phase 3).

920

AIR7403 PH3

Additional KC-30A Multi-Role Tanker Transport

AIR 7403 Phase 3 has acquired two A330-200 aircraft and converted them to KC-30A Multi-role Tanker Transport aircraft, with the first additional aircraft delivered with associated spares and support equipment.

888

SEA1448 PH2B

Anzac Class Anti-Ship Missile Defence Upgrade

The Anti-Ship Missile Defence upgrade SEA 1448 Phase 2 project has provided the ANZAC Class Frigates with an enhanced level of self-defence against modern anti-ship missiles. Phase 2B of the project, has introduced an indigenous, leading edge technology, phased array radar and missile illuminator collectively referred to as the Phased Array Radar System.

679

SEA3036 PH1

Pacific Patrol Boat Replacement

This project will acquire 21 vessels to replace the existing 22 Pacific Patrol Boats gifted to 12 Pacific Island Countries between 1987 and 1997 and two boats for Timor-Leste; as part of Australia’s Pacific Maritime Security Program.

501

AIR9000 PH7

Helicopter Aircrew Training System

The project will provide a new Helicopter Aircrew Training System to prepare Navy and Army aircrew for conversion to operational aircraft.

481

SEA1439 PH4A

Collins Class Submarine Replacement Combat System

The project was established to provide each of the six Royal Australian Navy Collins Class submarines with an initial installation of the United States Navy AN/BYG-1 Combat and Weapon Control System, minor improvements to the combat system augmentation sonar, and shore facilities for integration, testing and training.

450

JP2072 PH2A

Battlespace Communications Systems (Land)

The project has delivered approximately 11,000 Combat Radios and ancillary equipment to replace the Wagtail, Pintail and Raven fleets for the majority of the Land Force. Phase 2A has also established the mature support system for the new generation Combat and Tactical Data Radios.

438

SEA1442 PH4

Maritime Communications Modernisation

The project will upgrade the communications capability in the eight Anzac Class Frigates and address communications system obsolescence in the Class, by modernising it with improved communications management, secure voice and tactical intercom, red/black switching, tactical radios and a high data rate line-of-sight capability.

438

SEA1429 PH2

Replacement Heavyweight Torpedo

This project has acquired a Heavyweight Torpedo for the six Collins Class submarines to replace the United States Navy’s Heavy Weight Torpedo previously in service with the Royal Australian Navy.

428

JP2008 PH5A Indian Ocean Region UHF SATCOM

This Project will provide the Australian Defence Force (ADF) with twenty 25kHz UHF SATCOM channels on a hosted payload on a

commercial Intelsat Satellite (IS-22), to provide coverage of the Indian Ocean Region, and associated ground infrastructure to provide network control.

420

SEA1439 PH3

Collins Submarine Platform Systems Improvements

This program upgrades Collins Class platform systems and shore infrastructure to improve the Class reliability, sustainability, safety and capability for each of the six submarines.

412

SEA1448 PH2A

Anzac Class Anti-Ship Missile Defence

The project has provided the ANZAC Class Frigates with an enhanced level of self defence against modern anti-ship missiles.

387

LAND75 PH4

Battlefield Command Systems

The Battle Management System Command and Control provides tactical and operational commanders with enhanced situation awareness tools, evolved command and control, extend the capability into the armoured fighting vehicle fleets and enrich training and simulation to provide a collaborative approach to complex warfighting.

368

JP2048 PH3 Amphibious Watercraft Replacement

Joint Project (JP) 2048 Phase 3 provides the Amphibious Deployment and Sustainment capability with a new breed of watercraft

that are organic to the two new Canberra Class Amphibious Assault Ships, the Landing Helicopter Dock (LHD), acquired under JP

2048 Phase 4A/4B. The craft are known as LHD Landing Craft (LLC). The LLC was purchased as an organic ship to shore

connector and will interface and operate with the LHD ships, enabling transport of personnel and equipment from the LHD ships

to the shore, including where there are no fixed port facilities or prepared landing facilities.

237

     

Note a: The budget figures are rounded to full figures.

Source: Auditor-General Report No.20 2018–19 2017–18 Major Projects Report, available from https://www.anao.gov.au/sites/default/files/Auditor-General_Report_2018-2019_20.pdf [accessed 7 March 2019] and Capability Acquisition and Sustainment Group Quarterly Performance Report June 2018.

Footnotes

1 Capability Acquisition and Sustainment Group, Quarterly Performance Report, 30 June 2018, p. 5.

2 Independent Assurance Reviews (previously known in Defence as ‘Gate Reviews’) are a review of a Defence program, product or project by a specifically chartered board of Defence personnel with relevant skills and experience. Reviews are normally conducted annually for acquisition projects, or periodically (one to three years) for sustainment products. These reviews were examined in Auditor-General Report No.52 2011-12, Gate Reviews for Defence Capital Acquisition Projects.

3 The ANAO’s recent series of governance audits noted that one of the central themes in notable Australian inquiries into organisational behaviour — the 2003 Royal Commission into the failure of HIH Insurance, the 2018 APRA Prudential Inquiry into the Commonwealth Bank of Australia and the 2019 Royal Commission into the financial services industry — is that those charged with governance cannot operate properly without having the right information. While the specific focus of those inquiries was on financial institutions, their key insights on culture and governance have wider applicability and provide lessons for accountable authorities in the public sector. The first report in the series is Auditor-General Report No.34 of 2018-19 Effectiveness of Board Governance at Old Parliament House, available from https://www.anao.gov.au/work/performance-audit/effectiveness-board-governance-old-parliament-house.

4 Capability Managers agree the quantities and levels of military equipment and supplies to be delivered by the Capability Acquisition and Sustainment Group in Materiel Acquisition Agreements and Material Sustainment Agreements. These customer-supplier agreements also establish budgets and key performance indicators.

5 Capability Acquisition and Sustainment Group, Quarterly Performance Report, 30 June 2018, p.5. Internal guidance on the Report states that:

The QPR aims to provide senior stakeholders within Government and the Department of Defence with a clear and timely understanding of emerging issues and risks in the Acquisition and Sustainment of capability for our end-users. These issues are highlighted so that Defence can respond in an effective, efficient, and coordinated manner to guide the conduct of remediation actions.

6 ANAO comment: Defence’s projects of concern regime was reviewed in Auditor-General Report No.31 2018–19, Defence’s Management of its Projects of Concern. That audit found that the regime is based on managerial discretion and professional judgement rather than explicit criteria. In that context, the Quarterly Performance Report is a key source of advice to the Secretary, Chief of the Defence Force and Defence Ministers to support the identification of projects and products in need of additional attention.

7 The Services also have their own reporting mechanisms. For example, Service Chiefs receive information on acquisition projects and sustainment products through various committee processes, including Senior Advisory Committees, and biannual fleet screenings. Fleet screenings are major sustainment activity and resourcing reviews undertaken by Capability Managers.

8 The Monthly Reporting System is Defence’s primary system for reporting on the performance of its acquisition projects.

9 The Sustainment Performance Management System is Defence’s primary system for reporting on the performance of its sustainment products.

10 The Investment Committee governs the delivery of the Integrated Investment Program and provides advice to the Defence Committee to assure the alignment of strategy, capability and resources. The members of the Defence Investment Committee are listed at Appendix 2.

11 Defence advice to the ANAO, April 2019.

12 The purpose of these committees, the frequency with which they receive the Quarterly Performance Report, and their level of engagement with the report is detailed at Appendix 3.

13 Appendix 4 outlines how traffic lights are determined for the Quarterly Performance Report.

14 Defence advised the ANAO in June 2019 that although discussed twice, the purpose of the performance summary sections is to report on key performance indicators agreed under the Materiel Acquisition and Material Sustainment Agreements. The purpose of the Projects and Products ‘of Interest’ and ‘of Concern’ is to highlight issues, risks, the mitigation strategy and plans related specifically to the areas of underperformance.

15 Auditor-General Report No.2 2017–18 Defence’s Management of Materiel Sustainment, paragraphs 16–17 and 3.10–3.20, available from https://www.anao.gov.au/sites/g/files/net5496/f/ANAO_Report_2017-18_02.pdf [accessed 4 October 2018].

16 ibid, paragraphs 3.20–3.28.

17 Hansard, 16 August 2017, p. 17.

18 ibid.

19 ibid.

20 Joint Committee of Public Accounts and Audit, Report 470: Defence Sustainment Expenditure, March 2018, p. viii, available from https://www.aph.gov.au/Parliamentary_Business/Committees/ Joint/Public_Accounts_and_Audit/DefenceSustainment/Report_470 [accessed 4 October 2018].

21 The ANAO’s recent series of governance audits noted that one of the central themes in notable Australian inquiries into organisational behaviour — the 2003 Royal Commission into the failure of HIH Insurance, the 2018 APRA Prudential Inquiry into the Commonwealth Bank of Australia and the 2019 Royal Commission into the financial services industry—is that those charged with governance cannot operate properly without having the right information. While the specific focus of those inquiries was on financial institutions, their key insights on culture and governance have wider applicability and provide lessons for accountable authorities in the public sector. The first report in the series is Auditor-General Report No.34 of 2018-19 Effectiveness of Board Governance at Old Parliament House, available from https://www.anao.gov.au/work/performance-audit/effectiveness-board-governance-old-parliament-house.

22 See Appendix 6 for a list of sustainment products sampled.

24 In its report on the inquiry, Report 470: Defence Sustainment Expenditure, the Committee recommended that the Auditor-General consider reviewing Defence’s new Monthly Reporting System and documentation including advising whether there have been any improvements in reporting to the Minister and the Secretary, and the information provided via reporting is timely and accurate (Recommendation 5, p. 13), available from https://parlinfo.aph.gov.au/parlInfo/download/committees/reportjnt/024113/toc_pdf/Report470DefenceSustainmentExpenditure.pdf;fileType=application%2Fpdf [accessed 27 March 2019].

25 The Auditor-General provides independent limited assurance over the status of selected Defence Major Projects as at 30 June each year through the Major Projects Report. Of the 43 acquisition projects listed in the June 2018 Quarterly Performance Report, 26 were included in the 2017–18 Major Projects Report. The total approved budget for the 26 projects in the 201718 Major Projects Report is approximately $59.4 billion, covering 57 per cent of the total Defence budget of active major and minor capital equipment projects of $103.5 billion. The 26 major acquisition projects from the 201718 Major Projects Report are listed in Appendix 7. The most recent Major Projects Report, Auditor-General Report No.20 2018-19 2017-18 Major Projects Report, is available from https://www.anao.gov.au/sites/default/files/Auditor-General_Report_2018-2019_20.pdf [accessed 3 April 2019].

26 These were Auditor-General Report No.6 2018-19 Army’s Protected Mobility Vehicle–Light presented on 11 September 2018; and Auditor-General Report No.14 2018-19 Joint Strike Fighter – Introduction into Service and Sustainment Planning presented on 5 December 2018.

27 The 2017-18 Major Projects Report also covered the period to June 2018.

28 This issue was reported on most recently in Auditor-General Report No.20 2018-19 2017-18 Major Projects Report. The Joint Committee of Public Accounts and Audit, in the context of its consideration of the report, sought further information on whether cost variations and the cost of retaining project staff might be reported annually in future Major Projects Reports. As reported in para 1.49 of the 2017-18 Major Projects Report, Defence advised the ANAO in November 2018 that current IT systems do not provide a direct mapping of personnel to projects, with personnel often working on multiple projects and sustainment activities at any given time. Defence further advised the ANAO that while it is not yet in a position to provide the staff cost component of projects, it has begun to collect information on the numbers of staff (including Australian Defence Force and Australian Public Service, but not contractors) for projects. It is expected that further information on staff numbers will be reported in the 2018–19 Major Projects Report. Source: https://www.anao.gov.au/sites/default/files/Auditor-General_Report_2018-2019_20.pdf [accessed 3 April 2019].

29 Including Land, Aerospace, Maritime and Joint domains.

30 Defence advised the ANAO in June 2019 that:

The minor delivery delay associated with the boots contract was a small element of the total CA39 ADF Clothing work program for the year and did not adversely affect the KPIs by which the fleet is measured.

31 Deficiencies in performance indicators were also identified in Auditor-General Report No.30 2018–19 ANZAC Class Frigates—Sustainment. For the ANZAC Class frigates, issues were identified with the accuracy and completeness of the performance measures used to assess the effectiveness of sustainment outcomes. The traffic light for the product’s availability in the Quarterly Performance Report was rated ‘green’, and the comment on availability in the Top 30 Sustainment Report noted that the target had been met. This may not be an accurate representation of the product’s availability if the performance measures are not effective. The ANAO recommended that: ‘Defence review the key performance measures for the ANZACS class frigates’ sustainment to ensure they are reliable and complete’. Defence agreed to the recommendation with qualification, noting that: ‘This will require consultation across Defence to review extant reporting metrics; as these are currently standardised across multiple warship classes’. See paragraphs 3.4 to 3.9 of the report, available from https://www.anao.gov.au/work/performance-audit/anzac-class-frigates-sustainment.

32 See Appendix 4 for a summary of how traffic lights are determined in the Quarterly Performance Report.

33 Defence advised the ANAO in April 2019 that:

It is not the difference in methodologies per se that mean the traffic light ratings can be different. It is that they are measuring different aspects of delivery. The performance summaries are specifically measuring agreed targets with set parameters. The projects/products of interest traffic lights are a qualitative assessment of the risks that have emerged in delivering to those targets (usually as a result of underperformance). It is possible to have an underperforming KPI, say, in cost which has triggered red. However, the risk to remediating that cost may be low or medium which would be green or amber.

34 Defence advised the ANAO in April 2019 that sustainment product budgets can be updated as necessary based on the results of biannual fleet screening, and the prioritisation of sustainment activities and budgets across products to respond to capability risks.

35 Of the seven sets of Investment Committee member clearance comments reviewed (from June 2017 to December 2018), the ANAO noted that only two comments resulted in changes made to the final Quarterly Performance Report — one in June 2017 and one in September 2018.

36 This has been observed in previous ANAO reports, including: Auditor-General Report No.6 2018–19 Army’s Protected Mobility Vehicle – Light; Auditor-General Report No.14 2018–19 Joint Strike Fighter – Introduction into Service and Sustainment Planning; and Auditor-General Report No.30 2018–19 Anzac Class Frigates – Sustainment.

37 Defence advice to ANAO, April 2019.

38 Defence further advised the ANAO in June 2019 that a reporting function is in software development to populate the trend data in a format that allows quarterly analysis by the Strategic Performance Reporting team (this is currently a manual process).

39 Such as information prepared for the annual Major Projects Report and information appearing in Independent Assurance Reviews.

40 Auditor-General Report No.2 2017–18 Defence’s Management of Materiel Sustainment, p. 38, available from https://www.anao.gov.au/sites/default/files/ANAO_Report_2017-18_02.pdf. The Joint Committee of Public Accounts and Audit subsequently observed at page 13 of its Report 470: Defence Sustainment Expenditure that internal reporting via the Quarterly Performance Report required improvement, available from https://parlinfo.aph.gov.au/parlInfo/download/committees/reportjnt/024113/toc_pdf/Report470DefenceSustainmentExpenditure.pdf;fileType=application%2Fpdf [accessed 6 March 2019].

41 Auditor-General Report No.2 2017–18 Defence’s Management of Materiel Sustainment was tabled on 11 July 2017.

42 The ANAO’s March 2019 Audit Insights focused on the importance of Quality Assurance Processes for government entities. In particular, quality assurance processes are important when developing policy and providing advice, because government expects that the advice provided will be evidence-based and reflect the needs of those impacted by the policy or advice. ANAO, Insights from reports tabled October to December 2018, 7 March 2019, available from https://www.anao.gov.au/work/audit-insights/insights-reports-tabled-october-to-december-2018.

43 See paragraph 2.4 regarding the impact of IT system limitations on the timeliness of reporting.

44 The Vice Chief of the Defence Force chairs the Investment Committee, which is focussed on the future force and is responsible for bringing the future force and supporting enablers into being. The Investment Committee governs the delivery of the Integrated Investment Program through review of proposals, taking into account force requirements, all necessary enabling functions and full-life costs. The Investment Committee provides advice to the Defence Committee to assure the alignment of strategy, capability and resources. Membership of the Defence Investment Committee is at Appendix 2.

45 For example, the scope of the report, data sources informing the report and risks and sensitivities that impact on the timeliness of the Capability Acquisition and Sustainment Group’s reporting.

46 These risks include: emerging trends and lessons learned in sustainment; and the provisioning of funds to support capability, key workforce capability and industry risks.

47 For example, Defence advised the ANAO in May 2019 that the Chief Information Officer Group is developing a performance report based on the Quarterly Performance Report.

48 As outlined in paragraph 1.2, the purpose of the Quarterly Performance Report is to provide senior stakeholders within Government and the Department of Defence with a clear and timely understanding of emerging risks and issues in the delivery of capability to Australian Defence Force end-users. These risks and issues are highlighted so that stakeholders can respond in a coordinated manner to guide the conduct of remediation actions.

49 Defence advice to the ANAO, April 2019.

50 This issue was commented on in Auditor-General Report No.2 2017–18 Defence’s Management of Materiel Sustainment and in the Joint Committee of Public Accounts and Audit’s Report 470: Defence Sustainment Expenditure.

51 Recommendation 2 was that ‘… the Department of Defence commence discussions with the Australian National Audit Office on updating Project Maturity Scores …’ available from https://www.aph.gov.au/Parliamentary_ Business/Committees/Joint/Public_Accounts_and_Audit/DefenceMPR15-16/Report_468/section?id= committees%2freportjnt%2f024078%2f25305 [accessed 26 March 2019].

52 Defence informed the ANAO that the qualitative assessment for these traffic lights are determined by the Project or Product Manager, with clearance from Division Heads.