The objective of the audit was to assess whether the Department of Defence’s Projects of Concern regime is effective in managing the recovery of underperforming projects.

Summary and recommendations

Background

1. The Department of Defence’s (Defence) Projects of Concern regime was established as a framework to manage the remediation of underperforming materiel acquisition projects. The objective of the regime is:

to remediate the project by implementing an agreed plan to resolve any significant commercial, technical, cost and/or schedule difficulties. Projects of Concern receive targeted senior management attention and must be reported regularly to the government.1

2. Of the 25 projects listed as Projects of Concern since 2008, Defence has cancelled two of these projects and returned most of the remainder to normal management arrangements. The period spent by individual projects on the list has ranged from a few months to over eight years. Thirteen are reported to have reached Final Operational Capability.2 As of December 2018, there were two projects on the Projects of Concern list.3

3. Entry to the Projects of Concern list, and exit from it, is decided by ministers. For most of the history of Projects of Concern, Defence has used specific criteria to provide a basis to recommend that a project be placed on the list. From 2017, a set of principles has been followed rather than specific criteria.

Rationale for undertaking the audit

4. The reason for undertaking the audit is that Projects of Concern include projects that contribute substantial capability to the Australian Defence Force and involve a major resource commitment by the Australian Government. As a mechanism for resolving difficulties with Defence projects, there is a clear link between the effectiveness of the Projects of Concern regime and Defence’s strategic priorities as stated in one of its purposes under the Public Governance, Performance and Accountability Act 2013 (PGPA Act): ‘Deliver and sustain Defence capability and conduct operations’.4 Further, the Projects of Concern regime regularly receives Parliamentary attention and this audit is intended to provide insight into how Defence operates and manages the Projects of Concern regime, comprising the small number of projects requiring an increased level of management and support.

Audit objective and criteria

5. The objective of the audit is to assess whether Defence’s Projects of Concern regime is effective in managing the recovery of underperforming projects. The following high-level criteria were adopted for the audit:

  • Defence has established an appropriate framework for the Projects of Concern regime, including processes for the entry, management and exit of projects.
  • Defence applies the Projects of Concern regime with an appropriate degree of consistency.
  • Defence has established appropriate internal and external reporting arrangements on the progress of Projects of Concern.
  • Defence can demonstrate that the Projects of Concern regime contributes materially to the recovery of underperforming projects and products.

Conclusion

6. While the Projects of Concern regime is an appropriate mechanism for escalating troubled projects to the attention of senior managers and ministers, Defence is not able to demonstrate the effectiveness of its regime in managing the recovery of underperforming projects. Defence remains confident of the regime’s effectiveness but its confidence is based on management perception and anecdotal evidence, as it has not attempted any systematic analysis. Over the last five years, the transparency and rigor of the framework’s application has declined.

7. Defence no longer has an appropriate framework for its Projects of Concern regime. The regime has two clear purposes: to resolve troubled capability development projects through remediation or cancellation with the explicit involvement of ministers; and to help keep ministers informed. However, its current implementation lacks rigour. From 2008 forward, ministers’ involvement heightened the focus on troubled projects and strengthened the regime. It was more fully developed in 2011, with the introduction of regular summit meetings chaired by ministers to review progress and stimulate action. Over the last five years, transparency has reduced, the level of formality has declined with explicit criteria replaced by unpublished principles, and processes have become less rigorous with a greater emphasis on maintaining relationships with industry.

8. There has been inconsistency in Defence’s application of its Projects of Concern regime. In particular, application of processes for entry onto the list have been inconsistent and summit meetings to address Projects of Concern have become less frequent. Greater consistency has been maintained in preparing remediation plans and removing projects from the list, though there have been exceptions to both.

9. Defence reporting on its Projects of Concern is appropriate, with regular reports provided to senior management within Defence, to ministers and to Parliament, as part of Defence’s Quarterly Performance Report. Reporting provides useful quantitative and qualitative data though Defence has acknowledged that the timing and quality of its Quarterly Performance Reports could be improved.

10. Defence has not evaluated its Projects of Concern regime over the decade it has been in place, nor set criteria for assessing success. There is no basis, therefore, for Defence to show that the Projects of Concern regime contributes materially to the recovery of underperforming projects and products.

Supporting findings

The framework for the Projects of Concern regime

11. The Projects of Concern regime has a clear purpose and scope. Its purpose is to help keep senior Defence leaders and ministers informed of materiel acquisition projects in difficulty and resolve problems in the project’s progress either through remediation or, where that is not practicable, cancellation. The regime has applied almost exclusively to underperforming materiel acquisition projects. Despite running in parallel for almost two decades, Defence has only recently sought to align its contractor performance data across its Performance Exchange Program and the Projects of Concern regime to ensure that views on contractor performance are consistent.

12. Defence has established a policy for its Projects of Concern regime and procedures for the entry, management and exit of projects. In 2009 and 2011, Defence’s approach was made more formal and rigorous at the instigation of ministers. In recent years, the transparency and formality of the process have diminished, regular summits chaired by ministers have been replaced by ad hoc meetings, and Defence no longer publishes its principles and procedures for Projects of Concern. Maintaining collaborative relationships with industry has become a more dominant element in the governance of the regime.

Application of the Projects of Concern regime

13. Defence does not apply a consistent process to the entry of projects to the Projects of Concern list with evidence of delays as well as advice being withheld from review processes and decision-makers. Procedures for Independent Assurance Reviews do not explicitly mention Projects of Concern even though such reviews are the primary occasion for nominating a project to be a Project of Concern. There was evidence that most reviews (75 per cent) had considered whether a recommendation should be made.

14. Broadly, Defence has applied a consistent process to the management of projects while on the Projects of Concern list. However, summit meetings involving the Minister, vendors and officials, a principal process devised in 2011 to help ensure that Defence can use its Projects of Concern regime to exert commercial pressure on vendors, are no longer regular and they have become less frequent. Another long-standing process, the preparation of remediation plans, has usually been followed.

15. Defence has generally applied a consistent process to the exit of projects from the Projects of Concern list. Defence’s practice has been to recommend removal of a project from the list only when it has both fulfilled a specified set of expectations (or removal criteria) and satisfied Defence that it is on a sound trajectory, making it unlikely to return to the list. A 2018 decision to remove a project (CMATS) has not observed the second condition.

Reporting on Projects of Concern and evaluating the regime

16. Regular reports are provided on Projects of Concern to senior management within Defence, to ministers and to Parliament which contain useful quantitative and qualitative data. Projects of Concern are also reported on publicly through Defence’s Annual Report and ministerial media releases. Defence has acknowledged that the quality of the data could be improved and that information technology systems have affected the timeliness of the reports. Notwithstanding its regularity, the reporting is not timely, taking nearly two months to complete.

17. Defence cannot demonstrate that the Projects of Concern regime contributes materially to the recovery of underperforming projects. Although Defence has consistently stated that its Projects of Concern regime is ‘one of the Department’s most successful management tools for recovering problem projects’ it has not evaluated the regime and this view is based on management perception and anecdotal evidence.

Department of Defence’s response

18. The proposed report was provided to the Department of Defence. The Department’s summary response is below and its full response is at Appendix 1.

Defence maintains that the Projects of Concern regime is a significant material factor, and a strong commercial lever to influence the positive recovery of underperforming projects and products.

Defence considers that the ANAO’s analysis and overall conclusion contained in the Proposed Report do not appropriately consider the evolving nature of the Projects of Concern regime, its role within the larger project management toolkit, and elevation of priority for attention by the Minister/Government of the day.

Defence does not agree with the ANAO’s statements inferring that it has avoided adding to the Projects of Concern list in the interest of trying to maintain a positive relationship with industry, nor has this resulted in a less robust governance arrangement. The reduction in the number of projects on the Project of Concern list is linked to the changing nature of the Capital Equipment Program as well as the close out of legacy projects.

Nevertheless, Defence acknowledges that there is room to enhance the administrative arrangements supporting this program and has agreed to both recommendations.

Recommendations

Recommendation no.1

Paragraph 3.29

Defence introduce, as part of its formal policy and procedures, a consistent approach to managing entry to, and exit from, its Projects of Interest and Projects of Concern lists. This should reflect Defence’s risk appetite and be made consistent with the new Capability Acquisition and Sustainment Group Risk Model and other, Defence-wide, frameworks for managing risk. To aid transparency, the policy and the list should be made public.

Department of Defence response: Agreed.

Recommendation no.2

Paragraph 4.32

Defence evaluates its Projects of Concern regime.

Department of Defence response: Agreed.

Key messages from this audit for all Australian Government entities

Below is a summary of key messages, including instances of good practice, which have been identified in this audit and may be relevant for the operations of other Australian Government entities.

Risk management

Governance and program evaluation

1. Background

Introduction

1.1 Defence established its Projects of Concern regime to manage the remediation of underperforming projects. The objective of the regime is:

to remediate these projects by implementing an agreed plan to resolve any significant commercial, technical, cost and/or schedule difficulties. Projects of concern receive targeted senior management attention and must be reported on more regularly to the Government.5

1.2 The Projects of Concern process is essentially a risk identification and management process which is intended to escalate troubled projects to the attention of senior managers and ministers.6 Defence states:

The Projects of Concern regime is a proven process for managing underperforming capability projects at a senior level. Once a project is listed as a Project of Concern, the primary objective of the regime is to remediate the project by implementing an agreed plan to resolve any significant commercial, technical, cost and/or schedule difficulties. Projects of Concern receive targeted senior management attention and must be reported regularly to the government.7

1.3 The Projects of Concern regime is managed by Defence’s Capability Acquisition and Sustainment Group (CASG which was formally known as the Defence Materiel Organisation). As of 30 September 2018, CASG had responsibility for the management of 117 major acquisition projects and 110 sustainment projects.

The Projects of Concern list

1.4 Since 2008, when the Projects of Concerns list was made public8, there have been 25 Projects of Concern in total. The list has included some of the most significant capital equipment projects and sustainment products in the portfolio, such as the Air Warfare Destroyer build, the Multi-Role Helicopter (MRH90) acquisition and the sustainment of the Collins Class Submarine fleet.9 As of December 2018, there were two projects on the Projects of Concern list — the Multi-Role Helicopter (MRH90) acquisition and the deployable air traffic management system.

1.5 Of the 25 projects listed as Projects of Concern since 2008, Defence has cancelled two of these projects and returned most of the remainder to normal management arrangements. Thirteen are reported to have reached Final Operational Capability.10 A complete list of projects that have appeared on the list since 2008 is set out in Table 1.1.

Table 1.1: Projects of Concern, from 2008 forward

 

Project

Entry and exit dates

Elapsed time on list from January 2008

Final Operational Capability

1

SEA 1411 — ANZAC Ship Helicopter Project (Super Seasprite helicopter)

Jan. 2008 – March 2008

2 months

Cancelled, March 2008

2

abAIR 87 — ARH Tiger Armed Reconnaissance Helicopter

Jan. 2008 – April 2008

3 months

14 April 2016

3

aLAND 106 — M113 Armoured Personnel Carrier Upgrade

Jan. 2008 – May 2008

4 months

19 December 2014

4

aSEA1390 Phase 2 — Guided missile frigate (FFG) upgrade

Jan. 2008 – Jan. 2010

2 years

3 June 2016

5

aAIR 5416 Phase 2A — Rotary Wing Electronic Warfare Self-Protection, Project Echidna

Jan. 2008 – July 2010

2 years and 6 months

1 January 2010

6

acJP 2043 — High Frequency Communications Modernisation

Jan. 2008 – June 2011

3 years and 5 months

Not yet reached.

7

aAIR 5333 — Air Defence Command and Control System ‘Vigilare’

Jan. 2008 – June 2011

3 years and 5 months

19 December 2012

8

abSEA 1448 Phase 2A/2B — ANZAC-class Anti-Ship Missile Defence (ASMD)

Jan. 2008 – Nov. 2011

3 years and 10 months

Not yet reached.

9

acAIR 5077 Phase 3 — Wedgetail, Airborne Early Warning and Control Aircraft

Jan. 2008 – Dec. 2012

4 years and 11 months

26 May 2015

10

aJP 2070 — Lightweight Torpedo Replacement Program

Jan. 2008 – Dec. 2012

4 years and 11 months

25 September 2013

11

JP 2048 Phase 1A — LPA Watercraft

July 2008 – Feb. 2011

2 years and 7 months

Cancelled

12

JP 0129 — Tactical Unmanned Aerial Vehicles

June 2008 – Dec. 2011

3 years

31 August 2014

13

JP 2088 Phase 1A — Air Drop Rigid Hull Inflatable Boat Trailers

July 2008 – Sept. 2009

1 year and 8 months

No FOC declared. Materiel Capability Acceptance 29 March 2010

14

bLAND 121 — Overlander, Medium and Heavy

July 2008 – Dec. 2011

3 years and 11 months

Not yet reached. Current forecast: 31 December 2023

15

CN10 — Collins Class Submarine Sustainment

Nov. 2008 – Oct. 2017

8 years and 11 months

[In sustainment — Not applicable]

16

cAIR 5402 — Air to Air Refuelling Capability

Feb. 2010 – Feb. 2015

5 years

1 July 2016

17

cAIR 5418 — Joint Air-to-Surface Standoff Missile (JASSM)

Nov. 2010 – Dec. 2011

1 year

20 January 2014

18

AIR 5276 — AP-3C Electronic System Measure System Upgrade

Oct. 2010 – April 2014

3 years and 7 months

6 May 2016

19

bAIR 9000 Phases 2/4/6 — MRH90 Multi Role Helicopters

Nov. 2011 – [continuing]

> 7 years and 1 month

Not yet reached. Current forecast: Dec 2021

20

LAND 40 Phase 2 — Direct Fire Support Weapon

Dec. 2012 – April 2016

3 years and 4 months

13 April 2018

21

JP 2086 Phase 1 — Mulwala Redevelopment Project

Dec. 2012 – April 2017

4 years and 4 months

15 December 2016

22

bSEA 4000 — Air Warfare Destroyer (AWD)

June 2014 – Feb. 2018

3 years and 8 months

Not yet reached. Current forecast: March 2021

23

JP 2008 Phase 3F — Defence SATCOM Terrestrial Enhancement

Sept. 2014 – August 2018

3 years and 11 months

Not yet reached.

24

bAIR 5431 Phase 3 Civil — Military Air Traffic Management System (OneSKY)

July 2017 – May 2018

10 months

Not yet reached. Current forecast 31 June 2023

25

AIR 5431 Phase 1 — Deployable Defence Air Traffic Management System

Aug. 2017– [continuing]

> 1 year and 4 months

Not yet reached. Current forecast 31 August 2019

         

Note a: In December 2007, Defence advised the incoming Minister for Defence that these ten projects — among some 215 projects then being managed within Defence by the Defence Materiel Organisation — were considered by Defence to be Projects of Concern. AIR 87 — ARH Tiger Armed Reconnaissance Helicopter and AIR 5077 Phase 3 — Wedgetail, Airborne Early Warning and Control Aircraft are known to have been on the Projects of Concern list from at least 2002.

Note b: These six are among the 27 projects reviewed in the 2016–17 Major Projects Report (Auditor-General Report No. 26 2017–18, 2016–17 Major Projects Report). The Project Data Summary Sheets in that report (p. 119 forward) give extensive detail on each project. Many of the projects that have been on the Projects of Concern list have been the subject of individual Auditor-General performance audits, including: Auditor-General Report No.26 2015–16, Defence’s Management of the Mulwala Propellant Facility; Auditor-General Report No.52 2013–14, Multi-Role Helicopter Program; Auditor-General Report No.22 2013–14 Air Warfare Destroyer Program, and Auditor-General Report No.23 2008–09, Management of the Collins-class Operations Sustainment.

Note c: These projects have been reviewed in earlier years’ editions of the Major Projects Report.

Note: No project which has been removed from the Projects of Concern list has returned to the list later. Defence advises that it has no rule, written or unwritten, prohibiting such an occurrence: it has simply not occurred.

Source: ANAO analysis of various Defence documents and ministerial press releases. Final Operational Capability (FOC) dates advised by Defence.

Length of time on the Projects of Concern list

1.6 Entry to the Projects of Concern list, and exit from it, is decided by ministers. The duration the 25 projects have spent on the Projects of Concern list differs widely, from two months (Super Seasprite helicopter) to eight years and 11 months (Collins Class submarine sustainment). The elapsed time is illustrated in Figure 1.1.

Figure 1.1: Projects of Concern, January 2008–September 2018

A bar graph showing the elapsed time on the Projects of Concern list for each of the 25 projects that have been, or were, projects of concern between January 2008 and September 2018.

Note: an asterisk identifies the ten projects on the original Projects of Concern list.

Source: ANAO analysis of Defence data.

1.7 On the face of it, the mean elapsed time on the list of this set of projects is three years and two months. However, it should be borne in mind that:

  • a range of projects had already been recognised by Defence as Projects of Concern before January 2008. For example, the Super Seasprite had been described internally as a Project of Concern in 2002; the Lightweight Torpedo replacement project had been described in this way in 2004; and the M113 Armoured Personnel Carrier went on the list in 2006. It is not known how long other projects had been there that were identified as being on the list at the start of 2008;
  • Collins Class Submarine Sustainment activity had been considered a Project of Concern earlier than November 2008, but that is the effective date accepted by Defence;
  • two projects remained on the list at the time this report was being prepared and their completed duration on the list will remain unknown until they are removed; and
  • the audit has identified instances of projects whose entry onto the list has apparently been delayed for various reasons (see Chapter 3).

1.8 Since 2008, the number of projects on the Projects of Concern list at any one time reached a maximum of 12 during 2008–09 and 2010–11, but has a declining trend, overall, from mid-2011 to the present (Figure 1.2 on the following page). Defence has advised the ANAO that this reflects the nature of the equipment program following the changes introduced by the Kinnaird and subsequent reviews of Defence procurement.11 These reviews led to a greater priority being given to military-off-the-shelf (MOTS) acquisitions and less on developmental acquisition projects.12

Figure 1.2: Number of concurrent Projects of Concern, by month, January 2008–December 2018

A line graph showing the number of projects concurrently on the Projects of Concern list by month from January 2008 to December 2018.

Source: ANAO analysis of Defence records.

1.9 The Projects of Concern list attracts a profile in Parliamentary consideration of Defence matters and in media discussion. For most of the history of Projects of Concern, Defence has used specific criteria to provide a basis to recommend that a project be placed on the list. The approach was then enhanced with further mechanisms to invigorate and formalise the process in mid-2011. From 2017, a set of (unpublished) principles has been followed rather than specific criteria (see Appendix 2).

1.10 Defence has also maintained separate Projects of Interest and Sustainment Products of Interest lists in various forms since around 2005 (see Appendix 3).13 A Project or Product of Interest is one which Defence management considers to be underperforming and in need of senior management attention and close monitoring to prevent it deteriorating to the point of becoming a Project or Product of Concern. The Projects of Interest and Products of Interest lists are not published by Defence.

1.11 Projects of Concern and Projects of Interest form only a small proportion of the capital acquisition projects that Defence has under management. Defence reported that it had 198 major and minor capital projects under way as at 30 June 2018.

Audit approach

Rationale for undertaking the audit

1.12 The reason for undertaking the audit is that Projects of Concern include projects that contribute substantial capability to the Australian Defence Force and involve a major resource commitment by the Australian Government. As a mechanism for resolving difficulties with Defence projects, there is a clear link between the effectiveness of the Projects of Concern regime and Defence’s strategic priorities as stated in one of its purposes under the Public Governance, Performance and Accountability Act 2013 (PGPA Act): ‘Deliver and sustain Defence capability and conduct operations’.14 Further, the Projects of Concern regime regularly receives Parliamentary attention and this audit is intended to provide insight into how Defence operates and manages the Projects of Concern regime.

Audit objective, criteria and scope

1.13 The objective of the audit is to assess whether Defence’s Projects of Concern regime is effective in managing the recovery of underperforming projects.

1.14 The following high-level criteria were adopted for the audit:

  • Defence has established an appropriate framework for the Projects of Concern regime, including processes for the entry, management and exit of projects.
  • Defence applies the Projects of Concern regime with an appropriate degree of consistency.
  • Defence has established appropriate internal and external reporting arrangements on the progress of Projects of Concern.
  • Defence can demonstrate that the Projects of Concern regime contributes materially to the recovery of underperforming projects and products.

Audit method

1.15 The audit was conducted by:

  • examining the frameworks used by Defence for identifying projects that require additional management attention as Projects of Concern;
  • examining documentation on the management, implementation, monitoring and reporting of Defence projects, scrutinising, in particular, the records of Defence Gate Reviews and Independent Assurance Reviews;
  • examining three prominent case studies, of which two comprise projects that have recently been removed from the Projects of Concern list and another which is expected to be removed within a year or so; and
  • interviewing key Defence officials and representatives of the defence industry. Contributions were also received from an Australian industry peak representative body and a representative of a major supplier organisation.

1.16 This audit was conducted in accordance with the ANAO Auditing Standards at a cost to the ANAO of approximately $398,000.

1.17 The team members were David Rowlands, Natalie Whitely, Sonia Pragt and Sally Ramsey.

2. The framework for the Projects of Concern regime

Areas examined

This chapter examines whether Defence has established an appropriate framework for the Projects of Concern regime, including a clear purpose and scope, and policy and procedures for entry, management and exit of projects from the list.

Conclusion

Defence no longer has an appropriate framework for its Projects of Concern regime. The regime has two clear purposes: to resolve troubled capability development projects through remediation or cancellation with the explicit involvement of ministers; and to help keep ministers informed. However, its current implementation lacks rigour. From 2008 forward, ministers’ involvement heightened the focus on troubled projects and strengthened the regime. It was more fully developed in 2011, with the introduction of regular summit meetings chaired by ministers to review progress and stimulate action. Over the last five years, transparency has reduced, the level of formality has declined with explicit criteria replaced by unpublished principles, and processes have become less rigorous with a greater emphasis on maintaining relationships with industry.

Does the Projects of Concern regime have a clear purpose and scope?

The Projects of Concern regime has a clear purpose and scope. Its purpose is to help keep senior Defence leaders and ministers informed of material acquisition projects in difficulty and resolve problems in the project’s progress either through remediation or, where that is not practicable, cancellation. The regime has applied almost exclusively to underperforming materiel acquisition projects. Despite running in parallel for almost two decades, Defence has only recently sought to align its contractor performance data across its Performance Exchange Program and the Projects of Concern regime to ensure that views on contractor performance are consistent.

Purpose

2.1 The Projects of Concern regime has two purposes:

  • to ensure that projects in serious difficulty attract senior (including ministerial) attention with a view to resolution through remediation or cancellation. The successful recovery of the project could involve a change in scope, cost or schedule15; and
  • to keep ministers informed both of projects in difficulty and their progress to resolution.
Drawing underperforming projects to the attention of decision-makers

2.2 Under the PGPA Act, the Secretary of Defence has a duty to keep the responsible minister informed of any significant issue that has affected Defence.16 This is a key accountability relationship between a department and its minister. Making and keeping ministers aware of problematic projects is also a prerequisite to their involvement in resolving difficulties for the project. The first stated purpose for the Projects of Concern regime is to ensure that projects in difficulty receive ministerial and senior management attention within Defence and engagement at the most senior levels of the vendor of the project.

2.3 Further, by making the project’s difficulty known to the public, Defence expects that commercial pressure on the vendor will facilitate action on the vendor’s part that will lead to or help substantially with remediation. Defence has put the view that it has ‘few better mechanisms for influencing commercial behaviour’ than the Projects of Concern list, including the threat to a vendor’s reputation of being put on it and the possibility of being excluded from future tenders.17 There is an underlying presumption in this mechanism that vendor performance is the aspect of the project most in need of attention. Only projects where an opportunity for improvement could flow from applying pressure to industry are regarded as potential candidates for nomination to the list.

Timely consideration of underperforming projects is important

2.4 Timely consideration of underperforming projects by decision-makers is important, particularly if cancellation may be an appropriate outcome where an underperforming project is unlikely to provide good value for money or there is a substantial risk of a waste of public resources. For example, the Super Seasprite helicopter project, a Project of Concern that was subsequently cancelled, cost over $1.4 billion and yielded no capability whatsoever.18

2.5 Delay in making the cancellation decision — or failure to make that decision — raises the risk of incurring additional sunk costs, as explained by a Senate Committee inquiry into Defence procurement:

a delayed or unsuccessful project creates a capability gap, fails to meet the government’s strategic requirements, damages Defence’s relationship with industry and undermines public and parliamentary confidence in Defence’s procurement program.19

2.6 Delaying the entry of projects onto the list, as noted in Chapter 3, may undermine the purpose of the list and result in wasted public resources if the project must subsequently be cancelled.

Keeping ministers informed

2.7 The Minister for Defence set out an additional purpose for Projects of Concern in 2010:

[The Minister for Defence Materiel] and I will announce later today that project AIR 5418, the acquisition of the Joint Air-to-Surface Standoff Missile (JASSM), has been added to the Projects of Concern list. This listing is not primarily because of industry delays or cost increases. It is because of our poor management, our failure to keep Government properly and fully informed about the Project and its difficulties.20

2.8 The Minister raised this matter with Defence on numerous occasions.21 Defence records show that Defence did not provide full and frank information in a timely manner on at least one occasion where a classified project with an approved $50 million facilities component had suffered a ‘cost blow-out’ to $150 million in those costs. After being aware of this for several months the relevant Defence groups could not agree to advise the Minister.22 Further, Defence records indicate concerns about providing frank assessments of Projects of Concern to the Minister, in particular, if that information might then also be available to the Auditor-General.23

2.9 In direct response to this finding, Defence advised the ANAO in February 2019 that ‘Defence is of the view that anything pre-2015 is of limited value due to the significant shift in the nature of the relationship with Industry and the Defence Operating Model as a result of the First Principles Review’.

Scope

2.10 Of the 25 projects listed as Projects of Concern between 2008 and 2018, 24 have been projects to develop and acquire military capability (acquisition projects).24 In addition, one ‘product’ under sustainment (Collins Class Submarine sustainment) has been listed.

2.11 Projects have generally been listed after ‘Second Pass’ approval of the project by government, where the vendor has been selected and contracts are in place or nearly so. In two cases, acquisition projects have entered the list before contracts have been signed with vendors. These have been AIR 5431 Phase 3, the Civil and Military Air Traffic System (CMATS), and LAND 40 Phase 2, Direct Fire Support Weapons.25

Performance Exchange Program

2.12 Since 2001, Defence has monitored and reported on the performance of major Defence contractors for capital acquisition and in-service support. Previously known as the ‘Company Scorecard Program’ and now known as the ‘Performance Exchange Program’, the program encompasses contracts that exceed the thresholds of $10 million (acquisition); $5 million (sustainment) and $5 million (services). Defence has advised that, ‘under the Performance Exchange Program, decision-makers across both Defence and industry now have a set of measures that are directly relevant to decision-making and performance across domains, military capabilities and tiers across the Defence industry sector’.26

2.13 The Performance Exchange Program and the Projects of Concern regime have a common objective, better performance, and they have proceeded in parallel for nearly two decades. Defence has raised concerns that inconsistent views can develop between the two processes. Defence has advised that:

… an updated Performance Exchange process has been developed and implemented to address this issue, as well as to deliver broader, and honest, two-way performance reporting between Defence and its contracted key industry partners.27

Has Defence established policy and procedures for the entry, management and exit of projects from the Projects of Concern regime?

Defence has established a policy for its Projects of Concern regime and procedures for the entry, management and exit of projects. In 2009 and 2011, Defence’s approach was made more formal and rigorous at the instigation of ministers. In recent years, the transparency and formality of the process have diminished, regular summits chaired by ministers have been replaced by ad hoc meetings, and Defence no longer publishes its principles and procedures for Projects of Concern. Maintaining collaborative relationships with industry has become a more dominant element in the governance of the regime.

2.14 The development of Defence’s approach to managing its Projects of Concern regime is reflected in the statements of processes, criteria and principles set out at various times since the commencement of the regime in 2001 (see Appendix 2):

  • In its earliest form (2001), Defence adopted a simple set of ‘hard’ criteria for making a project a Project of Concern, relating to governance, capability, schedule and budget. The criteria at this point were substantially numerical and the regime was largely internal to Defence.
  • Ministers became involved in 2008; the subsequent statement of criteria in 2009 moved away from numerical triggers and introduced a greater element of judgement. It introduces procedures for entry to the list, and exit, both by ministerial approval. A requirement for a plan for resolution (through cancellation or remediation) was required from this point.
  • The 2011 statement, announced in a press release by the Minister for Defence, introduced substantial additional elements, including incentives for industry to fix problem projects; a new ‘Early Indicators and Warnings’ system to flag potential problem projects; the use of Gate Reviews to analyse each such project; more detailed requirements for remediation plans and greater ministerial involvement through bi-annual Projects of Concern reviews (summits), being face-to-face meetings with the minister ‘to hold responsible individuals to account’.
  • In 2017, the ‘Statement of Principles’ introduced an emphasis on a ‘collegiate culture’ and included no reference to incentives for industry. There was no mention of review meetings (summits) with the minister.
  • The 2018 principles set out more procedural activity, including reporting to government and Parliamentary committees. The principles again mention summit meetings with the minister but these are to be held ‘when required’.

Ministerial involvement in strengthening the Projects of Concern regime

2.15 In mid-2008, the General Manager, Programs, Defence Materiel Organisation flagged ‘a major shift in thinking’ at ministerial level to his colleagues. He told them that ‘Projects of Concern are a serious issue for Defence and are currently attracting high public and political visibility and could be subject to precipitous decisions’. He stated:

From my meetings with the Parliamentary Secretary and the staff in the Minister’s office it is clear to me there has been a major shift in thinking about these projects. The new default way forward for a “Project of Concern” is to assume that the project will be cancelled unless sufficient justification is provided each month that the project can deliver its scope dependably within any agreed (revised) schedule and funding. Failure to do so will result in the project being recommended for cancellation by the Minister and Parliamentary Secretary. This puts the onus on the project office (assisted by the sponsor and capability manager) to show that they have an acceptable way forward to delivering the approved materiel capability within the agreed schedule, budget and risk profile.28

2.16 With projects identified as Projects of Concern to be subject to ‘intense management’, with a prospect of cancellation, Defence required a remediation plan that set out for each Project of Concern ‘an acceptable way forward to delivering the approved materiel capability within the agreed schedule, budget and risk profile’. Ministers became more involved, including by the Minister for Defence Materiel deciding entry to the list and exit, on Defence’s advice.

2.17 The General Manager emphasised that ‘While cancellation may be a suitable outcome in a small number of cases, in most cases it would be very unacceptable from a capability and reputation point of view’. He also advised Service Chiefs that if a Project of Concern were cancelled, they should not assume there would be a replacement program. He also stated that it was important to reduce the Project of Concern list to zero in the shortest possible time.

2.18 Defence then set up a Projects of Concern unit to report to the Parliamentary Secretary on progress with remediation:

The unit has been tasked with providing assistance and advice to the projects to help them get back on track, while also drawing detailed information from each project and providing it to Government on a monthly basis. This has allowed the Government to become more active in the management of these projects and has informed its decision making when it comes to how we should handle these projects.29

2.19 The process was further strengthened in 2011 with the Minister for Defence engaging with industry on how to improve the process. Industry advice formed the basis for reforms announced on 29 June 2011 by the Minister for Defence and the Minister for Defence Materiel. The reforms, which applied from 1 July 201130 included:

  • incentive arrangements for industry to focus on fixing problems — through weighing a company’s performance in remediation when evaluating tenders for other projects;
  • a formal process for adding projects to the list — using the triggering of Early Indicators and Warnings (discussed below) as a basis for advising ministers and potentially leading to a Gate Review;
  • a requirement for formal remediation plans — prepared by Defence and industry;
  • a process for removing projects from the list — involving government decision; and
  • increased ministerial involvement — through bi-annual reviews with Defence and industry.
Early Indicators and Warnings

2.20 Earlier concern that Defence senior management — and ministers — be kept aware of projects in difficulty led Defence to develop a system of mandatory ‘Early Indicators and Warnings’ on project progress. The triggering of one or more of these criteria was expected to lead to a Gate Review and consideration of whether the project should be nominated for inclusion on the Projects of Concern list.31

2.21 The Early Indicators and Warnings system emphasised early identification of risks and problems, as the name indicates.32 The Ministers’ announcement stated:

Defence assesses that 80 per cent of problems with Defence capability projects occur in the first 20 per cent of the project’s life.

That is why it is important to pick up problems early.

One of the biggest challenges in Defence procurement is projects running late. The earlier these issues are picked up, the earlier the problem can be fixed.

The Government will implement an Early Indicators and Warning System. This system will help identify and correct potential problems with projects.33

2.22 Before the 2011 reforms, nomination depended upon project staff identifying that the project had crossed a threshold and/or senior managers in Defence identifying the underperformance from internal reporting. A significant change to the process of nominating projects flowed from the use of Gate Reviews and their successor, Independent Assurance Reviews, to provide a critical review and considered approach. That process continues.34

Operation of the Projects of Concern regime in 2018

2.23 Defence has stated in evidence to the Senate Foreign Affairs, Defence and Trade Legislation Committee that the Projects of Concern regime is a disciplined process, both for entry and exit.35 In December 2018, that process was set out in Defence’s operating principles on its intranet site.36 In contrast to the 2011 principles, Defence does not make the 2018 principles public.

2.24 There has been a change over the years in the character of the criteria used to nominate projects to the Projects of Concern list (see the sets of criteria at various points of time set out in Appendix 2). In the earlier years, entry to the Projects of Concern list was based on a range of ‘hard’ criteria, such as a percentage change in expected cost or schedule being exceeded. A more recent characterisation is that ‘There are no set quantitative measures or thresholds …’ (2017). The rationale has been that the complexities that characterised the projects under consideration cannot be captured in such criteria.

2.25 A significant change is that, whereas the 2011 process included the Minister for Defence Materiel holding bi-annual reviews of Projects of Concern (summits) with Defence and Industry representatives, that element had been omitted from the 2017 edition (see Appendix 2) and referred to in the current principles as occurring ‘when required’. There is also a greater emphasis on collaboration between Defence and industry and less on mechanisms such as providing incentives for industry to fix problem projects. Defence has stated to the ANAO that ‘The main focus is on collaboration to obtain the best possible capability outcomes within the schedule and cost constraints of the program’.37

2.26 In February 2019, Defence advised that project cancellation remains an option:

Placing a project on the PoC list aims to provide additional support to remediate the project and deliver capability and value-for-money outcomes to the Department of Defence. If these outcomes cannot be achieved, project or contract cancellation may be considered.

Industry view

2.27 The ANAO received a submission from the Australian Industry Group, on the Projects of Concern process stating:

Industry feedback noted that the Projects of Concern process can be valuable; however, the current system lacks transparency. Industry comments stated that the criteria used to establish a Project of Concern are unclear, as well as the processes used to manage and resource the project once on the list.38

2.28 There is no public information on either the Defence Internet site or the Defence Annual Report 2017–18 on how the Projects of Concern process operates.39 The Australian Industry Group stated:

Overall, our members’ view is that the Projects of Concern process should have a high degree of visibility and status, a clear and transparent set of criteria and scheme for elevation, and be part of the regular conversation between Defence and industry.

Management of Projects of Concern following the First Principles Review

2.29 Following the First Principles Review, Defence introduced its Integrated Investment Program to govern investment in Defence’s strategic goals. The First Principles Review also led Defence to introduce its new Capability Life Cycle and the notion of ‘capability streams’:

To allow effective high level prioritisation and communication with Government, the Integrated Investment Program is structured into six Capability Streams which were developed through the 2015 Force Structure Review process. Defence capabilities (Programs) were defined and mapped to the Capability Streams.40

2.30 The Air Warfare Destroyer build case study undertaken as part of this audit shows that a risk which came to attention during the project — declining shipyard productivity — may now be more significant to the progress of the wider capability stream Maritime and Anti-Submarine Warfare than to the remainder of the Air Warfare Destroyer build project itself.41

2.31 However, the operation of the Projects of Concern regime is not well-defined in the context of the new Capability Life Cycle. There is no reference to Projects of Concern in: Defence’s Capability Life Cycle — detailed design; Capability Acquisition and Sustainment Group’s Complex Procurement Guide (March 2018); the 2016 Defence White Paper or the accompanying 2016 Integrated Investment Program and 2016 Defence Industry Statement. The absence of clear guidance on the operation of the Projects of Concern regime as part of the risk management framework gives the appearance that the regime is of marginal importance in the context of the Capability Life Cycle.

2.32 As a result of the First Principles Review, a proposal was developed within CASG in early 2017 for the potential extension of the Projects of Concern regime to align it with the One Defence framework. Such an extension could usefully encompass the issue of broader application of risks identified in any particular project to the capability stream of which it forms a part.42 In October 2018, Defence indicated that, following a discussion by its Investment Committee in September 2018, it supports expanding the scope of the Projects of Concern regime to encompass all projects managed across Defence, and this would be consistent with the One Defence approach.43

3. Application of the Projects of Concern regime

Areas examined

This chapter examines whether Defence has applied its Projects of Concern regime with an appropriate degree of consistency, including the entry, management and exit of projects from the Projects of Concern list.

Conclusion

There has been inconsistency in Defence’s application of its Projects of Concern regime. In particular, application of processes for entry onto the list have been inconsistent and summit meetings to address Projects of Concern have become less frequent. Greater consistency has been maintained in preparing remediation plans and removing projects from the list, though there have been exceptions to both.

Area for improvement

This chapter recommends that a formal policy with supporting procedures be introduced for the Projects of Concern regime to improve consistency in managing entry to, and exit from, its Projects of Concern list and that Defence follow through on its risk assessment reform program.

Further to the recommendation, the introduction of guidance for risk assessment of projects by Independent Assurance Reviews would help Defence improve the effectiveness of the regime.

Does Defence apply a consistent process to the entry of projects to the list?

Defence does not apply a consistent process to the entry of projects to the Projects of Concern list with evidence of delays as well as advice being withheld from review processes and decision-makers. Procedures for Independent Assurance Reviews do not explicitly mention Projects of Concern even though such reviews are the primary occasion for nominating a project to be a Project of Concern. There was evidence that most reviews (75 per cent) had considered whether a recommendation should be made.

Procedures for recommending a project as a Project of Concern

3.1 From June 2011 forward, the Projects of Concern processes require a recommendation that a project be considered for inclusion on the Projects of Concern list to originate from a Gate Review or Independent Assurance Review.44 One of the primary purposes for Independent Assurance Reviews is the early identification of problem projects and sustainment products.

3.2 The existing 19-page formal instruction governing the operation of Independent Assurance Reviews does not instruct an Independent Assurance Review Board to consider whether a project should be nominated for inclusion on the Projects of Concern.45 As Projects of Concern are not directly mentioned in the instruction, there is a risk that troubled projects might not be drawn promptly to the attention of senior management for consideration for listing as a Project of Concern. There would be merit in updating instructions for Independent Assurance Reviews to reflect their role in recommending whether a project should be a Project of Concern.

Consideration of projects as Projects of Concern

3.3 From a review of records of 191 Gate Reviews and Independent Assurance Reviews conducted since 2015, the following challenges in the management of entry of projects to the Projects of Concern list were identified:

  • in a minority of cases, no record of whether consideration was given by an Independent Assurance Review as to whether a project warranted nomination to the list;
  • potentially delayed entry of projects to the list;
  • delays in advice to ministers of a project in difficulty;
  • restrictions on information available to an Independent Assurance Review;
  • deciding not to nominate a project because it might attract additional workload;
  • incidental identification of a potential Project of Concern; and
  • an inconsistent approach to publicising additions to the Projects of Concern list.

3.4 These instances are discussed below.

Inconsistent consideration as to whether a project warrants nomination

3.5 Of the 191 Gate Review Boards and Independent Assurance Review Boards reviewed, in 144 cases (75 per cent) the Board both considered whether the project should be recommended for Project of Concern status and recorded its view (as outlined in Table 3.1). Of the remaining 47 cases, there were:

  • 27 cases where the project was in the early or late stage; and
  • 20 cases (10 per cent) with no record of consideration of whether the project should be nominated as a Project of Concern or as a Project of Interest (see paragraph 1.10).

Table 3.1: Gate Review and Independent Assurance Review consideration

 

SEA

LAND

AIR

JOINT

Total

No. of Gate Reviews/IARs examined

31

41

72

47

191

No. where consideration of Project of Interest/ Project of Concern status was not recorded

4

11

26

6

47

No. where a rationale was not apparent for omitting consideration of this status.

2

1

14

3

20

           

Source: ANAO analysis of Defence records

Delayed entry to the Projects of Concern list

3.6 Defence records indicate that there have been occasions when considerable time elapses between the first record of Defence contemplating the nomination of a project to be entered onto the Projects of Concern list and its actual entry. Any unnecessary delay in action leading to the resolution of projects carries a number of risks. Some of these risks are that:

  • as Defence is aware (paragraph 2.21), the more advanced the project the greater the costs of any corrective action;
  • delays can reduce the capacity of Defence to influence, on terms favourable to the Australian Government, the vendor’s actions in support of problem resolution; and
  • delays increase reputational risk associated with more radical options available to Defence (such as project cancellation) and increase the risk of the public resources being wasted as observed, for example, in the Super Seasprite project.46

3.7 Instances of lengthy periods, typically two years, elapsing between Defence’s observing problems with a project and the project entering the Projects of Concern list are set out in Box 1.

Box 1: Projects of Concern with delayed entry to the list

  • SEA 4000 Phase 3, Air Warfare Destroyer47 — Problems were identified with the project in the three years before its Project of Concern listing, including: evidence of low productivity in 2011, schedule slippage in 2012, and cost increases in 2013. The project was added to the Projects of Concern list in June 2014.
  • AIR 9000 Phases 2, 4 and 6, Multi-Role Helicopter48 — Defence held internal discussions in September 2010 about listing the project as a Project of Concern. Defence briefed the Minister for Defence Materiel in late 2010, recommending that the project become a Project of Concern because of continuing delays, the plan to stop aircraft acceptance and issues including engine failure. Gate Reviews in February, September and October 2011 identified additional shortcomings. Following the October 2011 Gate Review, ministers were advised that: ‘the program has triggered schedule and contractor performance Early Indicator and Warning thresholds, and [Defence] recommended that the project be listed and managed as a Project of Concern by Defence.’ The project became a Project of Concern in November 2011.
  • AIR 5431 Phase 3, Civil Military Air Traffic Management System — A November 2015 Gate Review identified problems in project governance and program management: ‘consideration of the elevation of the [project] to a Project of Concern is warranted.’ The Deputy Secretary, Capability Acquisition and Sustainment Group, concluded that the project was: ‘a strong candidate for a Project of Concern’ but sought the view of the CEO, Airservices Australia. As a result, the decision was postponed. Ministers decided that CMATS would be a Project of Concern in May 2017, some 19 months later.49
  • AIR 5276, AP-3C Electronic System Measures upgrade — The Defence Materiel Organisation’s Head of Aerospace Systems stated in July 2009 that this project had been on the margins of the Projects of Concern list since the vendor announced a 12 month schedule slippage in July 2008. In July 2009, a further six months’ slippage was announced. Defence decided to continue to use the threat of listing the project as a Project of Concern ‘until it was no longer useful’. The project entered the list in October 2010, a year later.
  • JP 2008 Phase 3F, Defence Satellite Communications Terrestrial Enhancement — A November 2011 Gate Review of the project identified schedule delays and contractor performance issues. The subsequent (August 2012) Gate Review noted further schedule slippage and that the Commonwealth and contractor had not been able to agree to a re-baselined schedule. The review identified a need for active senior leadership direction and oversight from both parties to resolve commercial issues, and that a Project of Concern listing should be considered if they were not resolved by September 2012. In 2012, a Deed of Settlement rebaselined the schedule; however, in September 2013 a Gate Review found that the reliability of the schedule lasted less than three months and the first milestone had not been achieved. It concluded that the project should: ‘be immediately recommended to the Minister for management as a [Project of Concern]’ and that the project: ‘commence development of a remediation plan’. In May 2014, Defence recommended that the project be added to the Projects of Concern list. Defence records indicate that the Minister was unwilling to do so as the government was considering involving the vendor in a remediation strategy for a much more significant project. In September 2014, the Minister asked that the project be added to the Projects of Concern list.
  • AIR 5431 Phase 1, Deployable Defence Air Traffic Management System — A November 2015 Gate Review for the project noted significant schedule slippage with a projected delay of 16 months for initial materiel release. In internal Defence documentation the project was identified in August 2016 as a potential Project of Concern due to contractor issues. A further Gate Review in November 2016 noted continuing schedule slippage and contractor underperformance. The review board did not recommend that the project be listed as a Project of Concern, but noted that: ‘the threatened or actual use of this option remains a credible means of incentivising [the contractor]’. The project was added to the Projects of Concern list in August 2017.

3.8 In relation to delayed entry of projects onto the list, Defence advised the ANAO in February 2019 that:

It needs to be acknowledged that there is a qualitative/management judgement aspect to [Projects of Concern] that has always existed and always will. Arguably, some projects could have been added sooner, however, Defence officials are always cognisant of the trade-off between allowing a company more time to ‘come good’ and the right time to add to the PoC list, including the cost/benefit analysis. A more detailed analysis would reveal that behind the scenes there is a considerable amount of engagement, negotiation, consultation, expectation setting, between the company (and/or Parent Company), Defence and often the Minister’s office. Usually, the time is related to companies working toward contractual milestones, or complex technical solutions which can be lengthy but have been negotiated in good faith. Defence often needs to wait for the expiry of timeframes (some of which are legally bound) before further firm action is taken. That does not imply that nothing is done, nor that the implication of becoming a project of concern has not been socialised at the senior level with a company.

Delays in advice to ministers

3.9 AIR 5431 Phase 1, mentioned among the case studies above, also represents a delay in Defence drawing the attention of ministers to a project that is underperforming. The November 2015 Gate Review, having observed a 16-month slippage only 11 months into the contract, recommended that government be informed of current project status, as the schedule was unlikely to be recovered.

3.10 A year later, the Defence Agendum paper prepared for the subsequent Independent Assurance Review (November 2016) noted that:

There has not yet been any specific updated advice to Government on the status of this project, despite the Gate Review recommendation last year. DGCP-AF [Director-General, Capability Planning–Air Force] advises that this project is a lower priority, both for Air Force and for Ministers, and that there is not an urgent need for such an update. Government will be advised ‘in due course’, as part of broader advice on the full range of air traffic control projects.50

3.11 As noted above, the project was added to the Projects of Concern list a year later, in August 2017, where it remained in December 2018.

Restricting information supplied to Independent Assurance Reviews

3.12 Given the important role of Defence’s Independent Assurance Reviews in identifying problem projects and products, including potential Projects of Concern, it is essential that they be supplied with all the information they need to carry out their work. The Independent Assurance Review conducted for SEA 1180 (Offshore Patrol Vessel) on 17 July 2017 reported as follows:

The team were restricted in being able to provide any meaningful detail, even when there was no suggestion that they provide the outcomes for each tenderer. Consequently, I am not able to provide an informed independent assessment of the deliverability of this project, nor of the appropriateness of the schedule, cost and risk assessments. I do feel that this is a missed opportunity for the project given the breadth of experience of the Board Members. Furthermore, I am not sure of the value of conducting an Independent Assurance Review for Gate 2 projects that have such restrictive information access …

Given the limitation for access to information for reviewers and the restrictions imposed on the project team and the Capability Sponsor’s ability to speak to the specifics of tender responses, particularly as they relate to cost, schedule and risk at the Board Meeting, I am not in a position to advise on the true status of the project, whether the cost estimates, schedule or risk assessments are a sound basis for progressing to Second Pass Approval, nor what more needs to be done to support the Approval Process or to enhance the actual delivery [Emphasis in original].

3.13 The reason for the restriction in supplying information to the Independent Assurance Review is not evident in the documentation. This approach has several consequences:

  • such restrictions reduce the value of holding an Independent Assurance Review;
  • the approach could keep from senior management risks that the Independent Assurance Review is best placed to identify. Those risks could then develop and crystallise; and
  • even if this instance is unusual, it may form an undesirable precedent that, if followed, could lead to failure to detect risks in other projects.

3.14 In response to this finding, Defence advised the ANAO:

This example is isolated and was particularly disappointing…However, it should be noted that there is no identified systemic issue of information being withheld from IAR reviewers.51

Deciding not to nominate a project because it might attract additional workload

3.15 An Independent Assurance Review for project JP 2008 Phase 5a, UHF SATCOM, took place on 1 August 2017. The Review Board considered that there were several significant issues that put the project in jeopardy. The Board concluded that ‘the Government must be advised about the schedule slippage to Final Operational Capability’.52 It expressed concern at the lack of senior executive oversight and found that the budget was a ‘major concern’.

3.16 Nevertheless, the Board’s ‘summary assessment’, considered and approved by senior officers (SES and star-ranking) in Defence, included the following finding:

The Board did discuss the possibility of elevating the project to a Project of Concern from a Project of Interest but agreed that this could create more work and challenges for a Project Office that is already overtasked.53

3.17 If the Projects of Concern regime is working effectively it should not be viewed by senior officers as creating bureaucratic overhead. Rather, it is intended to resolve problems that are proving intractable at lower levels of management and decisions not to employ the regime may result in delays and lost opportunities to progress a troubled project.

Incidental identification of a potential Project of Concern

3.18 An Independent Assurance Review of LAND 2072 Phase 2B, Integrated Tele-communications System — Land, took place in July 2017. The Board found that the project had made good progress, was ahead of schedule and the project team was handling issues professionally. Nevertheless, there were issues beyond the team’s control that presented contingencies to the project’s success. One of these was the delivery of the next generation of deployable local area network, a project being undertaken as a sustainment item. The Board was advised that the vendor’s performance had been ‘extremely disappointing’.

3.19 The Board concluded:

I am concerned by the eDLAN [enhanced deployable local area network] situation. It is unclear why an undertaking of this obvious complexity is being performed in a sustainment environment with inadequate governance and oversight. HJS [Head, Joint Systems] made an observation that if eDLAN was a Project, it would be ACAT II [Acquisition Category II, the second most significant] and on the current performance it would be a Project of Concern. CASG should review carefully the lessons being learned from the eDLAN experience and should engage with other stakeholders to ensure that follow-on DLAN projects are subject to more appropriate governance and management arrangements.

3.20 Even though there is a precedent for sustainment products being elevated to the Projects of Concern list (for example, CN10, Collins Class Submarine), there is no evidence of any further consideration within Defence of this finding.

Inconsistent approach to publicising additions to the Projects of Concern list

3.21 Defence has indicated that the operation of the Projects of Concern list depends substantially on the commercial pressure on vendors created by the public listing as a Project of Concern of the project they are involved in (paragraph 2.3). However, there have been occasions when a project has been added to the Projects of Concern list but this fact has not been made public, or at least, not until some months later, limiting the effect intended by elevating the project to that status. Instances include:

  • JP 2008 Phase 3F, Defence SATCOM Terrestrial Enhancement.
  • JP 2088 Phase 1A, Air Drop Rigid Hull Inflatable Boat Trailers.
  • AIR 5402, Air-to-Air Refuelling Capability.
  • JP 2048 Phase 1A, Landing Platform Amphibious Watercraft.

Reforming risk management practices

3.22 Following a review of the effectiveness of the CASG’s risk management framework, practices, systems and methodologies, Defence approved a Risk management reform program and implementation plan in June 2017 to remodel the Group’s risk management.54 This reform program noted that CASG had, at that time:

no standardised approach to risk identification, assessment, risk reporting or risk reduction. [Thirty-seven] different risk management software tools were identified alongside hundreds of different templates, methods and formats for capturing and presenting risk.55

3.23 The reform program set out priorities including to ‘develop business rules regarding Projects of Interest and Projects of Concern’. These would:

enhance the means by which Projects of Interest and Projects of Concern are identified, assessed and managed as part of the CASG Risk Management Model. The goal of this effort is to proactively move projects with this status to a level where the risk is in line with the organisational appetite for risk.56

3.24 In terms of Projects of Concern, the approved program of work set out the following:

For the Specialist Risk Area of Projects of Interest or Projects of Concern, develop a suitable module that will integrate and connect with the CASG Risk Management Model for this application. This module may include the “Practice Guide” for this Specialist Risk Area (p. 8).57

3.25 Subsequently, in August 2017, the Major Programs Control Directorate advised those leading the work on the Reform Program that the governance and processes for Projects of Interest and Projects of Concern already existed and that specific risk management practice was not required. There is no record of any further action having been taken on the planned work.

3.26 A subsequent Defence Internal Audit ‘Management of Selected Projects of Interest’ (April 2018) found that formal documentation of the process for Projects of Interest was not yet complete.58 The process relied on ‘professional judgement’. With no standard approach to risk identification, assessment, risk reporting or risk reduction, there can be no assurance that Projects of Concern — or Projects of Interest — are being identified appropriately and in a timely way.

3.27 Regardless of the detailed reasons for the difficulties that arise in any individual case, every project approved by government has a defined estimate of cost and schedule, and is expected to deliver a specified set of capabilities. These will all have been agreed by government and understood at the time of that approval. It should be possible to set tolerances — a risk appetite — which, when exceeded, form a clear and sound basis for concern, and a trigger to alert ministers to emerging risks.

3.28 The Projects of Concern regime could benefit from establishing a simple set of primary entry, management and exit criteria based on cost, schedule and capability, with further technical detail supplied to ministers and senior managers as and when required. These could be embodied in a short, transparent, formally-approved set of procedures which would help provide confidence that the process is being administered rigorously, fairly and consistently.

Recommendation no.1

3.29 Defence introduce, as part of its formal policy and procedures, a consistent approach to managing entry to, and exit from, its Projects of Interest and Projects of Concern lists. This should reflect Defence’s risk appetite and be made consistent with the new Capability Acquisition and Sustainment Group Risk Model and other, Defence-wide, frameworks for managing risk. To aid transparency, the policy and the list should be made public.

Department of Defence response: Agreed.

3.30 Defence agrees to this recommendation noting that Defence will endeavour to provide this formal policy as the Project Management specialist risk discipline is developed as part of the new CASG Risk Model. This work will build on the current quantitative measures against scope, schedule and cost to potentially include lead indicators of project performance. The Defence Projects of Concern list will continue to be made public.

Does Defence apply a consistent process to the management of projects while on the Projects of Concern list?

Broadly, Defence has applied a consistent process to the management of projects while on the Projects of Concern list. However, summit meetings involving the Minister, vendors and officials, a principal process devised in 2011 to help ensure that Defence can use its Projects of Concern regime to exert commercial pressure on vendors, are no longer regular and they have become less frequent. Another long-standing process, the preparation of remediation plans, has usually been followed.

3.31 Defence has used three principal processes to manage Projects of Concern while they are on the list: the preparation of remediation plans (commenced in 2009); the holding of regular summit meetings with the minister, attended by Defence officials and vendors (from 2011); and the application of commercial pressure through contract arrangements (from 2011).

Remediation plans

3.32 Since 2009, there has been a requirement that when a project is placed on the Projects of Concern list, a remediation plan would be prepared and agreed between the parties. Remediation plans were to be agreed at, or prepared in light of, Projects of Concern summits. For example, the department advised the Minister for Defence in October 2011 that:

During the recent Projects of Concern summit, 27–28 September 2011, you directed that a Remediation Plan and Action Log be developed by each project in consultation with the companies … Remediation Plans define the agreed strategy to remediate a project and the basis for removing the project from the Projects of Concern list. The Action Log defines the commitments made by the project manager, company CEO and others in each six month period between Projects of Concern summits.59

3.33 With an agreed remediation plan, it becomes feasible to measure progress against that plan with the action log. Remediation plans were to include the basis on which a project would be removed from the Projects of Concern list (‘removal criteria’). The prospect of further summit meetings at a regular frequency provides an incentive for participants to discharge their commitments before the next summit.

3.34 Defence’s June 2018 Projects of Concern report provided Defence senior leaders and ministers with:

  • a summary of remediation expectations for the short, medium and long term and removal criteria for all three projects on the list, a forecast date for removal and statement of progress towards that; and
  • a more detailed description of the expected path towards remediation but not specific remedial actions and accountability. Some of the activities are expressed in general terms such as ‘engage with the vendor’, rather than setting out specific remedial action and who is to do it.

3.35 The audit examined whether remediation plans had been prepared for the three projects selected as case studies (Air Warfare Destroyer, MRH90 helicopters and Collins Class submarine sustainment) and the other projects that remain on the Projects of Concern list or which have been removed over the preceding twelve months (AIR 5431 Phase 3, CMATS; AIR 5431 Phase 1, Deployable Air Traffic Management and Control System; JP 2086 Phase 1, Mulwala Redevelopment Project; and JP 2008 Phase 3F, Defence SATCOM Terrestrial Enhancement).

Remediation plans for case study Projects of Concern

3.36 Remediation plans had been prepared for each of the three projects comprising the major case studies:

  • For Air Warfare Destroyer (see Appendix 4, paragraph 19 forward) the remediation plan flowed from the Winter-White Review of the project, commissioned by both Defence and Finance, before the project entered the Projects of Concern list. The project is now expected to be completed with Initial Operating Capability forecast for December 2019. However, a primary basis for the project entering the Projects of Concern list, shipyard productivity, has deteriorated.
  • For MRH90 helicopters (see Appendix 5, paragraph 10 forward) the remediation plan was centred on two deeds to the acquisition contract. These, too, were agreed before the project entered the Projects of Concern list. Evolving remediation requirements and action were then discussed at subsequent Projects of Concern summit meetings. Remediation is not complete and the project remains on the list.
  • Collins Class submarine sustainment (see Appendix 6, paragraph 9 forward) was declared a Project of Concern before the remediation plan was prepared. The major element of that plan comprised the Coles Review. Remediation has been successful in terms of boat availability (materiel-ready-days) though sustainment costs remain an ongoing concern.
Remediation plans for other projects

3.37 For the other projects that remained on the Projects of Concern list at the time that audit fieldwork was undertaken or had been removed over the preceding twelve months, the approach to preparing remediation plans varied:

  • Defence documentation indicates that, whereas no formal remediation plan was agreed with the vendors for the Mulwala Redevelopment Project, Defence was satisfied that the project had successfully completed Defence’s objectives for remediation.
  • In the case of the Defence SATCOM Terrestrial Enhancement, a remediation plan was developed by Defence as an agenda item for the summit with the Minister on 20 July 2015. Defence and the vendor continued to engage on the remediation plan thereafter. However, the project was not completed as originally envisaged and a deed of settlement was signed with the vendor in November 2017.
  • In the case of CMATS, which entered the Projects of Concern list in July 2017, no remediation plan was prepared upon it becoming a Project of Concern. Following contract execution, CMATS was removed from the list.60
  • For the Deployable Air Traffic Management and Control System, a remediation plan had been developed in October 2016, before the project entered the Projects of Concern list. However, an Independent Assurance Review in November 2017 found that the remediation plan had not delivered the expected performance and the risk profile was bleak. Defence stated in February 2019 that a further remediation plan had been agreed in-principle on 11 September 2018. However, a contract change plan was yet to be finalised as Defence awaited outstanding documentation from the vendor.
The status of remediation plans

3.38 In October 2018, Defence advised the ANAO that the role of remediation plans set out in paragraph 3.32 no longer describes the process:

Since [the First Principles Review], industry is a Fundamental Input to Capability and remediation plans are agreed by all parties and progress reported against. The main focus is on collaboration to obtain the best possible capability outcomes within the schedule and cost constraints of the program.

3.39 On the current status of remediation plans for Projects of Concern, Defence further advised the ANAO in October 2018 that :

some were drafted during the [2010–13] era and some were drafted for the last full-on summit with [the then Minister] but not endorsed by any of the parties. It is important to keep in mind that PoC is not about introducing additional overhead, it is about getting the appropriate help for the project.61

3.40 This advice conveys a diminished view or expectation of remediation plans within Defence and leaves their current status uncertain, even though they remain a requirement under the current principles for the Projects of Concern regime.

Summit process

3.41 Under the Projects of Concern processes adopted in 2011, an important element was a six-monthly review or ‘summit’ meeting with the responsible minister. These meetings were held regularly from 2011 through to 2013, usually over a two-day period, and chaired by the responsible minister.62 They were attended by Defence officials and senior industry representatives. Their purpose was to address the problems of the Projects of Concern and prepare a remediation plan.

3.42 Typically, these summit meetings took place at Parliament House, Canberra. An hour or more was allocated for each Project of Concern, with successive groups of vendor representatives for respective projects in attendance, each only for the relevant period, along with senior Defence staff responsible for that project. The Chief Executive Officer of the (then) Defence Materiel Organisation stated:

I have observed many cases where significant progress has been achieved in the days leading up to the summit, demonstrating that their frequency is key to sustaining momentum.63

3.43 By late 2014, however, no summit had been held for over a year, which had led, in Defence’s view, to ‘a stagnation of this key element in the Projects of Concern regime’. Defence advised the Minister in November 2014 that:

Should the summits not proceed in the near term, there is a risk that some of the commercial leverage and momentum developed over the past six years would be lost. This is particularly important given the recent addition of the $8 billion Air Warfare Destroyer (AWD) programme to the list. A summit will provide a forum where the relevant senior stakeholders can be brought together to discuss the unique commercial, schedule and cost sensitivities involved, and begin the remediation work at the most senior levels.

3.44 Defence took the view that the Minister’s presence generated gravitas, which was a key element of the success of the program. Defence wished to maintain what it saw as the momentum of the program. A summit meeting was subsequently held, on 20 July 2015.

3.45 Defence has advised that two project-specific summits have been held since that time:

  • 12 September 2018 for AIR 5431 Phase 1, Deployable Defence Air Traffic Management and Control System; and
  • 16 August 2017 for AIR 5431 Phase 3, Civil Military Air Traffic Management System.64

3.46 As noted earlier, the Projects of Concern principles of 2017 (see Appendix 2) and those currently in effect (see paragraph 2.23) do not refer to six-monthly or any other review or summit meetings with the minister. There is no longer an expectation that there be regular summit meetings. Defence has advised that they are ‘held on an “as required” basis, depending on the nature of the issues with the project and the wishes of ministers.’65 A submission provided to the ANAO in November 2018 by an industry peak body stated that:

From a practical point of view, one company involved in the process noted that attendance and scheduling of the Projects of Concern meetings tended to be ad hoc. Similarly, the structure/ agenda for the meetings could vary greatly dependent on the Ministers at the time. The most positive meetings were when the Minister personally engaged with all the participants in an open and direct conversation.66

Using Project of Concern status to apply commercial pressure

3.47 Under the processes adopted in 2011, for Defence to employ its Projects of Concern regime to best commercial effect, the Department was required to take account of these projects in other dealings with vendors, in particular, ‘a negative weighting of tenders’ when contemplating further acquisitions (see Appendix 2).

3.48 To strengthen the consequences for companies that significantly underperform on Defence contracts, Defence incorporated requirements into its contract (ASDEFCON) templates.67 This was intended to place commercial pressure on vendors to rectify projects on the Project of Concern list.

3.49 There are two elements to this mechanism:

  • tenders are evaluated based on tender evaluation criteria which include taking account of performance on past and present contracts including those that are, and those that have been, a Project of Concern; and
  • the Commonwealth has an express discretionary right to exclude a tender from further consideration during a tender process if that tenderer is materially involved in a Project of Concern.68

3.50 These arrangements have been in place since 2011.

3.51 In 2012, Defence wrote to tenderers highlighting these conditions:

Should [Vendor’s name] tender be submitted for [the project], you can expect that a key factor that the Commonwealth will consider in exercising its rights under this clause during the tender process will be the progress [the Vendor] has made in remediating [an earlier project] as a Project of Concern, including the content of and timeliness in agreeing the remediation plan in the first instance, and then the subsequent implementation of the plan.69

3.52 Defence records included formal tender evaluation plans that explicitly included consideration of any Project of Concern connections. However, Defence advised the ANAO in July 2018 that it is not aware of any company being excluded from any further tender until a Project of Concern has been remediated.70 The audit did identify an occasion when Defence elected not to exclude a vendor from subsequent tenders. This was in relation to the Civil-Military Air Traffic System (CMATS) project where this decision was attributed to the unusual circumstances of the project.71

Does Defence apply a consistent process to the exit of projects from the Projects of Concern list?

Defence has generally applied a consistent process to the exit of projects from the Projects of Concern list. Defence’s practice has been to recommend removal of a project from the list only when it has both fulfilled a specified set of expectations (or removal criteria) and satisfied Defence that it is on a sound trajectory, making it unlikely to return to the list. A 2018 decision to remove a project (CMATS) has not observed the second condition.

Removing projects requires more than meeting remediation objectives

3.53 In 2010, removal from the Projects of Concern list depended on ‘both a completely clean overall bill of health/risk retirement and achievement of remediation objectives’.72 Defence has stated that it still requires a high degree of confidence in the outlook for the project before recommending to ministers that the project be removed from the list. In particular, ministers want to be ‘very much assured that something we have identified as being a project of concern is not going to come back’.73 That is, when making a recommendation to ministers to remove a project from the list, Defence management needs assurance that the project is unlikely to underperform again to the point that Defence will recommend it be returned to the list.

3.54 There is evidence that Defence has sought to keep a project on the Projects of Concern list after all the specifically articulated remediation objectives have been satisfied. In a particular case, CN10 Collins Class submarine sustainment, where all remediation objectives had been met, Defence senior managers took the view that residual risk remained with a remote possibility of the project coming back on the list. As a result, the project remained on the list for reassessment some six months or so later, after which it was removed. No specific milestones were set to form the final basis for removal. Defence has advised:

The removal criteria for CN10 Collins Class Submarine sustainment was consistent achievement of submarine availability in accordance with the international benchmark as per the Coles review. Post all remediation objectives were being met, CN10 remained on the PoC list for another six months to ensure consistent submarine availability was being achieved prior to recommending the removal from the PoC list. This is a good business practice for such a complex capability.74

3.55 In practice, under the various versions of the regime, removal of a project from the Projects of Concern list has consistently turned on a management judgement that risks have been sufficiently contained as well as the satisfaction of a specific metric or similar criteria. An instance where this approach has not been observed is the Civil-Military Air Traffic Management System (see Box 2, below).

Box 2: The removal of AIR 5431 Phase 3, Civil-Military Air Traffic Management System (CMATS), from the Projects of Concern list

Civil-Military Air Traffic Management System (CMATS) is a project managed jointly by Defence and Airservices Australia (with the latter as the lead) to replace separate, legacy systems with a unified air traffic management system.

Second pass project approval was given in December 2014, but contract signature was delayed.75 A review in September 2015 found that seeking a single, unified Civil and Military system had been a very significant driver in increasing the project complexity.76 After an Independent Assurance Review in November 2015, Defence proposed it become a Project of Concern but, on the advice of the CEO, Airservices Australia, postponed the decision.

With Defence’s advice that the vendor’s performance had been poor, but was improving, Ministers decided on 31 May 2017 to place the project on the Projects of Concern list. Defence later advised the Senate Foreign Affairs, Defence and Trade Committee that CMATS had been placed on the list because ‘negotiations were taking longer than expected, the costs were increasing significantly, and the schedule was not being achieved’.

In July 2017, Defence advised the vendor that:

In place of the remediation plan which would usually be required to document a Project of Concern’s path to resolution, Defence will instead be working with Airservices to develop a path to contract execution; the support of your team in working collaboratively to get to contract execution will be vital. The project’s placement on the Project of Concern list will be reconsidered following the successful execution of the acquisition and support contracts.

The commitment here was for ‘reconsideration’, not automatic removal upon contract execution. However, this commitment appears to have been interpreted subsequently as ‘automatic removal upon contract execution’. A Defence Independent Assurance Review in November 2017 stated:

AIR 5431 Phase 3 is presently a Project of Concern (PoC). I understand a commitment has been made to [Airservices Australia] to remove AIR 5431-3 from PoC status on contract signature. Given the overall performance seen to date, I would recommend it remains a Project of Concern until an agreed Governance system is in place …

On 26 February 2018, following contract signature, Ministers announced that the project would be removed from the Projects of Concern list.

Defence had not provided a briefing before the Ministers’ announcement and subsequently prepared documents to ‘formalise’ the decision. As it did so, reservations held by senior Defence officials about removing CMATS from the list became apparent, including a view that the vendor should be required to ‘get some runs on the board’ before removal.

Given the commitments already made and vigorous representations received from Airservices Australia, internal Defence consensus was reached to proceed with documentation for formal removal. However, the view that there remained a risk of the project returning to the Projects of Concern list was accommodated by making it a Project of Interest.

High element of judgement in selection of projects for removal

3.56 Selection of a project for removal from the Projects of Concern list has not always flowed from any apparent objective criteria, as exemplified by a case in late 2009. In this instance, the decision to recommend removal turned on the desire within Defence to give the vendor (which had other projects on the Projects of Concern list) some relief.

3.57 There is no reason documented by Defence for giving the particular vendor ‘some relief’ nor deliberation on the merits (or otherwise) of doing so in relation to the effectiveness of the Projects of Concern regime. The correspondence identified by the audit indicates a highly discretionary approach. The apparent fluidity with which project status could be varied would likely become apparent to the vendor, and possibly other vendors, thereby risking the undermining of the value of the entire mechanism.

4. Reporting on Projects of Concern and evaluating the regime

Areas examined

This chapter considers whether Defence has established appropriate internal and external reporting arrangements on the progress of Projects of Concern and can demonstrate that the Project of Concern regime contributes materially to the recovery of underperforming projects and products.

Conclusion

Defence reporting on its Projects of Concern is appropriate, with regular reports provided to senior management within Defence, to ministers and to Parliament, as part of Defence’s Quarterly Performance Report. Reporting provides useful quantitative and qualitative data though Defence has acknowledged that the timing and quality of its Quarterly Performance Reports could be improved.

Defence has not evaluated its Projects of Concern regime over the decade it has been in place, nor set criteria for assessing success. There is no basis, therefore, for Defence to show that the Projects of Concern regime contributes materially to the recovery of underperforming projects and products.

Area for improvement

This chapter recommends that Defence evaluate the contribution of the Projects of Concern regime to the recovery of underperforming projects. Defence’s reporting on Projects of Concern would be made more useful if Defence were to include the original approved costs and milestones as well as the current approved schedule and budget for the projects.

Has Defence reported internally and externally on the operation of the Projects of Concern list?

Regular reports are provided on Projects of Concern to senior management within Defence, to ministers and to Parliament which contain useful quantitative and qualitative data. Projects of Concern are also reported on publicly through Defence’s Annual Report and ministerial media releases. Defence has acknowledged that the quality of the data could be improved and that information technology systems have affected the timeliness of the reports. Notwithstanding its regularity, the reporting is not timely, taking nearly two months to complete.

Internal Reporting

4.1 As well as bimonthly reporting on acquisition projects and sustainment products to the Deputy Secretary, Capability Acquisition and Sustainment Group (CASG), CASG reports to senior Defence leaders and ministers on Projects of Concern through:

  • Quarterly reporting to the Secretary and the Chief of the Defence Force through the CASG Quarterly Performance Report.
  • Biannual reporting to the CASG Executive Advisory Committee.
  • Papers presented to the Defence Audit and Risk Committee.
Quarterly Performance Report

4.2 CASG’s primary method for internal reporting on Projects of Concern is through the CASG Quarterly Performance Report77 which includes a status update and a description on Defence’s progress towards remediation for each project on the list. The report includes a ‘traffic light’ reporting system with supporting quantitative and qualitative information.

4.3 The quantitative information reported is drawn from Defence’s information technology reporting systems for acquisition projects and sustainment products — the Monthly Reporting System and the Sustainment Performance Management System respectively.78 Qualitative information reported by Project and Product Managers outlines the key challenges relating to the project and progress towards remediation. The reports contain appropriate information, describing the project and what went wrong, providing (among other data) an update on each current Project of Concern, progress towards remediation and an expected date for removal from the list.

4.4 After an internal approval process, which includes noting by the Defence Investment Committee and approval by the Secretary of Defence and Chief of the Defence Force, the report is generally provided to the Minister within two months of the end of each quarterly reporting period. In August 2017, the Deputy Secretary, CASG acknowledged to a Parliamentary Committee that the report was not sufficiently timely and that he had concerns about its accuracy and completeness.79 In early 2018, Defence records indicate that the timeliness of the report remained an issue due to limitations with Defence’s information technology systems.80

4.5 In June 2018, Defence began developing a new Quarterly Performance Report section within the Sustainment Performance Management System. The new section is to be populated with a summary of key product information on budget, year to date spend, and Australian industry capability which cannot be drawn easily from the system. The data from that section would then be used to populate the Quarterly Performance Report. In October 2018, Defence advised the ANAO that the section was used initially for reporting on sustainment products in the April-June 2018 Quarterly Performance Report and for all top 30 sustainment products in the July-September 2018 Quarterly Performance Report.81

External Reporting

Reporting to Parliament

4.6 Defence has reported on Projects of Concern (project number and name, and the date it was added to the list) in its Annual Report including changes to the list during the reporting period (for example, the removal or addition of a project).82 Changes to the Project of Concern list are often also announced via ministerial media release.83

4.7 Projects of Concern and Projects of Interest are reported on as part of the Portfolio Budget Additional Estimates Statements. The Portfolio Budget Statement Outcome 2 is:

We protect and advance Australia’s interests through the provision of strategic policy, the development, delivery and sustainment of military, intelligence and enabling capabilities, and the promotion of regional and global security and stability as directed by Government.

4.8 The associated performance criterion is: ‘Capability proposals, once approved by Government, meet agreed schedule and are delivered within agreed costs and scope’. For 2017–18, the information included under the section on performance in achieving the criterion is:

Of the 123 post-second pass approved major capital equipment projects, four projects (or 3.3%) had issues with capability, schedule or cost which were significant enough to be included in the Projects of Concern report. A further nine projects (or 7.3%) were identified as Projects of Interest with risks associated with capability, schedule or cost that warrant attention from senior executives.

4.9 Defence has reported to Parliament on the Projects of Concern regime both at Senate Estimates and other Parliamentary committee hearings and in a more structured way through a regular statement to the Senate Standing Committee on Foreign Affairs, Defence and Trade.84 These reports include current Projects of Concern, any changes to the list since the previous report, and the previous Projects of Concern including the outcome (for example, cancelled or remediated) and the date removed from the list. The report also includes a one page summary for each Project of Concern that provides an overview of the project’s current status.

Reporting to Ministers

4.10 Defence reports to ministers on Projects of Concern regularly and has done so since, at least, 2007. Briefs initially included detailed information on each of the identified Projects of Concern including: project schedule with the original approved and rebaselined dates; expected risks to milestones; original approved budget and any cost increases over the life of the project; technical issues and their impact; current and planned remediation activities; commercial sensitivities; and any likely Ministerial guidance or decisions required in the future.

4.11 Over time, the reports have become more focused. In March 2018, Defence reduced the frequency of reports for Projects of Concern from monthly to quarterly and incorporated those reports within its Quarterly Performance Report.85 This reflected an internal view that the content of the Projects of Concern report did not change significantly from one month to the next and the reporting frequency may not be beneficial for the Ministers or the projects involved.

4.12 Reporting does not provide an indication of how cost or schedule may have shifted over time, and in some cases during the project’s time on the Projects of Concern list, such as the MRH90 helicopter. Including the original approved costs or milestones as well as the current approved schedule and budget for the Projects of Concern would make reporting more useful.

4.13 The Minister may also receive updates on specific projects at key milestones or decision points, and briefing materiel in preparation for Projects of Concern summits, and any outcomes of the summits.86 The Minister is provided a brief to approve a project’s addition or removal from the Projects of Concern list.

Can Defence demonstrate that the Projects of Concern regime contributes materially to the recovery of underperforming projects?

Defence cannot demonstrate that the Projects of Concern regime contributes materially to the recovery of underperforming projects. Although Defence has consistently stated that its Projects of Concern regime is ‘one of the Department’s most successful management tools for recovering problem projects’ it has not evaluated the regime and this view is based on management perception and anecdotal evidence.

4.14 Defence has consistently stated that its Projects of Concern regime is ‘one of the Department’s most successful management tools for recovering problem projects’ and that ‘the Projects of Concern regime is a proven process for managing seriously underperforming projects at a senior level’.87 Senior involvement in Projects of Concern is evident in:

  • the flow of documentation that must be put to ministers to obtain a decision on entry or exit from the list; and
  • ministerial summit meetings, and their associated processes (such as remediation plans).

4.15 Defence has not set any criteria by which it could evaluate or assess the Projects of Concern regime, nor has it conducted any systematic evaluation of the regime nor any analysis to highlight those features of the process that contribute most to achieving a successful outcome. When asked whether any persuasive analysis had been done that shows that the Projects of Concern process yields a better outcome than otherwise would have eventuated, Defence advised the ANAO that it was not aware of any.

4.16 The three major case studies (see Appendices 4 – 6) do not provide a basis for concluding that elevation of the project (or product) to the Projects of Concern list has had a significant effect on the progress or recovery of the project:

  • The Air Warfare Destroyer build project was placed on the Projects of Concern list well after the project’s progress had been extensively reviewed, had attracted senior engagement and a remediation plan developed (Appendix 4).
  • Similarly, remediation plans for the MRH90 helicopter were already in place when it was placed on the Projects of Concern list in 2011. The Minister regarded the latter step as ‘somewhat empty’. The project remained on the list in early 2019 with a prospect of removal during the year (Appendix 5).
  • Project of Concern status and senior engagement did precede development of the remediation plan for sustainment of the Collins Class submarines (Appendix 6).

4.17 There are positive and negative views of the Projects of Concern regime as outlined below.

Positive views of the effectiveness of the Projects of Concern regime

4.18 The audit encountered a range of positive views about the effectiveness of the Projects of Concern regime. For example, evidence in late 2014 from previous ministers in the Defence portfolio shows that they viewed the Projects of Concern list as good practice and useful in helping the Defence Minister and the Department to press industry for better service.88

4.19 Also, on occasions in the past, senior staff in Defence have recorded a view indicating that the Projects of Concern regime can be an effective mechanism:

In my experience the only way we get traction on these matters is to let them know that their behaviour will threaten new business. [A particular vendor] refused to come to the table on [a project on the Projects of Concern list] … until they lost [a second project] contract.89

4.20 Notably, both the comments from previous ministers and those from senior staff relate to the Projects of Concern regime as it operated before 2014. Their reflections encompass the period when regular summits and greater ministerial involvement were a part of the process.

4.21 There is written evidence of some vendors paying attention to Project of Concern status. However, both of these instances date from the earlier years of the Projects of Concern regime.

  • The chief executive of a major vendor issued a public press release which stated, ‘It is important to emphasise that, contrary to some reports, the [name of the asset] has not been placed on the list of projects of concern.90
  • In a very early example, the managing director of another vendor wrote to colleagues in his own organisation advising that a particular project was a Project of Concern: ‘the consequences of this are straight to reputation … If we have a problem with schedule deal with it now take the punishment and then make up and move forward. If this means buying back schedule then so be it’.91

4.22 The Australian Industry Group put this view to the audit:

The general sense from industry is that a Projects of Concern process is a valuable part of the project management and assurance process between Defence and industry. The system is useful as it provides a direct and open feedback mechanism to Government on major projects.

4.23 The Australian Industry Group’s submission to the ANAO passed on the view of a member organisation, a large Defence industry company, as stating:

the POC [Projects of Concern] process serves a useful purpose in the management, mediation and resolution of our very challenging defence projects when they are in trouble. It is very rare for a POC defence project to be completely the fault of either industry or defence — invariably there is a co-dependence on the root causes, and in reality the effective treatment and resolutions.

I would encourage that the POC process remains available in the toolkit for effective governance of defence programs.

Possible negative consequences of the Projects of Concern regime

4.24 Defence documents have raised the question of whether Projects of Concern had an ‘adversarial element inconsistent with [Defence’s] aim of improving its relationship with industry’ and whether the public naming of vendors in Defence industry had a negative impact. This was because ‘there is the potential for industry to perceive that they are being primarily targeted as the cause for all underperformance’.92

4.25 Concern about the negative effects of the Projects of Concern process on the relationship with specific vendors has arisen. In one case in 2018, the reputational implications of a project remaining a Project of Concern were highlighted as having negative consequences for the future of the project.

4.26 A senior Defence committee was advised in mid-2017:

While the companies listed as Projects of Concern have provided advice that the last summit was a useful and collaborative activity, there is a risk that the public may not view the process as positively and may still view it as a ‘name and shame’ list … Consideration should also be given to whether listing the project as a Project of Concern will likely have a positive impact and improve outcomes on the project overall.93

4.27 These concerns indicate a degree of ambiguity in perceptions within Defence as to whether the whole Projects of Concern mechanism has a benefit, overall, for Defence in its relationship with industry. In February 2019, Defence advised the ANAO that:

listing a project on the PoC report is not a ‘threat’ or a ‘name and shame’ exercise. It is a management tool for projects or products with significant risks or issues with regards to capability, schedule and cost.

4.28 Evidence indicates that in at least one case, concerns have been raised about inappropriate responses to Project of Concern status. In March 2018, an Independent Assurance Review Board recommended that a particular project be considered a candidate for the Projects of Concern list. Defence officials subsequently expressed concern that the vendor appeared to have focused on removal of Project of Concern nomination rather than delivery of an executable and value-for-money plan for the project’s Initial Operational Capability milestone.

Lessons learned

4.29 Following a briefing Defence provided to a Parliamentary Committee on Projects of Concern on 30 November 2017, Defence was asked, among other questions: ‘What is the systemic approach to Projects of Concern, e.g. how are the lessons learnt from remediating individual projects being applied systematically?’ Defence responded that ‘… CASG have established and are building a Lessons Learned Program …’ Review of this material indicated it was still in early development and contained no specific analysis of Projects of Concern or lessons flowing identifiably from these projects or their history.

Future evaluation

4.30 After a decade or more of operation of this high profile process, Defence has made no systematic assessment of its operation nor established criteria by which it could assess whether making an underperforming project a Project of Concern adds value to the particular project’s recovery or limits the project’s further deterioration. Defence has pointed out that there are challenges in doing so, given the complexity of large materiel acquisition projects and the difficulty of establishing a counterfactual:

There is no counterfactual for ANAOs argument – we have no way of definitely determining how much worse these projects could have been if they hadn’t been subject to additional oversight/reporting/implementation of remediation plans etc.94

4.31 An opportunity for establishing criteria to judge success or failure of the process may lie in formalising the practice of agreeing a remediation plan and remediation criteria as part of a future Projects of Concern policy.

Recommendation no.2

4.32 Defence evaluates its Projects of Concern regime.

Department of Defence response: Agreed.

Appendices

Appendix 1 Department of Defence response

Department of Defence entity response page 1

Department of Defence entity response page 2

Appendix 2 Projects of Concern — processes, criteria and principles, at various dates from 2001 to 2018

ANAO note: The set of criteria listed below for 2001 applied at a time when the Projects of Concern regime was managed as a largely ‘internal’ process by the Defence Materiel Organisation. For the sets of criteria/principles listed for 2009 through 2018, the regime has involved decision-making by ministers.

(1) Projects of Concern criteria, July–August 2001


CORPORATE OVERVIEW

Exception Reporting

The Corporate Overview is a monthly report, which the USDM [Under Secretary for Defence Materiel] presents to the Defence Committee. The document is also used for reporting in the Defence Matters Scorecard, the DMO Scorecard and is the foundation for a report, which goes to the Minister each month.

Recently the requirements for the Corporate Overview have changed. Not only does the DMO report on the Top 20 projects, two new sections have been added: “Projects of Concern” and “Significant Issues”.

The details below outlines draft criteria to determine if a project needs to be reported as a Project of Concern or Significant Issue.95

1. Governance:

• If a project is anticipating a potential Major Breach of Contract or

• Contractor Default.

2. Capability

• Any essential/critical requirements now forecasted that won’t be met.

3. Schedule

• If the schedule has slipped by more than 10% from the contracted baseline schedule.

• On Projects NYTC [not yet through committee] but approved if the schedule slippage is greater than 10% from the contract baseline schedule.

4. Budget

• The Project may go over budget.

• A change to the forecasted expenditure of 10% or greater.

• Project contingency money being used.

• If there is a risk that the remaining contingency is less that what is required for the project.


(2) Criteria for Listing and Removal from the List of Projects of Concern, September 2009

1. Entry to and exit from the list of Projects of Concern is decided by the Minister for Defence Personnel, Materiel and Science. There are two ways that a project will be removed from the list:

  1. Project or contract cancellation — only exercised in the most extreme cases, e.g. Seasprites and JP 129; or
  2. Project remediation.

2. Specific criteria have not been established for listing as a Project of Concern or for removal from the list. The complexities which characterise many of the listed projects tend to cross a number of aspects which are not always likely to captured entirely within specifically defined criteria. As general guidance an indicative set of principles for listing might include a synthesis of some or all of the following issues:

  1. Uncontained instability in schedule, cost, requirements or risk;
  2. Scope outside or non-compliant with Government approval;
  3. Unlikely to meet Capability Manager’s directed level of capability;
  4. Potential adverse impact on current or known future deployments;
  5. Unresolvable technical issues;
  6. Unresolvable commercial, or relationship with contractor, issues — eg: an unresolved dispute over 3 months duration; or
  7. Insufficient platform availability for installation and test.

3. Underlying any set of principles or criteria, a common quality is that issues or risk are unbounded or uncontained and lack any demonstrable or sufficiently assured path to closure or completion. Establishment of a demonstrably assured plan and path to resolution should be an essential precursor to consideration of a project’s removal from the list of Projects of Concern.

(3) Projects of Concern processes and criteria, June 2011


Project of Concern Reforms

Incentive for Industry to focus on fixing problem projects:

Where a company has a project on the list, Government and Defence will weigh their performance in remediating the project when evaluating their tenders for other projects.

When a company is not satisfactorily implementing an agreed remediation plan, this will result in a negative weighting of tenders received from the company, and in extreme circumstances could result in exclusion from further tenders until the project is remediated.

Formal process for adding projects to the list:

The process for determining whether a project should be added to the Projects of Concern list will be as follows:

  • When an Early Indicator and Warning is triggered [see below], Defence will advise Ministers, including whether a full diagnostic review (Gate review) of the project is required.
  • If a Gate review is to be conducted, Ministers will write to the Chair/CEO of the prime contractor advising them that the project has triggered an early warning, requesting their involvement in the Gate review, and emphasising the potential for the project to be added to the Projects of Concern list.
  • Following the Gate review, Defence will provide Government with recommendations on how to fix the problems with the project. If the problem is very serious, it may be listed as a Project of Concern immediately.
  • Alternatively, Defence will propose a series of actions that it and the company involved will undertake to fix the problems. This will include timelines, targets and thresholds which if not met will trigger a further Gate review to consider listing the project as a Project of Concern.

The decision to add a project to the list will be made by the Minister for Defence and Minister for Defence Materiel.

Remediation plans:

The Defence Materiel Organisation (DMO) and Industry will develop formal remediation plans for all projects that undergo Gate reviews.

In the case of projects confirmed as Projects of Concern, these plans will:

  • identify remediation objectives;
  • identify key milestones and the timeline for their achievement; and
  • detail an end-state for remediation and the agreed basis on which a project will be removed from the Project of Concern list.

Where DMO and Industry cannot agree a satisfactory remediation strategy, DMO will provide formal advice to Government on whether the project should be cancelled.

For all existing Projects of Concern, formal remediation plans will be developed and agreed with Industry. These will include the basis on which these projects will be removed from the current list.

Removal of projects from the list:

There are two events that will enable a project to be removed from the Project of Concern list:

  • Government satisfaction that remediation is completed in accordance with the agreed plan; or
  • a decision is taken by Government to cancel the project.

Increased Ministerial Involvement:

Ministerial involvement has been a cornerstone in driving improved outcomes for Project of Concern projects.

The Minister for Defence Materiel will hold bi-annual reviews of Projects of Concern with Defence and Industry representatives.

Biannual face-to-face meetings with the Minister will ensure responsible individuals are being held to account for the progress of projects, and will give the Minister a better understanding of the progress of remediation strategies.


Early Indicators and Warnings: Criteria and thresholds for post-second pass projects

Performance criteria and thresholds for post-second pass projects against Government-endorsed milestones were as follows:

  1. Schedule. Projects where the schedule for meeting Initial Operating Capability (IOC) or Final Operating Capability (FOC) will be delayed by:
    1. 10% or more for acquisition of an off-the-shelf capability,
    2. 20% or more for acquisition of a modified off-the-shelf capability, and
    3. 30% or more for acquisition of a developmental capability.
  2. Cost. Projects with costs that are forecast to exceed the approved budget.
  3. Contractor Performance. Projects where the delivery contractor is identified as not meeting:
    1. capability targets,
    2. schedule milestones, or
    3. approved costs.
  4. Policy or Legislation Impacts. Projects where policy or legislative changes are likely to increase the schedule or cost.
  5. Capability. Projects where an essential capability requirement will not be met.
  6. Emerging Requirements. Emerging requirements or regulatory or safety standards that are different to those in place when the Project was approved by Government, and that will materially affect the outcome.
  7. Industry Capacity. Industry engaged in the Project does not have:
    1. sufficiently skilled and/or qualified workforce,
    2. financial capacity, or
    3. management commitment to meet contracted project milestones.
  8. Risk. Project risks have increased beyond the parameters agreed by Government

(4) Projects of Concern principles, February 2017


Projects of Concern

Statement of Principles

1. Projects of Concern are a the most challenging projects in the Defence portfolio – they are:

  • at the extreme end of underperformance;
  • at risk of failure;
  • unable to manage their way out of difficulty through standard reporting chains; and
  • require buy-in from Senior stakeholders.

2. PoC provides senior intervention to underperforming projects. This intervention will supplement the existing standard senior management oversight in a more targeted and focused manner. This senior intervention will focus Defence and Industry stakeholders to align and concentrate their efforts on solving underlying issues and causes.

3. There are no set quantitative measures or thresholds to determine whether a project should be added to the Projects of Concern list:

  • Qualitative measures that take into account stakeholder relationships and emerging risks, and the extent to which these factors impact the prognosis for future performance are considered.
  • The Quarterly Performance Report also identifies underperforming projects and sustainment products, and these may be referred for further investigation.
  • Once identified as a candidate, CASG Independent Assurance Reviews provide a mechanism for assisting in determining whether a project would benefit from being added to the Projects of Concern list.

4. A non-advocate assessment provides the independence and perspective required to see the project or product’s risks and issues. Remediation plans must be centred on a clear strategy with identified milestones against which progress can be checked.

5. Behaviours: PoC supports a ‘hands up’ culture.

  • A collegiate approach allows projects to be remediated and deliver the best value for money to Defence.
  • The regime is built on trust, not blame.
  • Innovation is at the core — new ways of solving problems is to be encouraged.

6. A project’s removal from the Projects of Concern list will be determined by successfully achieving remediation outcomes, where there are no untreated risks or issues, and approval by the responsible Minister. If remediation of a Project of Concern is not likely to be achieved, there may be no alternative but for Government to cancel the Project or its contract.

7. The Projects of Concern regime provides innovative solutions to complex and serious performance issues. As part of this regime, CASG will capture, understand, and apply lessons learned from previous Projects of Concern.


(5) Projects of Concern principles, 201896


Projects of Concern

The Projects of Concern (PoC) team is part of the Business Analysis and Reform Branch (BARB). Projects or sustainment activities identified as Projects of Concern have very significant technical, cost, and/or schedule difficulties that are beyond the normal project team management.

The Projects of Concern regime is one of our most successful management tools for recovering challenged projects. This methodology includes:

  • independent diagnostics;
  • developing an agreed remediation plan; and
  • monthly project reporting.

Since its introduction in early 2008, the Capability Acquisition and Sustainment Group (formerly DMO), has worked closely with Industry, Defence, and the Government to successfully remediate these projects with the goal of returning them to the standard project management framework. It is important to note that the Projects of Concern list allows for all parties to form a common plan of action and avoids ‘blame’.

Projects of Concern will continue to have strong Ministerial engagement and hold summit meetings when required. The summits focus Industry’s attention on Projects of Concern, and provide the Commonwealth with real commercial leverage for improved project outcomes.

Key activities

The Projects of Concern team will assist listed projects with their reporting requirements and the implementation of their agreed remediation plans through these key activities:

  • Prepare and analyse data for the Projects of Concern monthly reporting to Government and Defence stakeholders
  • Prepare and submit a Projects of Concern statement to the Joint Standing Committee on Foreign Affairs, Defence, and Trade, immediately prior to each Senate Estimates Hearing
  • Coordinate oversight of Projects of Concern including face to face briefings and reporting
  • As and when required, coordinate the Projects of Concern summits with senior stakeholders from Government, Defence, and Industry to discuss and agree the way forward for challenged projects
  • Engage with individual projects of concern, particularly in relation to:
    • Ministerial submissions.
    • Stakeholder forums including Project Management Stakeholders Groups (PMSGs), CASG Independent Assurance Reviews (CASG IARs); and
    • Progress against agreed remediation objectives.

Policy

Entry to and exit from the list of Projects of Concern is decided by the Minister for Defence (MINDEF), and the Minister for Defence Industry (MDIND) either at the recommendation of the DEPSEC CASG or at the Minister’s own instigation.

The formal process for determining whether a project should be added as a Project of Concern begins when the Quarterly Performance Report (QPR) highlights a Project of Interest and an Independent Assurance Review is conducted for a diagnostic review. Following this, DEPSEC CASG will provide a recommendation to the MINDEF and MDIND to list the project or sustainment activity as a Project of Concern.

Projects are removed from the list through either remediation or cancellation with the approval of the MINDEF and MDIND.

Appendix 3 Projects of Interest

1. Projects of Interest are identified by Defence as those underperforming at any point in the capability lifecycle that require management action to avoid becoming a Project of Concern. Defence has not published a list of Projects of Interest; however, Projects of Interest have been identified in the annual Major Projects Report.97

2. The current Projects of Interest list is updated quarterly, and reported in the Capability Acquisition and Sustainment Group Quarterly Performance Report.98 The graph below illustrates the number of Projects of Interest over the past four years. Before July 2016, Projects of Interest were referred to as ‘underperforming projects’ or ‘underperforming products’. Sustainment Products of Interest have been reported from March 2016.

Figure A3.1: Numbers of concurrent Projects/Products of Interest July 2014–March 2018

List of projects and products on the Projects of Interest list as of December 2018

Source: ANAO analysis of Defence documentation

3. As of December 2018, there were eleven acquisition projects with an approved budget of $22 billion, and nine sustainment products with an annual approved budget of $975 million on the Projects of Interest list. In terms of the role of the list as a means to manage project and product performance, the Secretary of the Department of Defence has stated that99:

These frameworks [the Projects of Interest and Projects of Concern frameworks] provide an escalation mechanism for increased management of capabilities under development for the Australian Defence Force. A project may be identified as a Project of Interest when scope, schedule or cost variances warrant heightened senior management attention.

4. The projects and products on the list as of December 2018 are included in the table below. The performance issues that have warranted further attention from senior executives for these projects and products are: schedule delays, risks to capability, and budget overspend.

Table A3.1: Projects and Products of Interest as at December 2018

Project/Product Name

ACAT/ MSCAT Level

Acquisition Projects of Interest

JP 2048 Ph 4 A Amphibious Ships

I

AIR 6000 Ph 2A/B New Air Combat Capability

I

AIR 5431 Ph 3 Civil Military Air Traffic Management System

I

AIR 5428 Ph 1 Pilot Training System

II

JP 90 Ph 1 ADF Identification Friend or Foe and Automatic Dependent Surveillance – Broadcast

II

JP 2008 Ph 5A Ultra High Frequency Satellite Communications

II

JP 2089 Ph 2A Tactical Information Exchange Domain

III

JP2097 Ph 1B Enhancements to Special Operations Capability

III

JP 1770 Ph 1 Rapid Environmental Assessment

III

LAND 200 Ph 2 Battlefield Command System

I

AIR 5440 Ph 1, C-130J Block upgrade

III

Sustainment Products of Interest

CN34 Canberra Class Landing Helicopter Dock

I

CA12 Armed Reconnaissance Helicopter Weapon System

II

CA48 Multi Role Helicopter (MRH90) Weapon System

II

CAF12 Air Traffic Management

II

CN09 Armidale Class Patrol Boat

II

CN38 Navy Guided Weapons

II

CA40 Command and Intelligence Systems

II

CAF14 Air Battlespace Management System (ABMS) Capability

III

CA59 Explosive Ordnance Army Munitions

III

  

Note: The approved budget and spend to date for sustainment products of interest is for 2017–18 only. The Acquisition Category (ACAT) framework is Defence’s method for categorising acquisition projects. The framework is based on strategic significance, project and schedule management complexity, and levels of technical difficulty, operating and support and commercial arrangements. Higher complexity projects with a greater degree of strategic significance are assigned an ACAT I level, with low complexity minor acquisitions assigned an ACAT IV level. Defence assigns a Materiel Sustainment Category (MSCAT) to sustainment products (similar to the ACAT). The criteria is based on sustainment budget, management complexity, technical difficulty, life cycle stage, demand and availability and commercial arrangements. The more complex and higher budget products are assigned a MSCAT I level.

Source: Defence documentation.

Development of Projects of Interest

5. Defence’s use of Projects of Interest as an identifiable category has evolved. There are references to Projects of Interest dating back to 2005, with ‘projects and products of emerging interest’ reported in Defence’s Annual Report.

6. In 2011, the then Minister for Defence and the then Minister for Defence Materiel jointly announced a set of accountability and procurement reforms aimed at addressing outstanding recommendations from the Mortimer Review.100 These included establishing an Independent Project Performance Office to review projects and assist project teams to solve problems. Part of these reforms also included the implementation of the Early Indicators and Warnings System.

7. The Independent Project Performance Office was established in the Defence Materiel Organisation in July 2011, and part of its role was to: ‘Implement the reforms … to the Projects of Concern process and oversee the remediation of all Projects of Concern’. The office also monitored Projects of Interest and provided information on these projects to Division Heads responsible for their management and senior leadership within the organisation.

8. The Early Indicators and Warnings System was aimed at identifying problems early in a project’s lifecycle and undertake appropriate remedial action. The system included a set of performance criteria around cost, schedule and risk. Any project that exceeded performance thresholds identified in the Early Indicators and Warnings System would be referred to senior management within Defence for review.

9. In 2013, Defence sought to formalise the process for identifying Projects of Interest and developed a set of business rules. These rules outlined the purpose of the Project of Interest list, the data sources used to determine whether a project should be included on the list and how the list should be used within Defence.

10. In 2014, the Early Indicators and Warnings System was replaced with the Quarterly Project Performance Report (now the Capability Acquisition and Sustainment Group Quarterly Performance Report). This report focused on the performance of major capital acquisition projects, and in late 2015 also included information on the performance of sustainment products.

11. Defence has advised the ANAO that the current approach for determining Projects of Interest has been in place since July 2016. The current approach relies on the Strategic Performance Reporting team101 using information from various data sources in Defence — including from performance reporting systems and Independent Assurance Reviews — to identify potential Projects of Interest. These potential Projects of Interest are confirmed with relevant stakeholders within Defence such as Project or Product Managers and cleared at the Division Head level.

12. Defence informed the ANAO that Independent Assurance Reviews form a key role in identifying potential Project of Interests. However an ANAO review of 191 Independent Assurance Reviews since 2015 found:

  • Inconsistent approaches across Independent Assurance Reviews — the reviews are inconsistent in recording whether they considered if a project should become a Project of Interest or Concern. There are also projects with significant delays in schedule and budget overspends but which have not been recommended for inclusion on the Projects on Interest list (see paragraph 3.3 forward).
  • A disconnect between information in the Quarterly Performance Report and Independent Assurance Reviews — there are instances where a project has been identified a Project of Interest through the Quarterly Performance Report, but the Independent Assurance Review has not made this recommendation. In some instances, the review has made a recommendation that the project be included on the Project of Interest list after the project has already been included on the list in the Quarterly Performance Report, or recommended that the project stay on the Project of Interest list after it has already been removed from the list in the Quarterly Performance Report.

Appendix 4 Major case study 1 — SEA 4000 Phase 3, Air Warfare Destroyer

What was the project about?

1. The project is to build and deliver three Air Warfare Destroyers and a support system.

2. Government commitment to Defence’s requirement for a long-range air-defence capacity in the fleet was set out in the 2000 Defence White Paper.102 The project, known as ‘SEA 4000’, was identified in the Defence Capability Plan 2001–10 as an ‘affordable Maritime Air Warfare capability’ with an estimated acquisition cost of $3.5 – $4.5 billion and an expected in-service date of 2013.103 In the 2004–14 edition of the Defence Capability Plan this was revised to identify separate phases to the project: Phase 3 of SEA 4000 would be the ‘build’ phase with an estimated cost for that phase in the range $4.5 – $6 billion.104

3. On 1 May 2005, Navantia was announced as the preferred designer of the ship. On 31 May 2005, ASC was selected as the preferred shipbuilder from among three responses to a request-for-proposal. The Minister for Defence said the Government made the decision on the basis that ASC offered a superior bid in terms of value for money. Following this, the government’s first pass approval, Phase 2 (design) commenced.

4. In June 2007, at second pass approval, it was announced that Navantia would work with the AWD Alliance (Defence Materiel Organisation, ASC and Raytheon Australia) to deliver three air warfare destroyers to the Royal Australian Navy at a cost of nearly $7.2 billion. SEA 4000 Phase 3 (detailed design and construction), to build three Australianised air warfare destroyers, based on the F-105 design in service with the Spanish navy and equipped with an Aegis combat system, then commenced.

How did the project become a Project of Concern?

5. Evidence of low productivity in the build project was evident in 2011, three years before it became a Project of Concern. The DMO Program Management Office reported to the Government–ASC Steering Committee meeting on 19 December 2011 that First Marine International (FMI), an international specialist consulting company, had been engaged to assess, independently, the actual and planned productivity at each of the Australian AWD shipyards in September–October 2011.105 DMO stated that FMI had found:

Productivity against each ship has only been assessed on an overall basis which means the data includes the productivity of the block subcontractors BAE and Forgacs. Although ASC-specific performance cannot be extracted from the report data, the overall assessment does reflect the ASC plan and budgets for overall ship production and, as shown by Figure 1, this shows that far from the Project achieving world class productivity, the plan and budget do not even approach [the] realistic target identified by FMI.106

6. Figure 1 from the FMI report, labelled ‘Current and target productivity’, and the associated notes are included below (see Figure A4.1):

Figure A4.1: Current and target productivity

Graph showing current and targeted productivity in man-hours per compensated gross tonne for ships one, two and three in the Air Warfare Destroyer fleet.

Note: This figure shows productivity measured in ‘man-hours’ per compensated gross tonne for ships ➀, ➁ and ➂ in the air warfare destroyer fleet. ‘Compensated’ means that account is taken of different ship types of varying complexity requiring differing amounts of effort per tonne of vessel built.

Source: Defence documentation.

7. The FMI report describes how shipyard productivity tends to be much lower for the first ship in a series and increase with each successive vessel (the heavy black curved arrow is described as the ‘learning curve’). FMI had concluded, based on its observations, that the potential core productivity of the enterprise was 60 man-hours/compensated gross tonne. However, FMI found this could not be expected to be achieved in a three-ship program: the realistic target was 80 – 85 man-hours/compensated gross tonne (the green band). The notation ‘Lead ship factor = 100%’ indicates that then current estimated at-completion productivity on the first ship of the series was 120 man-hours/compensated gross tonne, which represents 100 per cent greater input than the potential productivity of 60 hours/compensated gross tonne estimated by FMI.

8. The DMO report went on to state:

Evidence of concerted efforts by ASC to improve productivity were not observed by FMI; they did not identify a plan to achieve improved productivity targets, nor did they witness a high degree of focus in improving productivity or evidence of productivity-based targets defined by appropriate metrics particularly at the production shop-floor level, where productivity can be implemented.

9. The report concluded that ‘the Alliance with ASC, will need to address … productivity, cost control, schedule control and risk for the shipbuilding and consolidation/ integration.’

10. In September 2012, ministers announced a ‘re-baselined production schedule’ — that is, project slippage — with a ‘keel-to-keel’ interval extended to 18 months.107

11. An update from FMI (8 February 2013) found that estimated productivity at completion for all three ships had declined, with the lead ship productivity now 130 man-hours/compensated gross tonne.

12. Further concerns about productivity and cost overruns came to light over the next twelve months:

  • On 17 December 2013, the 2012–13 Major Projects Report was tabled.108 It identified the approved budget for the Air Warfare Destroyer build program as having risen through price indexation to $7.9 billion and noted ‘emerging concerns … around cost overruns’. The report identified shipbuilding productivity, among other items, as a major project risk.
  • In March 2014, Auditor-General Report No.22 2013–14, Air Warfare Destroyer Program was tabled in Parliament. It reported a continuing decline in construction productivity.109

13. The secretaries of the Departments of Defence and Finance commissioned a three-month independent review of SEA 4000 by two experts (Professor Donald C. Winter and Dr John White), in March 2014. The review would:

recommend remediations [sic] and mitigations to improve the cost and schedule performance of the AWD Program and to realise the national security benefits of the program and the long term benefits of the program for the Australian shipbuilding industry. The Review will also identify the risks to the future completion of the AWD Program, and provide recommendations to mitigate those risks.110

14. The Winter–White report, completed in May 2014, identified ‘two principal direct causes for the AWD Program difficulties, along with three significant contributing causes and two systemic issues’.111

The two direct causes that were both categorised as findings are:

1) The initial program plan for AWD development and production was unrealistic in its cost and schedule estimates.

2) The Alliance, as structured, composed and staffed, has been unable to effectively manage the AWD Program.

The contributing causes include:

1) Systems engineering on the AWD Program has been of limited effect.

2) The AWD Alliance and ASC were unable to effectively manage the AWD block subcontractors.

3) The oversight provided by the Commonwealth of Australia has been of limited effect.

The two systemic issues, which could affect any other naval shipbuilding program in Australia, were identified as:

1) The limited base of shipbuilding activity in Australia materially impacted the AWD Program.

2) The Commonwealth of Australia has not developed a long term shipbuilding plan that can cost effectively support the needs of the RAN while sustaining the Australian industrial shipbuilding base’.112

15. The Review recommended inserting a new management team from a suitable company. It evaluated ‘a number of potential sources’ and found that only BAE Systems met all of the identified requisite criteria for such a role.113

16. The Winter–White Review was focused as much on future naval shipbuilding in Australia as on addressing the Air Warfare Destroyer build: it saw the two as strongly linked. In their letter to ministers setting out their early findings (4 April 2014) the authors recommended that ‘no final commitments are made beyond the planning phases of other inter-related major naval industry programs until the AWD Program issues are showing clear signs of resolution’.114 Their final report also found that ‘If appropriate internationally bench-marked performance levels are not achieved through the AWD Program, consideration could be given to future naval surface combatants being procured overseas’.115

17. In light of the Auditor-General’s performance audit and the Winter–White Review, Defence proposed to the Minister that the Air Warfare Destroyer project be placed on the Projects of Concern list. It advised him that these reports had confirmed that the project’s cost estimates would be ‘significantly exceeded due to poor shipbuilding productivity’, and:

Unless addressed quickly, the ongoing poor performance on the Project could also have a wider detrimental effect on Australia’s maritime sector, including projects such as SEA 5000 — Future Frigate — that are likely to rely on a high quality and productive ship building capability for success.

18. Defence considered that listing was warranted ‘because of the high profile of the Project and particularly its significance to the future of Australian naval shipbuilding in Australia’. The Minister agreed, and announced his decision in June 2014.116

What was the remediation plan?

19. At the same time, the Minister for Defence and Minister for Finance announced a ‘way forward for the $8.5 billion Air Warfare Destroyer program’.117 They stated that the Government was ‘totally committed to put this important Defence project back on track and to stop the growing cost and schedule overruns by implementing the recommendations for a Reform Strategy made by the Winter Review.’118

20. The ministers also announced that the Reform Strategy would:

  • Improve shipbuilding productivity at the Air Warfare Destroyer shipbuilder ASC and its subcontractors BAE Systems, Forgacs and Navantia;
  • Include the urgent insertion of an experienced shipbuilding management team into ASC; and
  • After augmented shipbuilding capacity has been put in place, pursue the reallocation of blocks between shipyards to make the Air Warfare Destroyer program more sustainable.

21. The reform strategy became the remediation plan for the Air Warfare Destroyer project. As mentioned above, a major element was to insert a new team to take control of production management.

22. At the Ministers’ press conference on the matter the Minister for Finance stated: ‘In summary, the reform strategy that Professor Winter has recommended to the Government will seek to improve ship building productivity at the Air Warfare Destroyer Ship Builder ASC and its sub-contractors’. The Minister for Defence referred directly to the analyses of productivity by FMI (discussed above): ‘We balanced baseline production at about 80 man hours per gross tonne for these three ships. Currently it is running at 150 man hours per tonne’.119

23. By October 2014, four months later, a Gate Review found ‘The schedule for delivery of the ships is to a large extent “unknown”’ and a cost increase of ‘around $310 million above the increase of $140 million in the 2012 baselining’.120

The Board noted that, commensurate with ‘pain-share’ under the Alliance agreement of 50/50 pain share, the Commonwealth (Defence) and the industry participants each bear half the cost of over-runs above the Alliance’s Total Cost Estimate. The Industry participants’ fee is reduced by the amount of industry’s pain share until all fee is exhausted. After that point is reached (an overrun of some $600m), the Commonwealth bears the full costs while industry receives no fee. The projected outcome, even with future productivity reforms, would likely see the overrun exceeding $600m and thus industry participants receiving no fee which is a challenging environment within which to incentivise the participants.

24. Nearly a year later, on 22 May 2015, the Minister for Defence and the Minister for Finance announced that an additional estimated $1.2 billion would be required to complete the project and that delivery of the destroyers had been ‘significantly delayed’.121 On 4 August 2015, the Prime Minister and the Minister for Defence announced that the additional $1.2 billion would be provided to deliver the program.122 The insertion of a Navantia management team to ASC to manage the project was announced in December 2015 and reforms to the structure of ASC were announced in October 2016.123

25. On 5 May 2016, the Joint Committee of Public Accounts and Audit tabled Report 458: Defence Major Projects Report (2014–15): Review of Auditor-General Reports No 16 (2015–16). This included a recommendation that Defence review Air Warfare Destroyer contractual arrangements, examining the distribution of liabilities for project problems. Defence’s response indicated that, under the ’pain share/gain share’ arrangements one partner had repaid the full amount it was liable to repay but that ASC had committed to repay its share, ($82.7 million) by 2019. Defence has informed the ANAO that it expects this to occur by 31 January 2019.

How did the project leave the Projects of Concern list?

26. Defence’s Independent Assurance Reviews are a primary opportunity to consider whether a project on the Projects of Concern list should be removed. Defence conducted an Independent Assurance Review of the Air Warfare Destroyer project in July 2016. The review noted that the project had been added to the list in June 2014 but did not identify the reasons, nor did it refer to any remediation plan. It concluded that the project ‘had come a long way in the past 12 months’ and recommended that ‘consideration be given to setting achievement of IOR [Initial Operational Release] for HMAS Hobart as a suitable milestone for removal of SEA 4000 Phase 3 from the Projects of Concern [list]’.

27. Defence conducted a further Independent Assurance Review of the project in July 2017.124 It drew positive conclusions about progress: ‘My assessment is that Phase 3 (the build) is now well on track to meet the re-baselined schedule, notwithstanding some defects on Hobart remain to be rectified’. It noted that Navy was not supportive ‘in light of the number of remaining risks and defects’ but, nevertheless, the Review recommended ‘SEA 4000 Phase 3 be considered for removal from POC list at IOR/commissioning as that milestone will demonstrate significant burn down of risks and the adherence to the re-baselined schedule …’ Moreover, ‘the Board considered that by IOR there would be sufficient evidence that the original concerns that triggered POC status (namely cost overruns, low productivity, technical and quality issues and subcontractor delays to block delivery) had been addressed effectively’.

28. Defence subsequently sought the agreement of ministers to remove the project from the Projects of Concern list. In its advice, Defence stated that ‘Through the long-term reform arrangements set in place by the Government in December 2015, the Project is now meeting its revised budget and schedule targets, with shipyard productivity improving.’ Ministers agreed, and their decision was announced on 1 February 2018.125

29. Shortly after the announcement, FMI provided a further update of its analysis of ASC productivity (2017 update, dated 15 February 2018).126 This shows that productivity in man-hours/compensated gross tonne estimated at completion — the measures quoted by the minister at the time the project went on the list — has improved from the first ship to the second ship, and the second to the third, as expected, but overall, productivity for each ship has declined over the life of the project.

30. The ‘adjusted’ productivity based on the 2017 EAC [estimated at completion] data is illustrated in Figure A4.2, below.

Figure A4.2: Air Warfare Destroyer build productivity (2018)

Graph showing adjusted productivity in man-hours per compensated gross tonne for ships one, two and three in the Air Warfare Destroyer fleet based on estimated at completion data for 2017.

Source: Defence documentation.

31. The FMI report explained this diagram thus:

The historical adjusted budgets for 2012 to 2016 are also shown. These illustrate how the expected out-turn productivity has continued to deteriorate with each successive year. The horizontal bands across the chart represent the historical and current budget-based productivity targets. FMI’s original (2011) suggested target for Ship 3 and the estimated core productivity of the AWD enterprise are also shown.127

32. This FMI report also provides evidence that some elements of the remediation strategy may not have worked as effectively as expected:

Navantia took control of production management in February 2016 … the goals of ASC Shipbuilding and Navantia are not wholly aligned. This misalignment has created friction which at the time of this assessment had led to deterioration in the relationships between senior managers in both teams. This now appears to be having a negative impact on performance. It is essential that the senior managers work effectively together and focus on the common goal of delivering quality vessels to schedule and under budget. Working together to achieve this in the remaining two years of construction is the best way to maintain the credibility of all parties involved.128

33. FMI also found that Navantia was ‘not incentivised to provide a shipbuilding continuum at Osborne’:

Thus it is only motivated to provide technology transfer and process improvements that will make a difference on AWD. On the other hand, ASC is focused on process improvements for the future and on further developing the local workforce.

34. Defence advised the ANAO that FMI benchmarking reports are not provided to its ministers.

Analysis of the case

35. Work on the analysis of the project (Winter–White report) and subsequent remediation plan began before the Minister placed the project on the Projects of Concern. Further, the concerns that developed before the Winter–White review began had a twofold focus:

  • the slow progress of the Air Warfare Destroyer build project and its increasing cost; and
  • the productivity of the ASC shipyard, where that build was taking place.

36. The question of productivity was relevant to the Air Warfare Destroyer build but was also a concern for future shipbuilding capability within Australia, one of the central concerns of the Winter–White review (see paragraph 12, above) and included in the review’s terms of reference.129

37. Defence’s Air Warfare Destroyer Project Management Office states that it was difficult to address the productivity problem when it first came to light and attributes credit to the Department of Finance’s involvement in remediation:

low productivity in itself did not initially cause concern as it was expected to improve through the build program. The rate of improvement, however, was well below expectation. The EV [earned value] data in the monthly PMRs [project management reports] made the deterioration of cost and schedule clear to the AWD PMO [Air Warfare Destroyer Project Management Office], but, as stated above, it was hampered in not being able to initiate remedial actions in the Shipyard and of ASC, quickly enough.

The Government’s Reform program brought the Department of Finance into the process, which allowed remediation to move forward.

38. When the Air Warfare Destroyer project was removed from the Projects of Concern list, Defence’s Independent Assurance Reviews concluded that, after the implementation of the reform program (including insertion of the Navantia management team) the prospects for completion within the available time and budget were good. In this light, there was a basis of substance from the perspective of the project that it be removed from the Projects of Concern list. This view does not seem to have changed subsequently, with provisional acceptance of the third and final ship expected in December 2019.130 Initial Operating Capability was forecast to be achieved in December 2018 (36 months behind the original planned schedule).

39. Notably, there was no reference in the Defence deliberations on removal from the list of the meeting of any milestones on a remediation plan or, indeed, any reference whatsoever to the remediation plan. The primary consideration was the prospect of completion of the required capability within the time and funding available.

40. On the other hand, the regular FMI analyses were not showing any increase in productivity of the shipyard. Defence has pointed out that ‘Even in mature shipyards, productivity levels for a first of class ship are always much lower than the shipyard’s core productivity, even up to 50%’.131 Thus, productivity in the build of Ship 2 and Ship 3 were progressively better than Ship 1 — as expected. Nevertheless, by FMI’s calculation, that productivity remained, even with Ship 3, over 120 man-hours/compensated gross tonne, twice the benchmark originally set by FMI of 60–65 man-hours/compensated gross tonne and well outside the modified target of 80 man-hours/compensated gross tonne.

41. A further consideration with Ship 3 in this case is that its construction may suffer from ‘last ship syndrome’ — where progress is adversely affected by labour anticipating a need for new work. This has been recognised by Defence as a risk in need of mitigation.

42. There is no indication that placing the Air Warfare Destroyer on the Projects of Concern list, of itself, made any difference to the remediation work, given that remediation was initiated independently and already had senior engagement. It is also highly probable that the project would have continued to command senior management and ministerial attention, given the project’s importance and cost and the fact that other work was under way on a future shipbuilding strategy.

43. On the matter of what difference being placed on the Projects of Concern list made to the Air Warfare Destroyer project, Defence’s Air Warfare Destroyer Project Management Office provided the following account:

While placing the AWD project on the Projects of Concern list in 2014 may have garnered the attention of Government and senior members in Defence and DMO, it made little difference to the operation of the AWD Program, as the Commonwealth and the Alliance were already working to remediate the project.

In 2012, internal reporting made it clear that there were issues within the project that needed to be addressed: the quality of work in the block subcontractors, and that budget and schedule were slipping. The Alliance made sure overseas shipbuilding expertise was inserted into BAE’s Williamstown shipyard and ASC personnel embedded at Forgacs, and the Alliance reallocated module work away from BAE to Forgacs and Navantia.

Even with these actions, cost and schedule were still slipping due to the inexperience of the shipbuilders, but the DMO Program Office was unable to take positive control of the deteriorating situation, in part due to the nature of ASC being a Government Business Enterprise, and in part due to the nature of the ABTIA [Alliance-Based Target Incentive Agreement] contract, which required unanimous approval at the AWD Project Board level before any remedial action could be taken.

It was the publication of the ANAO report and the publicity surrounding the Winter-White report, rather than being placed on the PoC list, that initiated the Reform program which culminated in Navantia being installed as shipbuilding manager in December 2015. Since then, the AWD budget and schedule have stabilised and shipyard productivity improved.132

From the AWD Program’s perspective, being placed on the PoC list appeared to be a necessary formality, required for administrative monitoring of DMO/CASG programs under stress and warranting increased senior stakeholder oversight.133

Appendix 5 Major case study 2 — AIR 9000 Phases 2, 4 and 6, Multi-Role Helicopter

What was the project about?

1. In 2002, Defence released the Australian Defence Force’s Helicopter Strategic Master Plan which sought to reduce the number of helicopter types in the ADF from nine to five with the aim of realising training, support and sustainment efficiencies. The acquisition of the Multi-Role Helicopter (MRH) formed a key component of the Strategic Master Plan as one of three fundamental classes of platform.134

2. The MRH Program consists of three phases of AIR 9000:

  • Phase 2 (12 MRH90 helicopters) is the acquisition of an additional Squadron of troop lift aircraft for the Australian Army;
  • Phase 4 (28 MRH90 helicopters) will replace Army’s Black Hawk helicopters in the Air Mobile and Special Operations roles; and
  • Phase 6 (6 MRH90 helicopters) will replace Royal Australian Navy (RAN) Sea King helicopters in the Maritime Support Helicopter role.

3. The Government approved AIR 9000 Phase 2 in 2004, and Phases 4 and 6 in 2006. The Phase 2 acquisition contract was signed in June 2005 for $912.5 million, and was amended in June 2006 to include the additional 34 MRH90 aircraft acquired under Phase 4 and 6, and their support systems, at a cost of $2,656.5 million. The sustainment contract, for $353 million, was signed in July 2005.

How did the project become a Project of Concern?

4. In September 2010, Defence discussed internally the MRH90 project becoming a Project of Concern. In November 2010, Defence had stopped accepting aircraft as they did not meet all the contractual requirements which, in turn, affected the MRH90 program’s ability to achieve capability milestones. Defence briefed the then Minister for Defence Materiel in late 2010 recommending that the project be included on the Projects of Concern list due to continuing delays, and outlining the planned action to stop aircraft acceptance and key system issues including instances of engine failure. The Minister did not approve the listing, requesting instead that Defence undertake a diagnostic gate review to enable further consideration of the matter.

5. A Defence Gate Review in February 2011 identified a number of issues with the project including: technical issues being complicated by contractual shortcomings, poor contractor performance, low aircraft availability, and value for money concerns. The Gate Review Board raised 18 action items to address these issues, of which the key focus was the renegotiation of the acquisition and sustainment contracts commencing with a deed to be negotiated with the Contractor by June 2011.

6. A Gate Review was held in September 2011, and was followed by a further meeting of the Gate Review Board in October 2011 with the prime contractor, Australian Aerospace.

7. Following the September and October 2011 Gate Reviews, the Defence Materiel Organisation provided advice to the Ministers for Defence and Defence Materiel that: ‘the program has triggered schedule and contractor performance Early Indicator and Warning thresholds, and recommend that the project be listed and managed as a Project of Concern by Defence.’

8. The Gate Review Assurance Board further recommended, if the Ministers agreed to the Project of Concern listing, that:

  • Industry should be invited to participate in an initial Project of Concern summit, if Defence is to achieve the desired impact.
  • Project of Concern remediation objectives should encompass whole of program aspects, including full rectification of technical issues and the delivery of a reliable and available capability at an affordable price through the capability life cycle.
  • Specific triggers and milestones would need to be achieved for the program to be removed from the Project of Concern list and these will need to be defined by Defence senior management in conjunction with the Minister and industry.

9. The MRH90 program subsequently became a Project of Concern in November 2011 due to: ‘schedule and contractor performance.’

What was the remediation plan?

10. The 2011 Gate Reviews that recommended that the project become a Project of Concern also identified a number of actions to remediate the project. Remediation was centred on the negotiation of two Deeds with the contractor: Deed 1, to rectify technical issues with the aircraft; and Deed 2, to negotiate the Commonwealth’s desired sustainment-related performance and value-for-money outcomes.

11. The first Deed was signed on 5 October 2011 just before the project became a Project of Concern. The Deed sought to address a number of technical, legal and commercial issues.135 Deed 1.1 was signed in March 2012 as a variation to Deed 1. Through this variation Defence agreed to accept an initial tranche of MRH90 aircraft at a lesser configuration than the contract required and, in return, Australian Aerospace agreed to an interim sustainment performance regime for 1 February 2012 to 31 January 2013.

12. A further deed, Deed 2, was signed on 9 May 2013. Defence and Australian Aerospace entered into Deed 2 to agree how certain issues between them would be resolved, vary the AIR 9000 program contracts in accordance with the terms of the deed, and release each other from certain claims each of them had at 1 July 2013.136

13. The two-Deed approach to remediating the MRH90 Program was already underway before the project became a Project of Concern in November 2011. A brief to the then Minister for Defence provided advice on the outcomes of the September 2011 Gate Review and noted that while the Gate Review Assurance Board had considered termination of the project due to excessive delays, the Board supported continuing the two-Deed approach and that the project would likely become a Project of Concern. In response the Minister commented that: ‘[Project of Concern] status seems somewhat empty if one has already gone down the Deed 1 [and] 2 route.’

Development of remediation criteria and action logs

14. In December 2011, Defence began developing the remediation criteria for the project. The Project Director of the MRH90 Program noted that the first remediation criterion should be ‘full access to costing and technical data’.

15. Remediation criteria and action logs for the project were developed as part of the Projects of Concern summits. The first summit after the project was listed as a Project of Concern took place in April 2012. Early remediation plans focused on the negotiation of Deed 2 and the achievement of Operational Capability Maritime 2137 by 30 August 2013. At the last Project of Concern summit in July 2015, remediation focused on testing of a sub-system to ensure deficiencies had been rectified; conducting an Airworthiness Board to achieve an Australian Military Type Certification and Service Release; and holding a Configuration Board to enable further aircraft acceptance.

16. Defence advised the ANAO in August 2018 that the CASG Quarterly Performance Reports contain a summary of short term, medium term and long term remediation plans. The June 2018 report for the MRH90 program outlines the short-term (1–3 months) remediation goal for the project. These include addressing issues with the Electronic Warfare system to allow operations in medium and high-threat environments, and to seek Army’s and Navy’s position in removing the program from the Projects of Concern list.

17. The Project of Concern monthly reports illustrate that the reason for the Project of Concern listing, expected removal dates and removal criteria have changed over time for AIR 9000 Phases 2, 4 and 6.

Removal from the Projects of Concern list

18. The project is still on the Projects of Concern list as of October 2018; nevertheless, there have been attempts to remove the project from the list over the years.

19. A November 2013 brief to the then Minister for Defence noted that MRH90 was one of three projects approaching a point where it would be appropriate to consider removing them from the list:

The signature of Deed 2 along with improved reliability in the helicopter, means that this project could be considered for removal from the list upon achieving Initial Operating Capability for the Navy. This milestone is expected in late 2013 or early 2014. While some challenges will remain, this project would be able to return to normal management arrangements.

20. There were no further attempts to remove the project from the Projects of Concern list until May 2018 when an internal Defence brief recommended that the project be removed due to a ‘demonstration of an acceptable level of capability for Navy and Army’ and that the ‘resolution of remaining issues are being managed through existing leverage under the contract and via high level engagement with Industry.’

21. The brief noted that the project may need to remain on the Projects of Concern list due to the delays to the Operational Capability Maritime 2 milestone; unresolved commercial matters relating to capability delivery; and cost of ownership concerns.

22. The MRH90 project team sought the Capability Manager’s position on the assessment of the Early Warning Self Protection System and the ability to progress declaration of further capability milestones. As of October 2018, Defence forecast removal of the project from the list during mid-2019.

Analysis of the case

23. The following sections provide an overview of the implications of the Project of Concern listing for the MRH90 Program.

Effect of the Project of Concern listing on budget, schedule and capability

24. The table below outlines the change in budget and schedule at key points for the MRH90 project. While the project’s budget has remained largely the same over the last 12 years, the scheduled Final Operating Capability for the project has slipped by 89 months (or around 7.5 years).

Table A5.1: Budget and schedule at key points for MRH90

 

At Second Pass Approval (June 2006)

At POC listing (November 2011)

Current (August 2018)

Budget

$3,522 milliona

$3,753 million

$3,764 million

Final Operating Capability (FOC) date

Dec 2012 (Navy)

Jun 2014 (Army)

Dec 2012 (Navy)

Jun 2014 (Army)b

December 2021c

    

Note a: This figure includes $953.9 million for Government Second Pass Approval for AIR 9000 Phase 2, and $2,565 million for Government Second Pass Approval for AIR 9000 Phases 4 and 6.

Note b: At the time of Project of Concern listing, the Final Materiel Release and Final Operational Capability dates were subject to review based on the rebaselining of aircraft delivery as part of Deed 2 negotiations.

Note c: December 2021 is the current forecast date for FOC; however, July 2019 remains the date in the current Materiel Acquisition Agreement and the approved date. An update is being progressed to Government seeking agreement for the 2021 FOC date (Defence advice of October 2018).

Source: ANAO analysis of Defence documentation.

25. Over the life of the program, operational capability has been affected by several technical issues including the cargo hook release mechanism, the fast roping and rappelling device, cabin seating and the self-defence gun mount. While capability over the life of the project may not have been reduced, the schedule has been delayed. For example, the initial remediation plan in April 2012 for the project sought to achieve Operational Capability Maritime 2 by 30 August 2013; however, as of July 2018 this milestone had not yet been achieved.

Leveraging industry

26. Defence have sought to use the Project of Concern listing for AIR 9000 Phases 2, 4, and 6, and also the potential listing of the project, to leverage industry.

27. The February 2011 Gate Review of the MRH90 project noted that:

The indicators for making the MRH Project a [Project of Concern] appear to have been triggered. However, [Australian Aerospace] are keen to avoid this action and have indicated a strong commitment to resolving engineering problems and raising aircraft capability … the Commonwealth should use the [Project of Concern] leverage to require [Australian Aerospace] to improve performance.

28. In April 2011 prior to the project being listing as a Project of Concern, the General Manager Systems of the Defence Materiel Organisation wrote to the Chief Executive Officer of Australian Aerospace regarding the project and the strategies needed to address various issues.

29. The General Manager Systems of the Defence Materiel Organisation also proposed that a joint steering group from Defence and Australian Aerospace be established to monitor and guide the parties. Defence sought written confirmation from the contractor to agree to this strategy and indicated that Defence would recommend to the Minister of Defence that he not list the Program as a Project of Concern.

30. Following this letter, a briefing paper to the Defence Capability and Investment Committee noted that the February 2011 Gate Review of the program had set a trigger date for consideration as a Project of Concern for September 2011 and that:

[Australian Aerospace] have been actively campaigning to not become a [Project of Concern] (reputational impacts), and the set time period will allow them time to demonstrate their commitment to remediation of the program.

31. In June 2018, Defence’s First Assistant Secretary, Helicopter Systems Division, wrote to the Chief of Navy and the Chief of Army to seek their agreement on the Project of Concern removal criteria for AIR 9000 Phase 2, 4 and 6. The letter noted that Airbus and Defence agreed that the project would be considered for removal from Projects of Concern when an acceptable level of capability for Navy and Army was achieved, commercial resolution of technical deficiencies was resolved and there was a demonstrated improvement in aircraft serviceability that supported achievement of Navy and Army capability milestones.

32. The letter further noted that a key technical issue for resolution is the Enhanced Cargo Hook System, and that Defence was managing the resolution of this issue through high level engagement with industry and a delay to the Final Acceptance Milestone payment. Defence assessed that tying the resolution of this issue to the project’s Project of Concern listing offered Defence: ‘the best chance to extract the last remaining leverage and further incentivise Industry beyond measures already in place’.

Use in other Defence acquisitions

33. In February 2012, Defence wrote to Australian Aerospace regarding the implication of the Project of Concern listing for AIR 9000 Phase 2, 4 and 6. The contractor was advised that if it were to tender for AIR 9000 Phase 7, Defence would be exercising its rights under the Conditions of Tender to ensure Australian Aerospace had made progress in remediating the project, including the content of and timeliness in agreeing the remediation plan in the first instance, and then its subsequent implementation.

34. The tender evaluation plan for AIR 9000 Phase 7, Helicopter Aircrew Training System, dated 6 March 2012, referred to Projects of Concern in its initial screening process:

Non-compliant Tenders, Tenders associated with Projects of Concern, and any Alternative Tenders will be identified and a recommendation will be provided to the Delegate regarding appropriate treatment, including whether any non-compliant Tenders or Tenders associated with Projects of Concern should be excluded and whether any Alternative Tenders should be considered.

35. Additionally, the RFT criteria included: ‘Past performance of contractual obligations of the Tenderer, including involvement in any contract that is or has been listed as a Project of Concern.’

Disagreement in Defence on the removal criteria and timeframe for the Project of Concern listing

36. There has been some internal disagreement within Defence on the timeframe and criteria for removing the MRH90 Program from the Projects of Concern list.

37. In November 2013, a brief to the Minister noted that it might have been possible to remove the project from the list in early 2014. A Gate Review in December 2014 concluded that, despite costs and schedule risks, the Deed 2 agreement had led to significant improvement and the project was proceeding well. The CEO of DMO noted that the findings of the Gate Review Board were in direct contrast to the views of Navy as Capability Manager: ‘I have recently received a minute from Chief of Navy which is much more critical of the deficiencies [with the project] than this review indicates.’ A July 2018 minute from the Chief of Navy to Helicopter Systems Division, Capability Acquisition and Sustainment Group, noted that he had very little confidence in the contractor’s ability to deliver and that Defence should consider in-house solutions for the cargo hook.

38. Defence has forecast that the MRH90 Acquisition Project is expected to be removed in mid-2019.138

MRH90 Sustainment

39. The sustainment ‘product’ corresponding to the acquisition project, CA48 Multi-Role Helicopters139, has been on the Projects of Interest list since March 2017.

40. The October 2017 Independent Assurance Review for CA48 identified a number of issues with sustainment comparable to the issues for the MRH90 acquisition project including contractor delays. The Board noted that the most significant issue was the ‘high operational maintenance burden and associated inability to achieve an acceptable ratio of maintenance-hours to flying-hours, and a lack of responsiveness by [the sub-contractor] in a number of areas.’

Summary

41. It is not apparent that making the MRH90 program a Project of Concern made a difference in remediating the acquisition project. The key remediation activities — negotiating Deed 1 and Deed 2 — were already underway at the time of the listing. As noted in paragraph 13, the Minister commented in 2011 that ‘[Project of Concern] status seems somewhat empty if one has already gone down the Deed 1 [and] 2 route’.

42. The Project of Concern listing for the MRH90 program was used by Defence Senior Management to try to leverage industry before the listing (paragraphs 30–32, above), and in attempting to influence industry to resolve the ongoing cargo hook issue. It is not apparent that any benefit flowed from this attempt to influence industry behaviour, given the continued lengthy delays to the project schedule, and the ongoing technical issues. For example, the cargo hook issue was first identified in early 2012, and remains unresolved as of October 2018. Defence advised the ANAO in October 2018 that it expects the issue will be resolved by June 2019.

Appendix 6 Major case study 3 — CN10, Collins Class Submarine Sustainment

What was the project about?

1. Navy has six Collins class submarines that fulfil the roles of maritime strike and interdiction, maritime surveillance, reconnaissance and intelligence collection, undersea warfare and Special Forces operations. Construction of the first of the six submarines, HMAS Collins, began in 1990 with delivery in 1996. The final submarine, HMAS Rankin, was delivered in 2003.140

2. In December 2003, Defence entered into a Through Life Support Agreement with the then Australian Submarine Corporation (now known as ASC). The agreement was worth $3.5 billion over 25 years (comprising a 15 year agreement with a further two five-year extension options).

How did the project become a Project of Concern?

3. The program was made a project of concern in November 2008 as the submarine fleet had failed to meet acceptable benchmarks for operational capability.

4. The Through-Life Support Agreement signed with ASC in 2003 had a number of shortcomings, and Defence noted in internal briefings that:

The terms of the [Through-Life Support Agreement] reflected, in part, the [Commonwealth of Australia’s] recognition of ASC’s weak financial position at the time and limited assured future work, and the strategic importance of having an Australian industrial capacity to provide submarine maintenance and capability enhancement.

5. Defence also noted that the Agreement provided little leverage to drive ASC to achieve greater efficiencies in sustainment of the submarines, but that achieving greater efficiency would be essential to the Commonwealth affording the submarine capability.

6. The Defence Budget Audit (Pappas Review),141 released in April 2009, indicated that the ASC sustainment activity was inefficient and that a 15 to 30 per cent increase in submarine availability could be achieved for the same cost.

7. By mid-2009, submarine availability had worsened — for the period from 19 to 21 June 2009142 there were no available submarines for training or operations, a situation Defence deemed unacceptable given the need to train submariners for the current and future capability. This prompted a review of submarine availability, completed in August 2009. The review identified a number of issues including that submarine availability was impacted by design defects, logistics support and poor industry performance. The review concluded there was a lack of understanding of the resource implications and management effort involved in Collins sustainment. This issue was further compounded by a lack of an integrated master schedule143, and inefficiencies in industry support.

8. The review identified that the key focus should be providing the greatest number of materiel-ready days144 to progress training in support of maintaining sufficient Navy submarine crew.

What was the remediation plan?

9. The Collins Class Sustainment program was regarded by Defence and its ministers as the ‘number one’ Project of Concern. In February 2010, a speech by the Minister for Defence Personnel, Materiel and Science noted that ‘Collins Class Submarine sustainment remains a challenge, and it is at the top of the list of Projects of Concern.’ The speech outlined the key remediation activities for Collins Class sustainment including: developing a new contractual approach with ASC; appointing a senior DMO executive to oversee Collins Class sustainment and establishing the Australian Submarine Program Office with the goal of improving submarine output.

10. The new contractual approach included the negotiation of an In-Service Support Contract, to replace the Through-Life Support Agreement with ASC. A Gate Review in June 2010 assessed the readiness of the program to enter the In-Service Support Contract. The Gate Review Board concluded that the submarine program was not ready and raised a number of significant issues with the program including continuing concerns with submarine availability, and significant financial shortfalls in the order of a half to one billion dollars across the coming decade.

11. The Gate Review listed 21 action items to assist in the program’s remediation, and recommended that the Submarine Program Manager conduct a fundamental review of Defence’s strategy for submarine sustainment before the In-Service Support Contract was negotiated.

12. The Collins Class Sustainment Review Phase 1 Report (Coles Review Phase 1) was completed in November 2011. Coles Review Phase 1 sought to make a number of early recommendations to improve the availability and overall program management of Collins Class submarines, before a more detailed Phase 2 report due for completion in late 2012. The Phase 1 report identified: poor availability of submarines caused by a crew shortfall; lack of spares and unreliable equipment; lack of cohesion in strategic leadership including a lack of clarity of accountability, authority and responsibility; thinly spread submarine domain knowledge; a lack of robustness of Navy’s contribution to manning and sustainment; no long term strategic plan for efficient asset utilisation; unclear requirements and unrealistic goals. The report recommended that the In-Service Support Contract should proceed as planned and should be used to drive the requirement for an early move to performance based contracting.

13. In June 2012, the In-Service Support Contract became operational. This contract was performance based and placed responsibility on ASC for delivering outputs using key performance indicators.

14. In November 2012, the Study into the Business of Sustaining Australia’s Strategic Collins Class Submarine Capability (Coles Review) was released. The study sought to identify the problems with Collins fleet sustainment performance and the initiatives that would address problems and resolve issues. The study assessed Collins Class sustainment against four categories: availability, planned maintenance duration, maintenance overrun and days lost to defects when not in maintenance. The report found that, compared to international benchmarks, the Collins Class: achieved 56 per cent of the benchmark for availability, spent 39 per cent more time on planned maintenance days, and had more than twice the expected maintenance overrun days and days lost to defects when not in maintenance.

15. The Coles Review made 25 recommendations aimed at remediating the Collins Class sustainment program including: implementing the In-Service Support Contract to encourage performance based behaviour, developing a through-life capability management plan, developing and implementing a workforce strategy to address skills shortages at the management level, and developing and implementing an information management strategy to inform decision making.

How did the project leave the Projects of Concern list?

16. In March 2014, the Study into the Business of Sustaining Australia’s Strategic Collins Class Submarine Capability (Coles Progress Review 1) was released, 15 months on from the November 2012 report, to assess the progress made towards improving submarine availability and performance.145 Overall, the report noted an improvement in submarine availability, and progress towards achieving benchmark availability.

17. Following the Coles Progress Review, Defence considered removing the program from the Projects of Concern list due to the improvement in sustainment performance. However, after further consideration, the program remained on the list until the international benchmark availability mentioned in the Coles Progress Review had been achieved and maintained for a two year period.

18. The Study into the Business of Sustaining Australia’s Strategic Collins Class Submarine Capability – Beyond Benchmark (Coles Progress Review 2) was released in May 2016. The review noted a steady improvement in submarine availability and performance against international benchmarks. The table below illustrates the progress in remediating the Collins Class sustainment program across the three Coles Reviews.

Table A6.1: Progress in remediating the Collins Class sustainment program across the three Coles Reviews

Criterion

International benchmark

Coles Review (November 2012)a

Coles Progress Review 1

(March 2014)

Coles Progress Review 2

(May 2016)

Availability (days)

1

0.56

0.71

0.86

Planned maintenance duration (days)

1

1.39

1.20

1.27

Maintenance overrun (days)

1

2.33

1.20

0.42

Percentage days lost to defects when not in maintenance (per cent)

1

2.43

2.00

0.60

Cost per Material Ready Daysb ($million per day)

1

2.50

1.50

1.40

     

Note a: This figure is the average from 2006–07 to 2010–11.

Note b: Defence defines a Materiel Ready Day as a day in which a submarine is not in planned maintenance or does not have defects that prevent it from being at sea. The Coles Reviews determined that cost per Material Ready Day was an indicator of cost effectiveness but noted in the 2014 review that increased cost effectiveness was a result of the increase in Materiel Ready Days and not a reduction in budget.

Source: ANAO analysis of: J.Coles, Study into the Business of Sustaining Australia’s Strategic Collins Class Submarine Capability, November 2012; J.Coles, Study into the Business of Sustaining Australia’s Strategic Collins Class Submarine Capability – Beyond Benchmark, May 2016; and J. Coles, Study into the Business of Sustaining Australia’s Strategic Collins Class Submarine Capability – Progress Review, March 2014.

19. The review also noted that 17 of the 25 recommendations made in the 2012 Coles Review had been implemented, with the remaining eight partially implemented.

20. The Coles Progress Review 2 noted that: ‘with benchmark availability on the verge of being achieved, the focus should now be on efficiency improvements and cost reductions across the sustainment program’.

21. Following the Coles Review, there were internal discussions within Defence to remove the Collins Class sustainment program from the Projects of Concern list in November 2016. In February 2017, the Projects of Concern report to the Minister for Defence noted that:

[The Collins Class Sustainment Program] has achieved significant improvement in areas identified by the 2012 Coles Report and has achieved international benchmark levels of availability. Removal may be considered if benchmark availability is maintained.

22. An Independent Assurance Review in May 2017 noted that while benchmark availability had been delivered for 2016–17, continued rigorous governance and reporting processes, including high level reporting to Government would be needed to keep them apprised of changing circumstances and action. The Independent Assurance Review Board recommended to the Minister that the Collins Class sustainment program be removed from the Projects of Concern list. The Minister did so in September 2017.

Analysis of the case

Effect of the Project of Concern listing on budget and capability

23. Despite being placed on the Projects of Concern list in late 2008, the costs of Collins Class sustainment continued to rise and the availability of submarines continued to decline. At the February 2011 Senate Estimates hearings Defence advised that, from 2004 to 2010, submarine sustainment costs went from $203 million per annum to $325 million per annum — a 60 per cent increase. At the same time, the number of submarines available was in decline. At the May 2011 Senate Estimates hearing, Defence advised that the sustainment cost had climbed by a further $90 million to $443 million per annum — a 25 per cent increase across five months and a 120 per cent increase over seven years.

24. In March 2011, a Defence Workforce and Funding Request was made to address the funding shortfall for Collins Class sustainment on behalf of the Chief of Navy. The proposal requested:

  • Additional funding for 2011–12 and 2012–13 for the delivery of Navy’s Collins Class Submarine availability to meet Defence strategic guidance.
  • That the In-Service Support Contract negotiation with ASC would require an additional $141.06 million be in place by 30 April 2011.
  • Additional funds of $96.65 million were required to be in place by 31 May 2011 for inventory and obsolescence remediation.
  • Further, the proposal noted a funding shortfall for 2016–21 of $1341.12 million.

25. The request noted that if funds were not made available, there was a risk to the schedule and operational availability of submarines.

26. The Coles Progress Review 2, released in 2016, noted that:

Attaining benchmark performance was a higher priority than efficiency. With benchmark availability on the verge of being achieved, the focus should now be on efficiency improvements and cost reductions across the sustainment program. Such cost reductions may be required to re-invest into inventory, obsolescence remedies, new infrastructure to manage an ageing fleet and the transition to future submarines.146

27. The August 2017 Projects of Concern monthly report to the Minister for Collins Class sustainment outlined the improvement to availability and the achievements of benchmarks as set out in the Coles Reviews. However, the report also noted that to maintain this level of performance the program budget needed to remain as set out in the current Materiel Sustainment Agreement (the 2017–18 budget was $570.8 million), and that there were several issues to resolve requiring funds from sources other than the sustainment budget, to ensure they did not affect Collins Class performance.147

Leveraging with Industry

28. During the program’s time on the Projects of Concern list, Defence used senior executive and Ministerial engagement to leverage with industry in an attempt to resolve issues. This included the Program Manager for Collins writing to the Managing Director of ASC in April 2011 to resolve key issues around the In-Service Support Contract, and meetings between the then Minister for Defence Materiel and the Managing Director and Chief Executive Officer of ASC to discuss progress on the Collins program, the status of the In-Service Support Contract and sustainment remediation. Explicit reference to the Projects of Concern listing was not used in correspondence with industry.

29. In a February 2015 brief to the Minister, Collins sustainment was used by Defence as an example of how Projects of Concern could be used to influence companies and see them apply significant efforts and resources to recover key programs. The brief noted that Collins sustainment is ‘a technically complex and challenging program requiring additional oversight to manage the sustainment of this strategically important capability. During its time on the Projects of Concern list, availability has improved and continues to become more reliable.’

Other Collins Class projects in Defence

30. In addition to the Collins Class Sustainment program, there are 13 major capital projects under management within the Collins submarine program to improve reliability and capability of the submarine fleet. Twelve of these projects have been approved by the government, and one has not yet been approved. Of these twelve, two are complete, and three are nearing completion and will be consolidated into one project.

31. One of the projects, SEA 1439 Phase 5B.1 Communications Antenna Capability Enhancement, was the subject of an Independent Assurance Review in May 2013. The review concluded that the management of the project was unsatisfactory resulting in the need for a significant cost increase and slippage to the original scheduled final operating capability of around 10 years. The project was not made a Project of Interest, or a Project of Concern and the Independent Assurance Review Board noted instead that: ‘While not an individual Project of Concern, SEA1439 Phase 5B.1 is part of the Collins Submarine Program, which is a Program of Concern as a whole.’

32. The relationship between Projects of Concern listing and remediation of the program is, on the face of it, more positive than in the other case studies. Defence has claimed that the listing for Collins Class sustainment played a significant role in the program’s remediation. Additionally, many of the Collins program reform initiatives began in 2009, after the November 2008 Project of Concern listing, including the establishment of the Australian Submarine Program Office, the agreement of the Integrated Master Schedule and the negotiation of the In-Service Support Contract with ASC.

33. However, the Coles Review Phase 1 report suggests that the remediation of the program and, in particular, the renegotiation of the Through Life Support Agreement, was prompted by the then Government’s announcement in 2008 that it would consider privatising the ASC. The Coles Review and its subsequent progress reports do not refer to the Project of Concern listing for the program, nor is it included as a consideration as part of the terms of reference for the reviews.

Footnotes

1 Department of Defence, Annual Report 2017–18, p. 123.

2 Final Operational Capability is the point where the relevant Capability Manager has certified their satisfaction that the delivered system has satisfied tests and evaluation, and performs as approved at Second Pass approval by government.

3 A complete list of projects that have been on the Projects of Concern list since 2008 is set out in Table 1.1.

4 Department of Defence, Annual Report 2016–17, Chapter 2.

5 Department of Defence, Annual Report 2016–17, p. 124.

6 Defence also maintains a list of projects exhibiting less serious risks, its Projects of Interest list, which is not within the scope of this audit. See paragraph 1.10.

7 Department of Defence, Annual Report 2017–18, p. 123.

8 The concept of ‘Projects of Concern’ is mentioned in Auditor-General Report No.24 2001–02, Status Reporting of Major Defence Acquisition Projects, December 2001. Defence refers to the Projects of Concern regime as having commenced in 2008, when it was expanded from a largely internal process to one involving ministerial decision-making. See Appendix 2.

9 These three Projects of Concern are examined in Appendices 4–6 respectively.

10 Final Operational Capability is the point where the relevant Capability Manager has certified their satisfaction that the delivered system has satisfied tests and evaluation, and performs as approved at Second Pass approval by government.

11 M. Kinnaird (Chair), Defence Procurement Review 2003, Canberra, 2003.

12 Defence advice to the ANAO, 1 February 2019. The effect of this change is analysed in Auditor-General Report No.20 2018–19, 2017–18 Major Projects Report, pp. 50–9.

13 Defence’s Projects of Interest regime is discussed in Appendix 3.

14 Department of Defence, Annual Report 2016–17, Chapter 2.

15 An example of a Project of Concern whose remediation involved a change in scope is AIR 5416 Phase 2A, Project Echidna, which aimed to deliver electronic warfare self-protection capability for rotary wing aircraft.

16Public Governance, Performance and Accountability Act 2013, section 19.

17 This view was advanced in advice to the Minister in November 2014. The advice did not identify any other mechanisms for influencing vendors’ behaviour.

18 Auditor-General Report No.41 2008–09, The Super Seasprite, p. 18.

19 Senate Foreign Affairs, Defence and Trade References Committee, Procurement Procedures for Defence capital projects, 30 August 2012, p. 13.

20 Speech by Minister for Defence, Address to the Department of Defence Senior Leadership Group, Canberra, 26 November 2010, and Transcript, Press Conference, Perth, 17 December 2010.

21 The Minister frequently commented on the time taken by Defence to advise government of delays affecting acquisition projects. See Auditor-General Report No.6 2013–14, Capability Development Reform, p. 244. In September 2012, the Chief Executive Officer, Defence Materiel Organisation, undertook to introduce personal accountability measures requiring senior executives to certify to him their performance against a range of criteria, including meeting remediation milestones for Projects of Concern.

22 Advice from the Defence Materiel Organisation to the Defence Secretary, May 2011.

23 The senior manager within the Defence Materiel Organisation had earlier sought internal legal advice as to whether Defence could decide lawfully not to provide certain information to the Auditor-General. The information in question comprised internally-produced documents provided to ministers in the regular Projects of Concern report. These documents, he stated, could contain: ‘quite frank assessments or statements about Projects which are really intended for Ministerial eyes and ears only. If this material is going to be routinely available to ANAO, [the Defence Materiel Organisation] might think twice about the form and content of such statements to the Minister’ (Request for advice from Special Counsel, Defence Material Organisation, 31 March 2009).

24 Some acquisition projects are managed under sustainment. An example is SEA 5510 Phase 1, a project to upgrade the MH-60R Seahawk helicopter’s combat system, sensors, weapons and counter-measures.

25 LAND 40 Phase 2 was to deliver two new Direct Fire Support weapons for use in close combat. Defence terminated discussions with the preferred vendor for one element, the Light Weight Automatic Grenade Launcher, in February 2011 when, during contract negotiations, Defence identified non-conformance with its requirements. The project had slipped by almost five years and Defence recommended it be added to the Projects of Concern list. The Minister agreed and it was added in December 2012. A new tender for the required equipment began in July 2014. The project was removed from the list in May 2016.

26 Defence advice to the ANAO, October 2018.

27 Defence advice to the ANAO, 1 February 2018.

28 Advice of the General Manager, Programs, Defence Materiel Organisation, to senior colleagues in Defence, 1 July 2008.

29 Parliamentary Secretary for Defence Procurement, Address to the Annual Australian Defence Magazine Congress, 17 February 2009.

30 Minister for Defence and Minister for Defence Materiel, media release MR 187/11, ‘Reforms to Projects of Concern’, 29 June 2011. The full details of the Projects of Concern regime established in June 2011 are set out in Appendix 2.

31 Defence developed this approach in late 2010 and it became available for incorporation in the process announced in May 2011. Minister for Defence and Minister for Defence Materiel, Media Release MR116/11, ‘Strategic Reform Program’, 6 May 2011.

32 Defence’s ‘smart buyer’ risk assessment and project execution process, introduced as a reform following the First Principles Review, also emphasises the importance of identifying project risks and mitigating them early.

33 Minister for Defence and Minister for Defence Materiel, media release MR 116/11, ‘Strategic Reform Program’, 6 May 2011.

34 Discussion of the results of the Gate Review/Independent Assurance Review consideration is in Chapter 3.

35 Evidence of Mr Kim Gillis, Deputy Secretary, Capability Acquisition and Sustainment Group, Defence, Senate Foreign Affairs, Defence and Trade Legislation Committee, 30 March 2017.

36 See ‘Projects of Concern principles, 2018’, Appendix 2. Appendix 2 sets out various editions of the Projects of Concern criteria or principles from 2001 to 2018. For 2018, the material shown is at it appears on the Defence Intranet on 3 December 2018.

37 Defence advice to the ANAO, 1 February 2018.

38 Advice to the ANAO from the Australian Industry Group, November 2018.

39 See Department of Defence, Annual Report 2017–18, p. 123 which provides two paragraphs on the purpose of Projects of Concern and which projects were added to or removed from the list during the year. There is no detail on process.

40 Department of Defence, Capability Life Cycle — detailed design.

41 See Appendix 4.

42 Defence has stated (October 2018) that it intends to review the capability streams construct and the subordinate program structure.

43 Defence advice to the ANAO, 10 October 2018.

44 See Appendix 2.

45 The Internal Assurance Review Policy Instruction (DMI (Exec) 00-0-009 – IARs for Projects and Sustainment Products ver.3) does not mention the Projects of Concern regime. There is some mention of Projects of Concern in a template associated with reporting Independent Assurance Reviews, which may not be an adequate means to convey policy.

46 See paragraphs 2.4-2.6.

47 Further information on SEA 4000 Phase 3 and its Project of Concern listing is included at Appendix 4.

48 Further information on AIR 9000 Phases 2, 4 and 6 and its Project of Concern listing is included at Appendix 5.

49 Defence notified the vendor of the project’s entry to the list in July 2017. A ministerial press release was released on 18 August 2017. A more detailed account of this project’s history on the Projects of Concern list is set out in Box 2 later in this chapter.

50 Department of Defence,’ Air 5431 Phase 1—Deployable Defence Air Traffic Management and Control System, Independent Assurance Review’, Thursday 3 November 2016, Agendum Paper.

51 Defence advice to the ANAO, 1 February 2019.

52 JP 2008 Phase 5A, UHF SATCOM, Independent Assurance Review, 1 August 2017. Defence advised the ANAO (October 2018) that it proposes to delay the final operating capability milestone by approximately one year from quarter four, 2017–18, to quarter four, 2018–19.

53 Ibid.

54 Department of Defence ‘A Strategy to Remodel the Management of Risk in Acquisition and Sustainment in Defence’, February 2017.

55 Department of Defence, Risk management reform program and implementation plan, June 2017, p. 3.

56 Ibid., p. 4.

57 Ibid., p. 8.

58 Defence advised the ANAO (1 February 2019) that it had finalised documentation formalising the Projects of Interest process.

59 Department of Defence, Ministerial submission, ‘Projects of Concern—Remediation Plans and Action Logs’, 31 October 2011. The Department attached detailed remediation plans and action logs with deadlines for four then-current Projects of Concern in fifteen pages of attachments to the submission.

60 A letter from the Deputy Secretary, Capability Acquisition and Sustainment Group, Defence, to the vendor, 28 July 2017, advised that ‘In place of the remediation plan which would usually be required to document a Project of Concern’s path to resolution, Defence will instead be working with Airservices to develop a path to contract execution …’

61 Defence advice to the ANAO, May 2018.

62 Meetings were held on: 15–16 February 2011; 2 March 2011, 27–28 September 2011; 23–24 April 2012; 19–20 November 2012; 7–8 May 2013.

63 Defence advice to the Minister, November 2014.

64 Defence states that they are now held only ‘as required,’ with only one specific vendor at a time, and at the direction of ministers. (Defence advice to ANAO, October 2018.) In relation to the summit held on 12 September 2018, Defence advised the ANAO (1 February 2019) that ‘there is no official meeting record due to the sensitive nature of the dealings with the contractor’.

65 Defence advice to the ANAO, 1 February 2019.

66 Advice to the ANAO from the Australian Industry Group, November 2018.

67 Defence advice to the ANAO, August 2018.

68 This right continues only so long as the project retains Project of Concern status.

69 Letter of 14 February 2012 from Defence to a vendor.

70 Defence stated that its procurement and contracting area is not aware of any project seeking contracting or legal advice on the application of the Projects of Concern exclusion. It also stated that it had not applied any weighting to tenderers nor excluded a tenderer for reasons related to involvement with a Project of Concern.

71 Letter of 28 July 2017 from Defence, to the vendor.

72 Email to colleagues from Director, Projects of Concern, Defence Materiel Organisation, December 2010.

73 Evidence of Mr Dennis Richardson, Secretary of Defence and Mr Kim Gillis, Deputy Secretary, Capability Acquisition and Sustainment Group, Defence, Senate Foreign Affairs, Defence and Trade Legislation Committee, 30 March 2017.

74 Defence advice to the ANAO, 1 February 2019.

75 The CMATS (or OneSKY) project has been the subject of two performance audits: Auditor-General Report No.1 2016–17, Procurement of the International Centre for Complex Project Management to assist on the OneSKY Australia Program, and Auditor-General Report No.46 2016–17, Conduct of the OneSKY Tender.

76 Mr Jim McDowell, Interim Report into the Progress of the OneSKY Australia Program Project to supply a Civil and Military Air Traffic Management System, September 2015.

77 The Quarterly Performance Report also includes an overview of acquisition projects (performance data for key acquisition projects and Projects of Interest) and an overview of sustainment products (performance data for the top 30 sustainment products and Products of Interest).

78 Defence plans to replace these reporting systems as part of a wider program to rationalise its IT applications across the department. The Enterprise Resource Planning Program aims to standardise business processes in Defence, and move to a SAP-based application for reporting. This program is expected to be finalised and implemented within the department by 2023.

79 The lack of timeliness for the Quarterly Performance Report was discussed in Auditor-General Report No.2 2017–18 Defence’s Management of Materiel Sustainment, paragraph 3.15 https://www.anao.gov.au/sites/g/files/net4981/f/ANAO_Report_2017-18_02.pdf [Accessed 7 September 2018]. The matter was discussed at a public hearing for the inquiry by the Joint Committee of Public Accounts and Audit, Defence sustainment expenditure, 16 August 2017.

80 Defence has identified limitations in the information technology systems including: the use of ‘lag’ rather than ‘lead’ indicators, the difficulty in identifying the appropriate information due to various layers of reporting, and inconsistency in methods of reporting by different areas in Defence. Defence has also given testimony that it would have a ‘much better reporting system’ by the end of 2017. (Mr Kim Gillis, Deputy Secretary, Capability Acquisition and Sustainment Group, Defence, transcript of evidence, the Joint Committee of Public Accounts and Audit inquiry into Defence’s Management of Materiel Sustainment, 16 August 2017, p. 10.)

81 Defence advice to the ANAO, October 2018.

82 See for example Department of Defence, Annual Report 2017–18, p. 123. http://www.defence.gov.au/annualreports/17-18/Downloads/DAR_2017-18_Comp... [Accessed 20 November 2018].

83 See for example: Minister for Defence Industry and Minister for Defence – Joint Media Release – Projects of Concern Update, 18 August 2017. https://www.minister.defence.gov.au/minister/marise-payne/media-releases/minister-defence-industry-minister-defence-joint-media-release [Accessed 6 September 2018].

84 There is also ad hoc reporting on Projects of Concern at Parliamentary Committee Hearings regarding priority of projects on list, or reasons for a project being added to or removed from the list.

85 Defence advice to the ANAO, 1 February 2019.

86 The most recent Project of Concern summit was held in July 2015.

87 Defence made the first claim regularly in its CASG Quarterly Performance Report. In its most recent Quarterly Performance Report this claim has been changed to ‘a successful management tool for recovering problem projects’. The second claim appears in the Department of Defence, Annual Report 2016–17, p. 124.

88 The views of previous ministers were recorded in interviews conducted for the First Principles Review of Defence.

89 View of a Defence SES Band 3 officer, November 2013, when the number of projects of concern had just passed an historical peak.

90 J Goennemann, (Australian Aerospace Limited), media briefing, Canberra, 8 February 2011.

91 Message dated August 2006.

92 Defence, submission to CASG Executive Advisory Committee, February 2017.

93 Defence, CASG Executive Advisory Committee, agenda paper 10, 12 May 2017.

94 Defence advice to the ANAO of 1 February 2019.

95 Although the above criteria are described as ‘draft’, no more definitive version has been found.

96 Defence Intranet, viewed 28 November 2018.

97 See, for example, Auditor-General Report No.26 2017–18, 2016–17 Major Projects Report, p. 12.

98 Before July 2015, known as the ‘Defence Materiel Organisation Quarterly Program Performance Report’.

99 Auditor-General Report No.26 2017–18, 2016–17 Major Projects Report, Part 2. Defence Major Projects Report.

100 D. Mortimer AO, Going to the next level: The report of the Defence Procurement and Sustainment Review, 18 September 2008 (known as ‘the Mortimer Review’) was an evaluation of the effectiveness of ongoing reforms in the then Defence Materiel Organisation and sought to identify further potential reforms to the acquisition and through life support of defence equipment. http://www.defence.gov.au/publications/mortimerReview.pdf [Accessed 27 July 2018]

101 The Strategic Performance Reporting area is location in the Directorate of Program Approvals and Agreements within the Capability Acquisition and Sustainment Group.

102 Commonwealth of Australia, Defence 2000: Our Future Defence Force (Defence White Paper 2000), pp. 89–90.

103 Department of Defence, Defence Capability Plan 2001–10 (public version), pp. 263–4.

104 Department of Defence, Defence Capability Plan 2004–14 (public version), pp. 159–60.

105 A similar study had been conducted in 2010. Defence undertook a quality check on FMI’s work in 2011 and concluded that it had been done in a thorough and professional manner and as contracted.

106 Defence, paper prepared for Government–ASC Steering Committee meeting, 19 December 2011.

107 In light of this announcement, the delivery schedule for the three Air Warfare Destroyers would be March 2016, September 2017 and March 2019.

108 Auditor-General Report No.12 2013–14, 2012–13 Major Projects Report.

109 At about that time, another update from FMI reported a further decline in estimated-at-completion shipyard productivity for all three ships.

110 Terms of Reference, Independent Review into the Performance of the Hobart Class Air Warfare Destroyer Program (Project SEA 4000), March 2014.

111 D. Winter and J. White, Report of the Independent Review into the Performance of the Hobart Class Air Warfare Destroyer Program, 8 May 2014.

112 Winter–White Report, p. 5.

113 Winter–White Report, p. 6.

114 Letter from Professor Don Winter and Dr John White to the Minister for Finance and Minister for Defence, 4 April 2014.

115 Winter–White Report, 19.

116 Minister for Defence, media release, ‘Air Warfare Destroyer added to Projects of Concern list’, 4 June 2014.

117 Minister for Finance and Minister for Defence, joint media release, ‘Putting the Air Warfare Destroyer program back on track’, 4 June 2014. The Minister for Finance was the shareholder Minister for ASC, which is a Government Business Enterprise and the builder of the Air Warfare Destroyers.

118 According to the reform plan, the Winter–White review had been adopted in principle by the Government. The subsequent AWD Reform Strategy aimed to achieve the following key objectives:

• Not limit the options for Australian naval shipbuilding;

• Ensure the viability of ASC’s submarine sustainment business;

• Focus on the importance of effective governance arrangements to reduce risk and provide frequent opportunity for Commonwealth oversight;

• Ensure a suitable, practical approach is developed, including an agreed transition plan is in place; and

• To ensure that the Commonwealth achieves these outcomes in a value for money manner.

119 Transcript of joint press conference, Minister for Finance and Minister for Defence, 4 June 2014. Defence’s Air Warfare Destroyer Project Management Office has pointed out (19 July 2018) that the Minister’s reference to 80 man-hours/compensated gross tonne was ‘not quite correct’. Rather, their view is that productivity between 80 and 85 man-hours/compensated gross tonne was ‘FMI’s opinion of a realistic productivity target for Ship 3 (FMI 2012 report, p. 10)’.

120 SEA 4000 Phase 3—Air Warfare Destroyer (AWD), Performance Gate Review, 2 October 2014. The Gate Review noted, however, that it was not aware of the deliberations and likely recommendations of the AWD Reform Group: ‘In particular, the eventual schedule and budget implications for the Project that will flow from the Reform Group recommendations cannot currently be quantified.’

121 Minister for Defence and Minister for Finance, joint media release, ‘Air Warfare Destroyer program still fixing serious legacy issues’, 22 May 2015. The schedule was then delayed to June 2017, September 2018 and March 2020 respectively for the three ships.

122 Prime Minister and Minister for Defence, joint media release, ‘The Government’s plan for a strong and sustainable naval shipbuilding industry’, 4 August 2015.

123 Minister for Finance, Minister for Defence Industry and Minister for Defence, joint media release, ‘Supporting Australia’s future shipbuilding capability’, 11 October 2016.

124 This Independent Assurance Review was conducted on 24 July 2017. It was not signed by the parties until October 2017. This was after the commissioning of the first ship, HMAS Hobart, on 23 September 2017.

125 The Minister for Defence agreed on 18 December 2017 and the Minister for Defence Industry agreed on 7 December 2017.

126 According to Defence advice (19 July 2018), FMI had had issues with collecting accurate data in earlier years’ work. Therefore, as it was difficult to get an overall view of the productivity levels being achieved, it was also difficult to manage improvement. However, under the reformed arrangements the use of Navantia shipbuilding tools in the shipyard had improved data reliability.

127 Defence, consultancy report by First Marine International, ‘Assessment of actual and planned shipbuilding productivity for the AWD projects’, 2017 Update, Final Report, 15 February 2018, p. 17.

128 Ibid., pp. 49–50.

129 The question of productivity in the Air Warfare Destroyer build has also been a central issue in the inquiry by the Senate Economics References Committee into the future of Australia’s naval shipbuilding industry. The Committee’s report was tabled on 28 June 2018.

130 If achieved, this will be three months ahead of the date announced by the Minister in May 2015.

131 [Defence footnote to its remarks:] ‘Core productivity of the AWD shipyards is defined as the best productivity that the shipyards could achieve, given a mature design depends on a number of factors, including the levels of technology and experience available in the yards. Core productivity for the project was assessed at 60–65 mhrs/CGT.’

132 Audit note: This statement by the AWD Project Office that shipyard productivity has improved needs to be read in conjunction with paragraph 29, above.

133 Defence advice to the audit, 19 July 2018.

134 The other two platforms were the Tiger Armed Reconnaissance Helicopter and the Heavy Lift Utility, Chinook.

135 Deed 1 included: the development of a remediation plan to rectify technical non-conformances by addressing engineering issues, Australian Aerospace agreeing to provide, at its own expense, specific modifications to address technical issues; defining guiding principles to address reliability and supportability issues for the aircraft and the development of a performance based contract; and listing a number of commercial matters which would be referred to a third party for adjudication. The guiding principles determined as part of Deed 1 would form the basis of negotiations for Deed 2.

136 The outcomes of the negotiation of Deed 2 included Defence being granted additional access to intellectual property; strengthened linkages between the acquisition and sustainment contracts; changes to the MRH90 acceptance process that did not compromise Defence’s ability to reject significant deficiencies; the extension of Defence’s access to liquidated damages; and re-baselining the acquisition contract in return for a range of improvements to the aircraft.

137 Operational Capability Maritime 2 is a requirement for two helicopters to be embarked concurrently, up to 90 days duration, in a medium threat environment. This threat level requires a fitted machine gun and ballistic protection in the aircraft. Operational Capability Maritime 2 is also contingent on the delivery and service release of an Enhanced Cargo Hook System and the delivery of a contractually compliant Electronic Warfare Self Protection system.

138 Defence advice of October 2018.

139 CA48 Multi Role Helicopter Weapons System is focused on the provision of through life support for the Multi Role Helicopter and associated training and support systems.

140 The six submarines and their delivery dates are: HMAS Collins delivered in 1996; HMAS Farncomb delivered in 1998; HMAS Waller delivered in 1999; HMAS Dechaineux delivered in 2001; HMAS Sheehan was delivered in 2001; and HMAS Rankin was delivered in 2003.

141 The Defence Budget Audit, delivered to Government in April 2009, was undertaken in parallel with the preparation of the 2009 Defence White Paper and associated companion reviews and the Mortimer Review of Defence procurement and sustainment. The audit examined the state of the Defence Budget and its major cost drivers, with a view to finding potential for efficiency gains and reinvestment opportunities. http://www.defence.gov.au/publications/docs/DefenceBudgetAudit.pdf [Accessed 24 August 2018].

142 The lack of available submarines was due to an incident on 19 June 2009 where HMAS Collins suffered the final of a series of unexpected serious propulsion defects; and at the same time HMAS Farncomb was rectifying hydraulic contamination and bow plane defects which were completed on 21 June 2009.

143 An integrated master schedule would document the planned submarine maintenance and availability periods, develop a stable baseline for Defence Industry to use in planning and organising submarine maintenance and operations, and take account of the need to provide critical to support Navy and its training and readiness requirements. Defence subsequently released a schedule in April 2010.

144 Defence defines a material ready day as a day when a submarine is not conducting planned maintenance and is not encumbered by defects that prevent it from proceeding to sea. Source: J.Coles, Study into the Business of Sustaining Australia’s Strategic Collins Class Submarine Capability, March 2014, p. 9. http://www.defence.gov.au/dmo/multimedia/coles_progress_review_march2014-9-5124.pdf [Accessed 24 August 2018]

145 J.Coles, Study into the Business of Sustaining Australia’s Strategic Collins Class Submarine Capability, March 2014. http://www.defence.gov.au/dmo/multimedia/coles_progress_review_march2014-9-5124.pdf [Accessed 24 August 2018]

146 J.Coles, Study into the Business of Sustaining Australia’s Strategic Collins Class Submarine Capability – Beyond Benchmark, May 2016, p. 5. http://www.defence.gov.au/casg/multimedia/coles%20beyond%20benchmark%20report.pdf [Accessed 29 August 2018]

147 In October 2018, the Australian Strategic Policy Institute released its report into the transition from the Collins Class submarine to the future submarine fleet. The report noted that there will be an extended overlap between the operation of the Collins Class and the future submarine fleet of around 20–25 years, and that Defence needs to ensure effective sustainment of the Collins Class during this period to remove the risk of a capability gap. Source: Australian Strategic Policy Institute, Thinking through submarine transition, October 2018. https://www.aspi.org.au/report/thinking-through-submarine-transition [Accessed 17 October 2018].