The audit objective was to examine Defence’s administration of Materiel Sustainment Agreements (MSAs) and the contribution made by MSAs to the effective sustainment of specialist military equipment.

Summary

Introduction

1. Materiel Sustainment Agreements (MSAs) have been used since 2005 as customer–supplier agreements formalising the relationship between the Department of Defence (Defence) and the Defence Materiel Organisation (DMO)1 for the sustainment of specialist military equipment.2 Through the agreements, the Defence Capability Manager3 undertakes to supply funding, and the DMO undertakes the sustainment of specific platforms (such as a ship or aircraft fleet), commodities (such as clothing or combat rations) and services (such as provision of maritime target ranges). Sustainment involves the provision of in-service support, including repair and maintenance, engineering, supplies, configuration management and disposal action. Effective sustainment of ships, aircraft and vehicles is necessary to maintain the preparedness of the Australian Defence Force (ADF) and enable the conduct of Defence operations.

2. In 2014–15, out of total funding of $35.781 billion, Defence budgeted $7.109 billion, or 20 per cent, for its Capability Sustainment Programme, with most of this funding transferred to the DMO.4 By way of context, the DMO’s sustainment expenditure has regularly exceeded its acquisition expenditure in the past decade.5

3. Sustainment activities are generally administered by the DMO’s Systems Program Offices (SPOs). The SPO serves as the single point of contact with industry, government entities, and other entities participating in the acquisition or sustainment of specialist military equipment. Generally, each major platform is managed by a single SPO, which may also manage the delivery of a commodity or a service.

Managing Defence’s sustainment function

4. The Australian Defence Organisation comprises the Department of Defence, the ADF and the DMO. The successful provision of Defence capability depends on the Defence Organisation as a whole collaborating effectively and the component parts meeting their respective functional responsibilities, including, for the DMO, acquisition and sustainment responsibilities. In September 2003, when the then Government approved the DMO’s transition to a prescribed agency under the Financial Management and Accountability Act 1997 (FMA Act)6, the intent was for the DMO to be more performance-oriented and business-like, so as to improve procurement and support practices, and establish a more transparent relationship between Defence and the DMO.7

5. The DMO’s February 2004 Business Model noted the primacy of Defence as the customer, with responsibility for setting requirements and determining priorities within agreed funding levels; and that the DMO would be funded and managed on the basis of agency agreements for the services it delivered. The agency agreements were to facilitate funding flows and delineate responsibilities and accountability between Defence and the DMO, and provide better visibility of the costs of procuring and sustaining Defence assets. The agreements were to be refined and improved over several annual cycles. The lowest tier of these agreements, for sustainment, would be the MSAs:

At the tactical level, the scope, price and time frame for specific services between Defence and DMO would be captured in simple agreements that describe the products and services flowing from DMO’s Outputs—projects, sustainment and policy advice and services. […] The products/services described in these agreements must be meaningful, manageable and measurable by both parties and facilitated by a principle of open books between the two with agreed underlying assumptions clearly stated and risk management measures built in.8

6. The implementation of these agreements led to MSAs consisting of two levels. The first level is the Heads of Agreement, which is an overarching document that covers a series of Product Schedules. The Heads of Agreement contain the high-level framework establishing the partnership between each Capability Manager and the DMO. The second level of each MSA is the Product Schedules. Each Product Schedule deals with the sustainment of a specific platform, commodity or service for the relevant Defence Service or Group. The Product Schedule defines: the supplies and services that will be provided by the DMO; the budget that is provided by the Capability Manager; and standards for matters such as responsiveness, availability levels, and maintenance timeframes.

7. Most of Defence’s $7.1 billion sustainment budget for 2014–15 is included in the DMO’s sustainment budget of $6.185 billion9, and of this, some $5.683 billion is to be expended through seven MSAs incorporating 116 Product Schedules. The main MSAs are those between the DMO and each of the three Services. For 2014–15, the Chief of Navy MSA includes 36 Product Schedules valued at $1.976 billion; the Chief of Air Force MSA comprises 28 Product Schedules valued at $1.976 billion; and the Chief of Army MSA includes 45 Product Schedules valued at $1.504 billion.10 Since 2012, the previous practice of complete annual revision of the MSAs has been replaced by partial revision as needed, particularly of the financial sections of Product Schedules.

Audit objective, criteria and scope

8. The audit objective was to examine Defence’s administration of Materiel Sustainment Agreements (MSAs) and the contribution made by MSAs to the effective sustainment of specialist military equipment.

9. Four high-level criteria were developed to assist in evaluating Defence’s administration of MSAs:

  • MSAs constitute an effective arrangement formalising the relationship between Defence Services/Groups and the DMO for sustainment activities.
  • MSA Product Schedules clearly identify costs, deliverables and appropriate Key Performance Indicators (KPIs).
  • Risks, issues and constraints to achieving effective sustainment are identified in MSA Product Schedules, and appropriate management strategies are in place.
  • Monitoring and reporting processes provide relevant and reliable information on sustainment activities.

10. The ANAO examined Defence’s MSA policy, procedures and systems, including reforms in these areas. The ANAO also examined Defence’s management of three Product Schedules. One Product Schedule was selected for each of the Navy, Army and Air Force:

  • Navy’s FFH ANZAC frigates (Product Schedule CN0211), as one of the first MSA Product Schedules to be revised after the Rizzo Report12;
  • Army’s Protected Mobility Vehicle (Bushmaster) fleet (Product Schedule CA04), which has been heavily used in Iraq and Afghanistan; and
  • Air Force’s AP-3C Orion fleet (Product Schedule CAF04), which was heavily involved in the search for Malaysian Airlines flight MH-370 in the Indian Ocean.

Overall conclusion

11. In 2014–15, Defence budgeted over $7.1 billion, or some 20 per cent of total Defence resourcing, for the sustainment of specialist military equipment operated by the ADF. The majority of Defence sustainment services are provided either directly or indirectly through the Defence Materiel Organisation (DMO), which applies some 50 per cent of its budget ($6.185 billion13) to sustainment. Defence has been using Materiel Sustainment Agreements (MSAs) since 2005 to formalise its requirements for sustainment services from the DMO, with the aim of facilitating effective and business-like relationships within the Defence Organisation. MSAs are contract-like arrangements that set out the level of performance and support required by Defence from the DMO, within an agreed price, as well as the Key Performance Indicators (KPIs) by which service delivery will be measured.

12. Over the past decade, Defence and the DMO have established and continued to refine a generally sound MSA framework to facilitate the management of sustainment activity for specialist military equipment. The framework has enabled the Defence Organisation to clearly identify roles and responsibilities at a functional level, and individual MSAs document funding, deliverables, risks and performance measures for each sustainment product. Further, the development and maintenance of the MSA framework has encouraged and facilitated collaboration between Defence and the DMO at both the management and operational levels. The MSA framework has evolved over time, in light of practical experience and the risk appetite of the parties to individual agreements, and there is an ongoing role for Defence senior leadership to shape the direction of the framework so as to realise its full potential. More generally, there remains scope for Defence to enhance its sustainment management through the implementation, use and refinement of newly developed performance measures.

13. As discussed, the MSA framework has continued to evolve since 2005, with relatively bureaucratic processes being replaced over time by simpler arrangements. In particular, when the DMO commenced an MSA reform process in 2012, Defence stakeholders observed that the practice of annually reviewing the entire suite of MSAs was long and tedious, with too little emphasis on sustainment planning and performance management to deliver the best outcome with available funding. The reform process resulted in revised procedures for the management of MSAs, including a move towards developing more enduring MSAs and a simplified process for amending them. Under the new process, the Services and the DMO collaboratively review sustainment progress at both the management and operational levels, focusing on capability issues and required funding changes.

14. A key strength of the MSA framework is the capacity for Capability Managers to adjust individual Product Schedules in light of assessed risks. Following Navy’s inability to supply vessels requested by the then Government to assist in the clean-up after Cyclone Yasi in 2011, Navy demonstrated a relatively conservative risk appetite which resulted in more detailed Product Schedules and a requirement to approve changes at higher management levels. While the preferred approach is a matter for Navy’s senior leadership, there is scope to review future settings, in light of delays experienced in processing changes to Product Schedules and the emergence of undocumented workarounds to manage those delays.

15. A key objective when introducing the MSA framework was to capture the scope, price and timeframe for the provision of specific services, and individual Product Schedules do so. Defence has also taken steps to enhance its sustainment performance management through the development of standardised suites of performance measures. Between 2012 and 2014, the DMO, Navy and Army developed new performance measurement frameworks, including measures of availability, cost, schedule, and materiel deficiencies, which are to be reported through a new DMO system. These performance measures will not be fully implemented until the DMO system is operational, and at the time of the audit, the first phase of the system rollout was scheduled for May 2015. While the new performance measures should provide a firmer basis for the evaluation and active management of sustainment performance and costs, their establishment remains at an early stage.

16. The delivery of ADF capability relies in large measure on effective collaboration between key elements of the Defence Organisation—including the three Services and the materiel sustainment arm. The MSA initiative introduced a structured framework for engagement on sustainment matters; an approach of continued benefit irrespective of specific organisational arrangements within Defence.14 The practical effectiveness of the MSA framework largely depends on active and timely management of identified risks by Capability Managers and the DMO, and in that respect, a robust MSA framework is an aid to management, not an end in itself. To build on the progress made to date through MSAs, the ANAO has made two recommendations focusing on: the review of change management processes for Navy Product Schedules and their level of detail, to support more flexible management of MSAs and avoid undocumented workarounds in their administration; and clarifying the internal treatment of acquisition and sustainment funding.

Key findings by chapter

Materiel Sustainment Agreements (Chapter 2)

17. In 2012, Defence stakeholders agreed that the MSA process was bureaucratic and inflexible. There was an overriding focus on the annual development and approval of MSAs, rather than on using the MSA framework to actively plan and manage sustainment funding and activities. The DMO led an MSA reform project and reached agreement with Defence Services and Groups on streamlined arrangements. There was a particular focus on treating MSAs as enduring documents rather than updating them annually, and delegating authority for updating sections of Product Schedules to line management. The November 2012 DMO Standard Procedure on MSAs formalised the new arrangements, and MSAs were generally revised in accordance with the Standard Procedure in a timely manner. Service Chiefs informed the ANAO that the MSA reforms helped strengthen collaboration between Defence and the DMO in sustaining specialist military equipment.

18. The ANAO reviewed the structure and content of a sample of MSAs as at 2014, based on criteria developed by the ANAO in 2010 for effective cross-entity agreements.15 For the most part, the MSAs examined by the ANAO met the key characteristics of well-structured cross-entity agreements. The MSAs outline governance arrangements, respective roles and responsibilities, sustainment deliverables, performance reporting and monitoring arrangements, sustainment issues and risks, and dispute resolution procedures. These features of the MSAs serve to clarify accountabilities, coordination arrangements and relevant processes. However, there remains scope to improve KPIs—well-structured cross-entity agreements will generally include reciprocal KPIs which recognise that one entity’s ability to perform work often depends on timely action by the partner entity16, whereas MSA Product Schedules tend to include performance measures related to only one party—the DMO. More generally, the DMO, Navy and Army have developed new sustainment performance measurement frameworks which are yet to be fully implemented.

Materiel Sustainment Agreements in Operation (Chapter 3)

19. Regular interaction between Defence and DMO personnel at both the management and operational levels is essential to developing a shared understanding of expectations and reaching agreement on how to effectively manage sustainment activity. During 2012–13, the Services introduced revised arrangements for Defence–DMO management review of sustainment, incorporating periodic strategic-level reviews, six-monthly multi-product reviews, and ongoing scrutiny at the working level. For the ANZAC ship, Orion aircraft and Bushmaster vehicle fleets, product and working-level reviews were effective in focusing the attention of management on capability planning and changes, and related changes in funding.

20. MSA Product Schedules are updated as necessary to formalise changes in sustainment arrangements and funding agreed between Defence and the DMO. Army and Air Force followed the DMO Standard Procedure in delegating authority for updating Product Schedule sections to line management, and their implementation of the revised process, discussed earlier, has been relatively smooth. Following Navy’s inability to supply vessels requested by the then Government to assist in the clean-up after Cyclone Yasi in 201117, Navy required Product Schedule updates to be approved at higher management levels, and included additional financial and maintenance detail in the documents. Navy’s management of MSAs demonstrated a relatively conservative risk appetite, reflecting its assessment of risk, and resulted in a higher number of Product Schedule changes and delays in approving them. An unintended consequence of Navy’s approach was the emergence of undocumented workarounds to overcome delays in processing Product Schedule changes. While a matter for Navy’s senior leadership, there is scope to review the change management process for Navy Product Schedules and their level of detail.

21. The MSA framework recognises the importance of risk management in sustaining specialist military equipment, and provides a structured process for the identification, assessment and management of sustainment risks by the Services and the DMO. The ANAO’s examination of Defence’s management of the ANZAC, Orion and Bushmaster fleets indicated that senior leadership was kept up-to-date about the risks to the relevant capabilities, and that risk mitigation strategies were generally in place. However, the practical effectiveness of the MSA framework largely depends on active and timely management of identified risks by Capability Managers and the DMO, and in that respect, a robust MSA framework is an aid to management, not an end in itself.18

Sustainment Funding and Cost Estimates (Chapter 4)

22. The DMO’s acquisition and sustainment activities are presented as separate programs in the Portfolio Budget Statements (PBS), and the 2006 Defence–DMO Memorandum of Arrangements documents certain constraints on the transfer of funds within the DMO between acquisition and sustainment activities. However, while the PBS suggests that a relatively clear-cut distinction exists between acquisition and sustainment activities and funding, that distinction is not as clear-cut in the Memorandum of Arrangements, and experience indicates that the distinction is not hard and fast in practice. While acknowledging that there can be a period of transition between the acquisition and sustainment phases of a capability, the ANAO noted a number of instances of overlap in the use of acquisition and sustainment funding.19 To clarify the internal treatment of acquisition and sustainment funding, Defence should review relevant business rules and guidance.

23. From the establishment of the DMO as a prescribed agency20 in 2005, control of sustainment funding rested with the CEO DMO, who was able to move funding between sustainment products as needs and priorities changed. Some years ago, Capability Managers were given renewed responsibility for controlling their sustainment budgets. Under this arrangement, transfers of funding between sustainment products can occur with the agreement of the relevant Capability Manager. The number of funding transfers increased from seven in 2011–12 to 55 in 2013–14, and the total value of these transfers across financial years increased from $170 million to $1.1 billion. Capability Managers informed the ANAO that they valued the ability to flexibly use sustainment resources according to operational and maintenance needs.

24. The three case studies examined by the ANAO indicated that there have been persistent inaccuracies in Defence’s sustainment cost estimates. Moreover, actual expenditure in 2013–14 for one third of all sustainment products (39 out of 118) varied from the budget estimate by over 25 per cent. Some departure from budget estimates can be expected due to flexible use of funding between sustainment products, and unforeseen factors such as the need to delay or bring forward maintenance work due to operational demands. However, variances of over 25 per cent are significant and suggest that there remains scope for the DMO to strengthen cost estimation techniques and understanding of cost variances. Improved cost estimation would strengthen the capacity of Defence’s Capability Managers to flexibly manage sustainment funding as informed purchasers of sustainment services.

Performance and Reporting (Chapter 5)

25. The 2008 Defence Procurement and Sustainment Review (the Mortimer Review) considered that sustainment performance would not improve unless it was measured, and reported that Defence did not have appropriate, quantifiable KPIs.21 In 2012, the DMO developed a sustainment performance measurement framework and commenced work on a new reporting system. In 2013 and 2014 Navy and Army developed revised sustainment performance measures which will be reported through the DMO reporting system. These include a suite of Navy KPIs and Key Health Indicators that assess sustainment performance and the state of each capability.22 However, the first stage of the rollout of the DMO’s new reporting system has been delayed from November 2014 until May 2015. It is only when the new measures have been reported for some time that their usefulness will be tested, and any need for refinement can be assessed.

26. Defence’s annual reporting on sustainment includes budget and expenditure data for the Top 30 sustainment products (representing some 77 per cent of current spending on sustainment), as well as an overview of the management of these products. While providing stakeholders with a basic summary of sustainment costs and activity, this information does not facilitate assessment of Defence and the DMO’s sustainment performance in terms of materiel availability, cost-effectiveness and key inputs such as inventory management, maintenance and configuration changes. Defence still has some way to go before it meets the intent of the recommendation of the Joint Standing Committee on Foreign Affairs, Defence and Trade for enhanced public reporting. Following a request by the Joint Committee of Public Accounts and Audit (JCPAA), in February 2015 the ANAO provided the JCPAA with options, developed in consultation with Defence, to enhance sustainment reporting to the Government and Parliament. The issue remains under consideration by the JCPAA at the time of preparation of this report.

Summary of entity response

27. Defence’s covering letter in response to the proposed audit report is reproduced at Appendix 1. Defence’s summary response to the proposed audit report is set out below:

Defence thanks the Australian National Audit Office (ANAO) for conducting the performance audit: Materiel Sustainment Agreements. The audit was conducted and completed in a positive and collegiate manner, with the ANAO and Defence staff working together to analyse performance of the selected Materiel Sustainment Agreements.

Defence is committed to the review of procedures around Materiel Sustainment Agreements and the internal treatment of acquisition and sustainment funding as noted in the recommendations. After the outcomes are known from the First Principles Review, Defence will be better positioned to meet the intent of, and implement, the recommendations from the report.

Recommendations

Recommendation No.1

Para 3.64

The ANAO recommends that Navy and the DMO review change management processes for Navy Product Schedules, and the level of detail in the Schedules, to support more flexible management of the Navy Materiel Sustainment Agreement.

Defence response: Agreed.

Recommendation No.2

Para 4.14

To clarify the internal treatment of acquisition and sustainment funding, the ANAO recommends that Defence review relevant business rules and guidance.

Defence response: Agreed.

1. Introduction

This chapter introduces the concept of sustainment and Materiel Sustainment Agreements. It also sets out the audit’s objective and scope.

Background

1.1 Materiel Sustainment Agreements (MSAs) have been used since 2005 as customer–supplier agreements formalising the relationship between the Department of Defence (Defence) and the Defence Materiel Organisation (DMO)23 for the sustainment of specialist military equipment.24 Through the agreements, the Defence Capability Manager25 undertakes to supply funding, and the DMO undertakes the sustainment of specific platforms (such as a ship or aircraft fleet), commodities (such as clothing or combat rations) and services (such as provision of maritime target ranges). Sustainment involves the provision of in-service support, including repair and maintenance, engineering, supplies, configuration management and disposal action. Effective sustainment of ships, aircraft and vehicles is necessary to maintain the preparedness of the Australian Defence Force (ADF) and enable the conduct of Defence operations.

1.2 In 2014–15, out of total funding of $35.781 billion, Defence budgeted $7.109 billion, or 20 per cent, for its Capability Sustainment Programme, with most of this funding transferred to the DMO, which either directly or indirectly provides the majority of Defence sustainment services.26 The specialist military equipment being sustained was valued at $41.270 billion in 2013–14.27

1.3 The DMO budget for 2014–15 comprises two major elements: $5.931 billion for acquisition projects, and $6.185 billion for its sustainment program. Figure 1.1 shows the DMO’s acquisition and sustainment expenditure between 2005–06 and 2013–14, as well as budgeted amounts for 2014–15 and the forward estimates. In five of the last nine financial years, sustainment expenditure exceeded acquisition expenditure, and budgeted sustainment expenditure for 2014–15 is at a record high level.

Figure 1.1: DMO acquisition and sustainment expenditure, 2005–14, and budgeted amounts for 2014–18

Source: ANAO analysis of expenditure and forward estimates for DMO Programme 1.1, Management of Capability Acquisition, and DMO Programme 1.2, Management of Capability Sustainment, in Defence Annual Reports and Portfolio Additional Estimates Statements 2014–15.

Note: The hollow columns represent the February 2015 Additional Estimates figures for 2014–15 and future years.

1.4 Sustainment activities are generally administered by the DMO’s Systems Program Offices (SPOs) and in particular through the Product Manager. The SPO serves as the single point of contact with industry, government entities, and other entities participating in the acquisition or sustainment of specialist military equipment. Generally, each major platform—such as an aircraft type or a class of ship (for example, F/A-18 Hornet aircraft, or the ANZAC Class frigates)—is managed by a single SPO, which may also manage the delivery of a commodity or service. On the customer side, a Capability Manager Representative liaises with the DMO on behalf of the relevant Defence Group or Service.

Managing Defence’s sustainment function

1.5 The Australian Defence Organisation comprises the Department of Defence, the ADF and the DMO. The successful provision of Defence capability depends on the Defence Organisation as a whole collaborating effectively and the component parts meeting their respective functional responsibilities, including, for the DMO, acquisition and sustainment responsibilities.

1.6 The then Minister for Defence approved the establishment of the DMO on 22 June 2000, through the amalgamation of the Department of Defence’s Defence Acquisition Organisation, Support Command Australia and part of its National Support Division. The objective of the amalgamation was to improve the delivery of equipment, systems and related goods and services to the ADF by integrating acquisition and through-life support into a whole-of-life capability management system. The DMO came into being on 1 July 2000, and related structural and organisational changes were made by December 2000.28

1.7 In September 2003, when the then Government approved the DMO’s transition to a prescribed agency under the Financial Management and Accountability Act 1997 (FMA Act), the intent was for the DMO to be more performance-oriented and business-like, so as to improve procurement and support practices, and establish a more transparent relationship between Defence and the DMO.29 The DMO became a prescribed agency under the FMA Act on 1 July 2005, with a separate Chief Executive Officer, financial accounts and annual reporting requirements, but staff provided by the wider Defence Organisation.30

1.8 The DMO’s February 2004 Business Model noted the primacy of Defence as the customer, with responsibility for setting requirements and determining priorities within agreed funding levels; and that the DMO would be funded and managed on the basis of agency agreements for the services it delivered. The agency agreements were to facilitate funding flows and delineate responsibilities and accountability between Defence and the DMO, and provide better visibility of the costs of procuring and sustaining Defence assets. The agreements were to be refined and improved over several annual cycles.

1.9 Agreements between Defence and the DMO are not legally binding, because both organisations are part of the same legal entity, the Commonwealth of Australia.31 Defence and the DMO have an overarching Memorandum of Arrangements for customer–supplier agreements that establishes the framework for their other bilateral agreements. The Memorandum of Arrangements outlines commitments that both parties agree to meet, in order to procure and sustain materiel for Defence.

1.10 Underneath the Memorandum of Arrangements, Defence and the DMO have entered into contract-like agreements for the provision of sustainment services:

Materiel Sustainment Agreements are between the Capability Managers and the Chief Executive Officer of the Defence Materiel Organisation. These agreements cover the sustainment of current capability, including goods and services such as repairs, maintenance, fuel and explosive ordnance.32

1.11 Each Materiel Sustainment Agreement (MSA) sets out the level of performance and support required by Defence from the DMO, within an agreed price, as well as the Key Performance Indicators (KPIs) by which service delivery will be measured. MSAs comprise a Heads of Agreement and Product Schedules for different platforms, commodities and services. Since 2012, the previous practice of complete annual revision of the MSAs has been replaced by partial revision as needed, particularly of the financial sections of Product Schedules. The content of the Heads of Agreement and Product Schedules is examined in Chapter 2.

1.12 As previously mentioned, the 2014–15 budget for DMO Programme 1.2, Management of Capability Sustainment, amounts to $6.185 billion33, of which some $5.683 billion is to be expended through seven MSAs incorporating 116 Product Schedules (Table 1.1).

Table 1.1: Materiel Sustainment Agreements and Product Schedules, as at February 2015

Capability Manager

Number of Product Schedules

Value of Services ($million)

Chief of Navy

36

1975.720

Chief of Air Force

28

1975.889

Chief of Army

45

1503.562

Chief Information Officer

4

76.728

Joint Health Command

1

45.941

Strategy Executive

1

20.908

Joint Operations Command

1

5.789

Not assigned to products

 

78.416

Total

116

5683.953

Source: DMO, MSA Product Budgets as at February 2015.

Note: Includes baseline funding, operations funding ($315 million) and expected Net Personnel and Operating Costs ($85 million).

Audit objective and scope

1.13 The audit objective was to examine Defence’s administration of Materiel Sustainment Agreements (MSAs) and the contribution made by MSAs to the effective sustainment of specialist military equipment.

1.14 Four high-level criteria were developed to assist in evaluating Defence’s administration of MSAs:

  • MSAs constitute an effective arrangement formalising the relationship between Defence Services/Groups and the DMO for sustainment activities.
  • MSA Product Schedules clearly identify costs, deliverables, and appropriate KPIs.
  • Risks, issues and constraints to achieving effective sustainment are identified in MSA Product Schedules, and appropriate management strategies are in place.
  • Monitoring and reporting processes provide relevant and reliable information on sustainment activities.

1.15 The ANAO examined Defence’s MSA policy, procedures and systems, including reforms in these areas. The ANAO also examined Defence’s management of three Product Schedules. One Product Schedule was selected for each of the Navy, Army and Air Force:

  • Navy’s FFH ANZAC frigates (Product Schedule CN0234), as one of the first MSA Product Schedules to be revised after the Rizzo Report35;
  • Army’s Protected Mobility Vehicle (Bushmaster) fleet (Product Schedule CA04), which has been heavily used in Iraq and Afghanistan; and
  • Air Force’s AP-3C Orion fleet (Product Schedule CAF04), which was heavily involved in the search for Malaysian Airlines flight MH370 in the Indian Ocean.

1.16 Assessing the overall effectiveness of the Product Schedules and their oversight involved engagement with: DMO staff responsible for the drafting and oversight of the Product Schedules; the project teams that carry out the sustainment of the selected military equipment; and staff assisting the relevant Capability Managers.

1.17 The 2010 ANAO performance audit on Effective Cross-Agency Agreements provided better-practice principles to assist in evaluating the MSAs, including discussion of consistency and clarity, guidelines, key provisions (such as achievable performance measures), and effective monitoring and review processes.36

1.18 The audit was conducted in accordance with the ANAO’s auditing standards, at a cost to the ANAO of approximately $493 000.

Structure of this Audit Report

1.19 The remainder of the Audit Report is arranged as follows:

Table 1.2: Structure of this Audit Report

Chapter

Overview

2.   Materiel Sustainment Agreements

Provides an overview of the Defence–DMO agency agreements framework. It then examines the structure and content of MSAs, and the reforms of the MSA framework that have been implemented in recent years.

3.   Materiel Sustainment Agreements in Operation

Examines management reviews of MSAs, the change management process for Product Schedules, and the management of sustainment issues and risks.

4.   Sustainment Funding and Cost Estimates

Examines Defence’s sustainment funding arrangements, and cost estimates for individual Product Schedules.

5.   Performance and Reporting

Examines internal and external sustainment performance reporting, including KPIs.

2. Materiel Sustainment Agreements

Provides an overview of the Defence–DMO agency agreements framework. It then examines the structure and content of MSAs, and the reforms of the MSA framework that have been implemented in recent years.

Introduction

2.1 Common drivers for formalising cross-entity arrangements include the need or desire to: promote a collaborative relationship between parties and demonstrate a commitment to joint work; establish a degree of control or assurance in relation to the activities or responsibility of another party; enhance accountability, transparency and efficiency; improve knowledge; and provide better services.37

2.2 In September 2008, the report of the Defence Procurement and Sustainment Review (the Mortimer Review) found that ‘the MSAs are a workable mechanism for Capability Managers and DMO to use’. The review also found that ‘the intent for DMO to become more business-like is not yet adequately reflected in a mature customer-supplier relationship between Defence and DMO’. The review concluded that the DMO needed to focus on being a business-like supplier of products and services rather than trying to accommodate all that was asked, and Capability Managers needed to become more informed customers.38

2.3 In this chapter, the ANAO examines:

  • the Defence–DMO agency agreements framework;
  • recent reforms of the MSA framework; and
  • the structure and content of MSAs.

Defence–DMO agency agreements framework

2.4 A map of the formal relationships between Defence and the DMO, as mediated through an agency agreements framework, is shown in Figure 2.1.

Figure 2.1: Defence–DMO agency agreements framework

Source: DMO, Agreements Manual (draft as at January 2013), Chapter 2, p. 5.

2.5 Under the framework, an overarching Memorandum of Arrangements provides for five types of agency agreement between Defence and the DMO:

  • Materiel Acquisition Agreements (MAAs);
  • Materiel Sustainment Agreements (MSAs);
  • Defence Services Agreements;
  • Shared Services Agreements, covering services such as payroll, accommodation and banking services provided by Defence, and contracting policy and advice provided by the DMO; and
  • Military Workforce Agreements, for posting of military personnel to the DMO.39

2.6 Defence and the DMO entered into the Memorandum of Arrangements for Customer–Supplier Agreements on 15 June 2005, and it was last revised on 30 June 2006.40 Under the Memorandum of Arrangements, in relation to sustainment:

  • Defence is responsible for setting clear performance requirements and priorities for products and services, and preparing government submissions for additional funding; and
  • the DMO is responsible for delivering the outputs agreed with Defence in the MSAs, controlling all resources, staffing and other inputs, providing appropriate evaluation and reporting, and setting policy instructions and governance arrangements for financial management of its funding and appropriations.

2.7 The Memorandum of Arrangements specifies that MSAs shall:

  • include individual schedules identifying the price and deliverables for each sustainment product;
  • separately identify baseline, supplemented and operations sustainment provided by the DMO to Defence41;
  • provide a budget forecast for the following 10 years; and
  • be renegotiated annually in conjunction with Defence’s financial planning process, known as the Defence Management and Financial Plan (DMFP).

2.8 In May 2009, in response to the Mortimer Review, the then Government formally committed to updating the Memorandum of Arrangements to clarify the respective authorities and responsibilities of Defence and the DMO.42 However, this has not yet occurred, and in respect of MSAs, the Memorandum of Arrangements has become increasingly obsolete. The outdated elements include:

  • the requirement for annual renegotiation of MSAs—this requirement has been overridden by a subordinate document, the 2012 DMO Standard Procedure on MSAs;
  • the discretion of the CEO DMO to move funds within the acquisition and sustainment areas—this discretion was restored to Defence Capability Managers some years ago;
  • a reference to the DMO Service Fee—which has not existed since
    2009–10, when the DMO began to receive its own appropriation for workforce and operating expenses; and
  • Product Schedules—the list of Product Schedules in the Memorandum of Arrangements is nine years out-of-date.43

2.9 The DMO has attempted to update the Memorandum of Arrangements on many occasions, and prepared an updated document for approval: between 2007 and 2009; in 2011 and 2012; and again in 2013. However, on each occasion, momentum was lost and the revised Memorandum was not formally approved, reflecting difficulty in reaching agreement across a large number of Defence stakeholders.

2.10 The long delay in updating the Memorandum of Arrangements—which is the capstone of Defence’s agency agreements framework—has resulted in a subordinate document, the DMO Standard Procedure on MSAs, now filling a policy gap. That document has increasingly been relied on as the Defence policy and procedure for sustainment activities.

2.11 While the Defence Organisation has shown a capacity to adapt, the DMO Standard Procedure is not mandatory policy for any of the Services, and the DMO relies on the Services’ ongoing commitment to the Standard Procedure to help bring coherence to sustainment arrangements. In contrast, an updated Memorandum of Arrangements would achieve an authoritative and ongoing basis for MSA policy and procedure across Defence Groups and Services.

2.12 Defence informed the ANAO in December 2014 that it was likely to experience organisational change flowing from internal and external reviews, including the First Principles Review44, and as a consequence it would be imprudent to update the Memorandum of Arrangements at this time. Defence further informed the ANAO that once the changes flowing from reviews are confirmed, Defence will shape an appropriate solution at the earliest possible opportunity.

Structure and content of Materiel Sustainment Agreements (MSAs)

2.13 As indicated in Figure 2.1, MSAs sit underneath the Memorandum of Arrangements in the Defence–DMO agency agreements framework. The first level of the MSA is the Heads of Agreement, which is an overarching document that covers a series of Product Schedules. The Heads of Agreement contain the high-level framework establishing the partnership between each Capability Manager and the DMO. Heads of Agreement are developed following a common template and can be tailored to the requirements of each Capability Manager. The main features of the Heads of Agreement template are set out in the text-box on page 4.

The Heads of Agreement template includes the following features:

  • the parties to the agreement;
  • the purpose and structure of Product Schedules;
  • funding model arrangements (but not prices);
  • future amendment only as required, and a requirement for review each time an incumbent to the signatory positions changes;
  • a requirement for each Product Schedule to be reviewed periodically, with each review to be recorded in a table within the Product Schedule itself;
  • establishment of a formal Change Proposal process to maintain each Product Schedule’s accuracy, currency and fitness for purpose;
  • establishment of a list of delegations, nominating the Service and DMO positions that may amend separate parts of the Product Schedules;
  • a list of the performance monitoring and relationship management activities that must occur at regular intervals in relation to both the MSA as a whole and individual Product Schedules.

Source: DMO, Materiel Sustainment Agreement Heads of Agreement Template, November 2012.

2.14 The second level of each MSA is the Product Schedules, a list of which is attached to the Heads of Agreement. Each Product Schedule deals with the sustainment of a specific platform, commodity or service for the relevant Defence Service or Group. In February 2015, the seven Heads of Agreement incorporated 116 Product Schedules. The number of Product Schedules for each Capability Manager is shown in Table 1.1.45

2.15 Product Schedules are where the specific sustainment requirements for each capability are found. The Product Schedule defines: the supplies and services that will be provided by the DMO; the budget that is provided by the Capability Manager; and standards for matters such as responsiveness, availability levels, and maintenance timeframes.

2.16 In 2013, the Product Schedules represented a significant body of paperwork, amounting to some 2600 pages for Navy, 1150 pages for Army, and 850 pages for Air Force. Since they are among the top planning documents for the expenditure of over $6 billion46, it is important to maintain a balance between statements of principle and the level of detail in these documents.

2.17 The standard structure of the Product Schedules for all the Capability Managers (except Navy) is shown in Table 2.1.

Table 2.1: Structure of non-Navy Product Schedules

Module/Annex Title(a)

Description

A: Product Description

Provides details of the product being supported, e.g. fleet/inventory size, variants.

B: Capability Requirements and Performance Indicators

Details performance outcomes sought, and Key Performance Indicators (KPIs).

C: Finance

Presents a price for services in the current and forward years, forecast monthly expenditure for the current financial year, operations funding, a statement on unfunded sustainment activities, and Strategic Reform Program (SRP)(b) savings targets.

D: Functions, Roles and Responsibilities

Sets out the functions, roles and responsibilities of the SPO, Lead Capability Manager, Supported Capability Manager, the End User, and those that are shared.

E: Issues, Risks and Constraints

Details issues affecting the Product, risks that may arise, and constraints that may limit effective sustainment.

F: Product Schedule Endorsement Delegations

Sets out the levels of delegation for amending the Product Schedule.

G: Reform and Continuous Improvement (optional)

When used, details reforms that will be undertaken and who will have responsibility for them.

H: Inter-dependent Product Schedules (optional)

When used, lists the associated Product Schedules that support the Product.

Source: DMO, Management of Materiel Sustainment Agreements including Product Schedules Standard Procedure, 6 November 2012, p. 4.

  1. Army uses the term Module, and Air Force uses the term Annex, for the different parts of the Product Schedule.
  2. On 2 May 2009 the then Government launched both the 2009 Defence White Paper and the SRP. Defence expected the SRP to improve accountability, planning and productivity and deliver savings of $20 billion over the decade 2009–10 to 2018–19, including $5.1 billion in sustainment savings through Smart Sustainment reform.

2.18 Navy Product Schedules are structured differently and are considerably longer than those of Army and Air Force. Their structure is shown in Table 2.2.

Table 2.2: Structure of Navy Product Schedules

Section Title

Description

1: Requirement

Outlines the nature of the capability, and Government and Navy requirements. Provides a list of the items (such as ships or facilities) that are being sustained.

2: Exceptions

Provides a framework for dealing with limitations on the DMO’s ability to deliver the required capability, and other circumstances that affect delivery of the capability.

3: Statement of Work

Contains a list of matters covered by the statement of work (Annex F).

4: Performance and Reporting

Details the theory behind KPIs and Key Health Indicators (KHIs), and the reporting framework to be used.

5: Constraints on Supply Variation

Outlines factors that influence the DMO’s ability to vary performance in response to adjustments sought by Navy.

6: Delegations and Authorities

Outlines the responsibilities of the lead Navy and DMO personnel (the Capability Manager Representative and the SPO Director), and the levels of delegation for amending the Product Schedule, including financial values at which delegations may be exercised.

7: Financial

Presents a whole-of-life cost plan, approved sustainment funding and forecast monthly expenditure for the current financial year (all with detailed breakdowns into line-items). Also includes a statement on unfunded sustainment, and tied funding support to operations.

Annexes

Description

A: Product Operating Profile

Details the purpose and various aspects of the use of the product.

B: Product Activity Plan

Presents planned Materiel Ready Days and scheduled maintenance periods for the years ahead.

C: Fleet Support Unit Capability and Capacity

Details the requirement for the DMO to offer maintenance work to the Navy’s Fleet Support Units in order to build up Navy capability.

D: Approved Capability Improvement, Sustainment and Retirement Initiatives

Lists the capital equipment acquisition projects associated with improving the capability, engineering changes addressing safety and/or supportability issues, and projects associated with capability retirement.

E: Accepted Materiel Capability Limitations and Risks

Lists the issues and limitations that Navy and the DMO recognise, and the risks that the SPO has identified, as having the potential to limit successful achievement of outcomes.

F: Statement of Work

States in detail the obligations upon the DMO and Navy in delivering the Product.

G: Key Performance and Health Indicators

Presents the KPIs and KHIs.

H: Product Schedules Supporting the Product

Lists the associated Product Schedules that support the product.

I: Transition

Provides detail of actions that are in progress at the time of drafting the Product Schedule, or are yet to be undertaken.

J: Acronyms and Glossary

Not always used.

Source: ANAO analysis of the ANZAC fleet Product Schedule as at July 2014, which was one of the models for subsequent Navy Product Schedules.

2.19 The nature of the products being sustained under the MSA framework is such that there is not necessarily a one-to-one relationship between a product and a Defence Service or Group, and a product may have relationships with other products. For example, the Bushmaster fleet, primarily ‘belonging’ to Army, has a small component of vehicles that are used by Air Force. Similarly, while the Bushmaster fleet is primarily sustained through Army’s CA04 Product Schedule, it could not operate without the fuel and lubricants supplied under another Product Schedule.47

2.20 In 2011, the three Services signed a Memorandum of Understanding (MoU) to provide clear guidance on Capability Manager roles and responsibilities where there are multiple users of a product. In effect, the Capability Manager who is the major user of a capability takes on the Lead Capability Manager role for the Product Schedule, and coordinates other Service or Group requirements.48

ANAO assessment of MSA quality

2.21 Key success factors for cross-entity arrangements include: clear roles, responsibilities and governance arrangements; a shared objective or outcome; clear funding arrangements; management of shared risks; and coordinated reporting and evaluation, with a clear focus on the shared objective as well as entity contributions.49

2.22 Table 2.3 considers the extent to which the three Services’ MSAs (comprising both the Heads of Agreement and Product Schedules) have regard to better practice in agreement-making.

Table 2.3: ANAO assessment of MSA quality

Characteristic

Navy

Army

Air Force

Clearly written – avoids legalistic language

Yes

Yes

Yes

Concise – only contains essential information

No

Yes

Yes

Appropriate overarching authority for agreement

No

No

No

Shared objective or outcome

Yes

No

No

Deliverables explicitly defined

Yes

Yes

Yes

Clear roles and responsibilities for both parties

Yes

Yes

Yes

Balanced performance indicators on both parties

No

No

No

Performance reporting and monitoring framework

Yes

Yes

Yes

Sufficient financial detail for informed oversight

Yes

Yes

Yes

Issues and risks documented

Yes

Yes

Yes

Bilateral governance and review arrangements

Yes

Yes

Yes

Dispute resolution procedures

Yes

Yes

Yes

Effective method of variation

No

Yes

Yes

Source: ANAO analysis of a random sample of six Product Schedules from each Service, based on criteria in ANAO Audit Report No.41 2009–10, Effective Cross-Agency Agreements, p. 54.

2.23 For the most part, the MSAs examined met the key characteristics of cross-entity agreements. The MSAs outline governance arrangements, respective roles and responsibilities, sustainment deliverables, performance reporting and monitoring arrangements, sustainment issues and risks, and dispute resolution procedures. These features of the MSAs serve to clarify accountabilities, coordination arrangements and relevant processes.

2.24 Some features of existing MSAs indicate scope for improvement in the MSAs. For example:

  • Each of the MSAs refers to the Memorandum of Arrangements as the overarching document that authorises the MSAs. However, as discussed in the early part of this chapter, the Memorandum of Arrangements is increasingly out-of-date, and in practice it has been bypassed by the DMO Standard Procedure on MSAs.
  • The establishment of shared objectives or outcomes as part of a cross-entity agreement assists in furthering individual entity outcomes, while focusing each entity on the overall intent and expected outcomes of the cross-entity initiative. The Navy MSA outlines its purpose and objectives, and supporting principles, whereas the other Services’ MSAs do not.50
  • While a bilateral agreement will generally include reciprocal KPIs which recognise that one entity’s ability to perform work often depends on timely action by the partner entity51, there is a tendency for MSA Product Schedules to include performance measures related to only one party—the DMO. More generally, the DMO, Navy and Army have developed new sustainment performance measurement frameworks which are yet to be fully implemented.
  • The Army and Air Force MSAs have workable change mechanisms that allow them to be kept reasonably current. Navy has experienced a greater administrative burden in establishing change mechanisms and keeping the documents current. This is a result of both the higher level of detail present in the Navy MSA (resulting in more frequent changes), and Navy’s requirement for changes to be approved at higher levels than required by Air Force and Army (resulting in slower changes).

Reform of the MSA framework

2.25 The MSA framework was introduced in 2005 and has continued to evolve. A 2012–13 Lean Project 52 represented a significant turning-point in the design and administration of the MSA framework.

2012–13 Lean Project

2.26 In February 2012, the DMO informed the Services that it had initiated an MSA Lean Project ‘to improve the MSA and MSA change proposal (MSACP) processes to make them straight-forward, easy to use, faster and more flexible.’ By June 2012, the DMO had conducted the initial, data-gathering phase of the Lean Project, and reported that:

Overall, stakeholders reported that the MSA process was long and tedious with little apparent value-add. The process currently has too much emphasis on the approval process and too little on planning and performance management. The focus needs to be on delivering outcomes which are achievable within the funding available in order to obtain the best outcome for the money available. The initial workshop findings included:

  • annual cycle to re-sign MSAs is not required and not supported;
  • annual cycle is not adding value to the content of the MSAs;
  • working level[53] stakeholders require more clarity of Capability Manager requirements earlier in the development process;
  • roles and responsibilities in the process are not clear;
  • ownership of the document is not clear;
  • current content and KPIs are not used for product management at any level;
  • loss of visibility of the MSAs after leaving the working level during the endorsement and signature process engenders lack of ownership of the final document; and
  • reporting on products is already a large burden.

2.27 In July 2012, the DMO organised five days of workshops to develop a streamlined MSA process with the intent to release capacity to focus on sustainment planning, management and performance. The DMO described the overall outcome to Capability Managers as follows:

We all agree that the current MSA process is too cumbersome, bureaucratic and inflexible. The good news is that the team conducting the Lean activity believe that it is possible to replace the annual cycle of MSA development stretched over months with a change process of less than five days. To achieve this level of reform will require us, as senior leaders, to consistently back this initiative at all levels of our organisations.54

2.28 The workshops were the culmination of several years of attempts to develop an effective and uniform MSA policy and practice. Representatives of Defence Services and Groups reached a consensus on key reform initiatives, including:

  • the Product Schedule being capability focused;
  • all sections of the MSA and Product Schedule being enduring, with built-in performance review periods;
  • a modularised Product Schedule template55;
  • introducing Product Schedule delegations to enable greater delegation aligned to line management accountability; and
  • simplified/accelerated workflows for changing/updating/introducing Product Schedule sections under the proposed delegation framework.

2.29 The broad implementation parameters from the workshop also included the need to redesign the MSA framework to focus on sustainment outcomes and information required by the O5/O6 level, and to make the Product Schedules more concise and focused on the changeable aspects under management.

The DMO issued a Standard Procedure on MSAs in 2012

2.30 In November 2012, following on from the Lean Project workshops, the DMO issued the Defence Materiel Standard Procedure entitled Management of Materiel Sustainment Agreements including Product Schedules Standard Procedure.56 The Standard Procedure addressed the key deliverables of the July 2012 workshops to develop a streamlined MSA framework. It sets out, essentially in four pages, the elements of the new approach to the MSA framework, and includes process flows for implementing scheduled financial decisions as well as other decisions.

2.31 The DMO asked Capability Managers to endorse the Standard Procedure as the ‘interim Defence and DMO policy’ for the management of MSAs, given the wide consultation involved in its development. This was seen as an expedient way of achieving Defence-wide commitment to the proposed MSA arrangements. Capability Managers endorsed the Standard Procedure before it was formally issued. The Standard Procedure included some exceptions from its requirements for Navy.

2.32 As a DMO policy, the Standard Procedure has no authority to direct the Services or other Defence Groups. It therefore explicitly states that it has been created ‘as an interim policy until a Defence Instruction (General) is considered by key stakeholders and released in early 2013.’57 However, in June 2013, Defence’s System of Defence Instructions (SODI) administrators advised the DMO that the creation of a Defence Instruction (General) for MSA policy was not appropriate because MSAs are non-legally binding, and they do not pertain to everyone in Defence.

2.33 By August 2014, it was agreed that the Standard Procedure would be updated, and Capability Manager Representatives would continue to acknowledge the Standard Procedure as the agreed MSA protocol between the DMO, Defence Groups and the Services. Further, the DMO would investigate the inclusion of an MSA chapter in a sustainment manual being written by its Standardisation Office.

Implementation of the new MSA framework

2.34 Work on implementing the new MSA framework continued until the formal closure of the Lean Project in April 2014. The biggest task involved the development of enduring Heads of Agreement and Product Schedules based on new templates. The revised documents and related governance and monitoring arrangements reflect an intention to eliminate rework and excessive review, and focus efforts on the higher-value areas of sustainment planning, management and performance. The transition of different MSAs and Product Schedules to the new model is shown in Table 2.4.

Table 2.4: Transition of MSAs to the new model, 2013–14

MSA

New Heads of Agreement signed

Last of new Product Schedules signed

Air Force

5 February 2013

19 August 2013

Army

8 July 2013

20 September 2013

Chief Information Officer

24 June 2013

13 August 2013

Joint Health Command

25 October 2013

25 October 2013

Joint Operations Command

Not yet signed

Not yet signed

Strategy Executive

9 August 2013

9 August 2013

Source: ANAO analysis of Defence records.

Notes: Taking a different approach, Navy had signed a new enduring Heads of Agreement on 9 August 2012, and completed its Product Schedule transition by 30 June 2013. Navy signed a revised Heads of Agreement on 20 January 2015.

 In a further round of revision, the Chief Information Officer Group and Army signed new Heads of Agreement with the DMO on 27 August 2014 and 3 September 2014 respectively.

As a result of the Rizzo Report, Navy took a different approach

2.35 While Navy representatives were involved in the DMO’s Lean Project, a different imperative drove the review and reform of the Navy–DMO MSA from 2011 onwards. In February 2011, Navy was unable to supply vessels requested by the then Government to assist in the clean-up after Cyclone Yasi. This was quickly followed by the early decommissioning of HMAS Manoora, the extended unavailability of HMAS Kanimbla and the temporary unavailability of HMAS Tobruk. These events resulted in the commissioning of the Rizzo Report.58

2.36 In July 2011, the Rizzo Report found that the events mentioned above were ‘reflective of on‐going systemic failure’, and made several observations pertaining to the then Navy MSA and its Product Schedules, namely that:

  • the MSA was critical for accountability, but was ‘currently poorly defined and weak’;
  • the KPIs in the Product Schedules were inadequate, with no consequences for non-compliance; and
  • the MSA should be used by the DMO to clearly define the obligations of Navy.59

2.37 The Rizzo Report made two recommendations (out of 24) that directly related to MSAs:

Recommendation 11. Capture Mutual Obligations: The Navy MSA should be transformed into an active ‘contract’ that meaningfully captures the mutual obligations of Navy and DMO, supported by business-like performance measures.

Recommendation 12. More Effective Information Exchange: Navy and DMO must improve their internal reporting by capturing direct, timely and candid, document-based information that draws on a rigorous set of metrics.60

2.38 Navy took the Rizzo recommendation for a more contract-like MSA to mean one that contained: a customer–supplier arrangement; clearly defined responsibilities, obligations, performance measures and deliverables; delegations of authorities and responsibilities; and appropriate management and reporting arrangements.

2.39 The Chief of Navy signed a new MSA with the DMO on 9 August 2012.61 This was three months before the DMO issued its Standard Procedure and new templates, in November 2012. The new MSA included a restructure of the 36 Navy Product Schedules, which were rewritten in the post-Rizzo format and approved progressively by 30 June 2013 as part of the Rizzo Reform Program.

2.40 In response to Recommendations 11 and 12 of the Rizzo Report, the Navy–DMO Heads of Agreement includes a section on the core obligations of Navy and the DMO, and requires a traffic-light system for monthly reporting in relation to both availability and price. Further, Navy’s new Product Schedules include a Statement of Work and a detailed breakdown of funding into line-items. The funding line-items are intended to give Navy visibility of where funding, below product level, is being spent.

2.41 The Standard Procedure on MSAs only requires details of overall baseline funding in Product Schedules; whereas Navy included 10 funding line-items in its post-Rizzo Product Schedules.62 Navy’s line-items were also not aligned with Defence’s corporate budgeting system, BORIS. These costs could therefore only be adjusted and reported through manual manipulation of data in spreadsheets.

2.42 In December 2014, Defence informed the ANAO that Navy and the DMO had agreed on a level of financial detail—for inclusion in Product Schedules—that could be budgeted and reported through BORIS. However, implementation of the solution had not yet occurred.

2.43 Another aspect of the Navy Product Schedules that differs from the DMO template is the inclusion of additional availability and maintenance information. Navy has responded to the non-availability of ships and the Rizzo Report by adopting a detailed approach:

  • a Product Activity Plan details the number of Materiel Ready Days required for the next year, as well as expected dates of maintenance for the next three years; and
  • the schedule for external (contractor) maintenance of each ship (where relevant) is set out for the following five years rather than just for the current financial year, as was previously the case.63

2.44 The revised Navy MSA includes additional detail on mutual obligations, expenditure and maintenance scheduling, following significant failures in Navy which led to the unavailability of supply vessels. This approach reflects Navy’s relatively conservative risk appetite, and is intended to enable close oversight of the DMO’s management of sustainment activity. The design features of the Navy Product Schedules have led to very lengthy documents in comparison to other Services’ Product Schedules. For example, in 2013, the average Army Product Schedule was 28 pages, the average Air Force Product Schedule was 33 pages, and the average Navy Product Schedule was 70 pages.

Conclusion

2.45 For the most part, the MSAs examined by the ANAO met key characteristics of well-structured cross-entity agreements. The MSAs outline governance arrangements, respective roles and responsibilities, sustainment deliverables, performance reporting and monitoring arrangements, sustainment issues and risks, and dispute resolution procedures. These features of the MSAs serve to clarify accountabilities, coordination arrangements and relevant processes. The MSA framework has evolved over time, in light of practical experience and the risk appetite of the parties to individual agreements, and there is an ongoing role for Defence senior leadership to shape the direction of the framework so as to realise its full potential.

3. Materiel Sustainment Agreements in Operation

This chapter examines management reviews of MSAs, the change management process for Product Schedules, and the management of sustainment issues and risks.

Introduction

3.1 Working across organisational boundaries presents many challenges, including harmonising different strategies and business processes to achieve the intended outcomes for government. The DMO–Defence customer–supplier relationship requires effective collaboration at both senior executive and operational levels, supported by efficient processes for MSA oversight, change management, and issues and risk management.

3.2 In this chapter, the ANAO examines management reviews of MSAs, the change management process for Product Schedules, and the management of sustainment issues and risks. The ANAO’s analysis draws on the three selected MSA Product Schedule case studies: ANZAC ships, Bushmaster Protected Mobility Vehicles and Orion aircraft.

Management reviews of MSAs

3.3 Although a system of MSA management reviews existed prior to 2012, the reviews were sometimes intermittent. When the Heads of Agreement template was released with the Standard Procedure on MSAs in November 2012, it included an indicative MSA review system. This consisted of three tiers of review: Strategic MSA Reviews held at least every two years at CEO DMO level; twice-yearly Strategic Product Schedule Reviews at DMO Division Head level; and monthly Product Schedule Meetings at working level.

3.4 During 2012–13, in renewing their MSAs, each of the Services also renewed the number and type of regular MSA reviews. Table 3.1 shows the tiers of MSA review that have been established by each of the Services.

Table 3.1: Tiers of MSA review

Type

Navy

Army

Air Force

‘Strategic review’/ ‘Deep Dive’

Strategic Review – as required (3-Star)

Deputy Chief of Army Product Schedule Review – at least once every three years for each Product Schedule (2Star)

Strategic MSA Review – at least once every two years (3-Star)

‘Fleet Screening’ – every six months

Biannual MSA Review (2-Star)

Capability Manager’s Product Schedule Screen (2-Star)

Principals Meeting (2-Star)

Force Element Biannual Progress Reviews (1-Star); these reviews were previously quarterly

Capability Sustainment Review (1-Star)

Sustainment Assessment Review (1-Star)

Working Level

Operational Sustainment Management Meetings - monthly

Working Groups – as required

Product Schedule Performance Meetings - monthly

Source: ANAO analysis.

Periodic reviews—‘Strategic Review’/‘Deep Dive’

Navy

3.5 The 2012 Navy–DMO Heads of Agreement established an Annual Strategic Review between the Chief of Navy and the CEO DMO to consider current and future high-level management issues and the overall performance of Navy sustainment. These Reviews had been proposed as part of the implementation of Rizzo Recommendations 11 and 12.64 However, to date, no meetings have taken place, and in January 2015 the frequency was changed to as-required.

Army

3.6 Army’s top level of review consists of ‘deep dives’ into Product Schedules (rather than strategic reviews), and these are formally known as Deputy Chief of Army (DCA) Product Schedule Reviews. The frequency of these deep dives varies with the level of risk identified for individual Product Schedules. Those deemed to present the highest risk are reviewed annually, with two and three-yearly reviews for those Product Schedules deemed to present medium and low levels of risk, respectively. These deep dives involve two to four hours of consideration of a single Product Schedule, and are intended to confirm the DMO’s management of Army materiel against the fleet management plan, Capability Manager priorities, budget constraints and the Product Schedule. The Product Schedule Reviews are empowered to approve funding priorities and transfers, and give in-principle agreement to inter-Service funds transfer for sustainment support. They can also authorise strategies to address particular sustainment issues.

Air Force

3.7 The first Air Force Strategic MSA Review occurred in June 2011, at Air Force’s initiative, with a view to establishing an annual forum of DMO and Air Force senior executives to discuss the broader strategic issues affecting the materiel sustainment of Air Force capability. At the time, Air Force was keen to use the MSA construct as a partnership rather than a customer–supplier relationship, acknowledging that it was only in recent times that Air Force had started to be involved at the appropriate depth of detailed understanding and engagement in the MSA and Product Schedule processes. During the June 2011 review, Air Force noted that its relationship with DMO was very good and continuing to mature, yet there were still opportunities for improvements, such as introducing a greater capability focus into Product Schedules, capability-driven KPIs and increased consistency in Air Force dealings across all DMO Divisions and SPOs. For its part, the DMO highlighted that in the past the MSAs had been ‘one way’, and that there was merit in pursuing a move to ‘two way’ arrangements and KPIs.

3.8 The 2013 Air Force–DMO Heads of Agreement provided for a Strategic MSA Review to be conducted at least once every two years, ‘to review the conduct of the MSA and the materiel sustainment relationship between the Air Force and the DMO’. No Strategic MSA Review has been held since 2011.

Six-monthly reviews—‘Fleet Screenings’

Navy

3.9 Navy conducts two levels of fleet screenings: Biannual MSA Reviews; and Force Element Biannual Progress Reviews, which consider the Product Schedules by Force Element Group.

3.10 The Biannual MSA Reviews form the most significant part of Navy’s MSA review system. These reviews consider the performance of all of the Navy Product Schedules for each DMO Division, and provide an opportunity for the Deputy Chief of Navy to manage sustainment funding between different Product Schedules, taking into account changing circumstances and operational needs. The reviews occur in February/March and September/October each year, to inform the development of the Commonwealth Budget. The reviews have generally been two-day events, and were well described in a recent Navy administrative instruction:

Cognisant of the findings of the Rizzo Report, the focus of the reviews has evolved beyond their original financial emphasis into a forum for DCN [Deputy Chief of Navy] and the relevant DMO Division/Group Head to consider how their respective organisations are meeting their obligations for whole of life management and sustainment of Navy capability. Financial planning and performance remains a key element of the reviews, and the composite view of Navy sustainment pressures provided by the biannual review activity enables DCN to make informed capability and resource allocation decisions.

3.11 The ANAO attended the October 2014 Maritime Systems Division fleet screening as an observer. The meeting demonstrated an open and collaborative relationship between Navy and the DMO, with attention given to the key issues affecting both parties.

Army

3.12 Army instituted its current system of biannual Product Schedule Screens in July 2012. The meetings provide an opportunity for Army to examine the current financial health of specific fleets; plan for the forward estimates; examine fleet and business management issues; move funding between products and assign priorities; and identify products to undergo a Product Schedule Review. The Product Schedule Screen is intended to occur over three days, and is chaired by the Deputy Chief of Army, who can adjust KPIs, approve funding priorities, and authorise management action plans to remediate issues.

Air Force

3.13 Air Force has the most intricate system of fleet screenings. Air Force’s Standing Instruction on MSAs notes that there are two review periods within each financial year: for the Defence Management and Financial Plan (DMFP) and the Mid-Year Review. The DMFP review period, between February and June each year, involves a strategic review of all materiel sustainment requirements to support Air Force capability. During the Mid-Year Review period, between September and December each year, Air Force and the DMO analyse the in-year sustainment and performance of all Air Force Product Schedules, and agree on the sustainment plan for the following financial year.

3.14 In each of the review periods, the same sequence of reviews occurs, namely Sustainment Assessment Reviews, Capability Sustainment Reviews, and Principals Meetings.65 Sustainment Assessment Reviews are led by the relevant DMO SPO and chaired by the DMO Branch Head. These reviews examine sustainment requirements, risks, issues, and the cost of achieving Air Force’s capability requirements. The reviews develop Management Options related to these areas, to feed into Capability Sustainment Reviews conducted by Force Element Groups. The Capability Sustainment Reviews analyse funded and unfunded sustainment requirements, risks and issues, and select specific Management Options for higher-level approval.

3.15 After these two reviews, an Air Force Capability Sustainment Plan is developed, and is endorsed by the Air Command Board in the lead-up to the Principals Meeting.66 In this way, Air Force is able to provide a
whole-of-sustainment proposal to the DMO from a capability-based perspective.

3.16 The Principals Meetings form the top tier of the six-monthly reviews. The DMFP Principals Meeting is chaired by the Deputy Chief of Air Force, and the Mid-Year Review Principals Meeting is chaired by the Air Commander Australia. The meetings consider and approve the Capability Sustainment Plan, omnibus Product Schedule Change Proposals67 for financial adjustments, and Change Proposals for any non-financial adjustments to a Product Schedule.

Ongoing review—Working Level Meetings

Navy

3.17 Navy’s working level meetings are termed Operational Sustainment Management Meetings. These meetings occur monthly, and include a comprehensive overview of the current status of the relevant Navy Product. The chief participants are the SPO Director and the Capability Manager Representative.

Army

3.18 The Army–DMO Heads of Agreement provides for working groups to be formed to address identified issues and engage with stakeholders, with the findings to be referred to a suitable forum for decision.68

3.19 In 2013, the DMO’s Head Land Systems Division initiated a new divisional level of review, referred to as Project and Product Review Boards. These Review Boards are intended to provide an opportunity for project and sustainment managers in various branches of the Division to inform the Head Land Systems about issues, and for him to provide direct input where necessary.

3.20 In August 2014, the DMO’s Land Systems Division informed the ANAO that:

There is a very strong (and regular) interaction between Land Systems Division and the Capability Manager that has created a much better shared understanding as to where each Product is at. This, linked with the DCA [Deputy Chief of Army] Reviews and new Head Land Systems Product Reviews, allows the twice-yearly screens to adopt a more strategic overview approach that focuses on the ‘big issues’. The layered approach, with multiple reviews and regular interaction, has allowed the six-monthly Product Screens to focus (as they should) on strategic 2-star issues: everything else gets picked up before these screens.

Air Force

3.21 The Air Force–DMO Heads of Agreement provides for Product Schedule Performance Meetings, to be conducted at least monthly, and to focus on the current and forecast performance of individual Product Schedules. The meetings are attended by SPO Directors and the relevant Air Force unit commanders and their teams.

Gate Reviews

3.22 In a significant step in the management and review of MSAs, Defence commenced Sustainment Gate Reviews69 in early 2015 for Product Schedules in Materiel Sustainment Categories I–III.70 Sustainment products will be subject to a Sustainment Gate Review once every three to five years in line with a major event (such as a re-contracting).71

3.23 During 2014, five Prototype Sustainment Gate Reviews were held, including one for the Bushmaster fleet in October 2014. The DMO advised the Gate Review Board that:

A simplified approach to reporting against this Product Schedule has been adopted by both DMO and Army, and refinement of the Product Schedule to rationalise performance measures and to address unfunded sustainment activities over the Forward Estimates and DMFP periods appears appropriate.

3.24 The ANAO has previously found that the DMO’s Gate Reviews for acquisition projects have the potential to improve the management of its projects.72 The reviews can also bring a valuable independent perspective to consideration of key issues and risks through the use of external members of Gate Review Boards.

Conclusion—MSA reviews

3.25 Overall, Defence has established a structured process for the review of each Service’s MSA and their Product Schedules, incorporating periodic strategic review, six-monthly reviews and ongoing scrutiny at the working level. However, strategic-level reviews of the MSA framework and sustainment performance, involving senior leadership, have not occurred as intended. On the other hand, the introduction of DMO Gate Reviews from early 2015 has the potential to bring a valuable independent perspective to the consideration of sustainment issues and risks.

Examples of recent Product Schedule reviews

3.26 This section examines the operation of the tiers of management review for three case studies: ANZAC ships, Bushmaster Protected Mobility Vehicles, and Orion aircraft. The analysis sheds light on whether the reviews provide a structured approach supporting the management of key sustainment issues.

ANZAC fleet

3.27 The most recent six-monthly fleet screening of Navy’s major platforms occurred in October 2014. In relation to the ANZAC fleet, the DMO was urgently seeking $2 million for a ship maintenance window of opportunity that was only three weeks away. This funding was necessary to keep the vessel seaworthy until its Anti-Ship Missile Defence (ASMD) upgrade, which had been postponed. As an emerging maintenance issue that required additional funding from Navy, this issue could not be dealt with at SPO/Capability Manager Representative level, and consideration of it was therefore elevated to the fleet screening.

3.28 The DMO made several bids for additional funding for the ANZAC fleet at the October 2014 fleet screening, amounting to $27.25 million, for items such as inventory, communications, ASMD, work health and safety and workforce augmentation. Further, the briefing material for the fleet screening gave detailed coverage of planned sustainment activities, risk management activities, achievement against KPIs and budgeting for the ANZAC fleet. Discussion took place during the fleet screening about delayed acquisition projects and their impact on ANZAC fleet sustainment more generally, with the DMO advising Navy that delays in three specific capability acquisition projects had led to DMO bids for $70.2 million in extra funding to maintain in-service capabilities until the new capabilities entered service.73

3.29 The October 2014 fleet screening, a result of the MSA framework, in this case provided a useful vehicle for bringing capability and funding issues to the attention of senior Navy and DMO management and enabling discussion of them.

Figure 3.1: The eighth ANZAC frigate, HMAS

Source: Australian Defence Image Library.

3.30 In another type of review, nine Force Element Quarterly Reviews have been held since September 2011.74 In relation to the ANZAC fleet, the June 2014 review discussed the need for a realistic schedule for the ASMD upgrade. The review also noted that movement of maintenance schedules had led to commercial pressures in the past, but maintenance staffing was currently high as a result of the attention paid to scheduling.

3.31 At the working level, by mid-2014, 28 Operational Sustainment Management Meetings had occurred between the ANZAC SPO and the relevant Capability Manager Representative, namely the Navy’s Frigate Group Capability Manager. Each meeting received a detailed report on ANZAC fleet sustainment activities, including performance against KPIs, engineering certification, finance, Product Schedule Change Proposals, risk management, and workplace health and safety. The regular management meetings and detailed reporting on sustainment activities support an effective Navy–DMO relationship at the working level.

Bushmaster fleet

3.32 Product Schedule Reviews of the Bushmaster fleet have occurred since 2012 at two-yearly intervals.75 They provide a comprehensive overview of the status of the sustained fleet, including financial data, performance against the KPIs, and consideration of specific issues.

3.33 The most recent Product Schedule Review occurred in June 2014 and generated four action items. The action items considered: the number of additional capabilities being integrated into the Bushmasters; the extra sustainment funding that would be required due to an increased fleet size as a result of new acquisitions; the commencement of Mid-Life Review planning; and the introduction of equipment acquired for operational purposes into the Bushmaster baseline configuration.

3.34 Defence records show that Product Schedule Screens covering the Bushmaster fleet Product Schedule (among others) have occurred every six months since 2012, and have examined detailed financial performance; fleet management activities and capability upgrades; KPI performance; and the health of the sustainment system.

3.35 Further, an Integrated Project Team, with representatives from Army, the DMO and the principal sustainment contractor, meets monthly to discuss the management of the capability.

Orion fleet

3.36 The review process during the Mid-Year Review period in late 2013 provides a good overview of how Air Force and the DMO negotiated their sustainment plans for 2013–14 and 2014–15. In October 2013, the Sustainment Assessment Review noted that while Orion aircraft serviceability had not met the KPI target, capability had not been impacted and Air Force had met its tasking obligations. The increased Rate of Effort (flying hours) required by government for Operation Resolute76 was being met within the existing funding allocation, which meant that no increase in funding was required for the operation. The Sustainment Assessment Review identified a requirement for an equity injection into the Orion fleet Product Schedule to take account of the transfer of responsibility for multi-platform avionics from another Product Schedule.

3.37 The Capability Sustainment Review conducted a few weeks later by Air Force’s Surveillance and Response Group considered the Sustainment Assessment Reviews of six Product Schedules. It endorsed the injection of funds for avionics into the Orion fleet Product Schedule, as well as the return of $4.3 million from Orion engine sustainment to Air Force due to the retirement of the C-130H fleet (which used the same engine).

3.38 A Capability Sustainment Plan was then drawn up by Air Command Board, covering all of Air Force’s Product Schedules and effecting a $20 million cut in Air Force sustainment costs during 2013–14. The plan noted that core support functions for each product remained as planned and ‘opportunities to delay activities, or their payment, into FY 2014–15 are the chosen mechanism to achieve an acceptable financial risk level in FY 2013–14’. A two-hour Principals Meeting in December 2013, involving the Air Commander Australia, Deputy Chief of Air Force and Head Aerospace Systems Division, endorsed the Capability Sustainment Plan.

3.39 A significant change had taken place for the Orion fleet by the next review period, April–June 2014, with the approval of an accelerated drawdown plan for most of the Orion fleet (changed from 2020 to 2019), changes to the asset life of major ground systems77, and a requirement to transfer $316 million from the Orion fleet Product Schedule budget so as to support the new Poseidon fleet.78

3.40 The June 2014 Capability Sustainment Plan noted that Air Force was significantly better placed to meet needs in 2014–15 and 2015–16, but also stated that a balanced sustainment budget could not be achieved in the following years, with the net shortfall exceeding $50 million in 2016–17, $70 million in 2017–18 and $75 million in 2018–19. For the Orion fleet Product Schedule, the plan noted that savings from a new contract for engine sustainment would be used to fund an extension to the life of the last Orion aircraft in the fleet.

Conclusion—MSA reviews for case studies

3.41 For the ANZAC ship, Orion aircraft and Bushmaster vehicle fleets, product and working-level reviews were effective in focusing the attention of management on capability planning and changes, and related changes in funding.

Change management

3.42 Product Schedules are a key component of the MSA framework because they constitute the core plans for sustainment of particular platforms, commodities and services. This section of the audit report discusses arrangements for revising MSA Product Schedules, firstly in terms of delegations authorising particular office-holders to effect a revision, and secondly in terms of the Change Proposal process set up by the DMO Standard Procedure on MSAs.

Delegations

3.43 In 2008, the Mortimer Review found that the process for revising MSAs was unnecessarily time-consuming, requiring Division Head (2-Star) approval for any changes to the agreements. The Mortimer Review observed that:

such a high level of control is at odds with commercial practice. In the private sector, authority and day-to-day management responsibility tend to be much more closely aligned. This should become the norm in DMO.79

3.44 The Mortimer Review recommended that:

Systems Program Office Directors should be empowered through greater delegation to deliver the performance levels set in Materiel Sustainment Agreements and, where necessary, to negotiate changes with Defence.80

3.45 One of the objectives of the DMO’s 2012 Lean Project was to improve the MSA Change Proposal process through appropriate delegations for updating Product Schedules. As part of the Lean Project, a delegation framework and revised Change Proposal process were agreed by stakeholders and incorporated into the Standard Procedure on MSAs. The Standard Procedure aptly describes the goals and limitations of the delegation framework:

Core to the success of a manageable Product Schedule is the balance between control and the delegation of authority and responsibility. The objective of the delegation of authority is to provide flexibility, speed of response within the bounds of the risk appetite and level of visibility sought by senior management.

[…]

It is accepted that [Capability Managers] and the DMO may operate differing authorities and levels of delegations as suits their respective organisations but both levels of authority must be clear to both organisations and the responsiveness must be achieved.

3.46 In following the Standard Procedure on MSAs, Army and Air Force now have almost identical delegation settings for changes to Product Schedules. The 2013 Army and Air Force Heads of Agreement contained a table of delegations by rank81, and each Army and Air Force Product Schedule includes a similar table detailing the specific office holder who has the delegation for approving different types of change for that Product Schedule. The types of change being delegated are expressed in generic rather than dollar terms—such as establishing a new Product Schedule, prioritising unfunded items, or incorporating the agreed outcomes of higher-level reviews. The delegations range from the Senior Executive Service Band 2/2-Star level down to the SPO Director/Capability Manager Representative level, with material changes dealt with at the Band
1/1-Star level and above. Minor changes that update financial arrangements, roles and responsibilities, and risks/issues/constraints can be effected at the working level.

3.47 The delegations for amending Army and Air Force Product Schedules reflect the wider body of applicable Defence delegations, such as the expenditure delegation of up to $5 million to persons at SPO Director or Capability Manager Representative level, or the DMO’s $20 million procurement delegation to SPO Directors.

3.48 Navy, by contrast, requires changes to be approved at higher levels under the Navy–DMO MSA. This approach reflects Navy’s relatively conservative risk appetite following the events of 2011, when Navy ships were unavailable for a requested deployment, and the subsequent Rizzo Report. The biggest difference in Navy’s approach compared to Army and Air Force is in the area of financial changes. For the ANZAC fleet, the SPO Director and Capability Manager Representative have been delegated authority to approve budget changes of up to $1 million. Changes between $1 million and $5 million require 1-Star approval, and changes over $5 million require 2-Star approval. This stands in contrast to the more general delegation levels in Defence outlined in paragraph 3.47.

3.49 As a result of having a higher level of delegation and including much more financial detail (line-items) in the Product Schedule, Navy has a greater administrative burden in managing its Product Schedules. Further, Navy Product Schedules are being used to control expenditure at a much lower level than is the case with other Capability Managers.

3.50 Since early 2013, the ANZAC SPO has made two attempts to have the level of financial delegation for the SPO Director and the Capability Manager Representative raised to $3 million, and to clarify the delegation for maintenance schedule changes. In January 2014, an email from ANZAC SPO advised Navy that:

The financial delegations impose an administrative burden on 1-Stars and above for financial changes. Presently any value between $1m and $5m requires 1-Star approval and above $5m 2-Star approval. [ANZAC fleet] budget is $250m this FY. $1m represents 0.4% of the entire budget. In terms of risk management, clearly there is no appetite for risk, and administratively this imposes serious constraints on business. The SPO Director and Capability Manager [Representative] should have more flexibility in their delegations to better manage the business. […] The constraints placed on the management and administration of Product Schedules [are] making the entire process burdensome and not what Mr Rizzo intended.

3.51 In July 2014, a two-week maintenance schedule change was approved by the ANZAC SPO Director and Capability Manager Representative, even though they did not have a clear delegation to make the change.

3.52 In mid-2014, Navy reviewed its MSA delegation framework and proposed to authorise 1-Star Force Commanders to make Product Schedule changes that would not affect the capability output delivered to government, to a value of $5 million. A further review of delegations for changes that could affect the capability output delivered to government was initiated at the Navy’s October 2014 fleet screening, with a view to establishing a graduated delegation framework within the Product Schedule. The review of Navy’s MSA delegations framework is expected to be completed in 2015. Navy informed the ANAO that:

There is disagreement regarding the interpretation of the [ANZAC fleet] Section 6 delegations table and the correct level of delegate approval for changes such as [scheduled maintenance periods]. The [Capability Manager Representative and SPO Director] consider they are entitled to approve such changes on the basis that they are minor changes to the product activity program whereas [Navy Strategic Command] has taken the view that changes that impact the number of [Materiel Ready Days] to be delivered under the CN02 ‘contract’ are effectively changing a key performance measure between Navy and DMO and the changes therefore require higher delegate approval/acknowledgment. This is an important issue given [Materiel Ready Days] underpin Unit Availability Days (UADs) that are reported to Government. […] Navy has agreed to review current delegation arrangements in this area to ensure there is an appropriate balance between the need to respond to maintenance change requirements at the waterfront in a timely manner and the need to ensure appropriate oversight of key performance measures at the corporate level which underpin Navy’s obligations to Government.

3.53 Mortimer recommended that responsibility be devolved to the extent possible, and this has generally been implemented by Defence. However, Navy’s risk appetite is lower as a consequence of the non-availability of Navy ships in early 2011 and the subsequent Rizzo Report. There continues to be an active debate within Defence around the appropriate balance to be struck in the delegation structure to streamline administration while maintaining appropriate senior leadership oversight by Navy.

Product Schedule Change Proposals

3.54 Product Schedules require amendment whenever acquisition projects, maintenance issues or operational activities, for example, affect funding requirements or other elements of the Product Schedule. Before 2012, 40 manual steps were involved in reviewing a Product Schedule and implementing a Change Proposal. As a result, it took an average of 3–4 months for Change Proposals to be finalised, with the result that some decisions were not communicated to stakeholders and documents were lost.

3.55 A key goal of the DMO’s Lean Project in 2012 was to make the process for revising MSAs ‘straight-forward, easy to use, faster and more flexible’. One of the key deliverables of the Lean workshops was a consensus on ‘simplified/accelerated workflows for changing/updating/introducing Product Schedule sections under the proposed delegation framework’. The Standard Procedure on MSAs outlines the process and its benefits:

Changes to the [Product Schedule] may be proposed by any participant at any time and must use the agreed [Product Schedule] Change process and Product Schedule Change Proposal (PdS-CP) form … Navy [Product Schedule] changes are to be actioned using the PdS-CP. However, the detailed process employed to achieve the change will be determined on a case by case basis. Changes only take effect from signature of both [Capability Manager] and DMO delegates. Amendments to the [Product Schedule] may be initiated as a result of a scheduled or unscheduled review or as a result of the financial / budget cycle. Each amendment proposed is to be incorporated into a new draft of the area affected by the change (i.e. [Product Schedule] Annex). Upon approval of the amendment the newly drafted [Product Schedule] annex is to be inserted into the master version of the [Product Schedule]. This process ensures that a single, up to date version of each [Product Schedule] is available for use by all stakeholders.

3.56 Table 3.2 shows the number and type of Change Proposals for the audit case study Product Schedules since they transitioned to the new MSA framework, as well as the net financial impact of all budget changes for a 10year period. The net financial impact is effected by Product Schedule Change Proposals, releases of funding for new capabilities, and Budget adjustments. The proportion of Army and Air Force Change Proposals approved at the 2Star level and above is a result of these Services’ preparing an omnibus Change Proposal to reflect all decisions at six-monthly reviews regardless of lower-level authority to update Product Schedules. Navy’s change process is discussed below.

Table 3.2: Analysis of Change Proposals for case study Product Schedules

 

ANZAC fleet

Bushmaster fleet

Orion fleet

Period

July 2012–September 2014

July 2013–September 2014

July 2013–September 2014

Number of Change Proposals

16(a)

10

4

Number approved at 2-Star or 3-Star level

12

8

3

Types of Change Proposals

Funding, schedule, capability upgrades, revision and reissue of Product Schedule

Funding, implementation of review outcomes, capability procurement/upgrades, KPI changes

Reissue of Product Schedule, funding

Net financial impact to June 2014 ($million)

651.014

-0.238

35.238

Source: ANAO analysis of Defence records.

  1. From February 2014, the ANZAC SPO Director and the Capability Manager Representative did not raise Change Proposals for amounts less than $1 million; by September 2014, 15 internal transfers of funding between line-items, amounting to $4.98 million, had been approved under this arrangement. Another two transfers totalling $1.44 million were made in January 2015. See discussion below at paragraph 3.60.
ANZAC SPO raised numerous issues about Change Proposals during 2014

3.57 During 2014, the ANZAC SPO and the Capability Manager Representative for the ANZAC fleet were attempting to handle the administrative complexity of keeping the 88-page ANZAC fleet Product Schedule up-to-date.82 These complexities included a Change Proposal processing backlog, the level of financial detail included in the Product Schedule, and the method of implementing decisions taken at the six-monthly review point. These issues are examined in the paragraphs below.

3.58 In February 2014, the ANZAC SPO advised the Capability Manager Representative that the administrative and management processes applicable to Product Schedules had become too complex, and were presenting risks to capability that were having to be managed by the SPO. The SPO was awaiting sign-off of an omnibus Change Proposal from November 2013, delaying the progress of three subsequent Change Proposals, two of which were to secure funding for the installation of new capabilities. In the interim, the SPO was providing funding of some $618 000 to fund these capability upgrades. This presented risk to the SPO’s ability to fund other commitments if the Change Proposals remained delayed for much longer. The SPO noted that Change Proposals have to be processed in the order in which they are submitted, which maintains the accuracy of Navy’s 10 financial line-items in Product Schedules.

3.59 The minutes of the June 2014 Force Element Quarterly Review noted that:

Overachievement in [Materiel Ready Days], operational expenditure, product schedule changes and issues with the process of [Change Proposals] are affecting the performance of the FFH [ANZAC] Fleet.

3.60 Further, the inclusion of 10 separate financial line-items within Navy Product Schedules means that any internal movement of funds by ANZAC SPO, such as from ‘Planned Maintenance’ to ‘Engineering Change’, nominally requires a Change Proposal, no matter how small the amount.83 In January and February 2014, by exchange of emails, the SPO Director and Capability Manager Representative agreed not to raise Change Proposals for transfers of funding between line-items amounting to less than $1 million (the level of their delegation under the ANZAC fleet Product Schedule), and to instead include a summary report each month to the Operational Sustainment Management Meeting so that the transfers could be endorsed there. The SPO Director commented that this was a practical approach that retained clear audit trails of decision making. By January 2015, 17 internal funding transfers amounting to $6.42 million had been approved in this way, including transfers from, for example, Materiel Support to Maintenance (Planned), or from Maintenance (Unplanned) to Maintenance (Planned).84

3.61 In January 2015, Navy updated its Heads of Agreement, including the Change Proposal process. The revised Heads of Agreement stipulates that all Product Schedules will be ‘comprehensively reviewed and updated as necessary during the financial year, to ensure the currency and accuracy of their content for ongoing application’. An Administrative Instruction outlining this process and the timetable for completing the annual review has also been published. Any Product Schedule changes instituted through the annual review process will not be approved through a Change Proposal.

3.62 Navy’s annual review process represents a departure from the modular approach to Product Schedules established by the Standard Procedure on MSAs, and reintroduces the pre2012 arrangements involving annual review. The revised approach may increase the difficulty of keeping up-to-date a large number of Product Schedules, all of which are now significantly longer post-Rizzo.

Conclusion—MSA Change Proposals

3.63 Implementation of the Standard Procedure’s revised change process has been relatively smooth in Army and Air Force. Navy’s management of MSAs demonstrated a relatively conservative risk appetite, reflecting its assessment of risk, and resulted in a higher number of Product Schedule changes and delays in approving them. An unintended consequence of Navy’s approach was the emergence of undocumented workarounds to overcome delays in processing Product Schedule changes. While a matter for Navy’s senior leadership, there is scope to review the change management process for Navy Product Schedules and their level of detail, to support more flexible management of MSAs and avoid undocumented workarounds in their administration.

Recommendation No.1

3.64 The ANAO recommends that Navy and the DMO review change management processes for Navy Product Schedules, and the level of detail in the Schedules, to support more flexible management of the Navy Materiel Sustainment Agreement.

Defence response: Agreed.

Risk management

3.65 Risk management has been a feature of MSA administration since 2005, when the first Product Schedules included a section on ‘Risks and Mitigation Strategies’. The MSA framework now includes a number of levels of risk management and reporting:

  • The 2012 Standard Procedure on MSAs provides for Annex E of Product Schedules to address the ‘Issues, risks and constraints’ affecting that product.85 Product Schedules are to outline the current management action for existing issues, and the risk mitigation strategy and contingency plans for identified risks.
  • The Army and Air Force Heads of Agreement similarly require their subordinate Product Schedules to contain an evaluation of ‘Issues, risks and constraints’. The Navy Heads of Agreement lists the types of interdependent materiel risks that the MSA seeks to address, and commits Navy and the DMO to ‘work towards a mutually accessible and integrated Enterprise Risk Management System’.
  • The regular fleet screening reviews receive a narrative outline of current risk management activity, and briefing material for the reviews includes lists of current risks. Further, monthly meetings between DMO SPOs and the relevant Service receive and discuss risk reports.
  • Through the DMO’s Monthly Reporting System, DMO Product Managers and Service Capability Manager Representatives provide information on materiel availability and operational matters to senior leadership.

3.66 This section focuses on risk management changes within Navy, and the treatment of issues and risks for the three case studies examined by the ANAO: ANZAC ships, Bushmaster Protected Mobility Vehicles and Orion aircraft.

Navy and the ANZAC fleet

3.67 By way of background, the June 2010 Product Schedule for HMAS Manoora and HMAS Kanimbla (two of the ships whose materiel condition prompted the Rizzo Report) had identified as almost certain the risk that these ships were operating beyond their service life, with severe consequences for performance. The Product Schedule also noted that ship configuration and maintenance data was inadequate to comply with the RAN Maintenance System, and that the materiel support regime for the two vessels was almost wholly reactive, with the bulk of maintenance being effected in response to breakdown events.

3.68 The DMO and Navy did not effectively mitigate the known sustainment risks for HMAS Kanimbla and HMAS Manoora, and consequently were unable to meet government requirements for use of the vessels, prompting the commissioning of the Rizzo Report. The 2012–13 ANAO Major Projects Report noted that risk management became a major focus on the sustainment side of the DMO’s business in response to the Rizzo Report.86 As part of the Rizzo Reform Program, the DMO and Navy have developed the Integrated Mission Management System (IMMS) to address the need for an integrated Navy/DMO risk management system, and introduce greater accountability in relation to risk management.87

3.69 IMMS was endorsed by the then Chief of Navy in June 2011 as a mechanism for him to have visibility of the Product Schedules, their KPIs and the interdependent risks shared by Navy and the DMO. The system is intended to record sustainment information on all of the products across Navy, in particular highlighting information that might be more readily apparent to Navy than the DMO, or vice versa. IMMS sits above existing DMO and Navy reporting tools and provides senior decision makers with management information. Defence informed the ANAO in March 2015 that the Deputy Chief of Navy has directed that IMMS be used at fleet screenings, and the Guided Missile Frigate SPO (FFG SPO) is to pilot the use of IMMS at the October 2015 Fleet Screening.

3.70 In relation to reviews of the ANZAC fleet, the briefing material for the Navy’s October 2014 fleet screening contained a detailed five-page analysis of the activities under way to manage risks identified in the Product Schedule, including risks related to obsolescence, supportability, communications systems and maintenance.

3.71 At the operational level, each monthly Operational Sustainment Management Meeting receives an update of the ANZAC SPO Strategic Risk Register and the Navy’s Frigate Group CN02 Strategic Risk Register, and discussion of progress is a standing agenda item. By May 2014, these Strategic Risk Registers identified seven major risks as having been escalated by being transferred to the ANZAC fleet Product Register within IMMS.

Army and the Bushmaster fleet

3.72 The Bushmaster Product Schedule identified five issues affecting the product in 2013–14 and 2014–15, including: engine obsolescence; absence of sustainment funding to support a Bushmaster fleet numbering more than 737 (see the text-box on pages 74–75)88; and power pack serviceability. In 2014–15, the Lead Capability Manager provided an assessment of the management action being undertaken to address each issue identified that year.

3.73 Similarly, the Bushmaster Product Schedule identified a total of five risks for 2013–14 and 2014–15, including: labour shortfalls and late delivery of services by the Support Services Contractor; and attrition rates affecting the post-Afghanistan size of the fleet. In 2014–15, three risks were assessed by the Lead Capability Manager as being ‘low’, and two as ‘medium’.

Figure 3.2: Bushmaster Protected Mobility Vehicle (PMV)—Troop variant

Source: Thales, by permission.

Plans for the Bushmaster fleet have expanded to meet the numbers available

The Bushmaster Protected Mobility Vehicle (PMV) is the first Australian-designed and developed combat vehicle since World War II. On 2 July 2012, the then Government announced the acquisition of a further 214 Bushmasters (Production Period 5), to maintain critical skills at the Thales factory in Bendigo that would be required for the possible production of the next-generation Hawkei vehicle. The Government stated at the time that:

The new Bushmaster vehicles will progressively replace the oldest Bushmasters in the present fleet and … provide ongoing protected mobility for the command and control of our artillery.a

In July 2013 and again in July 2014, the Product Schedule for the Bushmaster fleet noted that the 214 new vehicles would:

… not form part of the sustained fleet. Any vehicles not funded (sustainment) by emerging projects will only be stored.

In July 2013, the Bushmaster Product Schedule contained no assessment by the Lead Capability Manager of the issue of lack of sustainment funding for these vehicles. By July 2014, the lack of sustainment funding was no longer listed as an issue, although elsewhere the Product Schedule repeated the 2013 statement about storage of new vehicles. By this time, 71 new vehicles were in storage, and none had been issued to Army.

Since the 2012 decision, Defence has expanded its requirements for Bushmaster vehicles, including through Plan Beersheba (a restructuring of Army units)b and through new projects that have adopted the Bushmaster vehicle as a solution to their requirements.

Army therefore informed the ANAO in December 2014 that the new vehicles announced in June 2012 would not be replacing older, existing vehicles. Instead, along with new vehicles from Production Period 4 that have not been used as attrition stock (as originally intended), they are to be incorporated into the Bushmaster fleet as follows:

  • 146 of the new vehicles will be sustained with funding from approved Defence projects;
  • 40 of the new vehicles will be sustained by Army to meet emerging requirements; and
  • six of the new vehicles are to be sustained by Air Force to meet emerging requirements.

Army further informed the ANAO that the storage of new vehicles is a temporary measure until sustainment funding is triggered through the ongoing projects. The currently expected end-state will see 66 vehicles surplus to Defence requirements by 2022–23, and this is intended to provide Defence with an appropriate attrition stock of Bushmasters to support training and future operations.

Remediation of old vehicles despite availability of surplus vehicles

Despite the availability of surplus attrition stock and new vehicles, in July 2013 the DMO approved a Transition and Remediation Plan for vehicles returning from the Middle East Area of Operations. The plan involves expenditure of $34.501 million to remediate 177 vehicles from 2013–14 to 2016–17, at an average cost of $194 920 per vehicle. The remediation is being funded through Operational Supplementation, under which Defence receives funding for operations on a no-win/no-loss basis.

Defence was unable to provide the ANAO with evidence showing that it considered the availability of surplus vehicles when it approved the Transition and Remediation Plan. Defence informed the ANAO in March 2015 that:

Given the decision to expand the requirements for the Bushmaster fleet of vehicles under Plan Beersheba, options were to consider remediation or acquisition of additional vehicles. Army, with assistance from DMO, conducted a broad cost assessment which indicated that the average cost of remediation per vehicle is $194 920 per vehicle. The cost per vehicle of the latest variant of Bushmaster is from $902 000 to $972 000, depending on the variant.

Note a: The Hon. Stephen Smith MP, Minister for Defence, More Bushmasters for Army, Hawkei development proceeds, media release, 2 July 2012.

Note b: By July 2014, Defence was planning for a Bushmaster fleet of 856 vehicles under Plan Beersheba.

Air Force and the Orion fleet

3.74 The Orion Product Schedule identified 11 issues affecting the product over the last two financial years, including: fleet and mission system availability, arising from spares shortage, obsolescence, ageing systems and supportability challenges; corrosion; and the expiration of the engine maintenance contract. Each issue in the Product Schedule has been accompanied by a detailed narrative of the management action being undertaken to address the issue, including the area responsible for the action.

3.75 The Orion Product Schedule also identified 11 risks over the last two financial years, including: structural ageing-aircraft issues; airframe fatigue life; spares supportability concerns; and hydraulics maintenance pending the signing of a new maintenance contract. In 2014–15, one risk was assessed as being likely, four risks were assessed as being possible, and the remainder were assessed as unlikely or remote. Each risk was accompanied by an outline of the mitigation strategy and contingency plan.

3.76 In March and April 2014, aircraft from the Orion fleet took part in the search for Malaysian Airlines flight MH370, under Operation Southern Indian Ocean. The ANAO sought DMO advice on any risk assessments made as a result of the operation, including the potential impact of the operation on the sustainment of the Orion fleet. The DMO informed the ANAO that, for the period of the operation, no additional risk or expenditure was noted outside that allowed for within capability requirements. The DMO also informed the ANAO that, although the Orion fleet flew 62 per cent more than the planned monthly flying hours for the period of the operation:

  • the total rate of effort for 2013–14 remained unchanged;
  • there was no effect upon Deeper Maintenance servicing inductions or the Planned Withdrawal Date of the Orion fleet; and
  • the operation established a new and enduring capability to detect Flight Recorders and Locator Beacons.

Conclusion—risk management

3.77 The MSA framework recognises the importance of risk management for sustainment activities, and provides a structured process for identification, assessment and management of sustainment risks by the Services and the DMO. The ANAO’s examination of the case study Product Schedules indicates that senior leadership was kept up-to-date about the risks to the relevant capabilities. However, the Navy’s experience relating to HMAS Manoora and Kanimbla, discussed at paragraphs 3.67 to 3.68, highlights that the practical effectiveness of the MSA framework largely depends on active and timely management of identified risks by Capability Managers and the DMO, and in that respect, a robust MSA framework is an aid to management, not an end in itself.

4. Sustainment Funding and Cost Estimates

This chapter examines Defence’s sustainment funding arrangements, and cost estimates for individual Product Schedules.

Introduction

4.1 Once it has received its annual appropriation, Defence has managed its acquisition and sustainment activities by transferring funding to the DMO. For management and reporting purposes, acquisition and sustainment funding is treated separately: DMO Programme 1.1 relates to Management of Capability Acquisition, and DMO Programme 1.2 relates to Management of Capability Sustainment. Sustainment funding is transferred to MSA Products according to their approved budgets.

4.2 Accurate estimation of Product Schedule sustainment costs is important because of the large sums and the long planning timeframes involved. A small percentage difference in costs, for example, can quickly amount to millions of dollars. Reliable cost estimates enable Defence’s Capability Managers to confidently use their current funding across different platforms, commodities and services as ‘informed purchasers’, and to move funding where it is most needed while still achieving required sustainment outcomes elsewhere.

4.3 In this chapter, the ANAO examines:

  • the provision of Defence funding for the DMO’s sustainment activities, and the application of the funding across capabilities; and
  • the accuracy of cost estimates for individual Product Schedules, focusing on the three case study Product Schedules: ANZAC ships, Bushmaster Protected Mobility Vehicles and Orion aircraft.

Defence funding for the DMO

4.4 To pay for the services to be provided by the DMO for acquisition or sustainment, Defence has made a prepayment to the DMO at the beginning of the financial year. The annual up-front payment—covering not only sustainment but also Major and Minor Capital Investment (that is, acquisition activities)—has been placed into a DMO Special Account.89 In 2014–15, the initial Defence prepayment amounted to $10.686 billion, of which $5.532 billion was for sustainment.90 Following the prepayment, the sustainment funding was distributed to the MSA sustainment products according to their approved budgets.

4.5 During the financial year, Defence’s prepayment to the DMO has been adjusted as needed, for reasons such as operational requirements or closure of acquisition projects and the transfer of unused funding to sustainment. During 2013–14, for example, the prepayment was adjusted five times. For sustainment, the changes included adjustments ranging from $4 million to $55 million. The final result was that the prepayment for 2013–14 relating to sustainment amounted to $5.064 billion.91

Distinguishing between acquisition and sustainment activity

4.6 As noted above, the DMO’s acquisition and sustainment activities are presented as separate programs in the Portfolio Budget Statements (PBS)—with Programme 1.1 covering the management of capability acquisition, while Programme 1.2 covers the management of capability sustainment.92 The DMO program structure broadly reflects the DMO Business Model approved by the Minister for Defence in September 2004, which identified three DMO outputs: acquisition projects; sustainment of ADF equipment fleets; and the provision of policy and advice services to Defence and government. The Business Model, and the related business rules agreed by the Defence Committee in October 2004, were both referred to in the 2006 Defence–DMO Memorandum of Arrangements, which documents certain constraints on the transfer of funds within the DMO between acquisition and sustainment activities.

4.7 However, while the PBS suggests that a relatively clear-cut distinction exists between acquisition and sustainment activities and funding, that distinction is not as clear-cut in the Memorandum of Arrangements. Further, experience indicates that the distinction is not hard and fast in practice, and that there is generally a transition period (illustrated in Figure 4.1), between Initial Operational Capability (IOC) and Final Operational Capability (FOC).93 The transition period may last for several years, and can present practical challenges for the internal management of acquisition and sustainment funding and expenditures.

Figure 4.1: Defence’s Capability Systems Life Cycle

Source: Extract from DMO Submission to Joint Committee of Public Accounts and Audit, Review of the 2011–12 Defence Materiel Organisation Major Projects Report, Submission 3, 5 April 2013.

Examples of acquisition–sustainment funding transfers

4.8 The DMO informed the ANAO that there were four transfers of funding from acquisition to sustainment in 2011–12, two in 2012–13 and one in 2013–14. Table 4.1 summarises such transfers from 2011–12 to 2013–14.94

Table 4.1: Detail of funding transfers from acquisition projects to sustainment products,
July 2011–June 2014

Date

Transfer from

Transfer to

Amount ($m)

Reason

Oct 2011

JP 2030 Phase 7B, Air Command Support System

CAF16, Air Component Command and Intelligence Systems

4.695

Project closed and In-Service Support provided.

Dec 2011

JP 5195 Phase 3C, GPS for ADF Aircraft

CAF16, Air Component Command and Intelligence Systems

3.400

Project was delivered in 2008, now closed and In-Service Support provided as per initial intent.

Feb 2012

JP 141 Phase 1, Chemical, Biological, Radiological Response

CA26, Chemical, Biological, Radiological, Nuclear and Explosive Equipment; CA39, ADF Clothing; JHC01, Health Systems; CA32, Satellite Terminal Communications

2.679

Project largely completed in 2009, now closed and remaining requirements transferred to various SPOs.

May 2012

LAND 58 Phase 3, Weapon Locating Radar Life of Type Extension

CA05, Radar and Short Range Air Defence

0.220

Project closed and three years In-Service Support provided.

Oct 2012

JP 2110 Phase 1A, Chemical, Biological, Radiological and Nuclear Defence

CA26, Chemical, Biological, Radiological, Nuclear and Explosive Equipment

2.500

Project about to be closed, In-Service Support provided for Years Two and Three of initial period.

Oct 2012

LAND 40 Phase 1, Direct Fire Guided Weapon

CA44, Army Explosive Ordnance

6.524

Project closed and procurement of additional Javelin missiles funded as per original intent.

May 2014

LAND 112 Phase 4, ASLAV Enhancement

CA02, ASLAV

3.072

Project about to be closed, some vehicle reassembly transferred.

Total

 

 

23.09

 

Source: ANAO analysis of DMO records.

4.9 In this audit, the ANAO noted a number of instances of overlap between acquisition and sustainment for the three case study Product Schedules:

  • in 2008–09, sustainment funding was used for the completion of an ANZAC ship;
  • in 2011–12, Army was uncertain whether anti-mine equipment and a gun mount required for Bushmaster operations in Afghanistan would be funded through Army Minors95 acquisition funding or through sustainment funding96;
  • in 2013–14, Army Minors acquisition project funding was used to provide extra cladding for Bushmaster vehicles being used on operations (ADF operations are normally funded through MSA Product Schedules);
  • in February 2014, Navy transferred $7.972 million from the Navy Minor Capital Equipment Program to the ANZAC fleet Product Schedule, for replacement of an obsolete Combat Management System at West Head Gunnery Range; and
  • the ANZAC fleet Anti-Ship Missile Defence (ASMD) refit and upgrade program is treated by Defence as composed of acquisition-funded (SEA 1448) and sustainment-funded (CN02) elements, but there are a number of indications that at least one sustainment-funded element, renewal of the Operations Room layout, was an integral part of SEA 1448, and should have been funded as such.97

4.10 In a similar example of project expenditure being classified as sustainment, the 2012 ANAO performance audit report on the M113 upgrade project reported that the preparation and extension of M113 hulls for upgrade incurred the bulk of the costs of the M113 Commercial Support Program, which was a sustainment program. An estimated $32.43 million of sustainment funding was expended over four years on initial work to upgrade some 80 per cent of the M113 fleet (350 of 431 vehicles). The DMO at one stage requested that Army spend more sustainment money so as to mitigate potential delays to the upgrade project.98

4.11 Defence informed the ANAO in March 2015 that:

There is a clear distinction regarding the constraints on the transfer of funds within the DMO between acquisition and sustainment activities, which is also reflected in the PBS as separate Programmes. Funding received from Defence under Programme 1.1 Management of Capability Acquisition is clearly linked to the Materiel Acquisition Agreements (MAAs) and deliverables that have been expressly stated under these documents. Funding received from Defence under Programme 1.2 Management of Capability Sustainment is again clearly linked under the Materiel Sustainment Agreements (MSAs). DMO is not able to utilise funding received from Defence for MAAs for MSA activities or vice versa. There are instances, however, where as part of a Project closure process there may be remaining activities that were in scope under the MAA and are transferred to a relevant Product Schedule under an MSA. In these instances it is clearly documented and agreed by all signatories to the MAA and MSA Product Schedule Change Proposal that Defence is agreeing that the activity funded initially through the MAA can now be transferred to the MSA.

Joint Committee of Public Accounts and Audit concerns

4.12 In May 2014, the Joint Committee of Public Accounts and Audit (JCPAA) expressed doubt about whether there is always financial and budgetary separation of acquisition and sustainment.99

4.13 As discussed, the clear distinction between acquisition and sustainment activities and funding shown in the DMO PBS is not as clear-cut in the Memorandum of Arrangements, and the underlying Business Model and business rules. In this light, there is scope to review current settings for clarity, recognising that there is often a transition period between the acquisition and sustainment phases of a capability. The timing of such a review will depend in part on the outcomes of the current First Principles Review of Defence.100

Recommendation No.2

4.14 To clarify the internal treatment of acquisition and sustainment funding, the ANAO recommends that Defence review relevant business rules and guidance.

Defence response: Agreed.

Funding transfers between sustainment products

4.15 From the establishment of the DMO in 2005, control of sustainment funding rested with the CEO DMO, who was able to move funding between sustainment products as needs and priorities changed. In 2009, the Government Response to the Mortimer Review proposed to strengthen the role of Capability Managers101 and some years ago Capability Managers were given renewed responsibility for controlling their own sustainment budgets. Under this arrangement, transfers of funding between sustainment products can occur with the agreement of the relevant Capability Manager.102

4.16 Table 4.2 shows that sustainment funding has been regularly transferred from one MSA product to another over the last three financial years, and that the number of transfers has increased significantly, reflecting increased use of the flexibilities available in the system. During the audit, Capability Managers informed the ANAO that they valued the ability to flexibly use sustainment resources according to operational and maintenance needs, which they considered a key strength of the sustainment framework.

Table 4.2: Funding transfers between sustainment products, July 2011–June 2014

Financial Year

Number of funding transfers

Value of transfers affecting current financial year ($m)

Value of transfers affecting nine following financial years ($m)

Total value of funding transfers ($m)

2011–12

7

17

153

170

2012–13

31

69

94

163

2013–14

55

178

891

1069

Totals

93

264

1138

1402

Source: DMO Finance.

Product Schedule cost estimates

4.17 Every year since the inception of the MSA framework in mid-2005, Defence and the DMO have agreed on the cost estimates for each MSA Product Schedule for the following 10 financial years—that is, the period covered by the Defence Management and Financial Plan (DMFP).

4.18 This section reviews the Product Schedule cost estimates for the three case studies examined by the ANAO. The focus is on the four-year period covered by the Commonwealth Budget, including the budget year and three years of forward estimates. A greater degree of accuracy can be expected for this period than for the full 10-year estimation timeframe included in MSA Product Schedules. The four-year estimates are compared to the actual spending that eventually occurred. The ANAO also examined the accuracy of estimates made just prior to, and during the budget year, for each year.

Navy

4.19 Figure 4.2 compares baseline cost estimates (that is, excluding operations funding) against expenditure for the ANZAC fleet Product Schedule, for each financial year from 2005–06 to the present. More specifically, Figure 4.2 illustrates:

  • the range of sustainment cost estimates for each financial year (the grey columns);
  • the estimate made at the beginning of each financial year (the blue circle);
  • the actual expenditure incurred in the financial year (the blue line); and
  • the range of estimates for the three financial years covered by the current forward estimates (2015–16, 2016–17 and 2017–18—the blue column/bars).103

4.20 In the past seven financial years, estimates made at the beginning of the financial year were well below actual expenditure for three years (2008–09, 2012–13 and 2013–14), while estimates for three years (2007–08, 2009–10 and 2010–11) were well above. For one year (2011–12), the estimate made at the beginning of the financial year was accurate. The figure also shows marked variation in actual expenditure from year to year.

Figure 4.2: ANZAC fleet sustainment cost estimates and outcomes for budget year and forward estimates, 2005–06 onwards

Source: ANAO analysis of CN02 Product Schedules, 2005–14.

Note: Defence was not able to supply an actual cost for 2005–06. The actual cost for 2006–07 is sourced from a hard-copy Sustainment Overview Report, rather than from Defence’s financial systems, and may reflect a slightly different cost basis.

4.21 Figure 4.3 shows the original and updated budget-year cost estimates for the ANZAC fleet Product Schedule for the last seven financial years. Two estimates are shown: for the annual Portfolio Budget Statement (in March); and Additional Estimates (in December). These estimates are shown as columns, and the actual expenditure for each financial year is shown by the blue line.

Figure 4.3: Budget-year ANZAC fleet cost estimates, and actual expenditure, 2007–08 to 2013–14

Source: DMO Finance, August 2014.

Notes: PBS—Portfolio Budget Statements (March); PAES—Portfolio Additional Estimates Statements (December).

 The blue column in this figure reflects an earlier estimate than the blue dot for the estimate made at the beginning of the financial year in Figure 4.2.

4.22 The figure shows that, in the last two financial years, actual spending has exceeded the March budget-year estimates, by $21 million (10 per cent) in 2012–13 and $39 million (17 per cent) in 2013–14.

4.23 The DMO informed the ANAO in October 2014 that:

  • The high actual costs in 2008–09 were a result of engineering changes to the ANZAC Class to meet the HMAS Perth build specification. SEA 1348 Phase 2 (Ship Build Project) did not have sufficient funding available to rollout all the progressive changes made during the ship build program, and consequently some changes (related to system issues or early obsolescence) had to be funded from the ANZAC fleet sustainment budget.104 Further, a logistics shortfall was identified to the then Government, and an additional $1 billion of funding was approved for the sustainment of the ANZAC Class.105
  • The low actual costs in 2010–11 were a result of the Strategic Reform Program (SRP) being initiated by Defence.106 In response to the SRP, the ANZAC SPO prioritised work and cancelled or reduced the value of contracts. Navy also put two platforms into extended readiness (that is, it would take longer to make them ready for service).

4.24 As indicated previously (paragraphs 4.6 to 4.14), there is a lack of clarity surrounding the financial and budgetary distinction between acquisition and sustainment. It is unclear why the use of sustainment funding for SEA 1348 Phase 2 was reported as actual expenditure against the MSA Product Schedule when it was used for acquisition rather than sustainment purposes.

4.25 As part of the Rizzo Reform Program, Navy developed a Whole of Life Cost Plan for each of its MSA products.107 Cost studies were still being undertaken when the revised ANZAC fleet Product Schedule was first agreed in August 2012. In late 2013, the Product Schedule included a provisional total Whole of Life Cost of $3.05 billion, somewhat less than the provisional approved funding of $3.17 billion. In July 2014, the Whole of Life Cost increased to match the approved funding, at $3.19 billion.

Army

4.26 Figure 4.4 compares baseline cost estimates (that is, excluding operations funding) against expenditure for the Bushmaster Product Schedule, for each financial year from 2005–06 to the present. More specifically, Figure 4.4 illustrates:

  • the range of the sustainment cost estimates for each financial year (the grey columns);
  • the estimate made at the beginning of each financial year (the blue circle);
  • the actual expenditure incurred in the financial year (the blue line); and
  • the range of estimates for the three financial years covered by the current forward estimates (2015–16, 2016–17 and 2017–18—the blue column/bars).108

4.27 In five out of the past seven financial years, actual sustainment expenditure has exceeded the range of estimates for the year, three times by a wide margin (2008–09, 2011–12 and 2012–13).109 In 2011–12, actual expenditure was more than twice the estimate made at the beginning of the financial year, and in 2012–13 it was 45 per cent higher. Actual expenditure was close to the estimate made at the beginning of the financial year in 2009–10 and 2013–14 only. The figure also shows marked variation in actual sustainment expenditure from year to year.

Figure 4.4: Bushmaster sustainment cost estimates and outcomes for budget year and forward estimates, 2005–06 onwards

Source: ANAO analysis of CA04 Product Schedules, 2005–14.

Note: Defence was not able to supply an actual cost for 2005–06. The actual cost for 2006–07 is sourced from a hard-copy Sustainment Overview Report, rather than from Defence’s financial systems, and may reflect a slightly different cost basis.

 At Additional Estimates in February 2015, the 2014–15 baseline estimate (that is, excluding operations funding) for sustainment of the Bushmaster fleet increased from $35.6 million to $47.8 million.

Budget-year cost estimates

4.28 Figure 4.5 shows the original and updated budget-year cost estimates for the Bushmaster fleet Product Schedule for the last seven financial years. Two estimates are shown: for the annual Portfolio Budget Statement (in March); and Additional Estimates (in December). These estimates are shown as columns, and the actual expenditure for each financial year is shown by the blue line.

Figure 4.5: Budget-year Bushmaster cost estimates, and actual expenditure, 2007–08 to 2013–14

Source: DMO Finance, August 2014, and further Defence advice, December 2014.

Notes: The blue column in this figure reflects an earlier estimate than the blue dot for the estimate made at the beginning of the financial year in Figure 4.4.

 In 2010–11, the original budget of $59.5 million was made in error, and was corrected to $17 million at Additional Estimates; the figure shows the corrected amount.110

4.29 The figure shows that actual expenditure has generally exceeded the original budget-year estimate as well as the mid-year revision. In 2012–13, the estimate increased markedly during the financial year to more closely align with actual expenditure. In contrast to earlier years, both cost estimates for 2013–14 were accurate.

4.30 The DMO informed the ANAO in June 2014 that:

  • 2010–11: The variance was primarily due to the cost of supporting Protected Mobility Vehicles procured by Land 116 Phase 3. Army accepted the financial liability of introducing the additional vehicles into service prior to the triggering of sustainment funding for the vehicles. Further, in 2010–11, the sustainment budget was managed by the DMO with Army on a ‘whole-of-Army’ basis, and not on an individual Product basis.
  • 2011–12: The variance was primarily due to SPARK II Mine Roller and Platt MR550 Ring Mount procurement activities.111 It was initially expected that Army would fund the SPARK II requirement using Minor Project funding controlled by the Land Capability Battleworthiness Board (LCBB), and the Platt MR550 Ring Mount using
    no-win/no-loss funding.112 Changes were made around this time regarding the application and eligibility for no-win/no-loss funding. Ultimately, both procurements were funded through Army baseline sustainment funds, with the variance managed on a whole-of-Army basis.113

4.31 As indicated previously (paragraphs 4.6 to 4.14), there is a lack of clarity surrounding the financial and budgetary distinction between acquisition and sustainment. It is unclear why acquisition funding was initially expected to fund the Spark II Mine Roller, which was an operational procurement.

Air Force

4.32 Figure 4.6 compares baseline cost estimates (that is, excluding operations funding) against expenditure for the Orion Product Schedule, for each financial year from 2005–06 to the present. More specifically, Figure 4.6 illustrates:

  • the range of the sustainment cost estimates for each financial year (the grey columns);
  • the estimate made at the beginning of each financial year (the blue circle);
  • the actual expenditure incurred in the financial year (the blue line); and
  • the range of estimates for the three financial years covered by the current forward estimates (2015–16, 2016–17 and 2017–18—the blue columns/bar).114

Figure 4.6: Orion sustainment cost estimates and outcomes for budget year and forward estimates, 2005–06 onwards

Source: ANAO analysis of CAF04 Product Schedules, 2005–14.

Note: The actual costs for 2005–06 and 2006–07 are sourced from a hard-copy Sustainment Overview Report, rather than from Defence’s financial systems, and may reflect a slightly different cost basis.

4.33 From 2007–08 to 2010–11, and in 2013–14, actual sustainment expenditure was within the range of the annual estimates.115 In three years (2006–07, 2010–11 and 2013–14) actual expenditure was close to the estimate made at the beginning of the financial year. In 2011–12 and 2012–13, actual expenditure significantly exceeded the estimate made at the beginning of the financial year, by 37 per cent and 17 per cent respectively.

Budget-year cost estimates

4.34 Figure 4.7 shows the original and updated budget-year cost estimates for the Orion fleet Product Schedule for the last seven financial years. Two estimates are shown: for the annual Portfolio Budget Statement (in March); and Additional Estimates (in December). These estimates are shown as columns, and the actual expenditure for each financial year is shown by the blue line.

Figure 4.7: Budget-year Orion cost estimates, and actual expenditure, 2007–08 to 2013–14

Source: DMO Finance, August 2014.

Note: The blue column in this figure reflects an earlier estimate than the blue dot for the estimate made at the beginning of the financial year in Figure 4.6.

4.35 The figure shows that actual expenditure has often been close to the original budget estimate, as well as to the mid-year revision. In 2011–12 and 2012–13, the estimate increased markedly during the financial year to more closely match actual expenditure.

4.36 The DMO informed the ANAO in October 2014 that:

  • the high actual expenditure in 2011–12 (the year with the most variance from estimates) was primarily due to the purchase of sonobuoys and imaging systems, and recovery of a maintenance backlog; and
  • in 2012–13, responsibility for multi-platform avionics support was transferred from another Product Schedule to the Orion Product Schedule, and a funding transfer was approved at the Mid-Year Review Principals Meeting.

Conclusion—case study cost estimates

4.37 In summary, the three case studies examined by the ANAO show persistent inaccuracy in Defence’s sustainment cost estimates. The ANZAC fleet estimates widely straddled actual expenditure, the Bushmaster fleet estimates were mostly below actual expenditure, and while the Orion fleet estimates were more accurate, they were still on occasions well below actual expenditure. Defence provided an explanation of variances in specific instances, including unanticipated changes in the allocation of funding across projects, products and sustainment activities. Defence also informed the ANAO that foreign exchange fluctuations and changes in the timing of contractual payments affect the accuracy of cost estimates. However, in no instances did Defence cite unexpected deployments or major materiel failure as a cause of inaccuracy in the estimates.

Sustainment cost estimates across all Product Schedules

4.38 The cost-estimation inaccuracy encountered for the case study Product Schedules is also evident for other sustainment products. Figure 4.8 shows that, out of 118 sustainment products in 2013–14 (the blue dots):

  • for 18 products (15 per cent), actual expenditure was over 25 per cent below the original budget cost estimates; and
  • for 21 products (18 per cent), actual expenditure was over 25 per cent higher than the original budget cost estimates.

Figure 4.8: Accuracy of Product Schedule cost estimates for 2013–14

Source: ANAO analysis of Defence data.

Notes: The order of the Product Schedules is different in each series. A blue dot may not represent the same Product Schedule as the adjacent grey dot.

 The figure does not include a number of variances that were off the scale:

• expenditure for five Product Schedules exceeded the PBS estimate by more than 100 per cent (105, 199, 335, 643 and 2636 per cent respectively);

• expenditure for three Product Schedules exceeded the PAES estimate by more than 100 per cent (110, 218 and 3613 per cent respectively).

4.39 Actual expenditure for one third of sustainment products (39 out of 118) varied from the original budget cost estimates by over 25 per cent. For another quarter of sustainment products (28 out of 118), actual expenditure varied from the original budget cost estimates by 10 to 24 per cent.

4.40 In relation to the mid-point of the financial year (Additional Estimates—the grey dots), actual expenditure for 39 products varied from the revised cost estimates by 10 per cent or more.

4.41 Figure 4.9 shows that, for the Top 30 sustainment products116, actual expenditure for seven products varied by 10–24 per cent from the PBS estimate (the blue dots), and for four products there was a variance of over 25 per cent. In relation to Additional Estimates (the grey dots), actual expenditure for seven of the Top 30 products varied from cost estimates by 10 per cent or more.

Figure 4.9: Accuracy of Top 30 Product Schedule cost estimates for 2013-14

Source: ANAO analysis of Defence data.

Notes: The order of the Product Schedules is different in each series. A blue dot may not represent the same Product Schedule as the adjacent grey dot.

 Defence reporting of the Top 30 sustainment products in 2013–14 combined certain cross-Service activities; for example, ‘Fuel & Lubricants – Air Force, Army and Navy’ was actually covered by three Product Schedules, and similarly, ‘Guided Weapons’ and ‘Explosive Ordnance’ (together covered by six Product Schedules), were shown as one line in the PBS and as two lines in the online supplement to the Annual Report. The ANAO analysis follows Defence’s practice: the chart shows 29 data points, and notes that the 30th (MRH90) is off the scale.

4.42 In response to a request about recent steps to analyse sustainment cost estimates or improve their accuracy, Defence informed the ANAO that in 2014 it:

  • began to classify Product Schedule Change Proposals by category, which should aid identification of reasons for cost variances117; and
  • released the first edition of the Defence Cost and Schedule Estimation Manual. The Manual prescribes policy and procedures for the development, validation and approval of cost and schedule estimates for all Defence resources (personnel, materiel, facilities, services and systems). The policy and procedures are to be applied for each stage of the capability life cycle, from identification of needs to disposal.

4.43 The Defence Cost and Schedule Estimation Manual was developed in response to Recommendation 1.2 of the 2008 Mortimer Review, which sought more disciplined cost, schedule and risk information for a project’s entry into the Defence Capability Plan. However, the May 2014 edition of the manual included only Chapter 1, containing the policy statement, roles and responsibilities, and definitions. While the manual was due to be completed by December 2014, no further sections had been published by February 2015.118

4.44 Despite the inherent uncertainty of financial forecasting, and the ambiguity introduced by the Capability Managers’ discretion to move funding between products, the extent of the variances in Product Schedule cost estimates indicates that Defence could do more to improve these estimates. This would in turn improve the capacity of Defence’s Capability Managers and the DMO to flexibly manage sustainment funding to achieve agreed outcomes.119

Conclusion

4.45 The DMO’s acquisition and sustainment activities are presented as separate programs in the Portfolio Budget Statements (PBS), and the 2006 Defence–DMO Memorandum of Arrangements, which documents certain constraints on the transfer of funds within the DMO between acquisition and sustainment activities. However, while the PBS suggests that a relatively clear-cut distinction exists between acquisition and sustainment activities and funding, that distinction is not as clear-cut in the Memorandum of Arrangements, and experience indicates that the distinction is not hard and fast in practice. While acknowledging that there can be a period of transition between the acquisition and sustainment phases of a capability, the ANAO noted a number of instances of overlap in the use of acquisition and sustainment funding. To clarify the internal treatment of acquisition and sustainment funding, Defence should review relevant business rules and guidance.

4.46 The three case studies examined by the ANAO indicated that there have been persistent inaccuracies in Defence’s sustainment cost estimates. Some departure from budget estimates can be expected due to flexible use of funding between sustainment products, and unforeseen factors such as the need to delay or bring forward maintenance work due to operational demands. However, variances of over 25 per cent are significant and suggest that there remains scope for the DMO to strengthen cost estimation techniques and understanding of cost variances. Improved cost estimation would strengthen the capacity of Defence’s Capability Managers to flexibly manage sustainment funding as informed purchasers of sustainment services.

5. Performance and Reporting

This chapter examines internal and external sustainment performance reporting, including KPIs.

Introduction

5.1 The 2008 Mortimer Review found that the ‘efficiency and effectiveness of DMO sustainment performance will not improve unless it is measured’, and that Defence did not have appropriate, quantifiable KPIs, nor did it accurately record sustainment costs. The Mortimer Review therefore recommended that:

DMO and Defence need to further develop the key performance indicators in Materiel Sustainment Agreements and the systems needed to record sustainment performance and costs.120

5.2 In this chapter, the ANAO examines Defence’s:

  • sustainment performance reporting; and
  • public reporting on sustainment activities.

Sustainment performance reporting

Defence currently reports sustainment performance through its Monthly Reporting System

5.3 Under the Memorandum of Arrangements and individual MSAs, the DMO is required to report monthly against all of the performance measures contained in Product Schedules. The DMO meets its sustainment reporting obligations through its Monthly Reporting System (MRS). There are two series of DMO sustainment reports produced each month:

  • The Sustainment Performance Report (SPR) covers the Top 30 sustainment products as listed in the PBS. The SPR is amended by DMO SPOs each month and cleared and endorsed by the respective DMO Branch and Division Heads.
  • The Navy Sustainment Performance Report (NSPR) is developed from the cleared DMO Division Head SPR and includes additional information to cover all Navy sustainment products. The NSPR was developed in response to the 2011 Rizzo Report, and is intended to provide a high level of visibility on sustainment performance across all Navy products.

5.4 For each sustainment product, the reports include a cost update (budget and year-to-date expenditure), an availability update (red/green/amber), and commentary on current performance by the relevant DMO Product Manager, Branch Head and Division Head. They also include advice on anything that has become known or has occurred in the reporting month that will either actually or potentially affect a product’s materiel readiness. This advice is intended to identify any issues similar to those experienced by Navy in 2011, when the supply ships HMAS Manoora and HMAS Kanimbla were unavailable for service.

5.5 The reports on sustainment product performance are reviewed by the CEO DMO, the Deputy CEO DMO, DMO General Managers, and the DMO Chief Finance Officer. The reports are also provided to the Government and central agencies.

5.6 Defence preparedness—the ultimate goal of Defence sustainment—is the subject of classified reporting to government:

Each quarter a Defence Preparedness Assessment (CDPAS) is undertaken, the report from which is considered by the Chief of Defence Force and the Secretary at their Strategic Command Group. […] The outcomes and key judgements from this assessment process are provided to the Minister as the classified Preparedness and Concurrency Ministerial Submission. This process has been refined considerably over the last four years, and provides a high level of assurance to Government as to the capability of Defence to meet current commitments and conduct future operations.121

Defence is developing a new sustainment performance reporting system

5.7 The MRS was originally designed for acquisition reporting, and has limitations in relation to sustainment reporting. In particular, the MRS can only report performance using relatively simple measures, and the system cannot present trend analysis. The Directorate of Sustainment Measurement and Analysis was created within the DMO Standardisation Office in July 2011, to work with DMO Divisions and Capability Managers, to recommend and implement an improved sustainment performance management system. The new system has now been designed and includes two components: a performance measurement framework—the Sustainment Performance Management Framework (SPMF); and a reporting system—the Sustainment Performance Management System (SPMS).

Sustainment Performance Management Framework (SPMF)

5.8 During the first half of 2012, the DMO reviewed the performance measures used for aviation and maritime sustainment products, and developed a framework involving four types of performance measures:

  • Outcome KPIs (for example, Mission Success Rate, Rate of Effort Achievement, or Mission Capable Days Achievement);
  • Output KPIs (for example, Aircraft Serviceability, Aircraft Availability or Materiel Ready Days Achievement);
  • Key Health Indicators (KHIs) to enable the DMO and Capability Managers to review a range of information to assess both the current and future health of a product (for example, maintenance schedule changes); and
  • Strategic Sustainment Analytics (SSA)—high-level health indicators used for cross-product comparison of performance (for example, DMO expenditure per Materiel Ready Day).
Sustainment Performance Management System (SPMS)

5.9 The SPMS has been developed since September 2012, with a total of $2.85 million in funding allocated for system development by February 2015. The SPMS is to generate dashboards for each sustainment product and reporting cycle, based on current and historical KPI, KHI and SSA data. Separate dashboards are to be designed for three levels of the Defence Organisation—System (DMO CEO/General Manager and Service Chief/Group Head), Division (Division Head/Fleet Commander and Service Deputy Chief), and Product (SPO/Product Operator). The dashboards are intended to assist Defence and the DMO to collaboratively manage current and future capability and sustainment risks.122

5.10 Pilot projects of the SPMF/SPMS were conducted in the maritime and aviation domains during late 2012–early 2013. Defence documentation indicated that, as at July 2014, the system build, training and documentation development were on track to support rollout to the Navy and the DMO’s Maritime Systems Division in November 2014. However, by January 2015, initial training and rollout had been delayed to April–May 2015. Rollout to the remaining Services and DMO Divisions is planned to occur over the following year. Further, Defence intends to progressively link SPMS to existing Defence systems. The aim is to enable complex analysis of financial and other data to identify risks.123

Navy’s new suite of performance measures

5.11 In September 2013, Navy established an MSA Performance Framework Project, with the objective of instituting a standard suite of KPIs and KHIs for its Product Schedules in response to Rizzo Recommendation 12.124 This work was intended to take advantage of the DMO’s development of the SPMS. The new framework and standardised performance measures were endorsed by the Navy Capability Committee on 4 March 2014, and by the Chief of Navy Senior Advisory Committee (CNSAC) on 8 May 2014. Eighty-nine performance measures across the 37 Navy Product Schedules were to be replaced with the standard suite of measures.

5.12 Navy’s standard measures include six KPIs and 15 KHIs (Table 5.1). For each KPI/KHI, Navy sets a tolerance, in terms of an acceptable difference between targeted and actual performance. Most of the measures will be used for each Navy platform. The services and commodities supported by Navy Product Schedules will generally be covered by two of the KPIs, and three or four of the KHIs.

5.13 The KPIs were introduced for Navy Product Schedules from 1 July 2014, and the KHIs will be implemented when the SPMS is rolled out to Navy and the DMO’s Maritime Systems Division, which is now planned for April–May 2015. The KPI data were recorded in the MRS, pending the transition to the SPMS.

Table 5.1: Navy performance measures

Key Performance Indicators

K1: Materiel Ready Days Achievement (for platforms)

The number of Materiel Ready Days achieved compared to the number planned, expressed as a percentage.(a)

K1.1: Service Level Achievement (for services and commodities)

The achievement of agreed service levels/requirements, expressed as a percentage.

K2: Achievement of External Maintenance Period planning milestones

The number of milestone failures per External Maintenance Period Event, and the impact of those failures on successfully achieving the individual External Maintenance Period completion date.

K3: Cost per Materiel Ready Day achieved(b)

A broad measure of the relative cost efficiency of the delivery of Materiel Ready Days. Cost per Materiel Ready Day Achieved is a rolling 12-months average of MSA Cost per Materiel Ready Day achieved over the term of the Usage Upkeep Cycle [maintenance cycle].

The rolling 12-months average cost per Materiel Ready Day is compared to the Cost per Materiel Ready Day projection calculated at the start of the platform operating cycle to identify whether relative cost efficiency is increasing or decreasing over time.

K4: Conformance to Operating Intent

A measure of conformance to operation within the Statement of Operating Intent, with particular focus on Operating Profile, Rate of Effort and Usage Upkeep Cycle.

K5: Price Reliability

The percentage variance of actual year-to-date expenditure versus planned year-to-date expenditure.

K6: Number of Priority 1 and Priority 2 Materiel Deficiency Reports raised

Reported individually for Priority 1 and Priority 2 Materiel Deficiency Reports as a numeric, and presented in graphical trend form over a period reflective of the platform operating cycle.

Key Health Indicators

Maintainability and Sustainability

Positions filled

Funding adequacy over forward estimates period

Open Priority 1 and Priority 2 Materiel Deficiency Reports

Inventory

Demand Satisfaction Rate by Class

External Maintenance Period Demands Supplied in Full (%)

Ship Allowance List (SAL) and Assembly Parts List (APL) Effectiveness

Cannibalisations (Number)

Obsolescence Liability

Maintenance

External Maintenance Period Cost Growth (%)

Organic Maintenance Completion to Plan (%)

External Maintenance Completion to Plan (%)

External Maintenance Effectiveness - Defects post External Maintenance Period

Configuration

Open Deviations – number

Open Configuration Change Proposals – average duration

Configuration Baseline Accuracy

Source: Navy, Navy MSA Performance Framework: Implementation, 26 May 2014, p. 11.

  1. A Materiel Ready Day is any programmed day where a platform is not in an External Maintenance period, undergoing defect repair, in an Extended Readiness, or subject to a Priority 1 Urgent Defect report that because of its nature prevents the ship from achieving its current tasking.
  2. In relation to Cost per Materiel Ready Day achieved, Navy’s intent is that an annual target should be set and trend data captured. Navy also intends to pursue international benchmarking, with Navy/DMO developing skills to better understand cost drivers and performance in comparison to international benchmarks.
ANAO assessment of Navy performance measures

5.14 In 2012–13, the ANAO developed a set of three criteria to evaluate the appropriateness of entity KPIs.125 The criteria examine whether KPIs are relevant (focused and understandable), reliable (measurable and free from bias) and complete (balanced and collective). While devised in the context of the Australian Government’s Outcomes and Programs Framework, the criteria may also usefully inform an assessment of other KPIs. The ANAO applied the criteria to Navy’s performance measures.

5.15 Overall, Navy’s KPIs/KHIs are appropriately designed and measurable. The KPIs provide balanced and usable coverage of Navy’s sustainment products in terms of availability, cost, schedule, and materiel deficiencies. The KPIs addressing Cost per Materiel Ready Day and Price Reliability are particularly noteworthy, and should generate better information on the DMO’s financial performance. Further, Navy’s KHIs address key factors affecting sustainment performance, including staffing levels, inventory management, maintenance performance and configuration changes.

5.16 While Navy’s performance measures are a step forward, there remain areas for improvement. None of Navy’s KPIs address outcomes, which is one of the four types of measures included in the DMO’s guidance. Measures such as Materiel Ready Days Achievement identify availability of platforms, but they do not indicate whether platforms were available when needed for operations. Further, some of the measures are not necessarily free from bias—that is, allowing for clear interpretation of results. The KPIs for Materiel Ready Days Achievement and Service Level Achievement are expressed in percentage terms against a plan or target, and changes in plans or targets may mask deteriorating performance. In presenting performance using these KPIs, Navy should be transparent about any changes in plans or targets.

5.17 Since the transition to the standardised system of KPIs in 2014, there has only been one KPI addressing Navy’s performance—Conformance to Operating Intent. In contrast, there were as many as seven KPIs of Navy’s contribution to sustainment in 2006, which addressed matters such as timely delivery of platforms to the DMO, timely notification of defects or damage to assets, and completion of mandatory organic-level maintenance. Some of these measures are now addressed by KHIs rather than KPIs.

ANZAC fleet KPIs

5.18 The ANZAC fleet Product Schedule transitioned to the new Navy KPI framework in July 2014. There are five KPIs addressing the DMO’s sustainment performance: Materiel Ready Days Achievement, External Maintenance Period Milestone Achievement, Cost Per Materiel Ready Day Achieved, Price Reliability and Priority 1 Materiel Deficiency Reports Raised.

5.19 Materiel Ready Days Achievement is an output measure. It is measured as a percentage achieved against planned days, and reflects the reliability of the Mission System and Support System. Figure 5.1 shows that the ANZAC fleet has been close to achieving its target for Materiel Ready Days over the last three financial years. However, Navy has varied the targeted number of Materiel Ready Days for the ANZAC fleet, and the KPI does not shed light on this reduction. The target was reduced by 20 per cent in 2012–13 and another 13 per cent in 2013–14, and then increased by 3 per cent in 2014–15.

Figure 5.1: ANZAC fleet achievement of Materiel Ready Days, as a percentage of target, June 2011 to January 2015

Source: ANZAC SPO.

Note: As at January 2015, a delay in HMAS Parramatta’s ASMD upgrade was inflating performance beyond planned levels, because the ship was materiel ready for longer than originally expected.

Army’s new Measures of Success framework

5.20 Army has independently developed a trial performance framework known as Measures of Success, and an associated reporting tool, which uses the same software as the DMO’s SPMS.126 The Measures of Success Framework is intended to address two primary goals: outline the required outcomes or performance for a specific aspect of a Product Schedule; and generate information to shape future actions relating to Product Schedule management.

5.21 The Measures of Success Framework is intended to provide Army Product Schedule drafters with a standardised suite of performance measures from which to select appropriate measures for a particular product. Once implemented, it will establish a hierarchy of measures, as shown in Table 5.2. The framework outlines specific measurement techniques for each measure.

Table 5.2: Army’s Measures of Success

Type of Measure

Examples

Critical Success Factors (4)

(to be applied to every Product Schedule)

  • Provision of sustainment activities to support preparedness
  • Effective and efficient financial planning
  • Ensuring security and diversity of suppliers
  • Communication and compliance with governance and reporting requirements

Defence Output Measures (10)

(Defence output for mandated government tasks; not subject to regular change)

  • Availability of equipment meets specified targets
  • Supply support satisfies demand
  • Utilisation of assets is appropriate to support capability with an agreed rate of effort

Health Indicators (31)

(measure of a particular aspect of performance)

  • Operational availability to meet designated requirements
  • Technical integrity
  • Actual versus planned Rate of Effort

Source: Army, Army Product Schedule—Measures of Success, version 0.2, March 2013.

Note: The 41 measures (10 Defence Output Measures and 31 Health Indicators) that support the Critical Success Factors comprise 21 that address DMO performance, 16 that address Army performance, and four that address shared performance.

5.22 Army’s different types of measures address outcomes, outputs and inputs. However, they are presented in a way that differs from the DMO’s guidance; for example, Army’s ‘Health Indicators’ are more akin to outcome measures. Ideally, the Services’ KPI frameworks would apply similar terminology to support Defence senior leadership and ministerial review.

5.23 From July 2013 to June 2014, Army conducted a pilot program of the Measures of Success Framework and the associated reporting tool. The primary objective of the pilot was to develop a management and reporting knowledge domain to monitor, control and improve how the Lead Capability Manager, Supported Capability Managers and the DMO collaboratively manage a Product Schedule. The pilot was conducted across seven Product Schedules in four DMO Divisions. Defence informed the ANAO in March 2015 that Army had developed a formal report on the pilot for submission to the Chief of Army and formal endorsement.

5.24 Army’s intent is to apply the Measures of Success Framework to all Product Schedules when the SPMS is implemented. In the meantime, Army’s existing KPIs, with minor amendments, will continue to be used. During 2013–14, Army and the DMO were working to bring together the separate development streams of Army’s Measures of Success Framework and reporting tool, and the DMO’s SPMS. The DMO intends to roll out the SPMS to Army late in 2015 incorporating the Measures of Success Framework.

Bushmaster fleet KPIs

5.25 Until January 2014, the main Bushmaster sustainment KPI was Operational Availability. Performance against a targeted level of Operational Availability was assessed using a traffic-light system: set at 70 per cent or above (Green), 60–70 per cent (Amber), and less than 60 per cent (Red). From February 2014, this KPI measure was changed to the ‘more appropriate’ metric of Functional Equipment—which measures a combination of Fully Functional and Restricted Use.127 The new measure, in the DMO’s view, better accounts for assets that remain taskworthy, albeit within certain operational parameters or restrictions.

5.26 The largest of the four Bushmaster fleets belongs to Army’s Forces Command, and Figure 5.2 shows the achievement of the main Bushmaster KPI for Forces Command since July 2011. The figure indicates better performance following the change in KPI from Operational Availability to Functional Equipment. This again highlights the importance of KPI measurement techniques and maintaining transparency about these to inform the pursuit of better performance and lower costs over time.

Figure 5.2: Operational Availability/Functional Equipment KPI for Bushmasters, Army’s Forces Command, 2011–14

Source: ANAO analysis of DMO data.

5.27 Since 2010, no KPIs have addressed Army’s performance in relation to sustainment of the Bushmaster fleet. The Measures of Success framework that will be applied to the fleet includes some Army-related measures.

Air Force

5.28 The Air Force Capability Management Manual (2012) provides guidance for developing performance measures for Air Force Product Schedules. The manual notes that the Capability Manager Representative should consider three distinct sets of performance measures, which are related to organisational responsibility: DMO only; Air Force only; and shared.

5.29 The Manual suggests that KPIs cover at least three of the following criteria: availability; reliability; sustainability; and maintainability. The Manual advises that the KPIs should: cover all the required outcomes; be limited in number (three to five being viewed as an appropriate amount); be objective rather than subjective measures where appropriate; strike the right balance between being tailored to the Product and standardised across Air Force; and be specific, measurable, achievable, relevant and time-based (SMART). The Manual also discusses risks associated with the use of common KPIs, and mitigation strategies.128 As indicated in paragraph 5.10, the SPMS is expected to be rolled out to Air Force in 2015–16.

Orion fleet KPIs

5.30 The Orion Product Schedule contains seven KPIs that address rate of effort and the DMO’s performance in relation to the availability of the aircraft, and the simulators and analytical equipment that support them. There is also one shared KPI—a measure of aircraft serviceability. Three of the KPIs are discussed below, highlighting the potential value in a set of appropriate KPIs, in terms of exploring sustainment performance and identifying issues.

5.31 Rate of Effort has been identified as a critical performance metric in several iterations of the Orion Product Schedule. Figure 5.4 shows the forecast annual Rate of Effort versus the actual Rate of Effort achieved. The figure shows that actual flying hours have generally been close to forecast flying hours, with a low of 91.02 per cent of forecast hours flown in 2005–06, and a high of 103.64 per cent of forecast hours flown in 2010–11.

Figure 5.3: An AP-3C Orion conducts Air Sea Rescue Kit training off the South Australian coast, October 2013

Source: Australian Defence Image Library.

Figure 5.4: Orion fleet rate of effort (forecast and actual), 2004–14

Source: ANAO analysis of Defence data.

Note: The DMO informed the ANAO that the forecast of 7900 flying hours for 2012–13 was revised to 7100 at the Mid-Year Review, due to a substantial reduction in funding.

5.32 Aircraft availability is a measure of the number of aircraft that have completed their SPO-level maintenance and have been given to the Air Force Wing that is operating them. Figure 5.5 shows the actual availability of the Orion fleet as a percentage of the targeted availability. In the three financial years from July 2011, aircraft availability as a percentage of the target averaged 97.05 per cent.

Figure 5.5: Orion fleet availability, as a percentage of target, 2011–14

Source: ANAO analysis of Defence data.

5.33 Serviceable aircraft are aircraft that have been made available to the Operational Maintenance Unit at Air Force’s 92 Wing to be rendered capable of flying missions. The target number of serviceable aircraft for this KPI has not been met since July 2011, even though the target was reduced in 2014. However, Defence informed the ANAO that the Orion aircraft are meeting their tasking requirements despite the serviceability target not being met.

Progress of sustainment performance measures

5.34 In terms of activities and processes, Defence has in recent years made a significant effort to establish new sustainment KPI frameworks and performance management systems, with relevant activity occurring across the Services and the DMO. Further, Defence records show that Navy has commenced reporting against its new set of KPIs, and Army has conducted a pilot of its Measures of Success.

5.35 On the whole, the new performance frameworks provide a firmer basis for the evaluation of Defence’s sustainment performance. However, there remain some inconsistencies in the definitions of measures adopted by the different Services. For example, while Navy has adopted Price Reliability as a KPI, Army has adopted a similar measure as a KHI.

5.36 The establishment of new DMO, Navy and Army performance measurement frameworks remains at an early stage. It is only when new measures have been reported against for some time that their usefulness will be tested, and any need for refinement can be assessed.

Transparency of external reporting

5.37 There is strong parliamentary interest in Defence’s public reporting of its sustainment performance, and two parliamentary committees have recently made recommendations and sought improvements in this area.129 The level of parliamentary interest reflects the importance of the state of readiness of Australia’s military capabilities, and the size of Defence’s sustainment budget, at over $7.1 billion. The ANAO reviewed Defence’s public sustainment reporting in light of other countries’ reporting on sustainment activities.

5.38 The DMO Portfolio Budget Statements (PBS) now include a brief description of sustainment arrangements for the Top 30 sustainment products by cost (Top 20 in 2012–13 and earlier years), and an outline of the focus of sustainment efforts for the budget year.

5.39 In its 2014–15 PBS, the DMO stated that its KPIs for Programme 1.2, Management of Capability Sustainment, vary with each sustainment product and are specified in the MSAs.130 However, the MSAs are not public documents. In the Defence PBS, each of the Services describes its deliverables related to capability performance, including Navy Unit Availability Days and Army and Air Force aircraft rate of effort. These measures are then reported against in Defence’s Annual Report.131

5.40 Defence also reports financial information for the Top 30 sustainment products in its Annual Report, namely the Budget estimate; the revised estimate; actual expenditure; the variation between the revised estimate and the actual expenditure; and an explanation of the variation. Table 5.3 shows three examples of the financial reporting on sustainment from Defence’s latest Annual Report.

Table 5.3: Sustainment financial reporting

Product

Budget estimate 2013–14 ($m)

Revised estimate 2013–14 ($m)

Actual spend 2013–14 ($m)

Variance ($m)

Reason for significant variation

Anzac Class Frigate

224

250

263

13

The variation is due to the increased costs to remediate ships entering refit as a result of their poor materiel state.

P-3C/ AP-3C Orion Weapons System

110

113

108

-5

The variation is due to lower than anticipated engine failure rates and lower than anticipated costs in transitioning to a new engine maintenance contract.

Fuels and Lubricants – Air Force, Army and Navy

507

493

520

27

The variation is due to increases to fuel prices, increased requirements by Air Force and Navy during Qtrs 3 and 4 2013–14, and the replenishment of fuel holdings to meet heightened operational activity.

Source: Department of Defence, Annual Report 2013–14, supplementary information for Chapter 6 [Internet], available from <http://www.defence.gov.au/AnnualReports/13-14/Supplementary Chapter6.pdf> [accessed November 2014], pp. 32–3.

Note: The Bushmaster is not among the Top 30 sustainment products, and so is not included in the table.

5.41 Defence also presents descriptive information on sustainment performance for the Top 30 sustainment products.132 Table 5.4 shows three examples of the descriptive reporting on sustainment from Defence’s latest Annual Report.

Table 5.4: Sustainment performance reporting

Product

Performance summary

Anzac Class Frigate (CN02)

Planned outcomes were the provision of continuing sustainment of materiel capability to meet the Navy’s operational requirements, the continued implementation of the Anzac class group maintenance contract, continued inventory management reforms and the continuation of anti-ship missile defence refit work on the designated ships under Project SEA 1448 Phases 2A and 2B.

Planned maintenance was completed, including additional pre-anti-ship missile defence work in HMAS Warramunga. Materiel support to capability was provided to the Anzac class frigates undertaking activities associated with operations Slipper, Sovereign Borders and Southern Indian Ocean. HMAS Arunta is undergoing sea trials following the completion of an anti-ship missile defence upgrade and refit program.

P-3C/AP-3C Orion Weapons System (CAF04)

The P-3 capability includes 17 aircraft and a range of ground-based systems. The capability is supported through the P-3 Accord and a range of commercial and foreign military support arrangements.

The fleet continues to be maintained under the more resource-intensive ‘safety-by-inspection’ program, comprising additional targeted structural inspections, repairs and/or structural element replacements. Boeing Defence Australia completed the P-3 aircraft repaint program in June 2014.

Fuels and Lubricants—Navy, Army and Air Force (CN26, CA43, CAF18)

Petrol, oil and lubricant products are procured under long-term contracts and provided to Defence operational and support elements and visiting foreign forces. The Fuels Technical Regulatory and Quality Control Framework is maintained for the conduct of Services’ operations and technical data integrity.

Provision of these products was completed to meet requirements. Under the implementation of the Wraith Review of the petroleum supply chain, Joint Logistics Command has assumed control of the fuel supply chain for the ADF. The DMO’s Joint Fuels and Lubricants Agency will now formally transfer to Joint Logistics Command.

Source: Department of Defence, Annual Report 2013–14, vol. 1, pp. 100, 105.

Note: The Bushmaster is not among the Top 30 sustainment products, and so is not included in the table.

5.42 The information in Defence’s Annual Report provides stakeholders with a basic summary of sustainment costs and activity. However, it does not facilitate any systematic assessment or analysis of Defence’s sustainment performance, and there may also be scope for additional standardisation and improved data consistency.

5.43 Defence informed the ANAO in March 2015 that, overall, reporting in the PBS and the Defence Annual Report needs to be carefully managed to avoid the disclosure of classified or other information.

Overseas public reporting of sustainment expenditure

5.44 In reviewing Defence’s public reporting on sustainment, the ANAO considered the extent of reporting on sustainment expenditure in Canada, New Zealand, the United Kingdom and the United States (Table 5.5). The table only considers sustainment-related reporting, and it does not address the overall quality of the countries’ reporting on defence.

Table 5.5: Overseas budgeting and reporting of defence sustainment expenditure

Overseas budgeting and reporting of defence sustainment expenditure

Canada

Budgeting

The Department of National Defence Report on Plans and Priorities 2014–15 (153 pages) was almost twice the size of the previous year (78 pages), partially in response to parliamentary requests. It also implemented a revised Outcomes and Programs framework for defence. It has one-page descriptions of ‘Engineering, Test, Production and Maintenance’ and ‘Inventory Management and Distribution’, including the overall budget and personnel numbers.

Reporting

The Department of National Defence Departmental Performance Report 2013–14 (November 2014) contains one-page descriptions of Infrastructure Maintenance and Equipment Maintenance in the maritime, land, aerospace and common spheres. Financial reporting remains at this level, and does not give detail at the platform or product level.

Summary

Public information on defence sustainment expenditure is brief and unspecific.

New Zealand

Budgeting

The New Zealand Defence Force is primarily responsible for managing capability while it is in service.a The Defence Force Estimates of Appropriations 2014/15 implemented a new structure under which total expenditure is identified for each Service, but is not broken down further. Capital expenditure for maintaining and upgrading capabilities is also identified only as a total amount.

New Zealand (continued)

Reporting

The New Zealand Defence Force 2013–2014 Annual Report was developed under the previous, more detailed, reporting structure. It identified expenditure at a lower level, such as Fixed Wing Transport Forces or Naval Patrol Forces. For each of these outputs, there was a two-page narrative, detailed performance reporting, and a breakdown of expenses, notably into personnel and operating costs. This did not enable specific sustainment expenditure to be identified.

Summary

Public information on defence sustainment expenditure is brief and unspecific.

United Kingdom

Budgeting

The Main Supply Estimates (similar to the Australian Government’s Budget Paper No. 4) include some 15 pages on the Ministry of Defence, and identify four line-items related to sustainment (Inventory Consumption and Equipment Support Costs for Capability and for Operations). There is no itemised budgeting for the Services.

The annual Defence Equipment Plan (30 pages in 2014) outlines expected expenditure over the next 10 years, showing amounts for support of new equipment and in-service equipment by broad category, such as air support, helicopters and ships.

Reporting

The Ministry of Defence Annual Report and Accounts 2013–2014 records the expenditure against the four line-items mentioned above. The four pages on the activities of the Defence Equipment and Support organisationb include only two brief mentions of sustainment activities.

The Departmental Resources 2013 statistical bulletin has a one-page breakdown of defence expenditure by commodity block (including Inventory Consumption and Equipment Support Costs) and a one-page breakdown of Estimated Equipment Expenditure in three categories: capital, support, and research and development.

The UK National Audit Office publishes an annual Major Projects Report on defence acquisition. The head of the UK National Audit Office observed in February 2014 that the Ministry of Defence ‘has not subjected the half of its equipment budget related to support to the same degree of scrutiny as its procurement costs.’c

Summary

Public information on defence sustainment expenditure is brief and unspecific.

United States of America

Budgeting

Sustainment is covered by the major Defense appropriation ‘Operation and Maintenance’.d The Department of Defense itemises proposed expenditure under this appropriation by Service, Budget Activity (4), Activity Group and Subactivity Group (line-item), such as Navy—Operating Forces—Ship Operations—Ship Maintenance.e Other examples of line-items are: Land Forces Systems Readiness; Land Forces Depot Maintenance; and In-Service Weapons Systems Support. The Department publishes a one to four-page overview at the Budget Activity level, describing the activity, program growth and major program changes. A departmental Data Book (the ‘Green Book’) gives actual and inflation-adjusted expenditure, personnel numbers, pay rises, etc as far back as 1945.

Each Service publishes its own overview of Operation and Maintenance, including summaries of personnel and of increases/decreases by line-item. Further, in a detailed outline of each line-item, some 10 pages describe: the operations financed; the force structure (such as number of ship overhauls, or cost and number of aircraft, vehicles or missiles to be maintained); detail of increases and decreases, including transfers between line-items; performance criteria (such as budget and numbers of overhauls); personnel details by Service; and a breakdown of expenditure under the line-item into classes (such as wages; transport; travel; supplies; rents; maintenance by contract; and facilities), with price and program changes from the previous financial year.

The Operation and Maintenance Data Book for each Service includes a significant amount of detailed information about the Service’s sustainment. For example, the US Navy, for its four shipyards, publishes several pages of financial and workforce data, including expenditure by platform. The Navy also publishes the maintenance schedules for individual vessels, including planned and actual start and end dates for aircraft carriers, nuclear submarines and frigates at these shipyards, over the last four years and the coming year. Similarly, the US Air Force publishes financial details of the Depot Maintenance appropriation by platform (though not by number of aircraft).

Procurement of weapons systems is itemised by Service, number to be acquired, and cost per platform.

Reporting

The Department of Defense Agency Financial Report Fiscal Year 2014 does not include a breakdown of sustainment costs, either by platform or by Service. It gives (at p. 43) a figure of US$272.780 billion for ‘Operations, Readiness & Support’ (and US$98.488 billion for ‘Procurement’).

A quarterly Operation and Maintenance Budget Execution report records expenditure to date by Service, Budget Activity and line-item.

United States of America (continued)

The Services’ Agency Financial Reports include a very brief overview and breakdown of Operations and Maintenance activity.

Selected Acquisition Reports provide annual updates on the development and procurement of major weapon systems.

Summary

Detailed sustainment budgeting and reporting occurs, mostly in the context of the United States Budget.

Source: ANAO analysis.

Note a: A separate appropriation is made for the New Zealand Ministry of Defence, which has primary accountability for defining what capability is needed to meet government objectives, and purchasing, replacing or upgrading major defence equipment. The Ministry of Defence has its own annual report.

Note b: The Defence Equipment and Support organisation became a bespoke trading entity within the Ministry of Defence from 1 April 2014, and will in future publish its own annual report, including a detailed financial statement. Its annual accounts will be consolidated with those of the Ministry of Defence.

Note c: National Audit Office, The Major Projects Report 2013 [Internet], London, 13 February 2014, available from http://www.nao.org.uk/report/ministry-of-defence-the-major-projects-report-2013/ [accessed December 2014].

Note d: The other major Defense appropriations are: Military Personnel; Procurement; Research, Development, Test and Evaluation (RDTE); Military Construction; and Family Housing. Overseas operations are appropriated separately.

Note e: The Budget Activity most relevant to sustainment is Operating Forces, with 116 line-items across Army, Navy and Air Force in Fiscal Year 2015. However, the other three Operation and Maintenance Budget Activities—Mobilization (15 line-items), Training and Recruiting (50 line-items) and Administration and Service-wide Activities (89 line-items)—also contain elements of sustainment funding, such as Depot Maintenance (under Air Force Mobilization and Air Force Training and Recruiting) and Planning, Engineering and Design (under Navy Administration and Service-wide Support).

5.45 The United Kingdom publishes very little detail on its defence sustainment expenditure, and the United Kingdom National Audit Office has observed that the equipment support budget has not received the same degree of scrutiny as the procurement budget. The United States Government spends some US$272 billion on ‘Operations, Readiness & Support’, compared to Defence’s $6 billion expenditure on sustainment. The operating context of the two countries also differs markedly. That said, the United States reports sustainment expenditure down to detailed line-items such as Ship Maintenance costs, as well as maintenance schedules for individual vessels. Australia has only a small number of platforms, and specific details on maintenance of any platform could potentially reveal details of capability.

Joint Committee of Public Accounts and Audit concerns

5.46 In March 2013, the Joint Committee of Public Accounts and Audit (JCPAA) questioned the DMO about sustainment reporting, and asked whether there were any reasons why an annual consolidated report on sustainment could not be compiled, in a similar vein to the annual Major Projects Report on significant Defence acquisition projects. The DMO responded that a consolidated sustainment report would be classified:

Where we run into highly sensitive matters in the sustainment area is that most of the performance metrics that we have for sustainment are against the [Chief of the Defence Force’s] preparedness directive, which is classified. If we were to do something similar in sustainment in terms of assessing performance against measures in a public fashion, it would be classified. […] If we were to put a consolidated report together of how we are performing against all the sustainment activities, that would give you a public indication of preparedness, which is something we cannot do because it is classified.133

5.47 In May 2013, the DMO undertook to increase its annual reporting to cover the Top 30 sustainment products (around 77 per cent of current spending on sustainment). This is consistent with the number of projects examined in the Major Projects Report and the Top 30 Major Capital Projects disclosed in the Defence Portfolio Budget Statement and Annual Report.134 The JCPAA welcomed the move to increase coverage, but noted that, given the increasingly large sum of public money devoted to sustainment, more needed to be done to increase the depth of sustainment reporting, not just the number of products covered.135

5.48 In its May 2014 review of the 2012–13 Major Projects Report, the JCPAA observed that the total budget for sustainment (over the next 10 years, the period covered by the Defence Management and Financial Plan) was not published, depriving the Major Projects Report of necessary context. The JCPAA recommended that the DMO should prepare, within six months, a suitable and separate methodology for reporting sustainment activity and expenditure. As discussed in paragraph 4.12, the JCPAA also expressed concern about the financial and budgetary separation between acquisition and sustainment.136

5.49 Defence responded to the JCPAA’s May 2014 request for a separate sustainment reporting methodology in December 2014, advising the Parliament that:

Defence’s position is that the current arrangements of Portfolio Budget Statements (PBS) and Defence Annual Report (DAR) reporting to Parliament, and Preparedness reporting to Government, balance effectively the obligation to allow Parliamentary scrutiny of the expenditure of Commonwealth funds on sustainment efforts, while protecting the classified information on capability readiness and availability which is associated with those sustainment efforts and which is separately provided to Government.137

5.50 In September 2014, the JCPAA also requested that the ANAO develop an options paper on sustainment reporting, and review other international work in this area. The ANAO provided an options paper to the JCPAA in February 2015, developed in consultation with Defence, and identifying four options for increasing the transparency of sustainment expenditure.138 The four options, which could be considered in isolation or in combination by the JCPAA, were:

  • Option 1: an annual in camera briefing for the JCPAA on Defence sustainment;
  • Option 2: expansion of sustainment reporting in the Defence Annual Report;
  • Option 3: expansion of the Major Projects Report to include further sustainment reporting; and
  • Option 4: development of a new sustainment report and limited assurance review.

5.51 In relation to the Defence Annual Report, the options paper observed that:

In addition to the reporting currently appearing in the Defence Annual Report, the new suites of sustainment Key Performance Indicators being developed by Defence for Materiel Sustainment Agreements offer opportunities for improved reporting. There may be merit in Defence disclosing against a small number of Key Performance Indicators for the Top 30 sustainment products, through measures that do not pose a [security] classification risk, e.g. measures of financial performance against estimates, performance as a percentage of a target, or Cost per Materiel [Ready] Day Achieved. The DMO could also report on other financial information in the Defence Annual Report, including measures such as cost growth across the capability life-cycle.139

5.52 The issue remains under consideration by the JCPAA at the time of preparation of this report.

Joint Foreign Affairs, Defence and Trade Committee concerns

5.53 The Joint Standing Committee on Foreign Affairs, Defence and Trade has also examined Defence’s annual reporting. In June 2013, the Standing Committee recommended that Defence enhance its public reporting by developing a more precise method for reporting capability acquisition and sustainment performance, including specific performance targets, how performance is assessed in relation to these targets, and the specific reasons why targets are, or are not, achieved.140

5.54 In October 2014, Defence agreed in principle to this recommendation, and advised the Standing Committee that it was expanding its reporting to cover the Top 30 sustainment products.141 However, the descriptive nature of Defence’s sustainment reporting (see Table 5.4) shows that Defence still has some way to go before it meets the intent of the recommendation.

Conclusion

5.55 Between 2012 and 2014, the DMO, Navy and Army developed new performance measurement frameworks, including measures of availability, cost, schedule, and materiel deficiencies, which are to be reported through a new DMO system.142 These performance measures will not be fully implemented until the DMO system is operational, and at the time of the audit, the first phase of the system rollout was scheduled for May 2015. While the new performance measures should provide a firmer basis for the evaluation and active management of sustainment performance and costs, their establishment remains at an early stage.

5.56 Defence’s annual reporting on sustainment includes budget and expenditure data for the Top 30 sustainment products (representing some 77 per cent of current spending on sustainment), as well as an overview of the management of these products. While providing stakeholders with a basic summary of sustainment costs and activity, this information does not facilitate assessment of Defence and the DMO’s sustainment performance in terms of materiel availability, cost-effectiveness and key inputs such as inventory management, maintenance and configuration changes. Defence still has some way to go before it meets the intent of the Joint Standing Committee on Foreign Affairs, Defence and Trade’s recommendation for enhanced public reporting. Following a request by the JCPAA, in February 2015 the ANAO provided the JCPAA with options, developed in consultation with Defence, to enhance sustainment reporting to the Government and Parliament. The issue remains under consideration by the JCPAA at the time of preparation of this report.

Appendices

Appendices

Please refer to the attached PDF for the Appendices:

  • Appendix 1: Entity Response
  • Appendix 2: Sustainment Products and Budgets

Abbreviations

ADF

Australian Defence Force

ASMD

ANZAC fleet Anti-Ship Missile Defence project

DMFP

Defence Management and Financial Plan

DMO

Defence Materiel Organisation

JCPAA

Joint Committee of Public Accounts and Audit

KHI

Key Health Indicator

KPI

Key Performance Indicator

MAA

Materiel Acquisition Agreement

MRS

Monthly Reporting System

MSA

Materiel Sustainment Agreement

NPOC

Net Personnel and Operating Costs

PAES

Portfolio Additional Estimates Statements

PBS

Portfolio Budget Statements

SPMF

Sustainment Performance Management Framework

SPMS

Sustainment Performance Management System

SPO

Systems Program Office

Footnotes

1 In April 2015, the Government announced that it had accepted the recommendation of a First Principles Review of the Defence organisation to disband the DMO and transfer its core responsibilities to a Group within Defence. This audit focuses on the relationship between Capability Managers and Defence’s sustainment arm irrespective of specific organisational arrangements within Defence.

2 Specialist military equipment is an accounting term that includes ships, vehicles and aircraft, and assets under construction.

3 A Capability Manager is responsible for raising, training and sustaining in-service capabilities through the coordination of fundamental inputs to capability. Capability Managers include the Service Chiefs, the Chief of Joint Operations and the Chief Information Officer.

4 Some $5.7 billion of the DMO sustainment budget was composed of funds transferred from Defence (specifically, transferred into the DMO’s Special Account). Australian Government, Portfolio Additional Estimates Statements 2014–15, Defence Portfolio, Canberra, February 2015, p. 145.

5 The DMO’s sustainment expenditure has exceeded its acquisition expenditure in five of the last nine financial years.

6 The acquisition and sustainment function had previously been part of the Department of Defence. The DMO, first established within the Department of Defence in 2000, became a prescribed agency on 1 July 2005, with a separate Chief Executive Officer, financial accounts and annual reporting requirements, but staff provided by the wider Defence Organisation. These general arrangements continued after the FMA Act was replaced by the Public Governance, Performance and Accountability Act 2013 (PGPA Act) on 1 July 2014. The PGPA Act also marked a change from using the term ‘agency’ within the Australian Public Sector to using the term ‘entity’. The DMO’s purposes as a listed entity under the Public Governance, Performance and Accountability Rule 2014 include:

 contributing to the preparedness of the Australian Defence Force and the Department of Defence   through acquisition and through-life support of military equipment and supplies.

7 Department of Defence, Annual Report 2003–04, Canberra, 2004, p. 304; Annual Report 2004–05, Canberra, 2005, p. 261.

8 Department of Defence, DMO Business Model, February 2004, p. 12.

9 Australian Government, Portfolio Additional Estimates Statements 2014–15, Defence Portfolio, Canberra, February 2015, p. 145.

10 The funding levels outlined here include funding for operations ($315 million) and expected Net Personnel and Operating Costs ($85 million).

11 MSA Product Schedules are identified by a prefix denoting the relevant Capability Manager—for instance, Chief of Navy (CN), Chief of Army (CA) and Chief of Air Force (CAF)—and a running number. A list of Product Schedules is included at Appendix 2.

12 The Rizzo Report responded to Navy’s inability to supply vessels requested in February 2011 by the then Government to assist in the clean-up after Cyclone Yasi.

13 Including $487 million for the DMO’s workforce and operating expenses.

14 At the time of the audit, the Defence First Principles Review was under way to ensure that Defence is fit for purpose and able to respond promptly to future challenges. Part of the review’s task was to examine the benefits and costs of different approaches to reforming the DMO. The review was released on 1 April 2015, and the Government accepted its recommendation to disband the DMO and transfer its core responsibilities in relation to capability delivery to a new Capability Acquisition and Sustainment Group. See http://www.defence.gov.au/Publications/Reviews/FirstPrinciples/.

15 ANAO Audit Report No.41 2009–10, Effective Cross-Agency Agreements, p. 54.

16 Ibid., pp. 50–1, 63.

17 See footnote 12.

18 The June 2010 Product Schedule for HMAS Manoora and HMAS Kanimbla (two of the ships whose materiel condition prompted the Rizzo Report) had identified as almost certain the risk that these ships were operating beyond their service life, with severe consequences for performance. The Product Schedule also noted that the configuration and maintenance data was inadequate to comply with the RAN Maintenance System, and that the materiel support regime for these two vessels was by then almost wholly reactive, with the bulk of maintenance being effected in response to breakdown events. However, the DMO and Navy did not effectively mitigate the risks, and consequently were unable to meet requirements of the then Government for use of the vessels.

19 Defence informed the ANAO in March 2015 that:

There are instances […] where as part of a Project closure process there may be remaining activities that were in scope under the [Materiel Acquisition Agreement (MAA)] and are transferred to a relevant Product Schedule under an MSA. In these instances it is clearly documented and agreed by all signatories to the MAA and MSA Product Schedule Change Proposal that Defence is agreeing that the activity funded initially through the MAA can now be transferred to the MSA.

20 See paragraph 4.

21Going to the Next Level. The Report of the Defence Procurement and Sustainment Review [Mortimer Review], Defence Materiel Organisation, Canberra, September 2008, p. 49.

22 Navy’s performance measures include KPIs such as ‘Materiel Ready Days’, ‘Cost per Materiel Ready Day’ and ‘Price Reliability’; and Key Health Indicators such as Demand Satisfaction Rates. Navy commenced reporting against its new suite of KPIs in July 2014, and is to report against Key Health Indicators from mid-2015 when the DMO’s sustainment reporting system is implemented for Navy products.

23 In April 2015, the Government announced that it had accepted the recommendation of a First Principles Review of the Defence organisation to disband the DMO and transfer its core responsibilities to a Group within Defence. This audit focuses on the relationship between Capability Managers and Defence’s sustainment arm irrespective of specific organisational arrangements within Defence.

24 Specialist military equipment is an accounting term that includes ships, vehicles and aircraft, and assets under construction.

25 A Capability Manager is responsible for raising, training and sustaining in-service capabilities through the coordination of fundamental inputs to capability. Capability Managers include the Service Chiefs, the Chief of Joint Operations and the Chief Information Officer.

26 Some $5.7 billion of the DMO sustainment budget was composed of funds transferred from Defence (specifically, transferred into the DMO’s Special Account). Australian Government, Portfolio Additional Estimates Statements 2014–15, Defence Portfolio, Canberra, February 2015, p. 145.

27 Department of Defence, Annual Report 2013–14, Canberra, 2014, vol. 2, p. 12.

 In December 2012, the Australian Accounting Standards Board granted Defence a two-year extension (until 30 June 2015) for implementing fair-value valuation of Defence weapons platforms (a subset of specialist military equipment), under new whole-of-government accounting standards (AASB 1049). The value reported for specialist military equipment in 2013–14 was assessed on a cost basis.

28 Department of Defence, Annual Report 2000–01, Canberra, 2001, p. 280.

29 Department of Defence, Annual Report 2003–04, Canberra, 2004, pp. 19, 304; Annual Report 2004–05, Canberra, 2005, p. 261.

30 These general arrangements continued after the FMA Act was replaced by the Public Governance, Performance and Accountability Act 2013 (PGPA Act) on 1 July 2014. The PGPA Act also marked a change from using the term ‘agency’ within the Australian Public Sector to using the term ‘entity’. The DMO’s purposes as a listed entity under the Public Governance, Performance and Accountability Rule 2014 include:

  contributing to the preparedness of the Australian Defence Force and the Department of Defence   through acquisition and through-life support of military equipment and supplies.

31 It is not legally possible for an Australian Government entity to enter into a contract with another Australian Government entity.

32 Australian Government, Portfolio Budget Statements 2014–15, Budget Related Paper No.1.4A, Defence Portfolio, Canberra, May 2014, Glossary, p. 226.

 A parallel framework of Materiel Acquisition Agreements (MAAs) covers the DMO’s acquisition services to Defence for both major and minor capital equipment.

33 Australian Government, Portfolio Additional Estimates Statements 2014–15, Defence Portfolio, Canberra, February 2015, p. 145.

34 MSA Product Schedules are identified by a prefix denoting the relevant Capability Manager—for instance, Chief of Navy (CN), Chief of Army (CA) and Chief of Air Force (CAF)—and a running number. A list of Product Schedules is included at Appendix 2.

35 The Rizzo Report responded to Navy’s inability to supply vessels requested in February 2011 by the then Government to assist in the clean-up after Cyclone Yasi.

36 ANAO Audit Report No.41 2009–10, Effective Cross-Agency Agreements.

37 ANAO Better Practice Guide, Public Sector Governance: Strengthening Performance through Good Governance, Canberra, June 2014, p. 59.

38Going to the Next Level. The Report of the Defence Procurement and Sustainment Review [Mortimer Review], Defence Materiel Organisation, Canberra, September 2008, pp. 47–8.

39 The Memorandum of Arrangements takes precedence over MSAs in delineating the functions and responsibilities of Defence and the DMO.

40 Among the changes introduced in 2006 was a move from annual review of the Memorandum of Arrangements to as-required review.

41 The inclusion of a separate line-item for operations funding was another change effected by the 2006 Memorandum of Arrangements. Consideration had been given to establishing a separate agreement to cover operations funding.

42 Department of Defence, The Response to the Report of the Defence Procurement and Sustainment Review, Canberra, May 2009, pp. 12, 38.

43 All MSA Heads of Agreement have contained a list of subordinate Product Schedules. The inclusion of these schedules in the higher-level Memorandum of Arrangements means that the document can become dated relatively quickly.

44 At the time of the audit, a Defence First Principles Review was under way to ensure that Defence is fit for purpose and able to respond promptly to future challenges. Part of the review’s task was to examine the benefits and costs of different approaches to reforming the DMO. The review was released on 1 April 2015, and the Government accepted its recommendation to disband the DMO and transfer its core responsibilities in relation to capability delivery to a new Capability Acquisition and Sustainment Group. See http://www.defence.gov.au/Publications/Reviews/FirstPrinciples/.

45 Since 2005, there has been little change in the number of MSAs, and a small but steady change in the number of Product Schedules, as capabilities come into or are withdrawn from service.

46 Including $487 million for the DMO’s workforce and operating expenses.

47 There are nine other MSA products that provide related items such as weapons, communications and navigation equipment, or medical equipment for the Bushmaster fleet.

48 The MoU documents the guiding principle—that the Lead Capability Manager funds baseline sustainment requirements, while any new requirements that are unique to another Capability Manager are funded by that ‘Supported Capability Manager’. The MoU then outlines the shared responsibilities of Capability Managers, as well as the separate responsibilities of the Lead and Supported Capability Managers, in relation to requirements, funding and communication.

49 ANAO Better Practice Guide, Public Sector Governance: Strengthening Performance through Good Governance, Canberra, June 2014, p. 60.

50 The Navy MSA 2015 states that its purpose is to:

 provide for the sustainment of systems and materiel utilised by Navy in its delivery of capability to: support Navy’s mission to ‘fight and win’ at sea; and support the Commonwealth policy of self-reliance.

 The objectives include providing clarity of Navy materiel requirements, for the whole of life of the capability, and integrated risk assessment to coordinate decision-making in relation to Navy assets.

51 ANAO Audit Report No.41 2009–10, Effective Cross-Agency Agreements, pp. 50–1, 63.

52 A Lean activity is intended to maximise customer value while minimising waste; for more information, see lean.org.

53 ANAO comment: Colonel/Lieutenant Colonel and equivalents.

54 Acting Head, Acquisition and Sustainment Reform Division, DMO, Lean MSA Project Update - July 2012, email to Capability Managers and other senior Defence officials, 6 July 2012.

55 The intent of modularising the Product Schedules was to ‘avoid updating every section every year, highlighting the areas which need to be updated annually and areas which do not necessarily need to change each year. This should reduce staff work involved in preparing the MSAs and focus attention on the important parts of the agreement.’ Sustainment Agreements Working Group, minutes, 17 August 2011, p. 4.

56 Issued under the System of Defence Materiel Instructions, this document, officially referred to as DMSP (A&S) 14-0-010, is referred to in this audit report as the Standard Procedure on MSAs. A number of subsidiary documents were issued with the Standard Procedure: templates for the Heads of Agreement and Product Schedules; guidelines on how to complete them; and a template for Change Proposals.

57 DMO, Management of Materiel Sustainment Agreements including Product Schedules Standard Procedure, 6 November 2012, p. 2.

58 Paul Rizzo, Plan to Reform Support Ship Repair and Management Practices, Department of Defence, Canberra, July 2011.

59 ibid., pp. 41, 46.

60 ibid., pp. 14, 49.

61 During late 2014, the DMO and Navy were preparing a revised Heads of Agreement in response to the appointment of a new Chief of Navy and to incorporate lessons learned from post-Rizzo management arrangements. This revision received final approval on 20 January 2015.

62 As at July 2014, the ANZAC fleet Product Schedule included a total of 120 expenditure amounts for 2014–15, and 240 expenditure amounts for the 10 financial years to 2023–24.

63 As at July 2014, the ANZAC fleet Product Schedule included a total of 182 dates for maintenance of the eight-ship fleet over five years.

64 For discussion of the Rizzo recommendations, see paragraph 2.37.

65 To minimise the administrative effort of preparing multiple reports and scheduling multiple meetings, and to reduce travel costs, where possible Sustainment Assessment Reviews and Capability Sustainment Reviews can be scheduled to occur either concurrently or on a back-to-back basis.

66 The Air Command Board governs Air Force’s Air Command, which has the mission of raising, training and sustaining Air Force’s capability.

67 That is, a Change Proposal that affects all of the Product Schedules under a particular Heads of Agreement.

68 An Army Logistics Working Group was established in 2012 to provide a twice-yearly forum for Army’s senior logistics officers to discuss, network and build consensus in resolving key issues affecting Army’s logistics community. Defence records suggest that the Working Group met twice in 2012, once in 2013, and has not met in 2014.

69 Gate Reviews are an internal DMO assurance process that involves a periodic assessment of a project, at key milestones during a project’s lifecycle, by a DMO‐appointed Gate Review Assurance Board. Previously, the DMO did not generally conduct Gate Reviews during the sustainment phase of a capability.

70 The DMO has been using an Acquisition Category (ACAT) framework since 2004, and categories have been published in the Defence Capability Plan since 2009, following a recommendation of the Mortimer Review. The Materiel Sustainment Category (MSCAT) framework has been in use since at least 2007, and uses a similar methodology to the ACAT framework. For example, as at September 2014, MSCAT I included three Product Schedules: the Hobart Class Air Warfare Destroyer, the Canberra Class Landing Helicopter Dock, and the Collins Class submarine.

71 ANAO Report No.14 2014–15, 2013–14 Major Projects Report, p. 87.

72 ANAO Audit Report No.52 2011–12, Gate Reviews for Defence Capital Acquisition Projects, p. 15.

73 The delayed projects were:

  • SEA 1408 Phase 2, Torpedo Self Defence;
  • SEA 500 Phase 3, Remediation of Navy Electronic Warfare; and
  • SEA 1442 Phase 4, Maritime Communications and Information Management.

Footnote continued on next page...

 Defence acquisition projects are not funded to provide compensation when delays in the projects result in higher sustainment costs for in-service capabilities, and Capability Managers must either shift funds from other areas for the relevant sustainment product, or obtain additional funding from the Defence Capability Plan.

74 Under the January 2015 revision of the Navy MSA, these meetings became biannual, and are called Force Element Biannual Progress Reviews.

75 The MSA Product Schedule CA04, Sustainment of Protected Mobility Vehicles, 2012, states (p. 4):

This Product Schedule is of high importance to Army and will undergo a DCA Product Schedule Review once every two Financial Years.

76 Operation Resolute is the ADF’s contribution to the whole-of-government effort to protect Australia’s borders and offshore maritime interests.

77 The service lives of the Orion flight simulator and the Systems Engineering Laboratory, for example, were reduced from 2025 to 2020 as part of the accelerated drawdown of the Orion fleet.

78 In late 2013, the Orion fleet Product Schedule included budget estimates to 2023–24, despite the Planned Withdrawal Date already having been set at 2020. The Orion fleet is to be replaced from 2017 by eight P-8A Poseidon aircraft, as well as a number of MQ-4C Triton Unmanned Aerial Vehicles (UAVs).

79Going to the Next Level. The Report of the Defence Procurement and Sustainment Review [Mortimer Review], Defence Materiel Organisation, Canberra, September 2008, p. 51.

80 ibid., pp. 50–1, Recommendation 4.5.

81 The revised Army–DMO Heads of Agreement signed in September 2014 excluded the table of endorsement delegations, leaving this detail to the Product Schedules.

82 These issues are additional to, but closely associated with, those discussed in the context of delegations, at paragraphs 3.48 to 3.53.

83 Army and Air Force Product Schedules include only one overall line of baseline sustainment funding. In the ANZAC fleet Product Schedule, the 10 line-items in 2014–15 range in value from $13 000 for Sustainment Travel to $88.5 million for Maintenance (Planned).

84 While this approach is practical, it is inconsistent with the Navy–DMO MSA.

 These internal transfers are made in Defence’s Task Management System (TMS), which is used for financial management and reporting, and they do not affect the overall product budget, which is recorded in BORIS. BORIS is only updated if the product’s budget value is increased or decreased and this action requires a journal.

85 Issues, risks and constraints affecting sustainment products are defined as follows:

issues are matters that currently affect the product;

risks are potential concerns to be monitored; and

constraints are factors that limit the ability to achieve or modify the agreed level of sustainment.

86 ANAO Report No.12 2013–14, 2012–13 Major Projects Report, pp. 39–40.

87 The latest ANAO Major Projects Report discusses IMMS in more detail in the context of the DMO’s enterprise risk management framework. ANAO Report No.14 2014–15, 2013–14 Major Projects Report, pp. 35–38.

88 As at June 2014, the Bushmaster fleet numbered 890 vehicles, of which 737 had sustainment funding in place.

89 A Special Account is not a bank account, but in effect a ledger entry against the broader pool of Commonwealth funds in the Consolidated Revenue Fund.

90 In addition, tied payments may be made by Defence to the DMO to support Defence operations through sustainment of a particular product, or to implement government budget measures.

91 On 9 February 2015, the DMO Chief Finance Officer advised his staff that Defence would be ceasing the endorsed funding model of a prepayment to the DMO from 1 July 2015, but that subsequent arrangements were yet to be finalised.

92 Programme 1.3 covers the provision of policy advice and management services.

93 The DMO advised the JCPAA in May 2014 of the transition from acquisition to sustainment:

There is a period when the acquired capability transitions into sustainment. This normally occurs when the Capability Manager declares Initial Operational Capability (IOC) meaning that the capability is sufficiently mature that it can be operationally deployed. Capability that can be operationally deployed is then managed through sustainment. Acquisition of the remaining materiel is managed through the project until completion of Final Materiel Release. The period between IOC and Final Operational Capability is commonly referred to as the transition period. The Major Projects Report (MPR) effectively provides transparency of the funding and expenditure for acquisition elements, which is managed separately from sustainment. Additional reporting on sustainment products is now provided through the Defence Annual Report.

 DMO Submission to Joint Committee of Public Accounts and Audit, Review of the 2012–13 Defence Materiel Organisation Major Projects Report, Supplementary Submission 3.1, [26] May 2014, p. [5].

94 Although the DMO tracks transfers of funding from acquisition to sustainment, the DMO informed the ANAO in January 2015 that it was not able to provide any data on use of sustainment funding for completion of acquisition projects.

95 A Defence Minor capital project is classified as having a low to medium risk, or low strategic significance, is nominally valued up to $20 million and generally will not exceed $100 million.

96 Sustainment funding was eventually used in this case. See paragraph 4.30 for further detail.

97 In particular, the following points indicate that the Operations Room layout should have been classified as SEA 1448 expenditure as opposed to sustainment expenditure:

  • in 2006, Saab, the major contractor for the SEA 1448 Phase 2A Combat Management System, received a contract for $23 000 (increased in 2011 to $439 601) for an Operations Room Layout Study;
  • the July 2009 ANZAC fleet Product Schedule stated that the Operations Room configuration did not meet the functional requirements for the ASMD upgrade—with the implication being that a renewed Operations Room was integral to SEA 1448;
  • as planned in 2009, SEA 1448 Phase 2A included upgrading ‘the Operations Room layout to accommodate a total of 10 Multi Function Consoles’;
  • the 2012 MAA for SEA 1448 Phase 2A includes the new Combat Management System and ‘platform modifications to all eight ships to support [that and other systems]’;
  • the Navy website states that ‘at the heart of the ASMD upgrade is the updated Mk3E Combat Management System (CMS) and the redesigned operations room layout that provides the Command Team with unprecedented situational awareness and weapon system coordination.’ Similar statements occur in other publications, including a statement by BAE Systems Director of Footnote continued on the next page...
    Maritime that the scope of the ASMD project included ‘an upgraded Saab Combat Management System (CMS), including an improved operations room layout.’ T Muir, ‘BAE Systems boasts success with ANZAC upgrades’, ADM Defence Week, 17 July 2014, p. 3.

98 ANAO Audit Report No.34 2011–12, Upgrade of the M113 Fleet of Armoured Vehicles, pp. 86–8.

99 Joint Committee of Public Accounts and Audit, Report 442: Inquiry into the 2012–13 Defence Materiel Organisation Major Projects Report, Canberra, May 2014, paragraphs 3.49–56, 3.70–71 and 3.74, pp. 22–9.

100 See footnote 44.

101 Department of Defence, The Response to the Report of the Defence Procurement and Sustainment Review, Canberra, May 2009, p. 12.

102 Where a proposal involves shifting funding from one Navy product, for example, to another, the Chief of Navy, as owner of both products, can approve the transfer of funding. Transfer of funding from a Navy product to an Army product, however, would require the agreement of both the Chief of Navy and the Chief of Army, making such transfers less easy to achieve.

103 The analysis includes two estimates made for 2016–17, and one estimate made for 2017–18, hence the smaller range for these two years.

104 SEA 1348 Phase 2 was the project phase that built the 10 ANZAC Class ships (including two for the Royal New Zealand Navy). Although HMAS Perth was commissioned in August 2006, minor configuration changes to the ships in service and the finalisation of outstanding design and installation issues were progressed during 2007–08, with implementation of solutions to all issues and first-of-class installations continuing into 2010–11.

105 The Department of Defence Annual Report 2008–09 (Canberra, 2009, volume 2, p. 71) describes the logistics shortfall:

Investment in the procurement of additional long-lead spares for the Anzac class frigates over the last two years has been facilitated by provision of logistic shortfall supplementation funds, providing contingency against increased operational tempo.

106 The SRP is discussed at footnote 47.

107 Recommendation 2 of the Rizzo Report was that:

Defence and DMO should ensure that decisions made during acquisition fully consider whole-of-life costs and capability, through a rigorous and formalised Asset Management process.

 Paul Rizzo, Plan to Reform Support Ship Repair and Management Practices, Department of Defence, Canberra, July 2011, p. 38.

108 The analysis includes two estimates made for 2016–17, and one estimate made for 2017–18, hence the smaller range for these two years.

109 The data does not include operational expenditure, which amounted to $137.8 million from 2010–11 to 2013–14.

110 Australian Government, Portfolio Additional Estimates Statements 2010–11, Defence Portfolio, Canberra, February 2011, p. 136:

The Budget Estimate included funding that was incorrectly allocated to this Product. The position was corrected at Additional Estimates. There is no impact on the Protected Mobility Fleet capability.

111 The SPARK II Mine Roller is attached to the front of a vehicle to roll along the ground and detonate any mines before the vehicle itself reaches them. The Platt MR550 Ring Mount is a protective turret for a gunner sitting above the roof of the vehicle.

112 No-win/no-loss funding is another term for, in this case, Operational Supplementation funding, under which unbudgeted Defence overseas operations are funded by government. Defence also receives no-win/no-loss foreign exchange funding.

113 The DMO further informed the ANAO that the operational procurement process was strengthened in 2012–13 and beyond to ensure the funding source and scope of work were locked in and finalised via MSA Change Proposals.

114 The analysis includes two estimates made for 2016–17, and one estimate made for 2017–18, hence the smaller range for these two years.

115 The large estimates range in 2008–09 reflects an early anomalous estimate made in 2005: the estimate then made for 2008–09 was much less than the expenditure then budgeted for the years preceding 2008–09.

116 Defence classifies the Top 30 sustainment products based on forecast expenditure at the beginning of the financial year. In 2013–14, actual expenditure on the Top 30 sustainment products ranged from $26 million to $590 million.

 Department of Defence, Annual Report 2013–14, supplementary information for Chapter 6 [Internet], available from <http://www.defence.gov.au/AnnualReports/13-14/Supplementary Chapter6.pdf> [accessed November 2014] pp. 32–3.

117 The categories are grouped under various headings, such as:

  • changes to customer demand: increased or reduced rate of effort, engineering change;
  • sustainment reform: inventory management, process reform, reduction to total cost of ownership;
  • industry performance: capacity restraints, non-performance, procurement lead times; and
  • Defence/DMO performance: delays in contract/order management, delays in receiving funding approval, movement between financial years (bring forward/slippage).

118 In developing the manual, Defence adopted a 12-step approach based on the United States Government Accountability Office’s GAO Cost Estimating and Assessment Guide (2009), which emphasises the need for cost estimates to be comprehensive, well-documented, accurate and credible.

119 The ANAO has previously reported that, in 2012, Defence identified a need for further work to refine its whole-of-life cost models for capability projects. ANAO Audit Report No.6 2013–14, Capability Development Reform, p. 188.

120Going to the Next Level. The Report of the Defence Procurement and Sustainment Review [Mortimer Review], Defence Materiel Organisation, Canberra, September 2008, Recommendation 4.2, p. 49.

121 Joint Committee of Public Accounts and Audit, Review of the 2013–14 Defence Materiel Organisation Major Projects Report, Government Response: Department of Defence, Executive Minute on Joint Committee of Public Accounts and Audit Report No. 442, Review of the 2012–13 Defence Materiel Organisation Major Projects Report [Internet], p. [4], Canberra, December 2014, available from http://www.aph.gov.au/Parliamentary_Business/Committees/Joint/Public_Accounts_and_Audit/2012-13_DMO_Major_Projects_Report/Government_Response (accessed 4 December 2014).

122 The SPMS is to be accessible to relevant Defence and DMO personnel for data input and dashboard use.

123 The SPMS is intended to link DMO sustainment data with Defence capability and risk management systems, that is, with Navy’s Integrated Mission Management System (IMMS), Air Force’s Air Command Capability Framework (ACCF), and Army’s Sustainment Management and Reporting Tool–Land (SMART–Land).

124 Recommendation 12 of the Rizzo Report was:

More Effective Information Exchange: Navy and DMO must improve their internal reporting by capturing direct, timely and candid, document-based information that draws on a rigorous set of metrics.

125 ANAO Report No.28 2012–13, The Australian Government Performance Measurement and Reporting Framework, p. 63; and ANAO Report No.21 2013–14, Pilot Project to Audit Key Performance Indicators, p. 41.

126 Army records indicate that the Measures of Success Framework was developed in consultation with staff from all of the DMO Divisions that manage Army Product Schedules.

127 For example, in the period from March 2012 to June 2013, the Operational Availability KPI was affected 11 times by failure to conduct technical inspections, a responsibility which rests with ADF units rather than the DMO. This failure often contributed to a red flag for the Operational Availability KPI.

128 For example, the Manual notes that significant disputes can result from inappropriately assigning activities to a responsible organisation, and that both parties must acknowledge their specific responsibilities in the delivery of a KPI, including agreed definitions.

129 Joint Committee of Public Accounts and Audit, Report 442: Inquiry into the 2012–13 Defence Materiel Organisation Major Projects Report, Canberra, May 2014, pp. 22–9.

 Joint Standing Committee on Foreign Affairs, Defence and Trade, Review of the Defence Annual Report 2011–2012, Canberra, June 2013, p. 90.

130 Australian Government, Portfolio Budget Statements 2014–15, Budget Related Paper No.1.4A, Defence Portfolio, Canberra, May 2014, p. 177.

 A similar statement describes the DMO’s KPIs for acquisition projects:

The key performance indicators are to deliver major and minor capital equipment within the agreed parameters for schedule, scope and budget. The detail varies with each project and is specified in each project’s Materiel Acquisition Agreement.

 Australian Government, Portfolio Budget Statements 2014–15, Budget Related Paper No.1.4A, Defence Portfolio, Canberra, May 2014, p. 147.

131 Figure 5.4 on page 112 provides an example of the type of data made public. The Defence Annual Report includes a single data point for the relevant financial year.

132 In 2009–10 and 2011–12, the description was accompanied by a rating consisting of one to three ticks—one tick for ‘partially achieved’, two ticks for ‘substantially achieved’, and three ticks for ‘achieved’.

133 Ms Shireane McKinnie, General Manager Joint, Systems and Air, DMO, quoted in Joint Committee of Public Accounts and Audit, Report 436: Review of the 2011–12 Defence Materiel Organisation Major Projects Report, Canberra, May 2013, p. 29.

134 DMO Submission to Joint Committee of Public Accounts and Audit, Inquiry into the 2011–12 Defence Materiel Organisation Major Projects Report, Submission 4, 13 May 2013, p. 1.

135 Joint Committee of Public Accounts and Audit, Report 436: Review of the 2011–12 Defence Materiel Organisation Major Projects Report, Canberra, May 2013, pp. 29–30.

136 Joint Committee of Public Accounts and Audit, Report 442: Inquiry into the 2012–13 Defence Materiel Organisation Major Projects Report, Canberra, May 2014, pp. 22–9.

137 Joint Committee of Public Accounts and Audit, Review of the 2013–14 Defence Materiel Organisation Major Projects Report, Government Response: Department of Defence, Executive Minute on Joint Committee of Public Accounts and Audit Report No. 442, Review of the 2012–13 Defence Materiel Organisation Major Projects Report [Internet], p. [4], Canberra, December 2014, available from http://www.aph.gov.au/Parliamentary_Business/Committees/Joint/Public_Accounts_and_Audit/2012-13_DMO_Major_Projects_Report/Government_Response (accessed 4 December 2014).

138 ANAO Submission to Joint Committee of Public Accounts and Audit, Review of the 2013–14 Defence Materiel Organisation Major Projects Report, Submission 1: Sustainment Reporting Options Paper, 18 February 2015, available from http://www.aph.gov.au/Parliamentary_Business/Committees/Joint/Public_Accounts_and_Audit/2013-14_DMO/Submissions (accessed 2 March 2015).

139 Ibid., p. 6. The ANAO also informed the JCPAA that it would consider the inclusion of further audits concerning sustainment activities in its performance audit forward work program, after the completion of this audit report on MSAs.

140 The full recommendation was as follows:

 The Committee recommends that the Department of Defence enhance its public reporting by:

 —Developing a more precise method for reporting performance on capabilities acquisition and sustainment, which would detail:

  Specific performance targets;

  how performance is assessed in relation to these targets; and

  the specific reasons why targets are, or are not, achieved;

 —Including some detail on emerging areas of concern and potential future issues;

 —Enhancing its reporting on the Defence budget and its implications for capabilities acquisition and  sustainment;

 —Undergoing a periodic review conducted by independent experts, similar to the United States’ Quadrennial Defense Review; and

 —Including information on operational readiness.

 Joint Standing Committee on Foreign Affairs, Defence and Trade, Review of the Defence Annual Report 2011–2012, Canberra, June 2013, p. 90.

141 Australian Government, Joint Standing Committee on Foreign Affairs, Defence and Trade. Inquiry into the Review of the Defence Annual Report 2011–2012: Government Response, Canberra, October 2014, p. [3].

142 Navy’s performance measures include KPIs, such as ‘Materiel Ready Days’, ‘Cost per Materiel Ready Day’ and ‘Price Reliability’; and Key Health Indicators such as Demand Satisfaction Rates. Navy commenced reporting against its new suite of KPIs in July 2014, and is to report against Key Health Indicators from mid-2015 when the DMO’s sustainment reporting system is implemented for Navy products.