The audit objective was to assess the effectiveness of the administration of the Commonwealth Scientific and Industrial Research Organisation's (CSIRO's) Gift to the Science and Industry Endowment Fund.

Summary and recommendations

Background

1. The Science and Industry Endowment Fund (the Fund) was established by the Science and Industry Endowment Act 1926 (the Act) to provide assistance to persons engaged in scientific research and in the training of students in scientific research. Since 1986, the Chief Executive of the Commonwealth Scientific and Industrial Research Organisation (CSIRO) has been the Trustee of the Fund. The Trustee is the Fund’s primary decision-making authority and is responsible for the effective administration of the Fund.

2. In June 2009, CSIRO’s Board agreed that the net proceeds from CSIRO’s wireless local area network (WLAN) licensing project—at that stage expected to be around $150 million—should be donated to the Fund. CSIRO’s stated aim in making the Gift to the Fund was to support nationally important research with the potential to create substantial long term benefits for Australia, beyond CSIRO’s usual business.

3. CSIRO’s Gift to the Fund was formalised on 15 October 2009 through a Deed of Gift (the Deed). The Deed sets out the purposes for which the Gift can be used, and the requirements relating to the Gift’s administration.1 At the same time, the Trustee and CSIRO entered into a Services Agreement covering the provision of administrative support to the Fund. On 20 October 2009, the renewed Fund was launched by the then Minister for Innovation, Industry, Science and Research.

4. At the end of December 2015, the Trustee had committed $142 million from CSIRO’s Gift to 26 projects across four funding programs. At the time of the audit, the 26 projects funded from CSIRO’s Gift have attracted almost $370 million in co-investment funding from around 130 collaborating organisations, including CSIRO. As well as playing a central role in the administration of the Fund, CSIRO is a participant in most of the projects that have received financial assistance from the three funding programs examined in this audit—the Research Projects, Research Infrastructure and Special Research programs.

Audit objective and criteria

5. The audit objective was to assess the effectiveness of the administration of CSIRO’s Gift to the Fund. To form a conclusion against the audit objective, the ANAO adopted the following high level audit criteria:

  • arrangements for the administration of the Gift were well-designed and have been implemented effectively; and
  • financial assistance from the Gift has been administered effectively and expected outcomes are being achieved.

6. In designing the arrangements for administering the Gift, CSIRO advised the ANAO that it had regard to the key principles for grants administration set out in the Australian Government’s grants administration framework.2 In addressing the audit’s second criterion, the ANAO considered CSIRO’s application of these principles, including: transparent decision making; consistent and equitable treatment of applications for financial assistance; and achieving outcomes and value with public money.

Conclusion

7. CSIRO’s Gift to the Fund has generally been well administered. In particular, the approach to designing the arrangements for managing the Gift was sound, and for the most part, these arrangements, including the processes for administering financial assistance from the Gift, have been operating as intended. To enable better assessments of the Fund’s performance and achievements, CSIRO and the Trustee should develop more qualitative and outcomes focused performance measures, and consider undertaking a formal evaluation. In addition, improved record-keeping would enhance the transparency of decision-making.

Supporting findings

Designing and administering the Deed of Gift

8. The design of the Deed of Gift supports the achievement of CSIRO’s objective in providing the Gift to the Fund—to provide funding for scientific research benefitting Australia. Of note, CSIRO and the Trustee: adopted a consultative and orderly approach to the design and administration of the Deed; developed a set of investment principles and strategic objectives to give practical effect to the broad purposes of the Gift; and established a number of robust accountability and ethical measures to support transparency in the Gift’s administration.

9. A well-designed risk management plan was developed and approved for the Fund—albeit three years after the payment of CSIRO’s first tranche of the Gift to the Fund. The plan’s ongoing accuracy and integrity has been regularly monitored, with refinements made to risk mitigation strategies as needed.

10. CSIRO has also put in place a series of structured measures to help manage conflicts of interest. The ANAO’s examination indicates that the established arrangements have, for the most part, been operating as intended. Some shortcomings were observed in the recording of conflicts of interest relating to the assessment of a small number of applications for funding from the Research Projects program, but these gaps did not affect the overall transparency of the decisions made.

11. The Fund’s website contains a range of useful promotion and communication information. Program guidelines developed for the Research Infrastructure and Special Research programs were well-designed and informative. Formal program guidelines were not produced for any of the four rounds of the Research Projects program, but relevant information about the program was provided on the Fund’s website. This information was provided in a timely manner, and overall, the information was well-presented and informative. However, the lack of clear eligibility criteria for the Research Projects program resulted in a degree of unnecessary assessment work by the Expert Panel, which affected administrative efficiency.

12. The three funding programs examined in this audit had separate and distinct application and assessment processes. The processes for assessing applications for funding under the Research Projects program were sound. In particular, the approach to assessing potential projects was refined and modified over the course of the four assessment rounds to reflect lessons learned. Under the Research Infrastructure program, detailed planning was undertaken to identify investment opportunities and to develop the associated assessment processes.

13. In contrast, a clear purpose was not established at the outset of the Special Research program. Rather, the decisions to seek proposals for the two projects funded under the program were largely unplanned and unrelated. Further, while assessment criteria were developed for each of the two proposals, details of these criteria were restricted to the organisations preparing the respective proposals and not made public.

14. The ANAO’s examination of a sample of assessments from the Research Projects program and each of the assessments undertaken for the Research Infrastructure and Special Research programs indicates that the assessments had appropriate regard to information provided by applicants and were undertaken in accordance with the established processes. There were, however, some shortcomings in relation to the accuracy and completeness of record-keeping. The ANAO suggests that in order to enhance the transparency and accountability of decision-making, the Advisory Council’s advice to the Trustee contain a greater level of detail supporting the basis of the Council’s recommendations.

Establishing funding agreements and measuring and reporting performance and outcomes

15. Standard funding agreements were well-designed and fit-for-purpose, and the sample of executed agreements examined by the ANAO contained terms and conditions consistent with the approved templates. Some protracted delays were experienced in the negotiation and execution of funding agreements in Rounds One and Two of the Research Projects program. However, sensible improvements in the assessment and funding offer processes were implemented that helped overcome these delays in the later rounds.

16. Arrangements for monitoring and assessing the progress and performance of the funded projects, including the achievement of project deliverables and outcomes, were well-designed and operating effectively. The ANAO also observed activity to address identified performance issues.

17. A range of useful management information, including information concerning the Fund’s financial position, the status of the funded projects, and the Fund’s results against its key performance indicators, has been captured, reported and utilised in administering the Fund. However, measuring the effectiveness of the Fund’s overall performance would be enhanced by developing more outcome-focused performance measures, and the ANAO has made a recommendation on this matter. Although processes are in place for capturing information on the contributions made by individual projects towards the Fund’s objectives, at the time of the audit, the Trustee had not undertaken a formal evaluation to assess the outcomes achieved by the Fund as a whole; or to identify opportunities for improvements to the Fund’s administration. In January 2016, CSIRO advised the ANAO that it anticipated an overall evaluation of the Fund would be undertaken by the end of 2016.

Recommendation

Recommendation No.1

Paragraph 3.22

To enable better assessments of the performance of the Science and Industry Endowment Fund, the ANAO recommends that the Trustee and the Commonwealth Scientific and Industrial Research Organisation develop and report against a set of key performance measures that include a focus on quality and outcomes.

Summary of entity response

18. CSIRO notes the audit and confirms it accepts the recommendation from the report and advises it has already begun implementing the recommendation, and notes the very professional audit.

1. Introduction

Background

1.1 The Science and Industry Endowment Fund (the Fund) was established by the Science and Industry Endowment Act 1926 (the Act) to provide assistance to persons engaged in scientific research and in the training of students in scientific research. Section 4 of the Act states that the Fund consists of:

  • the amount of one hundred thousand pounds appropriated to the Fund from the Consolidated Revenue Fund and the income derived from investment of that amount; and
  • any gifts or bequests given or made to the Fund, as well as the income derived from investment of those gifts or bequests.

1.2 Since 1986, the Act has provided that the Chief Executive of the Commonwealth Scientific and Industrial Research Organisation (CSIRO) is the Trustee of the Fund. In this role, the Trustee is responsible for the effective administration of the Fund and is the Fund’s primary decision-making authority.

1.3 In October 2007, the CSIRO Board decided that any proceeds from the organisation’s wireless local area network (WLAN) licensing project3 should be applied to the advancement of scientific research in Australia. In June 2009, the Board endorsed a recommendation from CSIRO’s management that the net proceeds from the WLAN project—at that stage expected to be around $150 million—should be donated to the Fund with the intention of:

[funding] nationally important research that otherwise could not be funded in the normal course of CSIRO’s business.

1.4 CSIRO’s Gift to the Fund was formalised through a Deed of Gift (the Deed), which was executed on 15 October 2009 by the Trustee and the CSIRO Board. The Deed outlines the arrangements to be adopted to administer CSIRO’s Gift, including setting out the:

  • purposes for which the Gift can be used; and
  • mechanisms for providing support and assistance to the Trustee in the exercise of their responsibilities.

1.5 CSIRO’s Gift of $150 million to the Fund was publicly announced by the then Minister for Innovation, Industry, Science and Research on 20 October 2009. At that time, the Minister observed that the Fund would support research activities ‘at Universities as well as at CSIRO’ and expressed the hope that CSIRO’s donation would be ‘augmented by donations from industry and other benefactors’. The Minister also announced the following allocations of financial assistance from the Fund, which had been stipulated by CSIRO as a condition of making the Gift:

  • subject to the receipt of an acceptable proposal, up to $10 million to CSIRO’s Information, Communications and Technology Centre to support further research into wireless broadband technologies;
  • subject to the receipt of an acceptable proposal, approximately $2 million to fund a Professorial Chair in Wireless Research (a joint appointment between CSIRO and Macquarie University); and
  • up to $7.5 million to establish a series of scholarship and fellowship schemes.

1.6 On the same day as the Deed was executed, the Trustee appointed CSIRO—under a Services Agreement—to assist in the administration of the Fund. The services provided to the Fund by CSIRO include:

  • managing the Fund’s investments;
  • legal and financial management services;
  • preparing management reports;
  • assisting in the design of the processes for providing financial assistance;
  • oversighting the preparation of funding agreements and the monitoring of the performance of the funded projects;
  • secretariat support to the Trustee and the Advisory Council; and
  • managing the Fund’s communication activities.

Nature of financial assistance

1.7 Prior to CSIRO’s Gift, the Fund’s contribution to Australia’s scientific research effort was modest. The total assets of the Fund in June 2009 amounted to around $530 000, and over the period 2005–06 to 2008–09 the Fund provided, on average, approximately $22 000 a year in financial assistance.4 Since October 2009, the Fund has provided financial assistance from CSIRO’s Gift through four programs—Research Projects; Research Infrastructure; Special Research; and Promotion of Science. Table 1.1 provides a summary of the each of these programs, including details of the amounts of financial assistance provided.5

Table 1.1: Summary of the Fund’s financial assistance programs relating to CSIRO’s GiftA

Fund program

Aim

Basis for allocating financial assistance

Number of projects funded

Funds approvedB ($m)

Funds expendedB ($m)

Research Projects

Support projects in the areas of: emerging science issues or priorities; developing solutions to science challenges or opportunities; and support for the delivery of scientific advances.

Open and competitive funding rounds

16

67.2

55.8

A condition of the Gift

1

10.0

10.0

Research Infrastructure

Support the creation or enhancement of nationally significant research facilities or equipment.

Application by invitation

3

31.6

16.6

Special Research

Support proposals identified by the Trustee and Advisory Council as aligning with the purpose and strategic objectives of the Fund.

Application by invitation

2

13.5

13.5

Promotion of Science

Support research undertaken by early career scientists, the appointment (or joint appointment) of scientists to university positions, and scholarships and fellowships.

Open and competitive funding rounds

3C

17.8

7.9

A condition of the Gift

1

2.0

1.2

Totals

 

26

142.1

105.0

Note A: CSIRO’s Gift was provided to the Fund in three tranches of $50 million—in October 2009, February 2010, and April 2011.

Note B: Funds approved and spent over the period October 2009 to December 2015.

Note C: This aspect of the Promotion of Science program comprises three broad project groups—Fellowships and Scholarships, Australian Academy of Science (Lindau) Fellowships, and STEM+ Business Fellowships. At the time of the audit, around 60 individual projects have been provided with financial assistance across these three project groups.

Source: ANAO analysis of the website of the Fund and of the Fund’s ‘cash pipeline’ reports.

The Fund’s objective

1.8 The stated objective of the Fund is to make strategic investments in scientific research that address issues of national priority for Australia. Specifically, the Fund aims to achieve this objective by investing in science that contributes to Australia’s sustainable future such as:

  • fundamental research into new paradigms for sustainable resource use, environmental protection and community health;
  • tactical research to fast-track solutions to national challenges;
  • collaborative research that brings together organisations capable of working together on solutions to national challenges; and
  • scholarships that create and sustain young researchers capable of addressing national challenges.
Collaboration and co-investment

1.9 The Deed provides that research funded from the Gift may be carried out ‘by or within one or more single institutions or within collaborative partnerships’. To give effect to this feature of the Deed, two important design elements of the financial assistance provided from the Gift are the:

  • promotion of collaborative research and related activities; and
  • leveraging of co-investment—either from the collaborators or from other parties, including industry.

1.10 Table 1.2 contains a summary of the levels of collaboration and co-investment associated with the 26 projects funded from CSIRO’s Gift at the time of the audit.

Table 1.2: Collaborator and co-investment levels

Fund program

Number of collaboratorsA

Amount of co-investment announced ($m)

Proportion of project funding sourced from co-investment (%)

Research Projects

59

119.0

60.7

Research Infrastructure

8

142.7

81.9

Special Research

5

98.7

88.0

Promotion of Science

62B

8.7B

31.0

Totals

134

369.1

72.2

Note A: A number of organisations are involved in more than one project. For the purposes of Table 1.2, these have been counted as a separate collaborator each time.

Note B: CSIRO advised the ANAO that the number of collaborators involved and the amount of co-investment in the Promotion of Science program will increase as further individual fellowships and scholarships are announced in the future.

Source: ANAO analysis of relevant Fund documentation.

Roles and responsibilities

1.11 The key roles and responsibilities relating to the operation of the Fund are illustrated in Figure 1.1.

Figure 1.1: Key roles and responsibilities

Source: ANAO analysis of relevant Fund documentation and the Deed.

Audit objective, criteria and approach

1.12 The audit objective was to assess the effectiveness of the administration of CSIRO’s Gift to the Fund. To form a conclusion against the audit objective, the ANAO adopted the following high level audit criteria:

  • arrangements for the administration of the Gift were well-designed and have been implemented effectively; and
  • financial assistance from the Gift has been administered effectively and expected outcomes are being achieved.6

1.13 CSIRO advised the ANAO that in designing the arrangements for administering the Gift it had regard to the key principles in the Australian Government’s grants administration framework set out in the Commonwealth Grant Guidelines (July 2009).7 In addressing the audit’s second criterion, the ANAO considered CSIRO’s application of these principles, including: transparent decision-making; consistent and equitable treatment of applications for financial assistance; and achieving outcomes and value with public money.

1.14 The audit reviewed processes and transactions occurring during the period October 2009 to August 2015, although some records covering the period prior to October 2009 were examined as part of the ANAO’s assessment of the design of the Deed. Specifically, the ANAO reviewed key documentation related to administration of the Fund, including: papers and minutes from the meetings of the Advisory Council; program guidelines; records of the assessment of applications and proposals for financial assistance; funding agreements; project performance reports; and management information. In addition, the ANAO interviewed the Chair of the Fund’s Advisory Council, key members of CSIRO’s service provider team, as well as representatives from a number of the funded projects and unsuccessful applications.

1.15 The audit was undertaken in accordance with the ANAO’s Auditing Standards at a cost to the ANAO of approximately $380 000.

2. Designing and administering the Deed of Gift

Areas examined

The audit examined the effectiveness of the processes associated with the design and ongoing administration of the Deed of Gift; including the processes for selecting projects for financial assistance.

Conclusion

The design and administration of the Deed of Gift support CSIRO’s objective in providing the Gift to the Fund—to provide funding to support scientific research which would benefit Australia. Notably:

  • suitable governance arrangements were established to manage the operation of the Fund;
  • CSIRO has, for the most part, effectively managed the risks associated with administering the Gift, despite delays in formalising some processes; and
  • while different processes were established to assess applications for financial assistance in the three programs examined by the ANAO, these processes were generally sound and have largely operated as intended. However, there were some shortcomings in recording details of conflicts of interest and in the level of documentation supporting key assessment decisions. Further, a clear purpose was not established at the outset of the Special Research program.
Area for improvement

The ANAO suggests that in order to enhance the transparency and accountability of decision-making, the Advisory Council’s advice to the Trustee contain a greater level of detail supporting the basis of the Council’s recommendations.

Introduction

2.1 CSIRO plays a central role in the administration of the Fund. In particular: it is the donor of the Gift; CSIRO’s Chief Executive is the Trustee of the Fund; and CSIRO has been engaged by the Trustee to provide a range of administrative services to the Fund. In addition, CSIRO’s Board determines the composition of the Trustee’s Advisory Council and, pursuant to section 10A of the Science and Industry Research Act 1949, is able to issue directions to the Trustee. During the period examined by the audit, CSIRO-related persons were also participants in the following governance and assessment bodies:

  • the five member Advisory Council—expanded to six in June 2015—included two CSIRO employees and a member of CSIRO’s Board; and
  • the four member Expert Panel—expanded to six in June 2012—included two Advisory Council members: the CSIRO Board member and an employee of CSIRO.

2.2 As shown in Table 2.1, CSIRO is also a participant in most of the projects that have received financial assistance from the three programs examined in this audit. In this regard, CSIRO has received more financial assistance than any other organisation; and has also contributed more co-investment funds to successful projects than any other organisation.

Table 2.1: Analysis of CSIRO’s involvement in approved projects

Fund program

Proportion of approved projects involving CSIRO

Amount of approved funding to CSIRO ($m)

Proportion of approved funding to CSIRO

Amount of co-investment from CSIRO ($m)

Proportion of co-investment from CSIRO

Research Projects

94%

46.2

59.9%

47.1

39.5%

Research Infrastructure

100%

22.3

70.7%

109.2

76.5%

Special Research

50%A

6.0

44.4%

8.7

8.8%

Totals

91%

74.5

60.9%

165.0

45.8%

Note A: The proportion of approved projects involving CSIRO does not include the Synchrotron project (worth $7.5 million) under which CSIRO is engaged to administer the project on behalf of the Fund. In particular, CSIRO is responsible for managing the project against the requirements of the funding agreement and allocating the financial assistance provided by the Fund to the beneficiary organisation.

Source: ANAO analysis of relevant Fund documentation.

Does the design of the Deed of Gift support scientific research that benefits Australia?

The design of the Deed of Gift supports the achievement of CSIRO’s objective in providing the Gift to the Fund—to provide funding for scientific research benefitting Australia. Of note, CSIRO and the Trustee: adopted a consultative and orderly approach to the design and administration of the Deed; developed a set of investment principles and strategic objectives to give practical effect to the broad purposes of the Gift; and established a number of robust accountability and ethical measures to support transparency in the Gift’s administration.

Development and administration of the Deed of Gift

2.3 Section 9 of the Act establishing the Fund8 states that:

The trustee shall deal with and apply so much of the Fund as represents gifts or bequests or the income arising from the investment thereof in accordance with the conditions upon which the gift or bequest was given or made …

2.4 In this context, the Fund offered CSIRO an opportunity to invest the proceeds from its WLAN licencing project in a way that would support the Board’s aim of funding significant scientific research activity for the benefit of Australia. In order to do so effectively, it was important for CSIRO, together with the Fund’s Trustee, to carefully consider the design and administration of the terms and conditions in the Deed of Gift.

2.5 CSIRO’s approach to the design and development of the Deed included consultation with a range of stakeholders, both internal and external to CSIRO. The consultation process included senior Ministers and the (then) Department of Finance and Deregulation. As a result of these consultations, the Deed sets out the ‘primary’ and ‘special’ purposes for which the Gift can be used. In particular, the Deed states that the primary purpose of the Gift is:

… [the Gift is to] be used for Research for the purposes of assisting Australian industry, furthering the interests of the Australian community or contributing to the achievement of Australian national objectives.9

2.6 The special purposes were designed to provide more guidance to the Trustee in applying the Deed’s broad primary purpose. In particular, CSIRO advised the ANAO that the special purposes were designed to ‘provide funding into a spread of…productive [research] activities’. To give practical effect to the special purposes and to help provide further insight into the nature of the activities that were likely to be funded, the Trustee adopted a set of investment principles.10

2.7 Through its consultations, CSIRO also considered and addressed a range of administrative, legal, ethical, procedural and financial matters relating to the operation of the Deed. In particular, as shown in Table 2.2, CSIRO put in place a series of measures designed to strengthen accountability.

Table 2.2: Accountability measures implemented by CSIRO relating to the Gift’s administration

Description of measures

Overarching specific conditionsA

The Gift is to be applied over a minimum period of five years and a maximum period of 10 years from the date of the Gift.

The maximum amount disbursed from the Gift in any one financial year is $25 million.

The Trustee must require the recipients of a grant to provide reports on the application of the grant, including the outcomes achieved.

Measures in the Deed

Requiring the provision of reports on the operations of the Fund to the Minister responsible for administering the Science and Industry Research Act 1949 and to the (then) Finance Minister.

Outlining the role, composition and standards of conduct of the Advisory Council, including disclosure of conflicts of interest.

Measures in the Services Agreement

The Services Agreement contains several measures relevant to accountability, including:

  • stating the standard of care to be exercised;
  • outlining the limitations of CSIRO’s powers;
  • listing the permitted investments;
  • requiring the maintenance of proper records and accounts; and
  • listing the management reports to be provided to the Trustee.

Note A: These conditions were imposed pursuant to clause 1.2 of the Deed. The conditions were reiterated by the CSIRO Board at the time of approving each of the three tranches of the Gift.

Source: ANAO analysis of the Deed and the Services Agreement.

Planning to implement the Deed

2.8 In June 2009, the CSIRO Board agreed to the following key steps to help manage the transition phase leading up to the Deed’s operation:

  • establish a sub-committee of the Board—the Board Endowment Committee11;
  • determine the quantum of the Gift;
  • determine any conditions attaching to the Gift;
  • execute the Deed;
  • appoint Service Provider(s);
  • establish the Advisory Council; and
  • provide the Trustee and Advisory Council with advice on a series of implementation issues—including establishing the mechanisms for making disbursements of the Gift from the Fund.

2.9 The ANAO’s analysis indicates that each of these steps was performed. With regard to the decision to appoint CSIRO as the Service Provider, the ANAO was advised that the Trustee’s decision took into account that the costs of administering the Fund had to be minimised and also that CSIRO had the skills necessary to manage the Fund. Further, the Board Endowment Committee was advised that the Trustee had considered the relative costs of engaging service providers and believed that the use of CSIRO’s services presented an economic and efficient solution. However, there was no documentation or analysis supporting the decision to engage CSIRO as an ’economic and efficient solution’.

Are processes for identifying and managing program-related risks effective?

A well-designed risk management plan was developed and approved for the Fund—albeit three years after the payment of CSIRO’s first tranche of the Gift to the Fund. The plan’s ongoing accuracy and integrity has been regularly monitored, with refinements made to risk mitigation strategies as needed.

CSIRO has also put in place a series of structured measures to help manage conflicts of interest. The ANAO’s examination indicates that the established arrangements have, for the most part, been operating as intended. Some shortcomings were observed in the recording of conflicts of interest relating to the assessment of a small number of applications for funding from the Research Projects program, but these gaps did not affect the overall transparency of the decisions made.

Risk management

2.10 A structured approach to the management of potential threats or risks is a key part of effective public administration. In particular, the early and active identification, assessment, treatment and monitoring of risks will promote accurate and well-informed judgements and plays an important role in successful program implementation, including securing value for money and supporting the achievement of policy objectives.12

2.11 The ANAO was advised that work on the development of a risk management plan for the Fund commenced in 2012—some three years after the Deed was signed and the first tranche of the Gift paid to the Fund. The resultant risk plan was approved by the Advisory Council in June 2013. Table 2.3 shows the four risk event groups identified in the approved plan.

Table 2.3: Risk event groups identified in the approved risk plan

Risk event group

Description

Compliance

Inappropriate governance of the Fund and lack of compliance with relevant requirements.

Financial management

Inappropriate financial management of funds, including investments.

Reputation

The Fund’s independence and reputation as a funder of integrity are challenged.

Grants

The grants made from the Fund do not achieve strategic impact.

Source: ANAO analysis of the Fund’s risk management plan.

2.12 The ANAO’s analysis indicates that the risks, controls and treatments identified in the approved risk management plan were relevant and appropriate. In addition, our examination also indicates that processes were in place, and operating as intended, to assess the continuing accuracy and integrity of the plan. As a result of these processes, a number of refinements and enhancements to the controls designed to reduce the impact of the identified risks or to address emerging risks have been incorporated into the plan.

Conflicts of interest

2.13 The management of conflicts of interest is an important part of the administration of Australian Government initiatives and services. Dealing effectively with conflict of interest situations typically requires the use of controls and measures that provide an appropriate level of assurance that decision making will not be, nor be perceived to be, compromised by a lack of impartiality.13 CSIRO’s central role in the Fund’s administration—as observed at paragraph 2.1—introduced increased risks of conflicts of interest that required careful management.

2.14 As shown in Table 2.4, CSIRO has put in place a series of formal and structured measures across key aspects of the Fund’s administration to address conflict of interest risks.

Table 2.4: Controls in place to address potential conflict of interest risks

Potential conflict of interest risk

Control measures in place

The Trustee is the CEO of CSIRO

CSIRO attempted to establish a separation of roles by appointing a delegate to act on CSIRO’s behalf in activities involving the SIEF when the CEO is performing in the role of Trustee.

Decisions to fund projects are made by the Trustee based on advice provided by the Expert Panel and Advisory Council.

CSIRO’s dual roles as administrator of the Gift funds and as recipient of SIEF funds

CSIRO staff record the time worked on Fund-related matters and this record forms the basis of CSIRO’s quarterly invoice to the Trustee.

CSIRO’s employees in the roles of the Fund’s Manager and the legal advisor to the Trustee are not involved in any other Fund-related activities on CSIRO’s behalf.

Before processing progress payments to the funded projects, CSIRO obtains the approval of the Trustee. The Trustee’s decisions are based on advice about the performance of the projects from members of the Advisory Council (Research Infrastructure and Special Research programs) and members of the Expert Panel and external reviewers (Research Projects program).

In March 2011, albeit some 18 months after CSIRO was appointed, the Trustee asked the Advisory Council to provide advice about CSIRO’s performance as service provider.

The Trustee authorises the payment of CSIRO’s quarterly fee invoices.

Professional and personal conflicts for Advisory Council members

Members agree, in writing, to abide by the ‘Guidelines for managing confidential information and conflicts of interest for Trustee Advisers’.

The Council maintains a register of members’ declared interests, and conflicts of interest is a standing item on Council meeting agendas.

The Deed prescribes standards of conduct for the Advisory Council (including to act in the best interests of the Fund) and also states that the majority of the members of the Council must not be employees of CSIRO.

Professional and personal conflicts for Expert Panel members

Members agree, in writing, to abide by the ‘Guidelines for managing confidential information and conflicts of interest for Trustee Advisers’.

The Panel maintains a register of members’ declared interests—albeit use of the register only commenced for the assessment process in Round Two (August 2010).

Advice on the management of conflicts of interest for the Research Projects program was included in the guidance provided to the Panel—albeit not until stage two of Round Three (May 2011).

Professional and personal conflicts for expert reviewers

The expert reviewers are asked to declare any conflicts of interest when assessing the funded projects’ progress and supplementary reports. Further, at the start of the major review discussions, the expert reviewers are asked to declare any conflicts of interest or to confirm conflicts that they have flagged in their report assessments.

Source: ANAO analysis of relevant documentation.

2.15 The ANAO’s testing indicates that the established arrangements for managing conflicts of interest have, for the most part, been operating as intended. In particular, the ANAO observed that:

  • all funding decisions by the Trustee were consistent with the recommendations from the Advisory Council;
  • members of the Advisory Council regularly updated their declarations and proactively brought potential conflicts to the attention of the Chair;
  • members of the Expert Panel were aware of the need to declare conflicts of interest and actively took steps to manage these issues—even before official procedures were established; and
  • the Council has considered CSIRO’s performance each year following the Trustee’s request and on each occasion had indicated satisfaction with the level of services being provided.

2.16 However, the ANAO’s examination of assessment processes for the Research Projects program identified a small number of cases where the recording of conflicts of interest was fragmented and inconsistent. The ANAO’s analysis indicates that these gaps in the records did not affect the overall transparency of the decisions made.

Was clear, consistent and timely information about the Fund, including details of how to access financial assistance, readily available?

The Fund’s website contains a range of useful promotion and communication information. Program guidelines developed for the Research Infrastructure and Special Research programs were well-designed and informative. Formal program guidelines were not produced for any of the four rounds of the Research Projects program, but relevant information about the program was provided on the Fund’s website. This information was provided in a timely manner, and overall, the information was well-presented and informative. However, the lack of clear eligibility criteria for the Research Projects program resulted in a degree of unnecessary assessment work by the Expert Panel, which affected administrative efficiency.

Promotion and communication

2.17 CSIRO undertook a range of communication and promotional activities designed to improve the visibility, positioning and marketing of the Fund, primarily via the Fund’s website. The ANAO’s analysis indicates that the website was easily navigable, had been kept up-to-date, and contained key information about the Fund’s activities, including:

  • the Fund’s objectives and purposes;
  • a description of the Fund’s financial assistance programs;
  • important dates;
  • details of the projects that have been funded;
  • copies of the Fund’s quarterly newsletters; and
  • details on how to subscribe to the Fund’s mailing list.

Program guidelines

2.18 Readily accessible program guidelines are a key means of informing prospective funding applicants about a financial assistance program, including the program’s selection criteria and how to apply for financial assistance. Well drafted guidelines also promote confidence in the equity and transparency of the funding process.

2.19 Separate guidelines were established for the Research Infrastructure program and for each of the projects invited to seek funding under the Special Research program. As shown in Table 2.5, the ANAO’s analysis indicated that the guidelines developed for these programs were well-designed and reflected elements of good practice.

Table 2.5: ANAO assessment of the content of the Research Infrastructure and Special Research programs’ guidelines

Good practice elementsA

Addressed in the Guidelines

 

Research Infrastructure

Special Research–Synchrotron

Special Research–ASKAPB

Program purposes, scope, objectives and desired outcomes

Yes

Yes

Yes

Total available funding

Yes

No

Yes

Eligibility criteria

Yes

Not applicableC

Not applicableC

Governance arrangements, including roles and responsibilities

Yes

Yes

Yes

Application process

Yes

Yes

Yes

Assessment/selection criteria and process

Yes

Yes

Yes

Review or appeal mechanisms

No

No

No

Note A: These elements are based on information contained in the ANAO’s 2010 Grants Administration Better Practice Guide.

Note B: ASKAP—the Australian Square Kilometre Array Pathfinder.

Note C: The Special Research program funding rounds were not open and competitive; rather proponents were invited to participate. As such, Special Research guidelines do not contain eligibility criteria.

Source: ANAO analysis of the programs’ guidelines.

2.20 In contrast, program guidelines were not produced for any of the four funding rounds of the Research Projects program. Rather, a range of information about the program was provided on the Fund’s website. The ANAO’s analysis indicates that details about the program were informative and had been provided in a timely manner. However, while the published program material explicitly defined an eligible applicant, there was some uncertainty as to what was an eligible project. For example:

  • it was difficult to clearly determine project eligibility because information that could be interpreted as being eligibility criteria was not clearly marked as such; and
  • details of available project funding were ambiguous. For instance, the program material implied that projects within a $3 million to $10 million range were eligible. However, only three of the sixteen projects funded over the four funding rounds of the program were approved for more than $5 million in funding, and none for more than $6 million.14

2.21 Eligibility criteria should be straightforward, easily understood and effectively communicated to potential applicants. Grouping all eligibility criteria and clearly identifying the requirements assists with transparency of the application and assessment processes. In this context, the introduction of clear project eligibility criteria may have benefited potential applicants by reducing the submission of uncompetitive Expressions of Interest. Further, clearer eligibility criteria may also have reduced the workload of the Expert Panel and improved administrative efficiency.

Were arrangements for assessing proposals for financial assistance well-designed and were assessments accurate, equitable and consistent?

The three funding programs examined in this audit had separate and distinct application and assessment processes. The processes for assessing applications for funding under the Research Projects program were sound. In particular, the approach to assessing potential projects was refined and modified over the course of the four assessment rounds to reflect lessons learned. Under the Research Infrastructure program, detailed planning was undertaken to identify investment opportunities and to develop the associated assessment processes.

In contrast, a clear purpose was not established at the outset of the Special Research program. Rather, the decisions to seek proposals for the two projects funded under the program were largely unplanned and unrelated. Further, while assessment criteria were developed for each of the two proposals, details of these criteria were restricted to the organisations preparing the respective proposals and not made public.

The ANAO’s examination of a sample of assessments from the Research Projects program and each of the assessments undertaken for the Research Infrastructure and Special Research programs indicates that the assessments had appropriate regard to information provided by applicants and were undertaken in accordance with the established processes. There were, however, some shortcomings in relation to the accuracy and completeness of record-keeping.

Design of assessment processes

2.22 Separate and distinct assessment processes are in place for each of the three programs examined in this audit—the Research Projects, Research Infrastructure and Special Research programs. Figure 2.1 provides a timeline of significant events relating to the assessment and selection of proposals for financial assistance under each of these programs. Under the Research Projects program, the assessment and selection of projects for funding was conducted through four open competitive funding rounds. In each of the other two programs, selected projects have been invited, at different times, to submit proposals seeking funding. The ANAO examined the design of the assessment processes for each program to consider whether they were likely to result in funding projects that represent value for money in the context of the Fund’s objectives and intended outcomes.

Figure 2.1: Timeline of assessment and selection decisions

Note A: The four funding rounds of the Research Projects program were open to the public over the following periods—Round One: December 2009 to March 2010; Round Two: June 2010 to August 2010; Round Three: August 2010 to March 2011; Round Four: May 2012 to July 2012.

Source: ANAO analysis of Fund documentation.

Research Projects program

2.23 The Expert Panel undertook much of the assessment work relating to the Research Projects program. The results of the Expert Panel’s assessments were considered by the Advisory Council which, in turn, made recommendations about funding to the Trustee.15

2.24 The assessment and selection processes developed for the Research Projects program, which were approved by the Trustee in June 2010, reflected a sound and sensible approach. Importantly, the processes were cogently structured, such that key steps in the process were designed to build on the results achieved in previous steps. To facilitate consistency in assessment processes, the Panel was provided with pre-formatted assessment scoresheets and guidance on the use of the scoresheets—including an explanation of the scoring methodology. In addition, CSIRO undertook a number of reviews through the course of its administration of the Research Projects program and introduced a number of refinements and improvements to the assessment processes. Among other things, these enhancements benefited applicants and the Expert Panel by providing clearer information and by streamlining and strengthening the processes.

2.25 As shown in Table 2.6, the ANAO’s analysis indicated that the selection criteria developed to assess applications for funding under the Research Projects program exhibited many of the characteristics of good practice.

Table 2.6: ANAO’s analysis of the selection criteria used to assess applications for funding under the Research Projects program

CharacteristicA

Result

Comment

Outcomes focused and aligned with program objectives

Yes

The criteria were consistent with the stated purposes of the Fund.

Promote additionality

Yes

The criteria explicitly sought to fund research that would not otherwise occur.

Easily understood

Yes

The criteria were clearly labelled and key terms were defined.

Internally consistent

Yes

The criteria were internally consistent.

Comprehensive

Partial

Factors not identified as ‘Assessment Criteria’ but nonetheless mentioned in the publicly available material were taken into account by the Expert Panel during assessments.

Objectively assessable

Partial

Assessment records suggest there were differing interpretations by members of the Expert Panel of the information provided by applicants, particularly regarding whether proposed activities met the criterion of ‘scientific research’.

Note A: These characteristics are based on the information in the ANAO’s 2010 Grants Administration Better Practice Guide.

Source: ANAO analysis of information published on the Fund’s website and assessment documentation.

Research Infrastructure program

2.26 In August 2010, the Trustee suggested to the Advisory Council that supporting research infrastructure from the Gift—one of the special purposes set out in the Deed—was likely to be more effective if the Fund adopted a purposeful strategy to identify potential projects for funding, rather than relying on applications received through competitive funding rounds.

2.27 In response to the Trustee’s observation, CSIRO and the Advisory Council undertook a range of tasks to support the establishment of the Research Infrastructure program. In particular, this included:

  • identifying the priority areas for investment;
  • developing a (potential) timeline for engaging with the key stakeholders in each of the priority investment areas;
  • establishing an understanding of the relative maturity of the governance arrangements and research infrastructure needs in each priority investment area; and
  • setting up the processes for eliciting and examining requests for financial assistance, including developing the selection criteria for assessing proposals.

2.28 The ANAO’s analysis indicates that the selection criteria developed for the Research Infrastructure program were well-designed and reflected elements of good practice (the characteristics in Table 2.6).

Special Research program

2.29 The Trustee approved financial assistance to support two research projects under the Special Research Program: access to the Australian Synchrotron by Publicly Funded Research Agencies—in May 2012; and increasing the capacity of the Australian Square Kilometre Array Pathfinder—in July 2013.

2.30 In contrast to the Research Infrastructure program—under which project proponents were also invited to submit proposals for funding—the Trustee did not adopt a structured process to establish a clear focus or purpose for the Special Research program. Rather, CSIRO advised the ANAO that ‘…proposals were invited in relation to areas that the Trustee considered would require a potential intervention to support nationally significant research activities…’. The ANAO found no evidence that the Trustee had sought to actively identity and assess the relative priorities or merits of the range of research activities that may have warranted support. Rather, each of the two decisions to invite proposals seeking financial assistance was largely unplanned and unrelated.

2.31 In order to support the preparation and assessment of the proposals submitted under the Special Research program, CSIRO developed a separate set of selection criteria. The ANAO’s analysis indicates that the selection criteria developed for each proposal were sound and, while separate, were broadly consistent with each other. Like the selection criteria for the Research Infrastructure program, the criteria were well-designed and consistent with the characteristics of good practice set out in Table 2.6. However, details of the selection criteria were restricted to the organisations preparing the respective proposals and were not made public—a decision that contrasts with the approach to transparency and consistency adopted in administering other aspects of the Fund. CSIRO advised the ANAO that it took the approach of limiting publicity about the Special Research program because:

… publication of the program was not necessary for soliciting applications…[and] reflected a pragmatic approach to managing expectations in relation to the availability of funding ….

Assessment decisions

2.32 The ANAO’s examination of a sample of 5016 assessments from the Research Projects, Research Infrastructure and Special Research programs the indicates that:

  • the assessments have been conducted in accordance with established processes;
  • for the most part, the records of assessment contained a sufficient level of evidence and analysis to justify the decisions made; and
  • the decisions made reflected the information provided by the applicants.

2.33 However, as shown in Table 2.7, there were some gaps in the level of documentation supporting the assessment processes.

Table 2.7: Documentation gaps in assessment records

Fund program

Description

Research Projects

The records of the Expert Panel’s assessments of applications in Rounds One and Two could not be located at the time of the audit.

In Round Four, while the basis of the Expert Panel’s selection of a shortlist of 21 Expressions of Interest for further consideration was clearly documented, the Panel’s reasons for deciding to exclude 13 of these proposals during its subsequent deliberations were not recorded.

On the small number of occasions when the Advisory Council’s decision did not align with the advice provided by the Expert Panel, the reasons were not clearly documented.

Special Research

In one case, although the records indicate that the Advisory Council assessed (and accepted) the proposal, the records did not set out the views of the Advisory Council in terms of the program’s selection criteria. The only comment recorded about the quality of the proposal was that the proposal was ‘clear and well-written’.

Research Infrastructure

For each of the three projects, the records of assessment would have been improved by more clearly and consistently demonstrating the views of the Advisory Council in terms of the program’s selection criteria. In particular, none of the records examined contained any references to discussions that clearly aligned with either criterion 3a (value for money) or criterion 4 (implementation and management).

Source: ANAO analysis of assessment records.

Funding advice to the Trustee

2.34 The Advisory Council’s funding advice to the Trustee for each of the programs examined by the ANAO contained a clear recommendation about which projects should be funded, for what amount, and whether any conditions should be attached. However, the Council’s advice generally did not outline the basis for the Council’s recommendations or describe the process by which the Council reached its funding recommendations. CSIRO advised the ANAO that much of the advice to the Trustee was provided orally during Advisory Council meetings17, and as a result, the approach adopted was to only record a summary of the issues being considered.

2.35 The ANAO has reported concerns about the adequacy of the documentation supporting key decisions in the administration of financial assistance programs in a number of recent audits.18 In this light, and with a view to enhancing the transparency and accountability of the Trustee’s decision making, it is suggested that the Council’s advice to the Trustee contain a greater level of detail supporting the Council’s recommendations, such as describing the:

  • basis for the recommendation—including: how the applications rated against the eligibility and selection criteria; and how they compared with each other; and
  • process by which the recommendation was reached—including: who was involved and how the assessment was conducted.

3. Establishing funding agreements and measuring and reporting performance and outcomes

Areas examined

The audit examined whether sound arrangements were in place, and operating effectively, for administering the funded projects—in particular, implementing and managing funding agreements, and measuring and reporting on the performance of the Fund.

Conclusion

The funded projects were generally well administered. In particular:

  • funding agreements were fit-for-purpose and, except for some delays in the early rounds of the Research Projects program, have been executed in a timely fashion; and
  • processes for monitoring and assessing the progress and performance of the funded projects were well-designed and operating effectively.

A range of useful management information was being captured, reported and utilised in administering the Fund. There remains scope to improve the ability of stakeholders and the Trustee to assess the overall effectiveness of the Fund by developing more outcome-focused performance measures and by undertaking an evaluation of the Fund’s performance.

Areas for improvement

The ANAO has made one recommendation aimed at enabling better assessments of the Fund by developing and reporting against a set of key performance measures that include a focus on quality and outcomes.

The ANAO has also noted that deferring a formal review of the Fund’s performance would limit opportunities to identify areas for improvement.

Introduction

3.1 Well-designed and structured processes for administering the financial assistance provided by a funding program will further support the effective and ethical achievement of the program’s benefits and outcomes. Such processes include: establishing funding agreements with the recipients of financial assistance; verifying the individual projects’ progress and performance against agreed milestones; and monitoring the performance of the program to gain insights into whether the program is effective—that is, delivering expected outcomes—and being administered efficiently.

Is the design of funding agreements fit-for-purpose and have agreements been executed in a timely manner?

Standard funding agreements were well-designed and fit-for-purpose, and the sample of executed agreements examined by the ANAO contained terms and conditions consistent with the approved templates. Some protracted delays were experienced in the negotiation and execution of funding agreements in Rounds One and Two of the Research Projects program. However, sensible improvements in the assessment and funding offer processes were implemented that helped overcome these delays in the later rounds.

Funding agreement design

3.2 Funding agreements can provide a sound basis for effective oversight of the funded projects. Well-designed funding agreements will set out: the terms and conditions under which funding has been provided; the objectives and expected outcomes of the funded project; the timing of key milestones and associated payments; and the requirements for reporting on progress and performance. As outlined in Table 3.1, the ANAO’s analysis indicates that the standard funding agreements developed for the Fund were fit-for-purpose.

Table 3.1: Analysis of the content of the Fund’s standard funding agreements

Good practice elementsA

Contained in funding agreement templates

Description of the project/purpose of the grant

Yes

Period covered by the funding agreement

Yes

Amount of funding

Yes

Schedule of payments

Yes

Description and expected completion date of project milestones

Yes

Rights of access

Yes

Requirements relating to insurance

Yes

Performance measures

NoB

Reporting requirements

Yes

Evaluation arrangements

Yes

Withholding funds in instances of insufficient progress/performance

Yes

Processes for varying the agreement

Yes

Processes for terminating the agreement, including recovery of unspent funds

Yes

Acquittal arrangements, including arrangements relating to unspent monies

Yes

Note A: These elements are based on information contained in the 2009 Commonwealth Grant Guidelines, the ANAO’s 2010 Grants Administration Better Practice Guide and Legal Briefing 83—Grants and Funding Programs (Australian Government Solicitor, November 2009).

Note B: The funding agreement templates used for the Research Infrastructure and Special Research Programs require the funded projects to develop and report against a series of performance measures—refer to further discussion at paragraph 3.16. On the other hand, the funding agreement template used for the Research Projects program did not require funding recipients to develop and report against performance measures.

Source: ANAO analysis of funding agreement templates.

3.3 The ANAO’s examination of a sample of ten executed funding agreements indicates that they each contain terms and conditions consistent with the approved templates. Our analysis also indicates that:

  • with one exception19, each agreement examined describes how the project is expected to contribute to the Funds’ purposes and strategic objectives;
  • the design of the reporting requirements in each agreement was appropriate to support the Trustee make objective assessments of the project’s progress against the agreed milestones; and
  • the scheduling of progress payments in the agreements aligns with the achievement of milestones and, for the most part, is evenly spread over the course of the projects’ terms.

Time to execute

3.4 The timely execution of a funding agreement with the recipient of financial assistance is an important part of effective administration. In this regard, undue delays in negotiating the terms and conditions associated with the delivery of the approved projects can affect the timing of the achievement of the funded projects’ milestones and objectives.20

3.5 Figure 3.1 illustrates the time elapsed between the dates of the Trustee’s approval of financial assistance and the execution of the relevant funding agreements for each of the four funding rounds of the Research Projects program, as well as for the Research Infrastructure and Special Research programs.

Figure 3.1: Number of months between Trustee approval and execution of funding agreement

Note: For three projects the Trustee did not record the approval until the respective funding agreements had been executed. For the purposes of Figure 3.1, the ANAO used the date of the Advisory Council’s recommendation to the Trustee as a proxy measure for the date of approval.

Source: ANAO analysis of Trustee approvals and signed funding agreements.

3.6 As demonstrated in Figure 3.1, there were protracted delays in the negotiation and execution of funding agreements in the first two rounds of the Research Projects program. In August 2011, CSIRO advised the Advisory Council that the time delays were, in part, a result of complexities associated with the development of ‘intellectual property agreements and commercialisation strategies’. The ANAO’s analysis identified that the delays were also due to the Trustee’s decision to approve financial assistance with certain conditions and/or to approve less funding than originally requested by the applicants. While this approach allowed the Trustee to mitigate some of the risks identified in funding these projects, it also resulted in proponents having to re-design or re-scope their projects.

3.7 A number of measures were introduced to help overcome and reduce the delays in negotiating funding agreements. This included:

  • requiring applicants to identify any potential changes to the standard funding agreement at the time their full proposal was submitted;
  • including a deadline for finalising the funding agreement in the Trustee’s funding offer;
  • requiring applicants to consider aspects of the management of intellectual property at the application stage; and
  • providing clearer and more timely information to those applicants invited to submit stage two proposals, including advice on the likely level of financial assistance.

3.8 As illustrated in Figure 3.1, the implementation of these measures resulted in a significant reduction in the average time to execute funding agreements in the later rounds of the Research Projects program.

Is the performance of the funded projects, including assessing the achievement of project outcomes, actively monitored?

Arrangements for monitoring and assessing the progress and performance of the funded projects, including the achievement of project deliverables and outcomes, were well-designed and operating effectively. The ANAO also observed activity to address identified performance issues.

Project monitoring

3.9 The primary mechanism used to monitor the status and performance of the projects funded from CSIRO’s Gift to the Fund—including whether the projects are performing as expected and that agreed milestones and objectives are being achieved—is the assessment of information contained in the projects’ progress reports. As shown in Table 3.2, the ANAO’s review indicates that progress reports were designed to capture a range of relevant and appropriate information. Our analysis also indicates that these reports were required to be submitted regularly—at least annually for each of the projects examined.

Table 3.2: Information captured through progress reports

Content of progress reports

Assessment of progress against milestones

Assessment of overall progress in terms of the project’s description and objectives

Details of any issues likely to impact on the completion of the project

Details of publications and early career researchers

Details of the project’s budgeted and actual-to-date income and expenditure

Confirmation of co-investment contributions

An operational planA

Note A: Operational plans are required to be submitted by the proponents of the three projects funded under the Research Infrastructure program and one of the two projects funded under the Special Research program. Among other things, the Operational plans contain details of: milestones planned to be achieved in the next reporting period; governance arrangements; performance indicators; and a schedule of risks and treatment strategies.

Source: ANAO analysis.

3.10 In addition to the progress reports, the projects funded under the Research Projects program were required to submit a supplementary report at the project mid-point—at which time proponents were also required to participate in a major review21—and upon completion of the project. The requirements relating to the submission of the supplementary reports and the major reviews were introduced in July 2013 in order to assist the Trustee to better assess:

  • whether milestones remain appropriate;
  • current and future plans for engagement with collaborators, industry and other end users;
  • progress towards the research’s adoption or application;
  • delivery of the project’s overall outputs and objectives; and
  • likely outcomes and impacts from the funding provided.

3.11 The ANAO’s examination of a sample of progress and supplementary reports indicated that these reports were being submitted in accordance with the reporting requirements. Further, our analysis also indicated that the established arrangements for the examination of these reports were working as intended, and that the assessment decisions made by the reports’ reviewers were reasonable and appropriate.

3.12 The ANAO also observed that CSIRO and the Advisory Council regularly took action to address identified performance issues. In this regard, a common method for dealing with concerns about the progress or performance of projects was to request a further report—known as an ‘ad-hoc’ report. Case Study 1 outlines an example of the measures taken by the Advisory Council in response to information reported in a project’s progress report.

Case study 1. The provision of transition funding

In November 2010—as part of Round One of the Research Projects program—the Trustee approved financial assistance of $2 million to support the first stage (develop a proof of concept) of a project. At that time, the Trustee observed that further funding might be provided to support the project’s second stage and set aside funds for the development phase should it proceed.

During the assessment of the project’s third progress report, some concerns about the ability of the project to fully achieve the objectives of the first stage in the required timeframe were identified. As a result, the Advisory Council decided to advise the project proponents of the basis on which any funding for the second stage of the project would be assessed, including the importance of stage one being successful.

In the event, the project proponents submitted a proposal seeking funding for the project’s second stage. After assessing that proposal, the Council recommended to the Trustee that funding should not be provided for the next stage. Among other things, the Council considered that the proposal had not sufficiently demonstrated the necessary outcomes, nor had it met expectations regarding co-investment.

Despite its views about the second stage of the project, the Council considered it ‘desirable to transition out of funding [the project] in a manner that gave scope for the first phase to be progressed further and to achieve impact’. Specifically, the Council recommended—and the Trustee approved—the provision of additional funding of $250 000 to support a three month extension to the project period.A

The assessments of the final progress and supplementary reports submitted by the project proponent at the end of the three month ‘transition period’ indicate that the project had made good progress during the extra period and had achieved the objectives of the additional funding.

Note A: The funding was offered subject to the project proponents providing an equivalent level of co-investment and the proponents submitting a further proposal setting out, among other things, details of the milestones to be achieved in the three month period and plans to secure potential alternative sources of funding for the project.

Source: ANAO analysis of Fund documentation.

Status of the funded projects

3.13 Five of the 22 projects funded from the three programs examined in this audit have been completed. The ANAO’s examination indicates that the final progress and supplementary reports for each of these projects had been assessed and accepted in accordance with established practice. Our analysis of these reports also indicates that decisions made about the project’s performance, including the achievement of deliverables, were reasonable and appropriate. The ANAO’s analysis of an assessment by CSIRO of the status of the remaining funded projects, which was provided to the Advisory Council meeting in August 2015, indicates that the remaining 17 projects were progressing in line with the key milestones set out in their respective funding agreements.

Is useful and instructive information about the Fund’s performance regularly captured, measured and reported?

A range of useful management information, including information concerning the Fund’s financial position, the status of the funded projects, and the Fund’s results against its key performance indicators, has been captured, reported and utilised in administering the Fund. However, measuring the effectiveness of the Fund’s overall performance would be enhanced by developing more outcome-focused performance measures. Although processes are in place for capturing information on the contributions made by individual projects towards the Fund’s objectives, at the time of the audit, the Trustee had not undertaken a formal evaluation to assess the outcomes achieved by the Fund as a whole; or to identify opportunities for improvements to the Fund’s administration.

Performance measures

3.14 Key performance measures can provide management, as well as other interested stakeholders, with insights as to whether a program is being managed well and is meeting its objectives. When designing performance measures, entities should aim to capture information concerning the outputs delivered and the outcomes achieved, as well as information about the effectiveness and efficiency of the delivery of the program.22

3.15 Measures to help assess the performance of the Fund have been developed at two levels. Firstly, as mentioned in Table 3.1, the projects funded under the Research Infrastructure and Special Research programs (but not the Research Projects program) were required to report against a set of key performance indicators. Secondly, a set of indicators has been developed to help measure the performance of the Fund as a whole.

Project-level performance measures

3.16 The ANAO’s analysis indicates that the model performance measures contained in the funding agreement templates for the Research Infrastructure and Special Research programs were well-designed. In particular, the measures were designed to capture useful information about the progress and performance of the funded projects. The ANAO’s analysis also indicates that the performance information reported by the projects funded under these programs aligned with the requirements in the projects’ funding agreements—albeit at the time of the audit, some performance indicators were still under development.

Program-level performance measures

3.17 A structure for assessing the Fund’s overall performance was first developed in 2011. At that time a series of key goals or result areas were identified, together with a set of key measures of performance. In 2013, the framework was updated to increase the focus on the Fund’s impacts. Details of the Fund’s key result areas—and attendant performance measures—which were endorsed by the Advisory Council in June 2013, are shown in Table 3.3.

Table 3.3: Measuring the Fund’s performance

Key result areas

Key performance measures

The Science and Industry Endowment Fund provides funds that enhance Australia’s strength in research to deliver high risk frontier and unique science outcomes to the benefit of Australian industry and the Australian community.

  1. Proportion of projects involving research in areas of national priority.A
  2. Proportion of projects involving more than one organisation.

Funded projects deliver new knowledge, products and services.

  1. Number of publications.

Building globally significant critical mass.

  1. Number of early career researchers funded.

Enduring productive collaborations are established.

  1. The financial contributions of partners.

Note A: CSIRO advised that the national priority areas mentioned in the first key performance measure are the Australian Government’s National Research Priorities, which were developed in 2002–03. CSIRO advised the ANAO that following the release of the Australian Government’s new Science and Research Priorities in May 2015, it will reconsider this key performance measure as part of its preparations for the 2016–17 Budget.

Source: Advisory Council Meeting No.15 (June 2013).

3.18 Results against the Fund’s key performance measures have been reported in each of CSIRO’s Annual Reports since 2011–12. Table 3.4 contains a summary of the reported results for each of the measures.

Table 3.4: Reported results against key performance measures for the period 2011–12 to 2014–15

Key performance measures

2011–12

2012–13

2013–14

2014–15

Proportion of projects involving research in areas of national priorityA

100%

100%

100%

100%

Proportion of projects involving more than one organisation

85%

90%

92%

92%

Number of publicationsB

79

158

226

276

Number of early career researchers fundedB

23

42

131

241

Financial contribution by partners

57%

69%

68%

70%

Note A: The results shown exclude projects funded under the Promotion of Science program.

Note B: The results reported for these measures are cumulative.

Source: ANAO analysis of CSIRO’s Annual Reports.

3.19 Overall, the ANAO considers that the key result areas—shown in Table 3.3—are sound and well targeted. In particular, they are outcome-focused and aligned with the strategic objective and primary purpose of the Fund. In contrast, as shown in Table 3.5, the ANAO’s analysis indicates that there is scope to improve the Fund’s key performance measures.

Table 3.5: Assessment of the Fund’s key performance measures against key characteristics

CharacteristicA

Assessment

Relevant

Yes. Each of the measures is designed to measure something that is relevant to the Fund’s objective.

Reliable

Yes. Each of the measures is reliable in that they can be objectively and readily measured, and performance can be tracked over time.

Complete

No, because:

  • the indicators are all quantitative in nature. There are no explicit measures of a qualitative nature;
  • three of the five indicators (one, two and five) are input measures while the other two (three and four) are output measures or potentially proxy measures.B None of the measures are designed to measure outcomes;
  • there are no benchmarks or targets for any of the measures; and
  • not all aspects of the Fund’s objectiveC are addressed by the measures. Specifically, there are no measures relating to the following aspects of the objective:
    • ‘fundamental research into new paradigms for sustainable resource use, environmental protection and community health’; and
    • ‘tactical research to fast-track solutions to national challenges’.

Note A: These characteristics are based on the criteria developed to evaluate the appropriateness of an entity’s key performance indicators contained in Audit Report No.28, 2012–13, The Australian Government Performance Measurement and Reporting Framework: Pilot Project to Audit Key Performance Indicators.

Note B: A proxy measure is a surrogate indicator that provides indirect insights into the contribution made towards achievement of the program’s objective. Proxy measures are valuable where a program’s objectives are difficult to measure because, for instance, the objectives have been set at a high level, their achievement is dependent on a number of external factors or the outcomes can only be achieved over the longer-term.

Note C: The Fund’s strategic objectives are shown at paragraph 1.8.

Source: ANAO analysis.

3.20 While the existing measures offer some useful insights into the performance of the Fund, it is not possible to determine from these measures alone the extent to which the Fund is achieving its overall objectives. CSIRO has previously recognised shortcomings in the Fund’s key performance measures:

There are limitations to the way in which the effectiveness of the SIEF in meeting its objectives can be measured because the contribution of research can only be measured in the long term.23

3.21 Nonetheless, incorporating outcome-focused performance measures would help the Trustee, and other interested parties, to better assess the effectiveness of the Fund. The Advisory Council’s intention to undertake an evaluation of the Fund—discussed further at paragraph 3.32—provides an opportunity to re-examine the design of the Fund’s performance measures. In this context, the Trustee could usefully draw on the qualitative and outcome-focused performance measures developed by the Australian Research Council (Programme 1.1—Discovery and Programme 1.2—Linkage).24

Recommendation No.1

3.22 To enable better assessments of the performance of the Science and Industry Endowment Fund, the ANAO recommends that the Trustee and the Commonwealth Scientific and Industrial Research Organisation develop and report against a set of key performance measures that include a focus on quality and outcomes.

Entity response:

3.23 CSIRO agrees with Recommendation one: CSIRO will work with the Trustee and Advisory Council to develop a set of key performance measures with a greater focus on quality and outcomes, and which includes both quantitative as well as qualitative measures addressing all the Fund objectives. The refined key performance indicators will be used to evaluate not just the outcomes and (potential) impact of completed projects, but also ongoing activities as well as the management framework.

Management reporting

3.24 There were well-established arrangements in place for capturing and reporting information about the operations of the Fund to the Trustee and the Advisory Council. Among other things, at each Advisory Council meeting, the members of the Council and the Trustee were provided with:

  • financial reports and supporting analysis, including details of actual and forecast receipts and expenditure, as well as statements of assets and liabilities;
  • advice on issues associated with the administration of the Fund—for instance, the Council was provided with annual updates on the Fund’s risk management plan and results against the Fund’s key performance measures; and
  • an update on each of the Fund’s financial assistance programs—typically, this includes: details of the status of the funded projects; and a discussion of issues concerning the funded projects’ performance (and, where relevant, underperformance).

3.25 On occasion, the Advisory Council had also been provided with information on the distribution of financial assistance from the Fund. For instance, at its meetings in March 2012 and August 2015, the Advisory Council was presented with analysis of the types of research supported by the 17 projects funded under the Research Projects program. The results of the analysis prepared for the Council’s August 2015 meeting are shown in Figures 3.2 (Fields of Research) and 3.3 (Socio-Economic Objectives).25

Figure 3.2: Fields of Research

Note A: The rows add to more than 100 per cent as some projects support research across more than one field.

Source: CSIRO’s analysis of distribution of Research Projects funding.

Figure 3.3: Socio-Economic Objectives

Note A: The rows add to more than 100 per cent as some projects support research across more than one objective.

Source: CSIRO’s analysis of distribution of Research Projects funding.

3.26 As mentioned in paragraph 2.15, each year the Advisory Council considers CSIRO’s performance as the Fund’s service provider. In March 2015, when presented with details of CSIRO’s personnel and operating costs, the Advisory Council observed that the ‘service costs were low relative to the level of funds managed per annum’. The ANAO’s analysis indicates that the fees paid to CSIRO to the end of June 2015 for administering the Gift represent approximately two per cent of the total amount of financial assistance approved by the Trustee.26

3.27 The ANAO’s analysis indicates that the management reporting provided to the Trustee and the Advisory Council aligned with the reporting requirements set out in the Services Agreement. ANAO’s analysis of the usefulness of the suite of management reporting is set out in Table 3.6.

Table 3.6: ANAO’s assessment of management reporting

Internal management reports include information and analysis intended to:

Result

Assess the Fund’s financial performance and position

Met

Provide insights into current issues facing the Fund

Met

Raise awareness about events and factors with the potential to influence the performance of the Fund

Met

Support effective program oversight

Met

Assess the Fund’s performance

Partly metA

Note A: The usefulness of management reporting in terms of assessing performance has been assessed as Partly met due to shortcomings with the design of the Fund’s performance measures.

Source: ANAO analysis.

Evaluation activity

3.28 Evaluations—which typically involve an assessment of the appropriateness, effectiveness and efficiency of a program—can provide valuable insights into program design and performance, including achievements against program objectives. Evaluations can also help to inform decisions about a program’s future direction, including identifying opportunities for improvement.27

3.29 There were a number of processes in place for capturing information on the contribution of individual projects towards the Fund’s objectives. For instance, useful information was captured through:

  • ‘project impact’ brochures—these provide a high level overview of each project, outlining: the challenge; response; collaboration; and projected impacts28;
  • supplementary reports and major review presentations; and
  • the key performance indicators in place for the five projects funded through the Research Infrastructure and Special Research programs.

3.30 Despite this range of information, the ANAO did not find evidence that the data collected on individual projects had been analysed to help gauge the performance of the Fund more broadly. Furthermore, no evaluation activity had been undertaken at the whole-of-Fund level.

3.31 In August 2013, the Advisory Council agreed to undertake a review of the Fund’s performance in late 2014 or early 2015. However, in December 2013, when considering an outline of a plan for the conduct of the review, the Council observed that:

In light of these objectives [that the review should be aimed at demonstrating the value of the program] the review should be deferred until more evidence is available and the majority of projects have delivered final outcomes.

3.32 Subsequently, at its August 2015 meeting, the Advisory Council was advised that the proposed examination of the Fund’s performance should be conducted from around mid-2017 because ‘most Research Projects were expected to be completed and the Research Infrastructure and Special Research projects would be well advanced.’ The Council observed the importance of the proposed evaluation activity to inform decisions about the future of the Fund and agreed that the timing of the review should be brought forward to late-2016 so as to maximise its impact.

3.33 While acknowledging that decisions on the optimal approach to, and timing of, the conduct of the review of the Fund is a matter for the Trustee, the ANAO notes that deferring such a review further limits the opportunity for the Trustee to identify areas for improvement in either the Fund’s administration or program design. In January 2016, CSIRO advised the ANAO that it anticipated an overall evaluation of the Fund would be undertaken by the end of 2016.

Appendices

Appendix 1: Entity’s response

Please refer to the attached PDF of the report for Appendix 1 (Entity’s response).

Appendix 2: Schedule of financial assistance from CSIRO’s Gift to the Fund

Fund program

Amount ($m)

Research Projects (condition of CSIRO’s Gift)

Rural Wireless Broadband Program (Ngara)

10.0

Research Projects – Round One

Expressions of Interest (18); Full Applications (5); Projects funded (5); Funds committed ($17.7m)

 

The Australian Imaging, Biomarker and Lifestyle Study of Ageing

5.2

Plant Breeding

4.5

Health Diagnostics

2.5

eReefs–Transforming the Science and Management of the Great Barrier Reef

4.0

Two Jump Start Projects for the Global Ecosystem Assessment Initiative

1.5

Research Projects – Round Two

Expressions of Interest (32); Full Applications (8); Projects funded (3); Funds committed ($16.0m)

 

Energy Waste Roadblock

6.0

Stem Cell Biology

6.0

Reversible Addition-Fragmentation Chain Transfer (RAFT) for Biomedical Applications

4.0

Research Projects – Round Three

Expressions of Interest (147); Full Applications (7); Projects funded (3); Funds committed ($13.5m)

 

Early Nutrition, the Epigenome and the Prevention of Disease

5.0

Adaptation to Climate Change

3.5

Renewable Chemicals

5.0

Research Projects – Round Four

Expressions of Interest (175); Full Applications (8); Projects funded (5); Funds committed ($20m)

 

Manufacture of a Small Demonstrator Aero-Engine

4.0

Big-Data Knowledge Discovery

4.0

The Distal Footprints of Giant Ore Systems

4.0

Forests for the Future

4.0

High Performance Solar Cell Technology

4.0

Total Research Projects

$77.2

Special Research

Access to Synchrotron Science

7.5

Increasing the capacity of the Australian Square Kilometre Array Pathfinder

6.0

Total Special Research

$13.5

Research Infrastructure

Advanced Resource Characterisation Facility at the National Resource Sciences Precinct

12.4

National Agricultural and Environmental Sciences Precinct

13.7

Biomedical Materials Translational Facility at the Manufacturing and Materials Precinct

5.5

Total Research Infrastructure

$31.6

Promotion of Science

Joint CSIRO-Macquarie University Chair (condition of the Gift)

2.0

Various Scholarships and Fellowships schemes (condition of the Gift)

9.3A

Australian Academy of Science (Lindau) Fellowships

0.6

STEM+ Business Fellowships

7.9

Total Promotion of Science

$19.8

Note A: In June 2011, the Advisory Council agreed to extend the John Stocker Postgraduate Scholarship and Postdoctoral Fellowship schemes (part of the Promotion of Science program) by 12 months at a cost of approximately $1.8 million.

Source: ANAO analysis of the Fund’s records.

Appendix 3: Special purposes and related investment principles

Special purposes

Investment principles

Emerging Research

Comprises projects in emerging areas of research of a special or priority character. Includes grants to establish new fields of research that are likely to have known applications for industry, society or the environment, response to newly arising issues or priorities, and funding to tackle scientific or technological problems of a new or unusual nature.

Strategic Research

Comprises research into scientific approaches to solutions to threats, challenges or opportunities arising in or for Australia. Includes devoting resources to national challenges or national priorities, and research into opportunities where significant benefits exist.

Supporting Research

For established or long term research programs which aim to deliver scientific advances in areas of benefit to Australia. Includes funding of demonstration plants or pre-pilot plants, but would not normally include grants for extensions to existing programs.

Research Infrastructure

Comprises the creation or development of nationally significant facilities for the conduct of research. Includes investment into national scale scientific equipment and special purpose facilities for the conduct of scientific research.

Promotion of Science

Comprises grants for scholarships and fellowships for research. Comprises research undertaken by early career scientists, the appointment or joint appointment of scientists to university positions, and scholarships and fellowships including extensions to existing schemes.

Source: ANAO analysis of the Deed and the investment principles of the Fund.

Footnotes

1 CSIRO’s Gift was paid to the Fund in three tranches of $50 million—in October 2009, February 2010, and April 2011. Pursuant to the Deed, at the time of each payment, CSIRO stipulated conditions regarding the use of its Gift.

2 As a former Commonwealth Authorities and Companies Act 1997 body, CSIRO was not subject to the Australian Government’s grants administration framework, which was first introduced in the 2009 Commonwealth Grant Guidelines.

3 Under this project, CSIRO agreed to grant licences to major technology companies to use the organisation’s WLAN technology.

4 This assistance was provided under the Creativity in Science and Technology (CREST) awards program.

5 A more detailed breakdown of the financial assistance provided by the Fund from CSIRO’s Gift is provided in Appendix 2.

6 The audit focused on three of the Fund’s four programs—Research Projects; Research Infrastructure; and Special Research.

7 As a former Commonwealth Authorities and Companies Act 1997 body, CSIRO was not subject to the Australian Government’s grants administration framework, which was first introduced in the Guidelines.

8 The Science and Industry Endowment Act 1926.

9 The primary purpose aligns with the first key function of CSIRO set out in section 9(1) of the Science and Industry Research Act 1949.

10 The ANAO examined the application of these investment principles as part of its review of the design of assessment processes—see further discussion at paragraph 2.22. The special purposes contained in the Deed and the investment principles adopted by the Trustee are shown in Appendix 3.

11 The Board Endowment Committee was established by the CSIRO Board in October 2009 to assist the Board to fulfil its governance responsibilities in relation to the Gift. The Board observed that the Committee provided a vehicle for timely decision-making in relation to the tasks associated with making the Gift to the Fund. The functions of the Endowment Committee were subsumed by the Board in June 2011.

12 See for example: ANAO, Better Practice Guide, Implementing Better Practice Grants Administration, June 2010, p. 87; and ANAO and Department of the Prime Minister and Cabinet, Better Practice Guide, Successful Implementation of Policy Initiatives, October 2014, pp. 29-34.

13 ANAO, Audit Report No.47, 2013–14, Managing Conflicts of Interest in FMA Agencies, p. 29.

14 On multiple occasions projects requesting above $6 million were regarded as ‘too big’ by the Expert Panel despite fitting within the publicised funding envelope.

15 Except in the case of stage one of Round One of the Research Projects program when the Expert Panel’s advice on the assessment of Expressions of Interest went directly to the Trustee, with a copy provided to the Advisory Council.

16 The ANAO’s sample comprised: the assessments undertaken for the three projects funded under the Research Infrastructure program; the assessments undertaken for the two projects funded under the Special Research program; and a sample of 45 assessments of applications for funding under the Research Projects program.

17 As noted in Figure 1.1, the Trustee attends the meetings of the Advisory Council.

18 ANAO, Audit Report No.37, 2012–13, Administration of Grants from the Education Investment Fund, p. 69.

19 In this case, the funding agreement simply states that the project is designed to address the Fund’s ‘Emerging Research’ special purpose.

20 ANAO, Audit Report No. 37, 2012–13, Administration of Grants from the Education Investment Fund, p. 77.

21 A major review entails project proponents discussing the performance of their project—through a formal presentation and interview—with a review panel comprising selected members of the Advisory Council and selected expert reviewers. At the time of the audit, 12 major reviews had been completed, with a further review scheduled for February 2016. Four of the six projects approved in Round One have not been subject to major reviews as these projects were either completed or substantially completed by the time that the major review process was introduced.

22 ANAO, Audit Report No.21, 2013–14, Pilot Project to Audit Key Performance Indicators, p. 60.

23Portfolio Budget Statements 2015–16, Industry and Science Portfolio, p. 147.

24Portfolio Budget Statements 2015–16, Education Portfolio, pp. 159–163.

25 Fields of Research (FoR) and Socio-Economic Objectives (SEO) are two of the classification groups developed for use in the measurement and analysis of research and experimental development activities undertaken in Australia and New Zealand. FoR categorises the activity according to the methodology used; while SEO classifies the activity according to the intended purpose or outcome of the research. Refer to Australian and New Zealand Standard Research Classification, 2008. Available from: <http://www.abs.gov.au/ausstats/ abs@.nsf/Latestproducts/1297.0Main%20Features12008?opendocument&tabname=Summary&prodno=1297.0&issue=2008&num=&view=> [Date Accessed: 30 September 2015].

26 The ANAO’s analysis also indicates that the total fees estimated to be payable to CSIRO over the life of the Gift represent approximately 2.7 per cent of the total amount of forecasted financial assistance.

27 ANAO, Audit Report No.31, 2014–15, Administration of the Australian Apprenticeships Incentives Program, p. 58.

28 The project impact brochures were developed as a means of providing insights at the individual project level into the Fund’s impacts, or intended impacts. At the time of the audit, project impact brochures had been developed for each of the 17 projects funded under the Research Projects program. These brochures were available on the Fund’s website. During the audit, CSIRO advised the ANAO that the development of the brochures would be extended to the Fund’s other financial assistance programs.

Related documents: