Background

1. The Science and Industry Endowment Fund (the Fund) was established by the Science and Industry Endowment Act 1926 (the Act) to provide assistance to persons engaged in scientific research and in the training of students in scientific research. Since 1986, the Chief Executive of the Commonwealth Scientific and Industrial Research Organisation (CSIRO) has been the Trustee of the Fund. The Trustee is the Fund’s primary decision-making authority and is responsible for the effective administration of the Fund.

2. In June 2009, CSIRO’s Board agreed that the net proceeds from CSIRO’s wireless local area network (WLAN) licensing project—at that stage expected to be around $150 million—should be donated to the Fund. CSIRO’s stated aim in making the Gift to the Fund was to support nationally important research with the potential to create substantial long term benefits for Australia, beyond CSIRO’s usual business.

3. CSIRO’s Gift to the Fund was formalised on 15 October 2009 through a Deed of Gift (the Deed). The Deed sets out the purposes for which the Gift can be used, and the requirements relating to the Gift’s administration.¹ At the same time, the Trustee and CSIRO entered into a Services Agreement covering the provision of administrative support to the Fund. On 20 October 2009, the renewed Fund was launched by the then Minister for Innovation, Industry, Science and Research.

4. At the end of December 2015, the Trustee had committed $142 million from CSIRO’s Gift to 26 projects across four funding programs. At the time of the audit, the 26 projects funded from CSIRO’s Gift have attracted almost $370 million in co-investment funding from around 130 collaborating organisations, including CSIRO. As well as playing a central role in the administration of the Fund, CSIRO is a participant in most of the projects that have received financial assistance from the three funding programs examined in this audit—the Research Projects, Research Infrastructure and Special Research programs.

Audit objective and criteria

5. The audit objective was to assess the effectiveness of the administration of CSIRO’s Gift to the Fund. To form a conclusion against the audit objective, the ANAO adopted the following high level audit criteria:

• arrangements for the administration of the Gift were well-designed and have been implemented effectively; and
• financial assistance from the Gift has been administered effectively and expected outcomes are being achieved.

6. In designing the arrangements for administering the Gift, CSIRO advised the ANAO that it had regard to the key principles for grants administration set out in the Australian Government’s grants administration framework.² In addressing the audit’s second criterion, the ANAO considered CSIRO’s application of these principles, including: transparent decision making; consistent and equitable treatment of applications for financial assistance; and achieving outcomes and value with public money.

Conclusion

7. CSIRO’s Gift to the Fund has generally been well administered. In particular, the approach to designing the arrangements for managing the Gift was sound, and for the most part, these arrangements, including the processes for administering financial assistance from the Gift, have been operating as intended. To enable better assessments of the Fund’s performance and achievements, CSIRO and the Trustee should develop more qualitative and outcomes focused performance measures, and consider undertaking a formal evaluation. In addition, improved record-keeping would enhance the transparency of decision-making.

Supporting findings

Designing and administering the Deed of Gift

8. The design of the Deed of Gift supports the achievement of CSIRO’s objective in providing the Gift to the Fund—to provide funding for scientific research benefitting Australia. Of note, CSIRO and the Trustee: adopted a consultative and orderly approach to the design and administration of the Deed; developed a set of investment principles and strategic objectives to give practical effect to the broad purposes of the Gift; and established a number of robust accountability and ethical measures to support transparency in the Gift’s administration.

9. A well-designed risk management plan was developed and approved for the Fund—albeit three years after the payment of CSIRO’s first tranche of the Gift to the Fund. The plan’s ongoing accuracy and integrity has been regularly monitored, with refinements made to risk mitigation strategies as needed.

10. CSIRO has also put in place a series of structured measures to help manage conflicts of interest. The ANAO’s examination indicates that the established arrangements have, for the most part, been operating as intended. Some shortcomings were observed in the recording of conflicts of interest relating to the assessment of a small number of applications for funding from the Research Projects program, but these gaps did not affect the overall transparency of the decisions made.

11. The Fund’s website contains a range of useful promotion and communication information. Program guidelines developed for the Research Infrastructure and Special Research programs were well-designed and informative. Formal program guidelines were not produced for any of the four rounds of the Research Projects program, but relevant information about the program was provided on the Fund’s website. This information was provided in a timely manner, and overall, the information was well-presented and informative. However, the lack of clear eligibility criteria for the Research Projects program resulted in a degree of unnecessary assessment work by the Expert Panel, which affected administrative efficiency.

12. The three funding programs examined in this audit had separate and distinct application and assessment processes. The processes for assessing applications for funding under the Research Projects program were sound. In particular, the approach to assessing potential projects was refined and modified over the course of the four assessment rounds to reflect lessons learned. Under the Research Infrastructure program, detailed planning was undertaken to identify investment opportunities and to develop the associated assessment processes.

13. In contrast, a clear purpose was not established at the outset of the Special Research program. Rather, the decisions to seek proposals for the two projects funded under the program were largely unplanned and unrelated. Further, while assessment criteria were developed for each of the two proposals, details of these criteria were restricted to the organisations preparing the respective proposals and not made public.

14. The ANAO’s examination of a sample of assessments from the Research Projects program and each of the assessments undertaken for the Research Infrastructure and Special Research programs indicates that the assessments had appropriate regard to information provided by applicants and were undertaken in accordance with the established processes. There were, however, some shortcomings in relation to the accuracy and completeness of record-keeping. The ANAO suggests that in order to enhance the transparency and accountability of decision-making, the Advisory Council’s advice to the Trustee contain a greater level of detail supporting the basis of the Council’s recommendations.

Establishing funding agreements and measuring and reporting performance and outcomes

15. Standard funding agreements were well-designed and fit-for-purpose, and the sample of executed agreements examined by the ANAO contained terms and conditions consistent with the approved templates. Some protracted delays were experienced in the negotiation and execution of funding agreements in Rounds One and Two of the Research Projects program. However, sensible improvements in the assessment and funding offer processes were implemented that helped overcome these delays in the later rounds.

16. Arrangements for monitoring and assessing the progress and performance of the funded projects, including the achievement of project deliverables and outcomes, were well-designed and operating effectively. The ANAO also observed activity to address identified performance issues.

17. A range of useful management information, including information concerning the Fund’s financial position, the status of the funded projects, and the Fund’s results against its key performance indicators, has been captured, reported and utilised in administering the Fund. However, measuring the effectiveness of the Fund’s overall performance would be enhanced by developing more outcome-focused performance measures, and the ANAO has made a recommendation on this matter. Although processes are in place for capturing information on the contributions made by individual projects towards the Fund’s objectives, at the time of the audit, the Trustee had not undertaken a formal evaluation to assess the outcomes achieved by the Fund as a whole; or to identify opportunities for improvements to the Fund’s administration. In January 2016, CSIRO advised the ANAO that it anticipated an overall evaluation of the Fund would be undertaken by the end of 2016.

Recommendation

Summary of entity response

18. CSIRO notes the audit and confirms it accepts the recommendation from the report and advises it has already begun implementing the recommendation, and notes the very professional audit.

Background

1.1 The Science and Industry Endowment Fund (the Fund) was established by the Science and Industry Endowment Act 1926 (the Act) to provide assistance to persons engaged in scientific research and in the training of students in scientific research. Section 4 of the Act states that the Fund consists of:

• the amount of one hundred thousand pounds appropriated to the Fund from the Consolidated Revenue Fund and the income derived from investment of that amount; and
• any gifts or bequests given or made to the Fund, as well as the income derived from investment of those gifts or bequests.

1.2 Since 1986, the Act has provided that the Chief Executive of the Commonwealth Scientific and Industrial Research Organisation (CSIRO) is the Trustee of the Fund. In this role, the Trustee is responsible for the effective administration of the Fund and is the Fund’s primary decision-making authority.

1.3 In October 2007, the CSIRO Board decided that any proceeds from the organisation’s wireless local area network (WLAN) licensing project³ should be applied to the advancement of scientific research in Australia. In June 2009, the Board endorsed a recommendation from CSIRO’s management that the net proceeds from the WLAN project—at that stage expected to be around $150 million—should be donated to the Fund with the intention of:

[funding] nationally important research that otherwise could not be funded in the normal course of CSIRO’s business.

1.4 CSIRO’s Gift to the Fund was formalised through a Deed of Gift (the Deed), which was executed on 15 October 2009 by the Trustee and the CSIRO Board. The Deed outlines the arrangements to be adopted to administer CSIRO’s Gift, including setting out the:

• purposes for which the Gift can be used; and
• mechanisms for providing support and assistance to the Trustee in the exercise of their responsibilities.

1.5 CSIRO’s Gift of $150 million to the Fund was publicly announced by the then Minister for Innovation, Industry, Science and Research on 20 October 2009. At that time, the Minister observed that the Fund would support research activities ‘at Universities as well as at CSIRO’ and expressed the hope that CSIRO’s donation would be ‘augmented by donations from industry and other benefactors’. The Minister also announced the following allocations of financial assistance from the Fund, which had been stipulated by CSIRO as a condition of making the Gift:

• subject to the receipt of an acceptable proposal, up to $10 million to CSIRO’s Information, Communications and Technology Centre to support further research into wireless broadband technologies;
• subject to the receipt of an acceptable proposal, approximately $2 million to fund a Professorial Chair in Wireless Research (a joint appointment between CSIRO and Macquarie University); and
• up to $7.5 million to establish a series of scholarship and fellowship schemes.

1.6 On the same day as the Deed was executed, the Trustee appointed CSIRO—under a Services Agreement—to assist in the administration of the Fund. The services provided to the Fund by CSIRO include:

• managing the Fund’s investments;
• legal and financial management services;
• preparing management reports;
• assisting in the design of the processes for providing financial assistance;
• oversighting the preparation of funding agreements and the monitoring of the performance of the funded projects;
• secretariat support to the Trustee and the Advisory Council; and
• managing the Fund’s communication activities.

Nature of financial assistance

1.7 Prior to CSIRO’s Gift, the Fund’s contribution to Australia’s scientific research effort was modest. The total assets of the Fund in June 2009 amounted to around $530 000, and over the period 2005–06 to 2008–09 the Fund provided, on average, approximately $22 000 a year in financial assistance.⁴ Since October 2009, the Fund has provided financial assistance from CSIRO’s Gift through four programs—Research Projects; Research Infrastructure; Special Research; and Promotion of Science. Table 1.1 provides a summary of the each of these programs, including details of the amounts of financial assistance provided.⁵

Table 1.1: Summary of the Fund’s financial assistance programs relating to CSIRO’s GiftA

Note A: CSIRO’s Gift was provided to the Fund in three tranches of $50 million—in October 2009, February 2010, and April 2011.

Note B: Funds approved and spent over the period October 2009 to December 2015.

Note C: This aspect of the Promotion of Science program comprises three broad project groups—Fellowships and Scholarships, Australian Academy of Science (Lindau) Fellowships, and STEM+ Business Fellowships. At the time of the audit, around 60 individual projects have been provided with financial assistance across these three project groups.

Source: ANAO analysis of the website of the Fund and of the Fund’s ‘cash pipeline’ reports.

The Fund’s objective

1.8 The stated objective of the Fund is to make strategic investments in scientific research that address issues of national priority for Australia. Specifically, the Fund aims to achieve this objective by investing in science that contributes to Australia’s sustainable future such as:

• fundamental research into new paradigms for sustainable resource use, environmental protection and community health;
• tactical research to fast-track solutions to national challenges;
• collaborative research that brings together organisations capable of working together on solutions to national challenges; and
• scholarships that create and sustain young researchers capable of addressing national challenges.

Collaboration and co-investment

1.9 The Deed provides that research funded from the Gift may be carried out ‘by or within one or more single institutions or within collaborative partnerships’. To give effect to this feature of the Deed, two important design elements of the financial assistance provided from the Gift are the:

• promotion of collaborative research and related activities; and
• leveraging of co-investment—either from the collaborators or from other parties, including industry.

1.10 Table 1.2 contains a summary of the levels of collaboration and co-investment associated with the 26 projects funded from CSIRO’s Gift at the time of the audit.

Table 1.2: Collaborator and co-investment levels

Note A: A number of organisations are involved in more than one project. For the purposes of Table 1.2, these have been counted as a separate collaborator each time.

Note B: CSIRO advised the ANAO that the number of collaborators involved and the amount of co-investment in the Promotion of Science program will increase as further individual fellowships and scholarships are announced in the future.

Source: ANAO analysis of relevant Fund documentation.

Roles and responsibilities

1.11 The key roles and responsibilities relating to the operation of the Fund are illustrated in Figure 1.1.

Figure 1.1: Key roles and responsibilities

Source: ANAO analysis of relevant Fund documentation and the Deed.

Audit objective, criteria and approach

1.12 The audit objective was to assess the effectiveness of the administration of CSIRO’s Gift to the Fund. To form a conclusion against the audit objective, the ANAO adopted the following high level audit criteria:

• arrangements for the administration of the Gift were well-designed and have been implemented effectively; and
• financial assistance from the Gift has been administered effectively and expected outcomes are being achieved.⁶

1.13 CSIRO advised the ANAO that in designing the arrangements for administering the Gift it had regard to the key principles in the Australian Government’s grants administration framework set out in the Commonwealth Grant Guidelines (July 2009).⁷ In addressing the audit’s second criterion, the ANAO considered CSIRO’s application of these principles, including: transparent decision-making; consistent and equitable treatment of applications for financial assistance; and achieving outcomes and value with public money.

1.14 The audit reviewed processes and transactions occurring during the period October 2009 to August 2015, although some records covering the period prior to October 2009 were examined as part of the ANAO’s assessment of the design of the Deed. Specifically, the ANAO reviewed key documentation related to administration of the Fund, including: papers and minutes from the meetings of the Advisory Council; program guidelines; records of the assessment of applications and proposals for financial assistance; funding agreements; project performance reports; and management information. In addition, the ANAO interviewed the Chair of the Fund’s Advisory Council, key members of CSIRO’s service provider team, as well as representatives from a number of the funded projects and unsuccessful applications.

1.15 The audit was undertaken in accordance with the ANAO’s Auditing Standards at a cost to the ANAO of approximately $380 000.

Introduction

2.1 CSIRO plays a central role in the administration of the Fund. In particular: it is the donor of the Gift; CSIRO’s Chief Executive is the Trustee of the Fund; and CSIRO has been engaged by the Trustee to provide a range of administrative services to the Fund. In addition, CSIRO’s Board determines the composition of the Trustee’s Advisory Council and, pursuant to section 10A of the Science and Industry Research Act 1949, is able to issue directions to the Trustee. During the period examined by the audit, CSIRO-related persons were also participants in the following governance and assessment bodies:

• the five member Advisory Council—expanded to six in June 2015—included two CSIRO employees and a member of CSIRO’s Board; and
• the four member Expert Panel—expanded to six in June 2012—included two Advisory Council members: the CSIRO Board member and an employee of CSIRO.

2.2 As shown in Table 2.1, CSIRO is also a participant in most of the projects that have received financial assistance from the three programs examined in this audit. In this regard, CSIRO has received more financial assistance than any other organisation; and has also contributed more co-investment funds to successful projects than any other organisation.

Table 2.1: Analysis of CSIRO’s involvement in approved projects

Note A: The proportion of approved projects involving CSIRO does not include the Synchrotron project (worth $7.5 million) under which CSIRO is engaged to administer the project on behalf of the Fund. In particular, CSIRO is responsible for managing the project against the requirements of the funding agreement and allocating the financial assistance provided by the Fund to the beneficiary organisation.

Source: ANAO analysis of relevant Fund documentation.

Does the design of the Deed of Gift support scientific research that benefits Australia?

Development and administration of the Deed of Gift

2.3 Section 9 of the Act establishing the Fund⁸ states that:

The trustee shall deal with and apply so much of the Fund as represents gifts or bequests or the income arising from the investment thereof in accordance with the conditions upon which the gift or bequest was given or made …

2.4 In this context, the Fund offered CSIRO an opportunity to invest the proceeds from its WLAN licencing project in a way that would support the Board’s aim of funding significant scientific research activity for the benefit of Australia. In order to do so effectively, it was important for CSIRO, together with the Fund’s Trustee, to carefully consider the design and administration of the terms and conditions in the Deed of Gift.

2.5 CSIRO’s approach to the design and development of the Deed included consultation with a range of stakeholders, both internal and external to CSIRO. The consultation process included senior Ministers and the (then) Department of Finance and Deregulation. As a result of these consultations, the Deed sets out the ‘primary’ and ‘special’ purposes for which the Gift can be used. In particular, the Deed states that the primary purpose of the Gift is:

… [the Gift is to] be used for Research for the purposes of assisting Australian industry, furthering the interests of the Australian community or contributing to the achievement of Australian national objectives.⁹

2.6 The special purposes were designed to provide more guidance to the Trustee in applying the Deed’s broad primary purpose. In particular, CSIRO advised the ANAO that the special purposes were designed to ‘provide funding into a spread of…productive [research] activities’. To give practical effect to the special purposes and to help provide further insight into the nature of the activities that were likely to be funded, the Trustee adopted a set of investment principles.¹⁰

2.7 Through its consultations, CSIRO also considered and addressed a range of administrative, legal, ethical, procedural and financial matters relating to the operation of the Deed. In particular, as shown in Table 2.2, CSIRO put in place a series of measures designed to strengthen accountability.

Table 2.2: Accountability measures implemented by CSIRO relating to the Gift’s administration

Note A: These conditions were imposed pursuant to clause 1.2 of the Deed. The conditions were reiterated by the CSIRO Board at the time of approving each of the three tranches of the Gift.

Source: ANAO analysis of the Deed and the Services Agreement.

Planning to implement the Deed

2.8 In June 2009, the CSIRO Board agreed to the following key steps to help manage the transition phase leading up to the Deed’s operation:

• establish a sub-committee of the Board—the Board Endowment Committee¹¹;
• determine the quantum of the Gift;
• determine any conditions attaching to the Gift;
• execute the Deed;
• appoint Service Provider(s);
• establish the Advisory Council; and
• provide the Trustee and Advisory Council with advice on a series of implementation issues—including establishing the mechanisms for making disbursements of the Gift from the Fund.

2.9 The ANAO’s analysis indicates that each of these steps was performed. With regard to the decision to appoint CSIRO as the Service Provider, the ANAO was advised that the Trustee’s decision took into account that the costs of administering the Fund had to be minimised and also that CSIRO had the skills necessary to manage the Fund. Further, the Board Endowment Committee was advised that the Trustee had considered the relative costs of engaging service providers and believed that the use of CSIRO’s services presented an economic and efficient solution. However, there was no documentation or analysis supporting the decision to engage CSIRO as an ’economic and efficient solution’.

Are processes for identifying and managing program-related risks effective?

Risk management

2.10 A structured approach to the management of potential threats or risks is a key part of effective public administration. In particular, the early and active identification, assessment, treatment and monitoring of risks will promote accurate and well-informed judgements and plays an important role in successful program implementation, including securing value for money and supporting the achievement of policy objectives.¹²

2.11 The ANAO was advised that work on the development of a risk management plan for the Fund commenced in 2012—some three years after the Deed was signed and the first tranche of the Gift paid to the Fund. The resultant risk plan was approved by the Advisory Council in June 2013. Table 2.3 shows the four risk event groups identified in the approved plan.

Table 2.3: Risk event groups identified in the approved risk plan

Source: ANAO analysis of the Fund’s risk management plan.

2.12 The ANAO’s analysis indicates that the risks, controls and treatments identified in the approved risk management plan were relevant and appropriate. In addition, our examination also indicates that processes were in place, and operating as intended, to assess the continuing accuracy and integrity of the plan. As a result of these processes, a number of refinements and enhancements to the controls designed to reduce the impact of the identified risks or to address emerging risks have been incorporated into the plan.

Conflicts of interest

2.13 The management of conflicts of interest is an important part of the administration of Australian Government initiatives and services. Dealing effectively with conflict of interest situations typically requires the use of controls and measures that provide an appropriate level of assurance that decision making will not be, nor be perceived to be, compromised by a lack of impartiality.¹³ CSIRO’s central role in the Fund’s administration—as observed at paragraph 2.1—introduced increased risks of conflicts of interest that required careful management.

2.14 As shown in Table 2.4, CSIRO has put in place a series of formal and structured measures across key aspects of the Fund’s administration to address conflict of interest risks.

Table 2.4: Controls in place to address potential conflict of interest risks

Source: ANAO analysis of relevant documentation.

2.15 The ANAO’s testing indicates that the established arrangements for managing conflicts of interest have, for the most part, been operating as intended. In particular, the ANAO observed that:

• all funding decisions by the Trustee were consistent with the recommendations from the Advisory Council;
• members of the Advisory Council regularly updated their declarations and proactively brought potential conflicts to the attention of the Chair;
• members of the Expert Panel were aware of the need to declare conflicts of interest and actively took steps to manage these issues—even before official procedures were established; and
• the Council has considered CSIRO’s performance each year following the Trustee’s request and on each occasion had indicated satisfaction with the level of services being provided.

2.16 However, the ANAO’s examination of assessment processes for the Research Projects program identified a small number of cases where the recording of conflicts of interest was fragmented and inconsistent. The ANAO’s analysis indicates that these gaps in the records did not affect the overall transparency of the decisions made.

Was clear, consistent and timely information about the Fund, including details of how to access financial assistance, readily available?

Promotion and communication

2.17 CSIRO undertook a range of communication and promotional activities designed to improve the visibility, positioning and marketing of the Fund, primarily via the Fund’s website. The ANAO’s analysis indicates that the website was easily navigable, had been kept up-to-date, and contained key information about the Fund’s activities, including:

• the Fund’s objectives and purposes;
• a description of the Fund’s financial assistance programs;
• important dates;
• details of the projects that have been funded;
• copies of the Fund’s quarterly newsletters; and
• details on how to subscribe to the Fund’s mailing list.

Program guidelines

2.18 Readily accessible program guidelines are a key means of informing prospective funding applicants about a financial assistance program, including the program’s selection criteria and how to apply for financial assistance. Well drafted guidelines also promote confidence in the equity and transparency of the funding process.

2.19 Separate guidelines were established for the Research Infrastructure program and for each of the projects invited to seek funding under the Special Research program. As shown in Table 2.5, the ANAO’s analysis indicated that the guidelines developed for these programs were well-designed and reflected elements of good practice.

Table 2.5: ANAO assessment of the content of the Research Infrastructure and Special Research programs’ guidelines

Note A: These elements are based on information contained in the ANAO’s 2010 Grants Administration Better Practice Guide.

Note B: ASKAP—the Australian Square Kilometre Array Pathfinder.

Note C: The Special Research program funding rounds were not open and competitive; rather proponents were invited to participate. As such, Special Research guidelines do not contain eligibility criteria.

Source: ANAO analysis of the programs’ guidelines.

2.20 In contrast, program guidelines were not produced for any of the four funding rounds of the Research Projects program. Rather, a range of information about the program was provided on the Fund’s website. The ANAO’s analysis indicates that details about the program were informative and had been provided in a timely manner. However, while the published program material explicitly defined an eligible applicant, there was some uncertainty as to what was an eligible project. For example:

• it was difficult to clearly determine project eligibility because information that could be interpreted as being eligibility criteria was not clearly marked as such; and
• details of available project funding were ambiguous. For instance, the program material implied that projects within a $3 million to $10 million range were eligible. However, only three of the sixteen projects funded over the four funding rounds of the program were approved for more than $5 million in funding, and none for more than $6 million.¹⁴

2.21 Eligibility criteria should be straightforward, easily understood and effectively communicated to potential applicants. Grouping all eligibility criteria and clearly identifying the requirements assists with transparency of the application and assessment processes. In this context, the introduction of clear project eligibility criteria may have benefited potential applicants by reducing the submission of uncompetitive Expressions of Interest. Further, clearer eligibility criteria may also have reduced the workload of the Expert Panel and improved administrative efficiency.

Were arrangements for assessing proposals for financial assistance well-designed and were assessments accurate, equitable and consistent?

Design of assessment processes

2.22 Separate and distinct assessment processes are in place for each of the three programs examined in this audit—the Research Projects, Research Infrastructure and Special Research programs. Figure 2.1 provides a timeline of significant events relating to the assessment and selection of proposals for financial assistance under each of these programs. Under the Research Projects program, the assessment and selection of projects for funding was conducted through four open competitive funding rounds. In each of the other two programs, selected projects have been invited, at different times, to submit proposals seeking funding. The ANAO examined the design of the assessment processes for each program to consider whether they were likely to result in funding projects that represent value for money in the context of the Fund’s objectives and intended outcomes.

Figure 2.1: Timeline of assessment and selection decisions

Note A: The four funding rounds of the Research Projects program were open to the public over the following periods—Round One: December 2009 to March 2010; Round Two: June 2010 to August 2010; Round Three: August 2010 to March 2011; Round Four: May 2012 to July 2012.

Source: ANAO analysis of Fund documentation.

Research Projects program

2.23 The Expert Panel undertook much of the assessment work relating to the Research Projects program. The results of the Expert Panel’s assessments were considered by the Advisory Council which, in turn, made recommendations about funding to the Trustee.¹⁵

2.24 The assessment and selection processes developed for the Research Projects program, which were approved by the Trustee in June 2010, reflected a sound and sensible approach. Importantly, the processes were cogently structured, such that key steps in the process were designed to build on the results achieved in previous steps. To facilitate consistency in assessment processes, the Panel was provided with pre-formatted assessment scoresheets and guidance on the use of the scoresheets—including an explanation of the scoring methodology. In addition, CSIRO undertook a number of reviews through the course of its administration of the Research Projects program and introduced a number of refinements and improvements to the assessment processes. Among other things, these enhancements benefited applicants and the Expert Panel by providing clearer information and by streamlining and strengthening the processes.

2.25 As shown in Table 2.6, the ANAO’s analysis indicated that the selection criteria developed to assess applications for funding under the Research Projects program exhibited many of the characteristics of good practice.

Table 2.6: ANAO’s analysis of the selection criteria used to assess applications for funding under the Research Projects program

Note A: These characteristics are based on the information in the ANAO’s 2010 Grants Administration Better Practice Guide.

Source: ANAO analysis of information published on the Fund’s website and assessment documentation.

Research Infrastructure program

2.26 In August 2010, the Trustee suggested to the Advisory Council that supporting research infrastructure from the Gift—one of the special purposes set out in the Deed—was likely to be more effective if the Fund adopted a purposeful strategy to identify potential projects for funding, rather than relying on applications received through competitive funding rounds.

2.27 In response to the Trustee’s observation, CSIRO and the Advisory Council undertook a range of tasks to support the establishment of the Research Infrastructure program. In particular, this included:

• identifying the priority areas for investment;
• developing a (potential) timeline for engaging with the key stakeholders in each of the priority investment areas;
• establishing an understanding of the relative maturity of the governance arrangements and research infrastructure needs in each priority investment area; and
• setting up the processes for eliciting and examining requests for financial assistance, including developing the selection criteria for assessing proposals.

2.28 The ANAO’s analysis indicates that the selection criteria developed for the Research Infrastructure program were well-designed and reflected elements of good practice (the characteristics in Table 2.6).

Special Research program

2.29 The Trustee approved financial assistance to support two research projects under the Special Research Program: access to the Australian Synchrotron by Publicly Funded Research Agencies—in May 2012; and increasing the capacity of the Australian Square Kilometre Array Pathfinder—in July 2013.

2.30 In contrast to the Research Infrastructure program—under which project proponents were also invited to submit proposals for funding—the Trustee did not adopt a structured process to establish a clear focus or purpose for the Special Research program. Rather, CSIRO advised the ANAO that ‘…proposals were invited in relation to areas that the Trustee considered would require a potential intervention to support nationally significant research activities…’. The ANAO found no evidence that the Trustee had sought to actively identity and assess the relative priorities or merits of the range of research activities that may have warranted support. Rather, each of the two decisions to invite proposals seeking financial assistance was largely unplanned and unrelated.

2.31 In order to support the preparation and assessment of the proposals submitted under the Special Research program, CSIRO developed a separate set of selection criteria. The ANAO’s analysis indicates that the selection criteria developed for each proposal were sound and, while separate, were broadly consistent with each other. Like the selection criteria for the Research Infrastructure program, the criteria were well-designed and consistent with the characteristics of good practice set out in Table 2.6. However, details of the selection criteria were restricted to the organisations preparing the respective proposals and were not made public—a decision that contrasts with the approach to transparency and consistency adopted in administering other aspects of the Fund. CSIRO advised the ANAO that it took the approach of limiting publicity about the Special Research program because:

… publication of the program was not necessary for soliciting applications…[and] reflected a pragmatic approach to managing expectations in relation to the availability of funding ….

Assessment decisions

2.32 The ANAO’s examination of a sample of 50¹⁶ assessments from the Research Projects, Research Infrastructure and Special Research programs the indicates that:

• the assessments have been conducted in accordance with established processes;
• for the most part, the records of assessment contained a sufficient level of evidence and analysis to justify the decisions made; and
• the decisions made reflected the information provided by the applicants.

2.33 However, as shown in Table 2.7, there were some gaps in the level of documentation supporting the assessment processes.

Table 2.7: Documentation gaps in assessment records

Source: ANAO analysis of assessment records.

Funding advice to the Trustee

2.34 The Advisory Council’s funding advice to the Trustee for each of the programs examined by the ANAO contained a clear recommendation about which projects should be funded, for what amount, and whether any conditions should be attached. However, the Council’s advice generally did not outline the basis for the Council’s recommendations or describe the process by which the Council reached its funding recommendations. CSIRO advised the ANAO that much of the advice to the Trustee was provided orally during Advisory Council meetings¹⁷, and as a result, the approach adopted was to only record a summary of the issues being considered.

2.35 The ANAO has reported concerns about the adequacy of the documentation supporting key decisions in the administration of financial assistance programs in a number of recent audits.¹⁸ In this light, and with a view to enhancing the transparency and accountability of the Trustee’s decision making, it is suggested that the Council’s advice to the Trustee contain a greater level of detail supporting the Council’s recommendations, such as describing the:

• basis for the recommendation—including: how the applications rated against the eligibility and selection criteria; and how they compared with each other; and
• process by which the recommendation was reached—including: who was involved and how the assessment was conducted.

Introduction

3.1 Well-designed and structured processes for administering the financial assistance provided by a funding program will further support the effective and ethical achievement of the program’s benefits and outcomes. Such processes include: establishing funding agreements with the recipients of financial assistance; verifying the individual projects’ progress and performance against agreed milestones; and monitoring the performance of the program to gain insights into whether the program is effective—that is, delivering expected outcomes—and being administered efficiently.

Is the design of funding agreements fit-for-purpose and have agreements been executed in a timely manner?

Funding agreement design

3.2 Funding agreements can provide a sound basis for effective oversight of the funded projects. Well-designed funding agreements will set out: the terms and conditions under which funding has been provided; the objectives and expected outcomes of the funded project; the timing of key milestones and associated payments; and the requirements for reporting on progress and performance. As outlined in Table 3.1, the ANAO’s analysis indicates that the standard funding agreements developed for the Fund were fit-for-purpose.

Table 3.1: Analysis of the content of the Fund’s standard funding agreements

Note A: These elements are based on information contained in the 2009 Commonwealth Grant Guidelines, the ANAO’s 2010 Grants Administration Better Practice Guide and Legal Briefing 83—Grants and Funding Programs (Australian Government Solicitor, November 2009).

Note B: The funding agreement templates used for the Research Infrastructure and Special Research Programs require the funded projects to develop and report against a series of performance measures—refer to further discussion at paragraph 3.16. On the other hand, the funding agreement template used for the Research Projects program did not require funding recipients to develop and report against performance measures.

Source: ANAO analysis of funding agreement templates.

3.3 The ANAO’s examination of a sample of ten executed funding agreements indicates that they each contain terms and conditions consistent with the approved templates. Our analysis also indicates that:

• with one exception¹⁹, each agreement examined describes how the project is expected to contribute to the Funds’ purposes and strategic objectives;
• the design of the reporting requirements in each agreement was appropriate to support the Trustee make objective assessments of the project’s progress against the agreed milestones; and
• the scheduling of progress payments in the agreements aligns with the achievement of milestones and, for the most part, is evenly spread over the course of the projects’ terms.

Time to execute

3.4 The timely execution of a funding agreement with the recipient of financial assistance is an important part of effective administration. In this regard, undue delays in negotiating the terms and conditions associated with the delivery of the approved projects can affect the timing of the achievement of the funded projects’ milestones and objectives.²⁰

3.5 Figure 3.1 illustrates the time elapsed between the dates of the Trustee’s approval of financial assistance and the execution of the relevant funding agreements for each of the four funding rounds of the Research Projects program, as well as for the Research Infrastructure and Special Research programs.

Figure 3.1: Number of months between Trustee approval and execution of funding agreement

Note: For three projects the Trustee did not record the approval until the respective funding agreements had been executed. For the purposes of Figure 3.1, the ANAO used the date of the Advisory Council’s recommendation to the Trustee as a proxy measure for the date of approval.

Source: ANAO analysis of Trustee approvals and signed funding agreements.

3.6 As demonstrated in Figure 3.1, there were protracted delays in the negotiation and execution of funding agreements in the first two rounds of the Research Projects program. In August 2011, CSIRO advised the Advisory Council that the time delays were, in part, a result of complexities associated with the development of ‘intellectual property agreements and commercialisation strategies’. The ANAO’s analysis identified that the delays were also due to the Trustee’s decision to approve financial assistance with certain conditions and/or to approve less funding than originally requested by the applicants. While this approach allowed the Trustee to mitigate some of the risks identified in funding these projects, it also resulted in proponents having to re-design or re-scope their projects.

3.7 A number of measures were introduced to help overcome and reduce the delays in negotiating funding agreements. This included:

• requiring applicants to identify any potential changes to the standard funding agreement at the time their full proposal was submitted;
• including a deadline for finalising the funding agreement in the Trustee’s funding offer;
• requiring applicants to consider aspects of the management of intellectual property at the application stage; and
• providing clearer and more timely information to those applicants invited to submit stage two proposals, including advice on the likely level of financial assistance.

3.8 As illustrated in Figure 3.1, the implementation of these measures resulted in a significant reduction in the average time to execute funding agreements in the later rounds of the Research Projects program.

Is the performance of the funded projects, including assessing the achievement of project outcomes, actively monitored?

Project monitoring

3.9 The primary mechanism used to monitor the status and performance of the projects funded from CSIRO’s Gift to the Fund—including whether the projects are performing as expected and that agreed milestones and objectives are being achieved—is the assessment of information contained in the projects’ progress reports. As shown in Table 3.2, the ANAO’s review indicates that progress reports were designed to capture a range of relevant and appropriate information. Our analysis also indicates that these reports were required to be submitted regularly—at least annually for each of the projects examined.

Table 3.2: Information captured through progress reports

Note A: Operational plans are required to be submitted by the proponents of the three projects funded under the Research Infrastructure program and one of the two projects funded under the Special Research program. Among other things, the Operational plans contain details of: milestones planned to be achieved in the next reporting period; governance arrangements; performance indicators; and a schedule of risks and treatment strategies.

Source: ANAO analysis.

3.10 In addition to the progress reports, the projects funded under the Research Projects program were required to submit a supplementary report at the project mid-point—at which time proponents were also required to participate in a major review²¹—and upon completion of the project. The requirements relating to the submission of the supplementary reports and the major reviews were introduced in July 2013 in order to assist the Trustee to better assess:

• whether milestones remain appropriate;
• current and future plans for engagement with collaborators, industry and other end users;
• progress towards the research’s adoption or application;
• delivery of the project’s overall outputs and objectives; and
• likely outcomes and impacts from the funding provided.

3.11 The ANAO’s examination of a sample of progress and supplementary reports indicated that these reports were being submitted in accordance with the reporting requirements. Further, our analysis also indicated that the established arrangements for the examination of these reports were working as intended, and that the assessment decisions made by the reports’ reviewers were reasonable and appropriate.

3.12 The ANAO also observed that CSIRO and the Advisory Council regularly took action to address identified performance issues. In this regard, a common method for dealing with concerns about the progress or performance of projects was to request a further report—known as an ‘ad-hoc’ report. Case Study 1 outlines an example of the measures taken by the Advisory Council in response to information reported in a project’s progress report.

Note A: The funding was offered subject to the project proponents providing an equivalent level of co-investment and the proponents submitting a further proposal setting out, among other things, details of the milestones to be achieved in the three month period and plans to secure potential alternative sources of funding for the project.

Source: ANAO analysis of Fund documentation.

Status of the funded projects

3.13 Five of the 22 projects funded from the three programs examined in this audit have been completed. The ANAO’s examination indicates that the final progress and supplementary reports for each of these projects had been assessed and accepted in accordance with established practice. Our analysis of these reports also indicates that decisions made about the project’s performance, including the achievement of deliverables, were reasonable and appropriate. The ANAO’s analysis of an assessment by CSIRO of the status of the remaining funded projects, which was provided to the Advisory Council meeting in August 2015, indicates that the remaining 17 projects were progressing in line with the key milestones set out in their respective funding agreements.

Is useful and instructive information about the Fund’s performance regularly captured, measured and reported?

Performance measures

3.14 Key performance measures can provide management, as well as other interested stakeholders, with insights as to whether a program is being managed well and is meeting its objectives. When designing performance measures, entities should aim to capture information concerning the outputs delivered and the outcomes achieved, as well as information about the effectiveness and efficiency of the delivery of the program.²²

3.15 Measures to help assess the performance of the Fund have been developed at two levels. Firstly, as mentioned in Table 3.1, the projects funded under the Research Infrastructure and Special Research programs (but not the Research Projects program) were required to report against a set of key performance indicators. Secondly, a set of indicators has been developed to help measure the performance of the Fund as a whole.

Project-level performance measures

3.16 The ANAO’s analysis indicates that the model performance measures contained in the funding agreement templates for the Research Infrastructure and Special Research programs were well-designed. In particular, the measures were designed to capture useful information about the progress and performance of the funded projects. The ANAO’s analysis also indicates that the performance information reported by the projects funded under these programs aligned with the requirements in the projects’ funding agreements—albeit at the time of the audit, some performance indicators were still under development.

Program-level performance measures

3.17 A structure for assessing the Fund’s overall performance was first developed in 2011. At that time a series of key goals or result areas were identified, together with a set of key measures of performance. In 2013, the framework was updated to increase the focus on the Fund’s impacts. Details of the Fund’s key result areas—and attendant performance measures—which were endorsed by the Advisory Council in June 2013, are shown in Table 3.3.

Table 3.3: Measuring the Fund’s performance

Note A: CSIRO advised that the national priority areas mentioned in the first key performance measure are the Australian Government’s National Research Priorities, which were developed in 2002–03. CSIRO advised the ANAO that following the release of the Australian Government’s new Science and Research Priorities in May 2015, it will reconsider this key performance measure as part of its preparations for the 2016–17 Budget.

Source: Advisory Council Meeting No.15 (June 2013).

3.18 Results against the Fund’s key performance measures have been reported in each of CSIRO’s Annual Reports since 2011–12. Table 3.4 contains a summary of the reported results for each of the measures.

Table 3.4: Reported results against key performance measures for the period 2011–12 to 2014–15

Note A: The results shown exclude projects funded under the Promotion of Science program.

Note B: The results reported for these measures are cumulative.

Source: ANAO analysis of CSIRO’s Annual Reports.

3.19 Overall, the ANAO considers that the key result areas—shown in Table 3.3—are sound and well targeted. In particular, they are outcome-focused and aligned with the strategic objective and primary purpose of the Fund. In contrast, as shown in Table 3.5, the ANAO’s analysis indicates that there is scope to improve the Fund’s key performance measures.

Table 3.5: Assessment of the Fund’s key performance measures against key characteristics

Note A: These characteristics are based on the criteria developed to evaluate the appropriateness of an entity’s key performance indicators contained in Audit Report No.28, 2012–13, The Australian Government Performance Measurement and Reporting Framework: Pilot Project to Audit Key Performance Indicators.

Note B: A proxy measure is a surrogate indicator that provides indirect insights into the contribution made towards achievement of the program’s objective. Proxy measures are valuable where a program’s objectives are difficult to measure because, for instance, the objectives have been set at a high level, their achievement is dependent on a number of external factors or the outcomes can only be achieved over the longer-term.

Note C: The Fund’s strategic objectives are shown at paragraph 1.8.

Source: ANAO analysis.

3.20 While the existing measures offer some useful insights into the performance of the Fund, it is not possible to determine from these measures alone the extent to which the Fund is achieving its overall objectives. CSIRO has previously recognised shortcomings in the Fund’s key performance measures:

There are limitations to the way in which the effectiveness of the SIEF in meeting its objectives can be measured because the contribution of research can only be measured in the long term.²³

3.21 Nonetheless, incorporating outcome-focused performance measures would help the Trustee, and other interested parties, to better assess the effectiveness of the Fund. The Advisory Council’s intention to undertake an evaluation of the Fund—discussed further at paragraph 3.32—provides an opportunity to re-examine the design of the Fund’s performance measures. In this context, the Trustee could usefully draw on the qualitative and outcome-focused performance measures developed by the Australian Research Council (Programme 1.1—Discovery and Programme 1.2—Linkage).²⁴

Management reporting

3.24 There were well-established arrangements in place for capturing and reporting information about the operations of the Fund to the Trustee and the Advisory Council. Among other things, at each Advisory Council meeting, the members of the Council and the Trustee were provided with:

• financial reports and supporting analysis, including details of actual and forecast receipts and expenditure, as well as statements of assets and liabilities;
• advice on issues associated with the administration of the Fund—for instance, the Council was provided with annual updates on the Fund’s risk management plan and results against the Fund’s key performance measures; and
• an update on each of the Fund’s financial assistance programs—typically, this includes: details of the status of the funded projects; and a discussion of issues concerning the funded projects’ performance (and, where relevant, underperformance).

3.25 On occasion, the Advisory Council had also been provided with information on the distribution of financial assistance from the Fund. For instance, at its meetings in March 2012 and August 2015, the Advisory Council was presented with analysis of the types of research supported by the 17 projects funded under the Research Projects program. The results of the analysis prepared for the Council’s August 2015 meeting are shown in Figures 3.2 (Fields of Research) and 3.3 (Socio-Economic Objectives).²⁵

Figure 3.2: Fields of Research

Note A: The rows add to more than 100 per cent as some projects support research across more than one field.

Source: CSIRO’s analysis of distribution of Research Projects funding.

Figure 3.3: Socio-Economic Objectives

Note A: The rows add to more than 100 per cent as some projects support research across more than one objective.

Source: CSIRO’s analysis of distribution of Research Projects funding.

3.26 As mentioned in paragraph 2.15, each year the Advisory Council considers CSIRO’s performance as the Fund’s service provider. In March 2015, when presented with details of CSIRO’s personnel and operating costs, the Advisory Council observed that the ‘service costs were low relative to the level of funds managed per annum’. The ANAO’s analysis indicates that the fees paid to CSIRO to the end of June 2015 for administering the Gift represent approximately two per cent of the total amount of financial assistance approved by the Trustee.²⁶

3.27 The ANAO’s analysis indicates that the management reporting provided to the Trustee and the Advisory Council aligned with the reporting requirements set out in the Services Agreement. ANAO’s analysis of the usefulness of the suite of management reporting is set out in Table 3.6.

Table 3.6: ANAO’s assessment of management reporting

Note A: The usefulness of management reporting in terms of assessing performance has been assessed as Partly met due to shortcomings with the design of the Fund’s performance measures.

Source: ANAO analysis.

Evaluation activity

3.28 Evaluations—which typically involve an assessment of the appropriateness, effectiveness and efficiency of a program—can provide valuable insights into program design and performance, including achievements against program objectives. Evaluations can also help to inform decisions about a program’s future direction, including identifying opportunities for improvement.²⁷

3.29 There were a number of processes in place for capturing information on the contribution of individual projects towards the Fund’s objectives. For instance, useful information was captured through:

• ‘project impact’ brochures—these provide a high level overview of each project, outlining: the challenge; response; collaboration; and projected impacts²⁸;
• supplementary reports and major review presentations; and
• the key performance indicators in place for the five projects funded through the Research Infrastructure and Special Research programs.

3.30 Despite this range of information, the ANAO did not find evidence that the data collected on individual projects had been analysed to help gauge the performance of the Fund more broadly. Furthermore, no evaluation activity had been undertaken at the whole-of-Fund level.

3.31 In August 2013, the Advisory Council agreed to undertake a review of the Fund’s performance in late 2014 or early 2015. However, in December 2013, when considering an outline of a plan for the conduct of the review, the Council observed that:

In light of these objectives [that the review should be aimed at demonstrating the value of the program] the review should be deferred until more evidence is available and the majority of projects have delivered final outcomes.

3.32 Subsequently, at its August 2015 meeting, the Advisory Council was advised that the proposed examination of the Fund’s performance should be conducted from around mid-2017 because ‘most Research Projects were expected to be completed and the Research Infrastructure and Special Research projects would be well advanced.’ The Council observed the importance of the proposed evaluation activity to inform decisions about the future of the Fund and agreed that the timing of the review should be brought forward to late-2016 so as to maximise its impact.

3.33 While acknowledging that decisions on the optimal approach to, and timing of, the conduct of the review of the Fund is a matter for the Trustee, the ANAO notes that deferring such a review further limits the opportunity for the Trustee to identify areas for improvement in either the Fund’s administration or program design. In January 2016, CSIRO advised the ANAO that it anticipated an overall evaluation of the Fund would be undertaken by the end of 2016.

Appendix 1: Entity’s response

Please refer to the attached PDF of the report for Appendix 1 (Entity’s response).

Appendix 2: Schedule of financial assistance from CSIRO’s Gift to the Fund

Note A: In June 2011, the Advisory Council agreed to extend the John Stocker Postgraduate Scholarship and Postdoctoral Fellowship schemes (part of the Promotion of Science program) by 12 months at a cost of approximately $1.8 million.

Source: ANAO analysis of the Fund’s records.

Appendix 3: Special purposes and related investment principles

Source: ANAO analysis of the Deed and the investment principles of the Fund.