CRI Capability Fund evaluation: Evaluation findings
Section 1 of the Evaluation of the Crown Research Institute (CRI) Capability Fund. Prepared for MoRST by Taylor Baines andAssociates, Christchurch. 56 pages, published in June 2008.
Processes for management and monitoring
5.1 Introduction
The second focus of the formative evaluation is on how CRIs have generated the higher-level strategic context for their capability planning (Section 5.2) and the adequacy and appropriateness of monitoring arrangements (Section 5.3). Related to monitoring are the reporting/guidance processes involving CCMAU and MoRST (Section 5.4).
5.2 Strengths of CRI processes for management
As noted in Section 1.1, the stated aim of the fund “is to restore CRIs to a position where their strategic role within the innovation system is clearly articulated, endorsed, and understood by all.” The 2007 Operating Framework32 states that CRIs have a role in continually developing their human capital in ways that support and are aligned with the long-term interests of NewZealand’s wider science and innovation system.
In this section we describe the status of CRIs’ strategy development efforts at the time the Capability Fund was introduced and the impact on these efforts with the advent of the Capability Fund in 2005. Other management processes include internal reporting and management procedures applied to Capability Fund investments to ensure ongoing alignment of capability achievements with strategic objectives. Monitoring practices are discussed in Section 5.3.
Existing strategy processes as context for Capability Fund scheme
All CRIs have developed formal strategic plans and strategic priorities for their organisations.33 Eight34 of the CRIs indicated that the strategic planning processes in place at present were established before the advent of the Capability Fund. Nevertheless, the introduction of devolved capability funding is widely reported as having impacted on these strategy processes in some manner:
“Investing the Capability Fund and Retained Earnings forms a central part of strategy development”
“Had already established a new strategic direction, so the Capability Fund links well with how to get there; enables us to fund some of our strategy”
“Ongoing strategy development and Capability Fund decisions work together; building capacity and connections in terms of more students and post-grads, and collaborations with universities and other CRIs”
“Capability Fund investment has become part of a much more strategic process; the Capability Fund was introduced and fits well within this process - but it was not the driver for establishing the process”
“The advent of the Capability Fund has made it easier to implement science-platform strategic plans; gives us the flexibility to look at new areas if strategic bids are not funded”
“Brought together our strategic plan and our major strategic investment (our people capability) into one place - it’s good for our owners to see it that way”
“It’s the funding that is available to assist us make strategic shifts. It’s the only discretionary dollars to put behind new directions - otherwise we are locked into existing paradigms”
There is a strong sense from CRIs as a group that the Capability Fund was the missing piece of the governance and management jigsaw, in terms of the operational model established for CRIs and their statutory role. However, one CRI cautioned against premature expectations - “it is expensive to do our big new ideas, therefore it will take time for the Capability Fund to have a practical influence on strategic science directions”.
The 2007 Operating Framework35 expresses the owner’s expectations of CRIs as follows - “A CRI’s research and development capabilities, which are ultimately geared to ensure benefit to NewZealand, should always be underpinned by a successful business so that those capabilities can continue to be delivered over the long term.” CRIs are subject to various standard business disciplines, including evaluation of the annual return on shareholder funds. In the context of this evaluation, it is pertinent to observe that most CRI statements of dividend policy make no explicit reference to capability development as a strategic criterion to be considered when a CRI Board determines its annual shareholder return. This is despite the fact that at least three CRIs indicate that they combine their retained earnings with their Capability Fund allocations in order to fund their capability investments. Two CRIs explicitly note in their statements of corporate intent that the need to ensure the maintenance of scientific capability is an explicit consideration to be taken into account when estimating the amount or proportion of annual tax-paid earnings that is recommended to be distributed to the Crown. The existence of such a policy could well be considered relevant in future when evaluating the effectiveness and appropriateness of CRIs’ strategic capability development processes.
Other CRI management processes
CRIs acknowledge the management strengths they have gained as a result of working within the contestable funding environment since 1992. Most use established processes for managing their Capability Fund investment projects36 . These are often closely related to internal reporting procedures used for managing Foundation-funded programmes, but some CRIs had developed processes specifically for Capability Fund investment projects. In either case, the rationale was to minimise administrative burdens on scientists while maintaining sufficient organisational oversight. All CRIs demonstrated quarterly or 6-monthly monitoring of Capability Fund projects, with annual evaluations.
Changes to management since NSOF
The majority (seven) of CRIs had not made substantial changes to management arrangements developed under NSOF. The following comments are indicative of the character of changes:
“Our systems have become more formal, robust and auditable as the Capability Fund allocation and Retained Earnings have increased”
“Not a radical rethink because the process had already been established; more a refinement of strategy to become sharper”
“Now more democratic and bottom up - used to be more top down”
“Our portfolio managers have evolved their strategy skills and processes - getting better over time”
One CRI signalled its intention to review processes in the coming year:
“2008 will be a year to refresh decision making, resource allocation and monitoring processes.”
As one CEO expressed it:
“We’ve set up good governance structures and management processes for our CRI but most of our government dollars come in ways we have little influence on or power to influence; in short, our board structure could be used more effectively than it currently is if the funding channels were different.”
Remarks in several other CRI interviews endorsed this sentiment: “I believe our internal processes are robust; we have a devolved decision role in a sophisticated management environment - and it’s auditable. There has to be an element of trust.”
Overall evaluation
The interview with CCMAU confirmed that CCMAU has so far been satisfied, on the basis of the content reported in the strategic planning documents (SCIs and Business Plans), that Capability Fund investments are driven by strategic considerations that are consistent with their CRI roles (see Section 5.4 below for more discussion).
The interview with MoRST personnel revealed that they now have a better understanding of CRIs’ strategic planning efforts as a result of their role in administering the Capability Fund. MoRST personnel have been able to observe two complete planning and reporting cycles for each CRI. Their conclusion is that the Capability Fund has indeed sharpened CRI focus on strategic planning, and that the CRIs’ work has justified the level of trust afforded by the scheme (see Recommendation 1).
For the majority of CRIs, capability planning is embedded in their processes of organisational strategic planning, and is generally adequately supported by the internal reporting and management procedures already in place. However, there are some important issues to resolve regarding Capability Fund monitoring (Section 5.3) and external reporting (Section 5.4). Nevertheless, experience from the first 2 years of operating the Capability Fund has put in place much of the managerial and administrative architecture and capacity for this devolved funding scheme (see Recommendation 2).
5.3 Monitoring progress toward capability goals - how well?
Although monitoring and reporting issues are linked, this report considers them separately.
Monitoring progress in organisational capability
This evaluation is concerned with how well the CRIs monitor progress toward strategic organisational capability goals. Assessment of individual scientists, teams or projects is not part of the scope of this work. Monitoring should simultaneously serve the interests of the individual CRI and the science system interests of MoRST.
Expectations for monitoring
The 2007 Operating Framework for CRIs describes owner (Government) expectations relevant to capability development that imply potential indicators of performance. These include output, outcome and process indicators. The reporting requirements in the Capability Maintenance and Development Agreements between each CRI and MoRST support MoRST’s monitoring interests.
The original Cabinet Paper (at para. 32) implies that indicators should relate to strategic priorities:
“CRIs will be asked to develop indicators that relate to capability development and clearly set out milestones and measures towards them. These should demonstrate a number of high level organisational achievements, such as links with end-users, and the quality and relevance of CRI Capability Fund activities. CRIs may choose to demonstrate these in a number of ways.”
Current practice
There appears to be considerable diversity of practice in reporting capability profiles. CCMAU and MoRST initially proposed that capability development in terms of investments and achievements would be reported using the 3-year capability profile template.37 All nine CRIs have adopted this approach for their public reporting, with several adopting some modifications. However, as noted in Section 4.6, only one CRI reported explicitly on ”capabilities at risk”, those that ”present opportunities for further development”, and “progress in developing overall capability and organisational performance”. Another CRI indicated that although it does not report these aspects of its capability profile publicly, it does monitor them internally. A third CRI used an internal capability mapping process to identify capabilities at risk. Three other CRIs pointed out that they did not rely simply on reported indicators, but use additional indicators in their internal monitoring. However, it was not clear whether these are at the level of Capability Fund project data or whether they involve indicators for monitoring organisational capability.
Traditional measures of science excellence and newer indicators such as science collaborations and stakeholder engagement38 are relevant capability outcomes, and are reported by most CRIs amongst their non-financial indicators. While CRIs use a range of non-financial indicators to report the extent of capability achievements, and also to indicate future targets, there is no explanation of the rationale behind the annual targets. Nor is there any way of differentiating the extent to which incremental, year-on-year achievements relate to specific capability investments (see Recommendation 2a).
Five CRIs had conducted staff climate surveys, covering matters potentially relevant to organisational capability, such as organisational performance,39 organisational leadership,40 rewards and recognition41 and teamwork;42 However, the surveys had not been analysed for any explicit acknowledgment of Capability Fund activities. They may, however, provide a useful vehicle for staff input into future evaluation of Capability Fund outcomes (see Recommendation 2c).
Issues regarding monitoring of progress towards strategic capability goals
CRIs differ in the scope of monitoring their Capability Fund investment outcomes. It may be that a lack of guidance from MoRST and CCMAU contributes to this situation.
Monitoring indicators may include a mix of output: e.g. 3-year capability profile data; outcome, e.g. extent of sectoral connectedness through collaborations; and process indicators, e.g. transparent processes for budgeting, forecasting, capital allocation, and risk management.
It is not clear that there has been a systematic approach by all CRIs in developing indicators consistent with the range of outcomes discussed in Section 3.3 (Interpreting Capability), i.e.:
- excellence of research;
- engagement with stakeholders:
- repositories of strategic science knowledge and RS&T skills;
- collaboration between science organisations
- capability to transfer knowledge to end-user;
- capability to promote awareness of the value of science; and
- human capital of the organisation.
The outcomes expressed in the 2007 Operating Framework (Appendix C), are also not clear. These include:
- Policies and procedures in place that create the environment required to attract, develop and retain the capabilities;
- Science careers are attractive, stable and rewarding;
- Staff engagement to address issues relevant to human capital development affecting all CRI staff; and
- Engagement with NewZealand and overseas science and education communities.
Some of these aspects of capability are represented in existing indicators, and most CRIs report a number of them already. However, there is not much evidence of a systematic approach across all dimensions of capability by any of the CRIs. Furthermore, there is no interpretation of what these indicators say about ”progress towards strategic capability goals” (see Recommendation 2a).
It is also not clear whether sufficient attention has been given to monitoring indicators at the level of organisational performance, as distinct from listing individual project outputs. This may lie behind the absence of reporting on capabilities “at risk” and capability “opportunities”.
While there is not a strong argument for uniformity in monitoring practices associated with a devolved funding scheme (see Recommendation 1), there is a need to consider how individual CRI monitoring contributes to MoRST’s science system monitoring. In this regard, it is interesting to record that even MoRST did not have a national data set indicating the allocative splits between existing capabilities, new and emerging capabilities and over-the-horizon capabilities, as reported in Section 4.2 (p. 11) of this report. Seven of the nine CRIs provided us with these data at short notice (see Recommendation 2d).
For MoRST’s national overview interests, another issue is that output reporting is currently used as a surrogate for process reporting. For example, the 3-year capability profiles (essentially output descriptors) are used to indicate the existence of a capability planning process. If capability goals require appropriate strategic processes, then it would be more transparent to enquire as to whether or not these are in place (see Recommendation 2b).
Overall evaluation
At this early stage in Capability Fund implementation, these monitoring issues are an area for future refinement. Current monitoring practices, particularly the absence of effective organisational-level indicators, mean that there is not a strong evidential basis for future decision making on the size of the Capability Fund.
5.4 Reporting/guidance process with CCMAU and MoRST
While CRIs officially report to CCMAU as their owner’s agent, CCMAU shares Capability Fund‑specific reporting information with MoRST, as described in the agreement between each CRI and the Minister for RS&T. The roles and process followed by CCMAU and MoRST for analysing and giving feedback on CRI plans and indicators have not been formalised or codified.43
This section is based on views expressed by CCMAU, MoRST and the CRIs, and so reflects the important perspectives of those parties preparing and receiving the reports.
The CCMAU perspective
CCMAU’s role44 is that of the Government’s CRI ownership agency responsible for providing advice on monitoring and ownership issues. The Memorandum of Understanding under which CCMAU operates specifies at several points aspects on which CCMAU is entitled to question CRI boards. These include strategic plans and statements of corporate intent (p. 4), investment decisions (p. 5), annual reports (p. 5), and other CRI official communications (p. 6).
We note the earlier finding of CCMAU that CRI strategy development processes were considered adequate, on the basis of the outputs reported, and that CCMAU has never had cause to question the direction of planned CRI investments under the Capability Fund. CCMAU confirmed that while SCIs/strategic plans have been sent back to CRIs for reconsideration in the past, this has never been for a Capability Fund-related issue.
One output from the exchange of information between CRIs and CCMAU is the annual production of an Operating Framework for CRIs. This document incorporates confidential company-specific addenda which provide evidence of Ministerial (CCMAU) feedback and level of satisfaction/concern directed at board decision making. We have reviewed the company-specific addenda for the last 2 years, where points raised were relevant to organisational capability development. The level of satisfaction indicated by the balance between commendation for achievement and suggestion for caution or improvement reinforces the above commentary from CCMAU that, generally speaking, organisational strategy development within CRIs is satisfactory.
The MoRST perspective
As noted in Section 4.6, MoRST’s role is that of the Government’s science policy agency responsible for administering the Capability Fund monies and ensuring the policy intent of the Capability Fund is preserved. The MoRST checks of CRI Capability Fund reporting45 (referred to previously at the end of Section 4.6) summarised the way each CRI reported its use of the funding in 2006/07, noted where requirements were not fully met, and recommended that MoRST and CCMAU review Capability Fund reporting requirements. MoRST found that “Every CRI annual report had a section headed ‘CRI Capability Fund Report’ or similar and some had additional material throughout their reports. This appears appropriate.” However, it found that three CRIs met the criteria for reporting if interpreted strictly, three almost complied and three did not comply. Four CRIs reported variances between achievements and intentions, and three gave adequate explanations of the variances. The main deficiency was the absence of commentary about progress on developing overall capability and organisational performance. The MoRST review therefore concluded “At the policy design stage, the CRIs themselves asked to report on the CRI Capability Fund through their annual reports. This approach does not seem to be working yet.” However, an earlier discussion point states that “If CRIs are audited (by their Auditors) for compliance and reporting against their SCI then we could safely let the present level of reporting stand.”
At several points the MoRST review mentions the dual compliance and marketing roles of annual reports. These two roles could be seen to have the potential for conflict. For example, an honest appraisal of compliance - if it recorded aspects of non-compliance - would not be viewed as a positive marketing message. On the other hand, clear signals about the development of new capabilities could have marketing benefits. This raises yet again the issue of whether the publicly tabled annual reports are the most appropriate vehicle for compliance reporting.
The CRI perspective
Interviews with the CRIs elicited both common and divergent themes.
CRIs see the main purpose of current reporting as being compliance-related. Furthermore, they generally accept the need for accountability and transparency when receiving such sums of taxpayer funding - “in line with our expectations of good practice.” However, the majority of CRIs are lukewarm about the usefulness of the reported information to their own strategic decision making. This reflects the CRIs’ unease with public reporting of some aspects of their Capability Fund investments.
They see the strength of current arrangements as displaying a high level of trust and imposing low compliance requirements. This they consider to be an essential characteristic of a highly devolved fund - “it’s a devolved fund in a sophisticated management environment - it is auditable”. All CRIs referred to the compliance requirements as not being onerous - “working well”; “not onerous”; “no complaints”. Indeed, several CRIs pointed out that they had initially offered much more detail in their confidential reporting to CCMAU. However, two CRIs do not regard these reporting requirements as effective - “not always clear what’s wanted”; “a waste of time”.
The main weakness of current reporting arrangements from a CRI perspective is that the required reporting does not convey an accurate picture of their use of the Capability Fund - “we don’t need more directive ‘belts and braces’ but we do want to give CCMAU assurances that we are using the Capability Fund effectively”; “not helpful to making a good argument about boosting the scale of Capability Fund dollars.”
CRIs have so far not pushed for change to this reporting regime, partly because there has been no signal from MoRST or CCMAU for any need to change46 and, for some, partly because they expressed confidence in their own internal monitoring and management systems, in which the reporting requirements are more extensive and more strategically-based.
While there has been no strong push for change from any of the contracting parties, if change is to be contemplated, CRIs agree that it should not compromise the existing strengths of being high-trust
low-compliance in character.
For the six CRIs that did suggest improvements to the reporting requirements, the primary rationale was to address the main weakness, i.e. to improve the strategic information content of reporting. Several expressed the view that fuller and more accurate reporting could be provided to CCMAU for use by MoRST on a confidential basis, without introducing more onerous requirements, since this reporting would be based on analyses already carried out internally.
Overall evaluation
All parties are committed to the principle of a high-trust low-compliance accountability regime for administering the Capability Fund (see Recommendation 1).
The current output-oriented reporting accepted by CCMAU and MoRST is not an onerous compliance measure. The original prescription for reporting organisational capability risks, opportunities and progress has not been insisted upon. There is no obvious reporting of CRI strategic planning processes that could give more explicit evidence on which to base the trust that is implicit in the contract (see Recommendations 2a and 2b).
Current reporting to CCMAU is not well developed and would provide a limited basis for policy decisions about changes to the overall level of the Capability Fund in future. Nevertheless, both CCMAU and MoRST have expressed views that the performance of the CRIs as a group does justify the high level of trust associated with this devolved fund (see Recommendation 1). However, discussions with ACRI personnel reinforce the importance of creating a stronger evidential base to inform future policy discussions - “we must move from outputs to impacts; it is not enough to be transparent and accountable.” (See Recommendations 2a and 2d).
All parties can see reasons to improve current reporting practice. MoRST is awaiting the findings of this evaluation before considering the need for change (see Recommendation 3).
5.5 Cases of unintended use of the Capability Fund
This section is based on self-reported information from CRIs, and is not a financial audit.
No evidence of uses inconsistent with the aims of the Capability Fund
No CRIs offered evidence of uses of the Capability Fund which were not intended, i.e. Capability Fund investments made without reference to science capability needs or the CRIs’ strategic priorities. There have been cases where the Capability Fund investments have been used to provide “bridging” support for science teams. However, as discussed in Section 4.3 (Points of similarity), all CRIs emphasise that they do not encourage - in fact, do not allow - longer-term dependence on the fund’s investments. We regard such uses as being entirely consistent with the intent of the scheme. It is interesting to note that no CRIs made references at all to any Foundation initiatives on ”transitional funding” in cases where existing Foundation funding was terminated.
National databases
At least four CRIs referred to support for national databases as a funding issue. The 2007 Operating Framework (third aspect of CRI role) states that “An important part of CRIs’ repository role involves the curation of databases. The Government is committed to increasing the availability of data for the generation of national benefit. CRIs have an important role in actively managing [and] disseminating significant science databases and curating data to appropriate standards, including standards for metadata that facilitate data sharing, for the benefit of NewZealand.” Other aspects of their “key repository” role are explicitly linked to the Capability Fund, but the 2007 Operating Framework gives no specific guidance about the use of funds to support the curation of databases.
MoRST provided the following comments on funding for national databases. Nationally important databases, including plant and Full-Time Equivalent staffing positions (FTEs) to maintain them, should be funded through other means (rather than the Capability Fund) via the Foundation. Research which uses nationally important databases can be funded through the Capability Fund. Where CRIs have a temporary shortfall in database funding, short-term use of the Capability Fund is acceptable. MoRST also noted that it is in the process of establishing ”backbone funding” which involves identifying and ring-fencing funding for these databases.
“Subsidising” other sources of funding
Three CRIs mentioned having experienced what they described as “a growing expectation” by some national or regional research funding agencies47 that CRIs should apply Capability Fund allocations to “prop up shortfalls from overbid or only partially funded bids” or to subsidise research contracts. In some instances, agencies did not appreciate that there are fund-based or CRI-based rules against this practice. Several CRIs indicated that they “push back vehemently from this blurring of purpose”. Others said that they would engage in this on a case-by-case basis, at their discretion. For its part, the Foundation maintains that the existence of the Capability Fund has no influence on the decision making of their advisory groups in contestable bidding rounds. It needs to be remembered that total Foundation funding generally does not keep up with inflation. Top-ups to the Foundation’s overall budget from time to time, to address this issue, tend to be targeted to specific research portfolios (for example, to energy research, on-farm research or bio-diversity research) rather than spread uniformly across all the portfolios that are bidding at any one time.
Overlapping CRI capabilities
From our perspective as evaluators, a potential issue could arise where one CRI develops what for it is a ”new” science capability that is in competition with a similar capability in another CRI. However, this issue was not raised by individual CRIs.
5.6 Conclusions
In most cases, systematic strategic planning processes developed during the period of NSOF have been applied to the Capability Fund. However, responsibility for Capability Fund investment has “sharpened” the strategic focus for some CRIs. CRIs have a good grasp of the requirements for strategic planning processes and internal project management processes. Their strategies provide the necessary context for applying their Capability Fund allocations and their project management processes strongly support its intended use (see Recommendation 1).
CCMAU and MoRST can be satisfied that the high-trust low-compliance model is delivering well-targeted investments of the Capability Fund by the CRIs (see Recommendations 1 and 3).
Monitoring each CRI’s progress in developing its organisational capability is the most uncertain aspect of the fund’s implementation. Thought still needs to be given as to what the most appropriate outcome indicators are for monitoring and reporting development at the organisational level in a way that is relevant to strategic considerations (see Recommendations 2a and 2d).
The fact that most CRIs have so far chosen not to report on the risk/opportunity/progress aspects of their organisational capability development, and the fact that MoRST has so far not insisted on receiving reports on these aspects, has led to a situation where the evidence base for future decision making on the size of the Capability Fund is not very useful. These aspects of monitoring and reporting, established in the original contractual agreements, remain appropriate.
No evidence was presented or discovered relating to uses of the Capability Fund which were not intended.
Footnotes
2007 Operating Framework on the CCMAU website. CRI role statement #7.
AgResearch - Annual Report 2006, pp. 22-27; C&FResearch - Annual Report 2007, pp. 8-9; ESR - Annual Report 2007, pp. 2-3; GNS - Annual Report 2006, pp. 8-9; HortResearch - SCI 2007-08, pp. 5-6; IRL - SCI 2005-2008, pp.5-10; SCI2007-2010, pp. 4-5; Landcare Research - Annual Report 2006, pp. 2-3; NIWA - Annual Report 2007, Strategic Directions; SCION - Annual Report 2007, pp. 2-3
One CRI indicated that its overall strategy was developed in association with the introduction of the Capability Fund in 2005.
See Endnote 32. p. 2.
Five CRIs cited these specifically, while two had Capability Fund-specific formats.
A template is attached to the Capability Maintenance and Development Agreements between the Minister and each CRI.
As for example in the Research Application Indicators, introduced in July 2006.
e.g. the CRI values its people, encourages innovation, manages intellectual capital well.
. e.g. organisational goals and objectives are appropriate, support for the company’s direction, trust in management.
. e.g. good performance is recognised, career opportunities are good, women have same career opportunities as men.
e.g. teamwork is good, different teams cooperate.
The Memorandum between CCMAU and MoRST (on p. 1) describes the informal nature of their roles and processes as follows: “The two parties to this MOU acknowledge that, from time to time, it will be necessary to respond to issues that cannot be fully anticipated at the time of drafting. A measure of goodwill will be required from all parties, as well as recognition that this MOU provides a good-practice guide rather than a prescriptive and exhaustive set of rules.”
See Footnote note 2.
See Footnote 30. Also an earlier version dated 27 November 2007.
MoRST did not review the first round of Capability Fund reporting, i.e. for the 2005/06 year, as it did for the second year.
This remark is explicitly not aimed just at the Foundation; other agencies such as regional councils and central government policy agencies were included.