Productivity Commission: Science and Innovation Study

Submission from Deakin University

The Commission will receive a range of inputs across the breadth of this study.  This submission addresses a small number of general issues that are of particular relevance to the university sector.
Background: Decision making and the Research Quality Framework
The national focus on the assessment of research in publicly funded bodies including universities, and the development of a Research Quality Framework (RQF) within which to assess research output, are already having a marked influence on the university sector, even though no particular assessment process has yet been adopted.  Much of this influence is around the increased emphasis on research quality and research impact, and how to link these to research planning and funding.  There is much to be gained by this change in emphasis, particularly after a long period in which institutional research has been assessed through simple metrics such as research income and number of research publications.  
The assessment of research is complex and involves a range of complex factors.  Assessment based on quality and impact will be appropriate in some cases, but other require a different approach.  An example where a different approach is needed is in the difficult area of climate change.  In considering how to react to evidence of global warming, the criterion for selecting an approach is that “it works” rather than it derives from the research of the highest quality.  The latter may well provide the best solutions in due course, but at this point we know too little about the problem to be choosy and any positive contributions will be important.  This situation – of being prepared to adopt any beneficial approach – is common in fields where the problems are complex or on a large scale.  The point I am making is that research strategies cannot always be driven by quality and impact alone.  However, the notions of research quality and impact are already driving planning and decisions, and this will continue to grow.  
There are two conditions that must be met in an assessment exercise.
•	The approach to the assessment must be sound, leading to well-accepted outcomes.
•	Those making the assessments must follow the agreed approach.  There must be no significant variations or additional factors introduced by assessors.
The first of these conditions is relatively easy to meet: the assessment method and criteria can be developed at a comfortable pace with consultation and input from experts as necessary.  The second condition can be difficult to enforce particularly if the assessment exercise is on a large scale, involves many assessors, and must be completed in a short time.  
This is the situation that will obtain in the assessment of research outputs under the RQF, in both the quality and impact components.  Consider the statistics of the RQF exercise in the form proposed by the RQF Expert Advisory Group at the end of 2005.  There are something over 36,000 (full time effective) academic staff in the university sector, of whom 44%, hold teaching-and-research or research-only appointments.  It is likely that about 12,000 of these are RQF-eligible, that is they each produce at least 4 research outputs in five years.  Simple gamesmanship indicates that each of these staff will submit 4 items of research output in an RQF assessment, leading to 48,000 items to be assessed.  If there are 12 assessment panels each with 15 members, and each item is seen by two panel members, a panel member will have to assess about 530 items.  The load on a panel member is thus the equivalent of examining each of 8 – 10 higher doctorate theses twice, once for quality and once for impact.  This is a big ask.  For an expert in a field to examine one such thesis  against well-known criteria in no more than a week would be unusual.
It is clear that RQF assessments will be of limited value,  Because of the pressure on the assessors they will be drawn to processes that offer a quick result, such as publication metrics (for example peer citations papers, and the “impact factor” of the journals in which they were published) rather than actually reading the paper and forming a judgement.  Anyone with experience of, for example, assessment processes of the Australian Research Council since work on the RQF began will know that judgements will be strongly influenced by the source of funding for the work including whether there was significant funding through schemes on the Australian Competitive Grants Register (see http://www.dest.gov.au/NR/rdonlyres/0DFABD49-DBEA-4405-968E-87D6DA045DDB/10030/LISTOFFUNDINGSCHEMESFOR2006ACGR.pdf).  Anyone with experience of trial RQF assessments which are in progress across the sector will know that the assessment of research impact is even more problematic than research quality.  The logical but counter-intuitive definition of impact, which excludes impact within the field of the research, is not understood by assessors who follow their intuition rather than the rules.
To summarise, this section seeks to make the following points:
•	the RQF criteria of quality and impact are not the only criteria for assessing research, and in some cases are not the most important;
•	the development of the RQF is already having a major influence over the way research is assessed;
•	assessors are preoccupied with research funded by national competitive grants (i.e. grant schemes listed on the Australian Competitive Grants Register);
•	the RQF assessment process as anticipated will, in the early years at least, see judgements based on a much smaller set of criteria than intended or desirable.
It is important, therefore, that public support for science and innovation  continues and grows, not only in the level of support but also in the number of agencies that manage that support so that different areas of research are subject to different and appropriate forms of assessment.
Impact of public support for science and innovation
The impact of public support for science and innovation is considerable, and not strongly linked to research quality.  Some of the most important areas with significant economic impact are low visibility implementations of well-known basic science rather than cutting edge research.  Consider, for example, the work going on between research institutes and the catchment management authorities – not exciting for those outside the field but fundamentally important nevertheless.  This kind of work will never rate highly in an assessment of research quality, but that says nothing about their importance to the country.
Benchmarking arrangements for assessing the outcomes of publicly funded science and innovation are limited.  Benchmarking will be assisted by the introduction of an RQF assessment process but, as argued above, this will focus on research excellence, which will not catch all the important research outcomes.  It is necessary to improve benchmarking, but the risk of diverting too many researchers from their research into research reporting is real.  It will be important in extending benchmarking to avoid crossing the point of diminishing returns.
Impediments to the effective functioning of Australia’s innovation system
The history of research shows that it is adversely affected by conservatism on the part of funding agencies, an excessive push for efficiency in the sense of high returns from little investment, excessive emphasis on competition ahead of collaboration, and risk-averse decision making.
Conservatism and efficient use of funds
Research is about creating new knowledge – that is, working in the unknown.  It is therefore to be expected that a high proportion of research projects do not achieve their goal, although there may be significant “unintended” benefits of the work.  Experienced research managers estimate that no more than one in ten research projects are successful, which leads to the conclusion that a success rate of more than one in ten indicates problems, such as total funding is inadequate or the funding decisions are too conservative.  When funding is limited, the efforts to use it “effectively” multiply.  Typically this sees the development of a peer review system dominated by senior experts.  Such approaches tend towards decisions favouring high quality but conservative research.  The decision processes of the Australian Research Council illustrate this approach well.
Competition rather than collaboration
Knowledge transfer is strongly affected by the attitude towards competition.  Excessive emphasis on competition can lead to good ideas and intellectual property being “locked up” in conservative structures.  This is addressed effectively in the Lambert Review of Business-University Collaboration, an independent review of Business-University collaboration presented to the British Government in December 2003 (see www.lambertreport.org.uk), which argued that there has been too much emphasis in the UK on developing university spinouts, leading to underfunded structures that sit on their intellectual property and do not have the resources to develop or market it effectively.
Risk-averse attitude and collaboration
Risk-averse behaviour is hindering collaborative activities between research providers and client organisations.  It has been suggested that this is being led by federal and state legal advisers to governments.  A particular manifestation is the use of open-ended indemnity provisions in contracts, where a party seeks indemnity against not only the activities of the other party but also the flow-on effects to third parties.  The provision of proper indemnities in a research contract is always difficult because the outcomes of the research are, by definition, unknown at the time the contract is executed.  It is even more difficult to address flow-on effects.  This issue is of concern to parties seeking to collaborate, and also to their insurance brokers.
Conclusions 
1.	Public support for science and innovation continues to be very important.
2.	Research of high economic impact in the community is not limited to research judged to be of high quality.  Important work can also derive from less exciting work that is focused on economically important issues and is timely.
3.	The development of the RQF is focussing attention on high quality research, and this is important.  However the implementation of RQF assessment principles will be difficult to control because of the large number of assessors that will be required and the non-intuitive nature of some of the definitions (e.g. impact).
4.	A range of schemes for providing public support for science and innovation is needed.  A diversity of funding criteria are needed going beyond quality and impact, so that important work of economic value is not excluded by inappropriate emphasis on quality or impact.