Allocative efficiency in public research funding: Can bibliometrics help?

The use of outcome control modes of research evaluation exercises is ever more frequent. They are conceived as tools to stimulate increased levels of research productivity, and to guide choices in allocating components of government research budgets for publicly funded institutions. There are several contributions in the literature that compare the different methodological approaches that policy makers could adopt for these exercises, however the comparisons are limited to only a few disciplines. This work, examining the case of the whole of the “hard sciences” of the Italian academic system, makes a comparison between results obtained from peer review type of evaluations (as adopted by the Ministry of Universities and Research) and those possible from a bibliometric approach (as developed by the authors). The aim is to understand to what extent bibliometric methodology, which is noted as relatively inexpensive, time-saving and exhaustive, can complement and integrate peer review methodology in research evaluation.     

科研团队的多样性对团队绩效的影响研究

虚拟化、综合化、多元化趋势是科研团队在新一轮科技革命中的重要特征,研究新趋势下影响科研团队绩效的因素对科技工作者的未来发展以及国家科技创新实力的提升具有重大意义。在广泛的文献调研基础上,探讨团队规模、团队成员机构多样性、国别多样性、跨学科性、资金来源丰富性以及团队不稳定性六个因素对科研团队绩效（学术论文产出数量与产出质量）的影响,并通过实证分析予以验证。研究表明：科研团队产出数量与团队规模、机构多样性、资金来源丰富性正相关,与国别多样性、跨学科性和团队不稳定性负相关;产出质量与资金来源的丰富性正相关,与机构多样性负相关。     

Do bibliometrics introduce gender,institutional or interdisciplinary biases into research evaluations?

Systematic evaluations of publicly funded research sometimes use bibliometrics alone or bibliometric-informed peer review, but it is not known whether bibliometrics introduce biases when supporting or replacing peer review. This article assesses this by comparing three alternative mechanisms for scoring 73,612 UK Research Excellence Framework (REF) journal articles from all 34 field-based Units of Assessment (UoAs) 2014–17: REF peer review scores, field normalised citations, and journal average field normalised citation impact. The results suggest that in almost all academic fields, bibliometric scoring can disadvantage departments publishing high quality research, as judged by peer review, with the main exception of article citation rates in chemistry. Thus, introducing journal or article level citation information into peer review exercises may have a regression to the mean effect. Bibliometric scoring slightly advantaged women compared to men, but this varied between UoAs and was most evident in the physical sciences, engineering, and social sciences. In contrast, interdisciplinary research gained from bibliometric scoring in about half of the UoAs, but relatively substantially in two. In conclusion, out of the three potential sources of bibliometric bias examined, the most serious seems to be the tendency for bibliometric scores to work against high quality departments, assuming that the peer review scores are correct. This is almost a paradox: although high quality departments tend to get the highest bibliometric scores, bibliometrics conceal the full extent of departmental quality advantages, as judged by peer review. This should be considered when using bibliometrics or bibliometric informed peer review.     

Academic breeding grounds: Home department conditions and early career performance of academic researchers

This study investigates how research group characteristics relate to the early career success of PhD candidates who are trained in the group. In particular, I study how the citation impact of early-career PhDs is related to the staff composition and funding of the group. Using data on a cohort of Swedish doctoral graduates in science, engineering, mathematics and medicine, two sets of findings are obtained. First, students who were trained in groups with a lower number of PhD students perform better in terms of academic productivity. From the perspective of research policy, this finding suggests a decreasing return to funding additional PhD student positions allocated to professors who are already maintaining larger research groups. Second, PhD students trained in groups with funding for PhD research that is conditioned by funder influence over the topic of thesis research are more likely to stay in academia. Controlling for career destination, however, PhDs from such groups have lower than average scientific productivity and citation impact. These results suggest that funders of PhD studies face a trade-off between the two different funding objectives of “getting what they want” in terms of research content and fostering successful scholars.     

Performance-based university research funding systems

The university research environment has been undergoing profound change in recent decades and performance-based research funding systems (PRFSs) are one of the many novelties introduced. This paper seeks to find general lessons in the accumulated experience with PRFSs that can serve to enrich our understanding of how research policy and innovation systems are evolving. The paper also links the PRFS experience with the public management literature, particularly new public management, and understanding of public sector performance evaluation systems. PRFSs were found to be complex, dynamic systems, balancing peer review and metrics, accommodating differences between fields, and involving lengthy consultation with the academic community and transparency in data and results. Although the importance of PRFSs seems based on their distribution of universities’ research funding, this is something of an illusion, and the literature agrees that it is the competition for prestige created by a PRSF that creates powerful incentives within university systems. The literature suggests that under the right circumstances a PRFS will enhance control by professional elites. PRFSs since they aim for excellence, may compromise other important values such as equity or diversity. They will not serve the goal of enhancing the economic relevance of research.     

Academic careers, patents, and productivity: industry experience as scientific and technical human capital

We examine career patterns within the industrial, academic, and governmental sectors and their relation to the publication and patent productivity of scientists and engineers working at university-based research centers in the United States. We hypothesize that among university scientists, intersectoral changes in jobs throughout the career provide access to new social networks and scientific and technical human capital, which will result in higher productivity. For this study, the curriculum vitae of 1200 research scientists and engineers were collected and coded. In addition, patent data were collected from the U.S. Patent and Trademark Office. The overarching conclusion from our analysis is that the academic scientists’ and engineers’ research careers we studied are quite different than characterized in the research productivity literature that is a decade or more old. The wave of center creation activity that began in the early 1980s and continues today has resulted not only in greater ties between universities and industry, but also markedly different academic careers.     

Peering inside research networks: Some observations on the effect of the intensity of collaboration on the variability of research quality

Although there is some general agreement that increasing levels of collaboration amongst academics produce research papers that receive more citations, and that larger numbers of citations often imply higher quality, the issue of collaboration and its effect upon research output remains a controversial area with a wide range of views of what role collaboration plays and its general implications for quality. This paper re-examines the process of collaboration within research networks. It considers the role of collaboration and its effect on quality by studying the relationship between the level of interaction within research networks (collaboration) and the extent of variability of quality within those research networks. Twenty-two scientific networks from Austria are examined. The findings of the analysis are that increasing levels of collaboration are strongly associated with lower levels of variability of quality within each network. The conclusion is therefore drawn that collaboration at the level of the research network acts upon research quality qua peer review and that this peer review effect is inherent throughout the research process.     

Extramural research grants and scientists’ funding strategies: Beggars cannot be choosers?

Although competitive funding of public research has been characterised as providing output incentives that raise efficiency and productivity, we know very little about whether the quality of a scientist's research is in fact the primary award criterion on which funding bodies base their grant decision. This paper provides insights into scientists’ strategies for obtaining project-based research funding in the presence of multiple funding opportunities. It draws a distinction between four types of grants, including the Sixth Framework Programme for Research and Technological Development (FP6), government, foundation, and industry grants. Based on a sample of more than 800 scientists at universities and public research institutes in Germany, the results indicate that scientist productivity measured in terms of publication and patent stock is a statistically significant determinant only for obtaining foundation and industry grants while the award of an FP6 or government grant is influenced by other characteristics. The results further show that the different grants are not complementary, i.e. scientists specialise in certain grants. In this respect, the analysis informs science, technology and innovation policy about potential discrepancies between policy rhetoric, stipulated award criteria, and actual funding outcomes which makes it possible to fine-tune the debate on how public research should be financed.     

同行评议公正性的影响因素分析

同行评议是科学评价的重要方式,也是政府基础科学资助机构资源配置的主要方式之一,公正高效的同行评议是保证科学质量的基础。本文将通过区分影响同行评议公正性的两种类型三个方面的因素,即评议过程本身的因素和评议过程以外的因素,以及评议过程中的制度性因素和非制度性个人因素,较为系统地分析影响公正性诸因素,以为制定公正性政策提供依据。     

“3＋X”同行评议方式的合理性分析

国家自然科学基金项目同行评议方式的设计是否合理直接关系到项目评审的质量。近年,国家自然科学基金项目数个学科试行"3 X"新的同行专家评议方式。这项新的同行评议改制更加科学、合理,使得基金项目的评审更加客观、公正,更能确保国家基金评审项目的质量。     

A helping hand from the government? How public research funding affects academic output in less-prestigious universities in China

Universities play an important role in any scientific and technological innovation system. Previous studies have indicated that more generous public research funding resulted in higher research output in universities. Our study, however, proposes that the positive impact of public funds is much weaker in less-prestigious universities than in prestigious ones, and that overdependence on public research funding in fact even hurts academic output quality in less-prestigious universities. We find evidence for this claim in a dataset from among Chinese 622 universities in the period 2010–2017. The negative correlation between high dependence on public research funding and academic output quality is not uniform but depends on specific conditions. It is likely to be weaker in less-prestigious research-oriented universities and in less prestigious universities dedicated to fundamental research. Moreover, we find that for them collaboration with top universities and a high proportion of senior scientists can mitigate the negative impact that high dependence on public research funding has on academic output quality and improve the efficiency in the spending of public funds. Our study contributes to the literature by highlighting differences between prestigious and less-prestigious universities in terms of how public funds affect academic output and by evaluating the impact of government involvement in scientific research at the university-level and it is the first study globally to do so.     

‘Impact’ in the proposals for the UK's Research Excellence Framework: Shifting the boundaries of academic autonomy

Evaluation of university-based research already has a reasonably long tradition in the UK, but proposals to revise the framework for national evaluation aroused controversy in the academic community because they envisage assessing more explicitly than before the economic, social and cultural ‘impact’ of research as well as its scientific quality. Using data from the 2009 public consultation on the proposals for a Research Excellence Framework, this paper identifies three main lines of controversy: the threats to academic autonomy implied in the definition of expert review and the delimitation of reviewers, the scope for boundary-work in the construction of impact narratives and case studies, and the framing of knowledge translation by the stipulation that impact ‘builds on’ research. Given the behaviour-shaping effects of research evaluation, the paper demonstrates how the proposed changes could help embed impact considerations among the routine reflexive tools of university researchers and enhance rather than restrict academic autonomy at the level of research units. It also argues that the REF could constitute an important dialogical space for negotiating science–society relations in an era of increasing heteronomy between academia, state and industry. But the paper raises doubts about whether the proposed operationalisation of impact is adequate to evaluate the ways that research and knowledge translation are actually carried out.     

Using the h-index to measure the quality of journals in the field of business and management

This paper considers the use of the h-index as a measure of a journal’s research quality and contribution. We study a sample of 455 journals in business and management all of which are included in the ISI Web of Science (WoS) and the Association of Business School’s peer review journal ranking list. The h-index is compared with both the traditional impact factors, and with the peer review judgements. We also consider two sources of citation data – the WoS itself and Google Scholar. The conclusions are that the h-index is preferable to the impact factor for a variety of reasons, especially the selective coverage of the impact factor and the fact that it disadvantages journals that publish many papers. Google Scholar is also preferred to WoS as a data source. However, the paper notes that it is not sufficient to use any single metric to properly evaluate research achievements.     

国家重点实验室科研成果的同行评议创新模式

在当前我国开展科技评价体制改革的背景下,对现行的国家重点实验室科研成果的同行评议制度进行分析,揭示当前我国国家重点实验室科研成果的同行评议现状和问题。结合当前同行评议模式的不足,顺应国家对优化科技评价体制的要求,提出一套基于科研生态系统视角的国家重点实验室科研成果同行评议创新模式。新的评价模式从科研活动系统出发考虑科研机构的投入,产出及科研环境等要素,以科研产出的学术创新度和贡献度为导向,对科研成果进行系统评价。评议结果更加客观完整,符合科研活动的客观规律。有利于构建良好的,可持续发展的科研生态。     

Funding model and creativity in science: Competitive versus block funding and status contingency effects

In many countries the scientific funding system is shifting from an internal block funding model toward a competitive project funding model. However, there is growing concern that the competitive project funding system favors relatively safe, conventional projects at the expense of risky, novel research. It is important to assess different funding models in order to design better funding systems for science. This paper empirically tests for differences in the novelty of funded outputs between internal block funding and competitive project funding, in the setting of Japan, where both funding models play a significant role. Combining survey data from a large sample of research projects in Japan and bibliometric information about the publications produced from these projects, we find that projects funded by competitive funds on average have higher novelty compared to those funded by internal block funds. However, such positive effects only hold for researchers with high status, such as senior and male researchers. In contrast, compared to internal block funding, competitive project funding has a negative relation to novelty for low status scientists (especially junior and female researchers). The findings suggest that the competitive project selection procedure is less receptive to novel ideas from researchers with low academic status and therefore discourages their novel research. These findings can serve as a warning about potential biases in competitive funding allocation procedures and suggest the importance of secure stable funding for allowing researchers with low status to pursue their novel ideas.     

Influencing scientists’ collaboration and productivity patterns through new institutions: University research centers and scientific and technical human capital

This paper analyzes the effect of university research centers on the productivity and collaboration patterns of university faculty. University research centers are an important subject for policy analysis insofar that they have become the predominant policy response to scientific and technical demands that have not been met by extant institutions, including academic departments, private firms, and government laboratories. Specifically, these centers aim to organize researchers from across the disciplines and sectors which, collectively as a research unit, possess the scientific and technical capacity relevant to scientific and technical goals of the sponsoring agencies. In this paper, we measure the productivity and collaboration patterns of university researchers affiliated with a relatively large-scale and “mature” university research center to discern the effects, if any, of the center mechanism on individual scientists and engineers. Based on an analysis of longitudinal bibliometric data, the results from this case study demonstrate affiliation with the center to be effective at enhancing overall productivity as well as at facilitating cross-discipline, cross-sector, and inter-institutional productivity and collaborations.     

Performance-based research funding: Evidence from the largest natural experiment worldwide

A performance-based research funding system (PRFS) is a nationwide incentive scheme that promotes and rewards university research performance through competition for government funding. The UK’s PRFS, currently the Research Excellence Framework (REF), is considered the oldest, largest and most developed payment-by-results system in academia worldwide. Surprisingly, and despite the strong criticisms, little has been done to quantitatively and casually evaluate the intended and unintended effects of the PRFSs. In this paper, we evaluate the incremental impact of the REF 2014 in the fields of Economics and Business. We use a synthetic control method to compare the performance of UK universities with their artificial counterfactual units constructed using data from US universities. Our analysis shows, on the whole, that the introduction of the REF had a significant and positive impact on the quantity and quality of the scientific research produced at UK universities. However, we do not find a significant effect on the per author measures, suggesting that the REF did not result in an increase in research productivity. We also show that the effects are more heterogeneous across universities than across academic disciplines. We do not find evidence of a shift of research focus from Economics to Business topics, as some feared. But our analysis indicates that the REF 2014 may have contributed to the concentration of research excellence in elite institutions.     

Facts and fads in academic research management: The effect of management practices on research productivity in Australia

As a response to competitive market forces and governmental steering policies, Australian universities have strengthened considerably their internal research management in the last two decades. This paper examines empirically the effect of management on academic research productivity. The results suggest that management practices indeed seem to have some positive effect on research productivity, and the effect is consistent in the earlier (1995–2000) and later (2001–2007) time period. Universities with a more intensive management approach not only have higher absolute level of research productivity but they demonstrate also faster growth in productivity. An omitted variable bias and robustness of the results to the choice of the output measure are under a particular attention and call for some caution in interpreting the results.     

Industry funding and university professors’ research performance

University research is to an increasing extent funded by industry, and the share of basic funding is decreasing. In the literature, there are optimistic and pessimistic views on the implication of this development. Based on data from a questionnaire study among all tenured university professors in Norway (N = 1967) we find that there is a significant relationship between industry funding and research performance: professors with industrial funding describe their research as applied to a greater extent, they collaborate more with other researchers both in academia and in industry, and they report more scientific publications as well as more frequent entrepreneurial results. There is neither a positive nor negative relationship between academic publishing and entrepreneurial outputs.