Funding,evaluation, and the performance of national research systems

Understanding the quality of science systems requires international comparative studies, which are difficult because of the lack of comparable data especially about inputs in research. In this study, we deploy an approach based on change instead of on levels of inputs and outputs: an approach that to a large extent eliminates the problem of measurement differences between countries. We firstly show that there are large differences in efficiency between national science systems, defined as the increase in output (highly cited papers) per percentage increase in input (funding). We then discuss our findings using popular explanations of performance differences: differences in funding systems (performance related or not), differences in the level of competition, differences in the level of university autonomy, and differences in the level of academic freedom. Interestingly, the available data do not support these common explanations. What the data suggest is that efficient systems are characterized by a well-developed ex post evaluation system combined with considerably high institutional funding and relatively low university autonomy (meaning a high autonomy of professionals). On the other hand, the less efficient systems have a strong ex ante control, either through a high level of so-called competitive project funding, or through strong power of the university management. Another conclusion is that more and better data are needed.     

A comparison of two approaches for measuring interdisciplinary research output: The disciplinary diversity of authors vs the disciplinary diversity of the reference list

This study investigates the convergence of two bibliometric approaches to the measurement of interdisciplinary research: one based on analyzing disciplinary diversity in the reference list of publications, the other based on the disciplinary diversity of authors of publications. In particular we measure the variety, balance, disparity and integrated diversity index of, respectively, single-author, multi-author single-field, and multi-author multi-field publications. We find that, in general, the diversity of the reference list grows with the number of fields reflected in a paper’s authors’ list and, to a lesser extent, with the number of authors being equal the number of fields. Further, we find that when fields belonging to different disciplines are reflected in the authors’ list, the disparity in the reference list is higher than in the case of fields belonging to the same discipline. However, this general tendency varies across disciplines, and noticeable exceptions are found at individual paper level.     

A novel methodology to assess the scientific standing of nations at field level

The formulation of national research policies would benefit greatly from reliable strategic analysis of the scientific infrastructure, aimed at identifying the relevant strengths and weaknesses at field level. Bibliometric methodologies thus far proposed in the literature are not completely satisfactory. This work proposes a novel “output-to-input-oriented” approach, which permits identification of research strengths and weaknesses on the basis of the ratios of top scientists and highly cited articles to research expenditures in each field. The proposed approach is applied to the Italian academic system. 2012–2016 scientific publications are analyzed, in the 218 research fields where bibliometric assessment is appropriate.     

On the Shapley value and its application to the Italian VQR research assessment exercise

Research assessment exercises have now become common evaluation tools in a number of countries. These exercises have the goal of guiding merit-based public funds allocation, stimulating improvement of research productivity through competition and assessing the impact of adopted research support policies. One case in point is Italy's most recent research assessment effort, VQR 2011–2014 (Research Quality Evaluation), which, in addition to research institutions, also evaluated university departments, and individuals in some cases (i.e., recently hired research staff and members of PhD committees). However, the way an institution's score was divided, according to VQR rules, between its constituent departments or its staff members does not enjoy many desirable properties well known from coalitional game theory (e.g., budget balance, fairness, marginality). We propose, instead, an alternative score division rule that is based on the notion of Shapley value, a well known solution concept in coalitional game theory, which enjoys the desirable properties mentioned above. For a significant test case (namely, Sapienza University of Rome, the largest university in Italy), we present a detailed comparison of the scores obtained, for substructures and individuals, by applying the official VQR rules, with those resulting from Shapley value computations. We show that there are significant differences in the resulting scores, making room for improvements in the allocation rules used in research assessment exercises.     

Assessing R&D efficiency using a two-stage dynamic DEA model: A case study of research institutes in the Chinese Academy of Sciences

Various studies have been devoted to the evaluation of the research and development (R&D) performances of universities and research institutes. However, existing studies tend to focus on static systems, that is, systems with no intertemporal effect. To tackle this issue, this study attempts to assess relative R&D efficiency of institutes from a dynamic perspective. The unified two-stage model proposed by Kao (2017) made a contribution to combining division efficiencies in the multiplier form with frontier projections in the envelopment form in a unified framework. We develop his model in a dynamic framework into which the effects of carry-over activities are embedded across the period. If the dynamic effects in the efficiency measures are not considered, the results will be biased. This is one of the few studies to examine dynamic effects within the framework of the R&D process. Our analysis is based on samples of 17 research institutes in the Chinese Academy of Sciences over the period of 2012–2015. When compared with the proposed data envelope analysis (DEA) model, results show that the static DEA model may underestimate the R&D efficiency scores. The institutes experienced significant improvements in system efficiency, mainly due to the improvements in transfer efficiency. However, there is still much room for improvement in transferring scientific and technological (S&T) achievements. We also find that the resource scale played an important role in influencing basic research. Finally, the projections of inefficient institutes indicate that most institutes had insufficient carry-over inputs (newly approved projects and management cost) based on the average four-year values, and existing slack resources for managers to improve the future performance.