University licensing and the flow of scientific knowledge

As university involvement in technology transfer and entrepreneurship has increased, concerns over the patenting and licensing of scientific discoveries have grown. This paper examines the effect that the licensing of academic patents has on journal citations to academic publications covering the same scientific research. We analyze data on invention disclosures, patents, and licenses from the University of California, a leading U.S. academic patenter and licensor, between 1997 and 2007. We also develop a novel “inventor-based” maximum-likelihood matching technique to automate and generalize Murray's (2002) “patent-paper pairs” methodology. We use this methodology to identify the scientific publications associated with University of California patents and licenses.Based on a “difference-in-differences” analysis, we find that within our sample of patented academic discoveries, citations to licensed patent-linked publications are higher in the three years after the license, although this difference is not statistically significant. We then disaggregate our sample into (a) patented discoveries that are likely to be used as “research tools” by other researchers (based on the presence of material transfer agreements (MTAs) that cover them) and (b) patented discoveries not covered by MTAs. Citations to publications linked to licensed patents in the latter subset (not covered by MTAs) are higher for publications linked to licensed patents, and this difference is statistically significant. In contrast, licensing of patented discoveries that are also research tools is associated with a reduction in citations to papers linked to these research advances, raising the possibility that licensing may restrict the flow of inputs to “follow-on” scientific research.     

Identifying breakthrough scientific papers

Citation analysis does not tell the whole story about the innovativeness of scientific papers. Works by prominent authors tend to receive disproportionately many citations, while publications by less well-known researchers covering the same topics may not attract as much attention. In this paper we address the shortcomings of traditional scientometric approaches by proposing a novel method that utilizes a classifier for predicting publication years based on latent topic distributions. We then calculate real-number innovation scores used to identify potential breakthrough papers and turnaround years. The proposed approach can complement existing citation-based measures of article importance and author contribution analysis; it opens as well novel research direction for time-based, innovation-centered research scientific output evaluation. In our experiments, we focus on two corpora of research papers published over several decades at two well-established conferences: The World Wide Web Conference (WWW) and the International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), containing around 3500 documents in total. We indicate significant years and demonstrate examples of highly-ranked papers, thus providing a novel insight on the evolution of the two conferences. Finally, we compare our results to citation analysis and discuss how our approach may complement traditional scientometrics.     

Research addressing emerging technological ideas has greater scientific impact

This study empirically examines the association between the extent of emerging technological ideas in a scientific publication and its future scientific impact measured by number of citations. We analyze metadata of scientific publications in three scientific domains: Nano-Enabled Drug Delivery, Synthetic Biology, and Autonomous Vehicles. By employing a bibliometric indicator for identifying and quantifying emerging technological ideas – as derived terms from the titles and abstracts – we measure the extent to which the publication contains emerging technological ideas in each domain. Then, we statistically estimate the size and statistical significance of the relationship between the publication-level technological emergence score and the normalized number of citations accruing to the publication.Our analysis shows that the degree to which a paper contains technologically emerging ideas is positively and strongly associated with its future citation impact in each of the three domains. An additional analysis demonstrates that this relationship holds for citations from other publications, both in the same field as, and in different fields from, the scientific domain of the focal publication. A series of tests for validation further support our argument that the greater the extent to which scientific knowledge (a paper) contains emerging ideas, the bigger its scientific impact. Implications for academic researchers, research policymakers, and firms are discussed.     

Ranking researchers: Evidence from Indonesia

In the span of three years, Indonesia went from being the second worst to the top producer of scientific journal articles in Southeast Asia. We investigate whether a transparent system of ranking every single researcher in the country based on publications and citations (SINTA) contributed to this turnaround. Using panel data from over 200,000 Indonesian researchers (and comparing to researchers from Thailand and the Philippines), we show that the implementation of SINTA coincides with changes in the production of publications by Indonesian researchers consistent with the weights used in the ranking formula. Although we see modest improvements in publication rates in top-ranked journals, 62 % of the observed increase in total publications is from conference proceedings. Because SINTA was launched around the same time as other policies that focused on increasing publications, isolating the precise impact of SINTA remains challenging. Nevertheless, after accounting for such policies, our results imply that a ranking and evaluation system for researchers can contribute to overall improvements in scientific capacity in low- and middle-income countries.     

Short-term incentives of research evaluations: Evidence from the UK Research Excellence Framework

We document incentive effects of the evaluation deadlines in the UK’s performance-based research funding system. Studying 3,597,272 publications by UK researchers, we find that publications just before assessment deadlines obtain substantially fewer citations and are published in venues with lower impact factors. These trends reverse abruptly after the deadlines. We discuss different factors that contribute to this observation and provide evidence that evaluation deadlines are likely to set incentives against investment in research quality and long-term topics. We conclude that where such shifts in research incentives are not intended, they might require balancing by additional incentives for exploratory, long-term oriented research.     

融入科研管理的科技报告质量控制机制研究

   提高科技报告质量是实现科技资源有效积累、促进科技信息高效传播、提高科技成果转化应用的重要方式。构建融入科研管理流程的科技报告质量控制机制,将科技报告撰写主体、管理主体、评价主体行为纳入统一的科研管理平台,对科技报告进行精细化审核与评价。审核、评价行为将直接决定科技报告质量、科研项目进程与科研绩效。将科技报告质量控制与科研管理相融合能够实现对科技报告撰写、审核、评价行为的有效统一与衔接,达到降低管理成本,提高工作效率,实现科技报告质量有效控制的目的。     

Assessing basic research: Some partial indicators of scientific progress in radio astronomy

As the costs of certain types of scientific research have escalated and as growth rates in overall national science budgets have declined, so the need for an explicit science policy has grown more urgent. In order to establish priorities between research groups competing for scarce funds, one of the most important pieces of information needed by science policy-makers is an assessment of those groups' recent scientific performance. This paper suggests a method for evaluating that performance.After reviewing the literature on scientific assessment, we argue that, while there are no simple measures of the contributions to scientific knowledge made by scientists, there are a number of ‘partial indicators’ — that is, variables determined partly by the magnitude of the particular contributions, and partly by ‘other factors’. If the partial indicators are to yield reliable results, then the influence of these ‘other factors’ must be minimised. This is the aim of the method of ‘converging partial indicators’ proposed in this paper. We argue that the method overcomes many of the problems encountered in previous work on scientific assessment by incorporating the following elements: (1) the indicators are applied to research groups rather than individual scientists; (2) the indicators based on citations are seen as reflecting the impact, rather than the quality or importance, of the research work; (3) a range of indicators are employed, each of which focusses on different aspects of a group's performance; (4) the indicators are applied to matched groups, comparing ‘like’ with ‘like’ as far as possible; (5) because of the imperfect or partial nature of the indicators, only in those cases where they yield convergent results can it be assumed that the influence of the ‘other factors’ has been kept relatively small (i.e. the matching of the groups has been largely successful), and that the indicators therefore provide a reasonably reliable estimate of the contribution to scientific progress made by different research groups.In an empirical study of four radio astronomy observatories, the method of converging partial indicators is tested, and several of the indicators (publications per researcher, citations per paper, numbers of highly cited papers, and peer evaluation) are found to give fairly consistent results. The results are of relevance to two questions: (a) can basic research be assessed? (b) more specifically, can significant differences in the research performance of radio astronomy centres be identified? We would maintain that the evidence presented in this paper is sufficient to justify a positive answer to both these questions, and hence to show that the method of converging partial indicators can yield information useful to science policy-makers.     

预算控制系统对科研经费支出绩效的影响

以预算控制系统理论为基础,运用问卷调查数据系统分析了预算控制系统因素（制度设计、内部监督和第三方评估审计）对科研经费支出绩效的影响机制,以及核算难度的调节作用。将科研经费支出绩效分为科研人员经费使用绩效和科研单位经费管理绩效两个层面。研究表明,制度设计和内部监督以预算执行力为中介显著正向影响科研经费支出的两个层面绩效;第三方评估审计以预算执行力为中介显著正向影响科研人员经费使用绩效,但对科研单位经费管理绩效无显著影响;同时,核算难度负向调节预算执行力与科研经费支出两个层面绩效之间的关系。     

New journal classification methods based on the global h-index

In this work we develop new journal classification methods based on the h-index. The introduction of the h-index for research evaluation has attracted much attention in the bibliometric study and research quality evaluation. The main purpose of using an h-index is to compare the index for different research units (e.g. researchers, journals, etc.) to differentiate their research performance. However the h-index is defined by only comparing citations counts of one’s own publications, it is doubtful that the h index alone should be used for reliable comparisons among different research units, like researchers or journals. In this paper we propose a new global h-index (Gh-index), where the publications in the core are selected in comparison with all the publications of the units to be evaluated. Furthermore, we introduce some variants of the Gh-index to address the issue of discrimination power. We show that together with the original h-index, they can be used to evaluate and classify academic journals with some distinct advantages, in particular that they can produce an automatic classification into a number of categories without arbitrary cut-off points. We then carry out an empirical study for classification of operations research and management science (OR/MS) journals using this index, and compare it with other well-known journal ranking results such as the Association of Business Schools (ABS) Journal Quality Guide and the Committee of Professors in OR (COPIOR) ranking lists.     

Organization of biomedical data for collaborative scientific research: A research information management system

Biomedical researchers often work with massive, detailed and heterogeneous datasets. These datasets raise new challenges of information organization and management for scientific interpretation, as they demand much of the researchers’ time and attention. The current study investigated the nature of the problems that researchers face when dealing with such data. Four major problems identified with existing biomedical scientific information management methods were related to data organization, data sharing, collaboration, and publications. Therefore, there is a compelling need to develop an efficient and user-friendly information management system to handle the biomedical research data. This study evaluated the implementation of an information management system, which was introduced as part of the collaborative research to increase scientific productivity in a research laboratory. Laboratory members seemed to exhibit frustration during the implementation process. However, empirical findings revealed that they gained new knowledge and completed specified tasks while working together with the new system. Hence, researchers are urged to persist and persevere when dealing with any new technology, including an information management system in a research laboratory environment.     

The impact of national research funds: A regression discontinuity approach to the Chilean FONDECYT

This paper analyses the role of national research funds in promoting scientific production in emerging economies. The study focuses on the impact of the Chilean National Science and Technology Research Fund (FONDECYT). The analysis uses data drawn from international sources of bibliometric information combined with the administrative records of the program's executing unit. To measure the program's impact, we implement a regression discontinuity (RD) design on principal researchers who applied for funding between 1988 and 1997 considering as outcomes both quantity (publications up to 2002) and quality (citations up to 2005) of their scientific production. Our results show significant and positive impact in terms of publications, but no impact in terms of quality of scientific production in the proximity of the program's threshold ranking.     

评价交叉科学研究人员的新思路

在当前利用科学计量学工具对研究人员的科研评价中,由于采用的基本是一种基于学科分类的评价方法,这使得交叉科学研究人员处于相对不利的地位。本文以交叉科学研究人员代表钱学森为典型案例,将其与专业领域作者的被引指标进行比较。为解决这一问题,文章最后提出了新的"学科扩散因子"的指标。     

广东农业产业研究、科研支撑及发展演变分析——基于2001-2018年万方文献数据

通过搜索和分析2001—2018年广东区域内研究机构发表的农业类核心期刊论文可以看到,随着产业结构的调整,论文产出、项目来源、研究机构、研究重点以及高被引等情况发生一系列的变化,这些变化促使科技资源配置和科技项目管理要作出相应调整,并提出相关建议。     

基于成熟度的科研项目管理评测及改进研究

摘要：通过对科研项目过程的管理和控制,有利于将研究者的科研能力最终转化成实际成果,而对科研项目管理成熟度的测评则是提高科研项目管理水平的有效措施。本文改进了作者先前提出的科研项目管理成熟度的定义及相应的评测指标体系、成熟度等级模型,在建立评测结果与科研项目管理成熟度等级的对应关系后,提出了持续改进科研项目管理成熟度等级的方法。算例结果说明本文提出的方法可以较好地实现科研项目管理能力评测工作,帮助组织找出影响科研项目管理能力的瓶颈因素,提升组织的科研管理能力。     

诺贝尔生理学或医学奖获得者论文影响力研究

利用国际权威数据库,根据科学计量学方法,从SC I论文数量、被引次数和期刊影响因子等三个方面,通过对1981~2005年诺贝尔生理学或医学奖获得者获奖前科学论文影响力的测度,发现世界顶级生物医学科学家的科学创造最佳年龄区可能由25-45岁移至31-55岁。同时,分析科学家个人研究水平与其影响力和科学家个人科学能力与其国家科技整体实力的关系。     

How does co-authoring with a star affect scientists' productivity? Evidence from small open economies

There is increasing policy interest in the recruitment and integration of star scientists as a mechanism to catalyse research productivity. We use rich data for three Small Open Economies (Ireland, Denmark, and New Zealand) on publications, citations and co-authorships to examine how co-authorship with a co-located star scientist affects the co-author's productivity, both including and excluding the output directly co-authored with the star. The latter effect provides a measure of the extent to which star collaborations crowd out/in other output. Event-study analyses reveal that star co-authorships are associated with economically and statistically significant increases in co-authors' output (measured by field-normalized total citations). Output in the three years after the initial star co-authorship is increased by 89.6 % when star co-authored publications are included and by 16.2 % when they are excluded. The results are robust to using an alternative measure of quality-adjusted output based on journal publication quality. We find co-authoring with a star increases the quality but not quantity of output when star co-authored publications are excluded. We explore heterogeneity by period, field and whether the authors have multiple star co-authorships. We conclude that policymakers' and institutions' efforts to promote access to star scientists may have substantial direct and indirect effects on the productivity of incumbent scientists within departments.     

Methods to account for citation inflation in research evaluation

Quantitative research evaluation requires measures that are transparent, relatively simple, and free of disciplinary and temporal bias. We document and provide a solution to a hitherto unaddressed temporal bias – citation inflation – which arises from the basic fact that scientific publication is steadily growing at roughly 4% per year. Moreover, because the total production of citations grows by a factor of 2 every 12 years, this means that the real value of a citation depends on when it was produced. Consequently, failing to convert nominal citation values into real citation values produces significant mis-measurement of scientific impact. To address this problem, we develop a citation deflator method, outline the steps to generalize and implement it using the Web of Science portal, and analyze a large set of researchers from biology and physics to demonstrate how two common evaluation metrics – total citations and h-index – can differ by a remarkable amount depending on whether the underlying citation counts are deflated or not. In particular, our results show that the scientific impact of prior generations is likely to be significantly underestimated when citations are not deflated, often by 100% or more of the nominal value. Thus, our study points to the need for a systemic overhaul of the counting methods used evaluating citation impact – especially in the case of researchers, journals, and institutions – which can span several decades and thus several doubling periods.     

Commercializing the laboratory: Faculty patenting and the open science environment

This paper investigates the relationship between patenting and publication of research results by university faculty members. Our study adds to the limited evidence on this topic with an empirical investigation based on a panel data set for a broad sample of university researchers. Results suggest that publication and patenting are complementary, not substitute, activities for faculty members. This is not consistent with recent concerns regarding deleterious effects of patenting on the research output of faculty members. Average citations to publications, however, appear to decline for repeat patenters, suggesting either a decrease in quality or restrictions on use associated in patent protection.     

定量分析支持的医学科研项目代表作评价研究

鉴于目前我国的代表性成果评价制度处于研究和探索阶段,缺乏具有推广基础的示范应用,为促进代表作评价方法和工具创新,更加科学、准确、客观地开展科研成果代表作评价,分析国内外科研项目评价研究与实践现状以及代表作评价的内涵和特点,设计提出基于客观数据和定量方法支持的医学科研项目代表性成果评价方法,并选取20个科研项目的代表作进行实证分析,依次分级计算投入和产出指标数据,并利用数据包络分析模型进行效率分析。研究结果显示,利用科学计量和数据包络分析方法综合计算得到的科研项目代表性成果的量化分析结果,可在一定程度上反映学术成果的质量和影响力,可作为专家定性评价的参考和依据。最后,在强调代表作形式的多样性、提倡代表作评价方式的多元性和利用新的方法和技术改进传统评价方法等方面作出思考并提出建议,为我国代表作评价制度的建设和完善提供参考。     

自然指数对高校科研能力评估的意义与思考——以南方科技大学为例

 作为2014年新推出的一项科研评价指数,自然指数能通过文章计量数和分数式计量数两个指标数据反映出国家、地区、科研机构和大学等高水平论文的产出能力、创新趋势、创新能力、优势学科和合作潜力,为我们提供了一种新的评估高校科研能力的方法。自然指数以12个月为统计时长,定期更新,对被评价机构在82种顶级自然科学类期刊上发表的研究型论文进行了统计分析。与基本科学指标数据库（ESI）评价体系需要监测十年的论文数据相比,自然指数的排名更能反映出被评价机构在短期内的科研产出情况,能更客观公正的评价新建高校科研产出的情况。以南方科技大学为例,该校建校未满十年,ESI数据与其他老牌高校还有一定差距,但从自然指数数据显示,该校高水平论文一直在飞速增长,上升势头非常明显,2018年已跃居内地高校第27位,深圳地区首位。数据还表明该校与海外大学的科研合作十分密切,发表的高水平论文以外部合作为主,其优势学科为化学和物理学科,生命科学和地球与环境科学相对比较薄弱,学科发展水平较不均衡。同样,自然指数评价体系也存在一些不足,如对学者的贡献程度没有区分,默认每位作者的贡献相等;统计的是论文数量和分数式计量的总量,没有以个人平均产出作为衡量标准等。随着自然指数的推广,入选自然指数统计源的各期刊,极有可能被科研管理部门在科研评价中赋予更高的权重,也会成为科研工作者投稿时选择顶级期刊的一个标准。自然指数以一种全新的视角评价了高校的科研产出情况,为今后的科研管理工作提供了一定参考。