Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK research excellence framework (REF)

Altmetrics have been proposed as a way to assess the societal impact of research. Although altmetrics are already in use as impact or attention metrics in different contexts, it is still not clear whether they really capture or reflect societal impact. This study is based on altmetrics, citation counts, research output and case study data from the UK Research Excellence Framework (REF), and peers’ REF assessments of research output and societal impact. We investigated the convergent validity of altmetrics by using two REF datasets: publications submitted as research output (PRO) to the REF and publications referenced in case studies (PCS). Case studies, which are intended to demonstrate societal impact, should cite the most relevant research papers. We used the MHq’ indicator for assessing impact – an indicator which has been introduced for count data with many zeros. The results of the first part of the analysis show that news media as well as mentions on Facebook, in blogs, in Wikipedia, and in policy-related documents have higher MHq’ values for PCS than for PRO. Thus, the altmetric indicators seem to have convergent validity for these data. In the second part of the analysis, altmetrics have been correlated with REF reviewers’ average scores on PCS. The negative or close to zero correlations question the convergent validity of altmetrics in that context. We suggest that they may capture a different aspect of societal impact (which can be called unknown attention) to that seen by reviewers (who are interested in the causal link between research and action in society).     

Do we measure novelty when we analyze unusual combinations of cited references? A validation study of bibliometric novelty indicators based on F1000Prime data

Lee et al. (2015) – based on Uzzi et al. (2013) – and Wang et al. (2017) proposed scores based on cited references (cited journals) data which can be used to measure the novelty of papers (named as novelty scores U and W in this study). Although previous research has used novelty scores in various empirical analyses, no study has been published up to now – to the best of our knowledge – which quantitatively tested the convergent validity of novelty scores: do these scores measure what they propose to measure? Using novelty assessments by faculty members (FMs) at F1000Prime for comparison, we tested the convergent validity of the two novelty scores (U and W). FMs’ assessments do not only refer to the quality of biomedical papers, but also to their characteristics (by assigning certain tags to the papers): for example, are the presented findings or formulated hypotheses novel (tags “new findings” and “hypothesis”)? We used these and other tags to investigate the convergent validity of both novelty scores. Our study reveals different results for the novelty scores: the results for novelty score U are mostly in agreement with previously formulated expectations. We found, for instance, that for a standard deviation (one unit) increase in novelty score U, the expected number of assignments of the “new finding” tag increase by 7.47%. The results for novelty score W, however, do not reflect convergent validity with the FMs’ assessments: only the results for some tags are in agreement with the expectations. Thus, we propose – based on our results – the use of novelty score U for measuring novelty quantitatively, but question the use of novelty score W.     

The validation of (advanced) bibliometric indicators through peer assessments: A comparative study using data from InCites and F1000

The data of F1000 and InCites provide us with the unique opportunity to investigate the relationship between peers’ ratings and bibliometric metrics on a broad and comprehensive data set with high-quality ratings. F1000 is a post-publication peer review system of the biomedical literature. The comparison of metrics with peer evaluation has been widely acknowledged as a way of validating metrics. Based on the seven indicators offered by InCites, we analyzed the validity of raw citation counts (Times Cited, 2nd Generation Citations, and 2nd Generation Citations per Citing Document), normalized indicators (Journal Actual/Expected Citations, Category Actual/Expected Citations, and Percentile in Subject Area), and a journal based indicator (Journal Impact Factor). The data set consists of 125 papers published in 2008 and belonging to the subject category cell biology or immunology. As the results show, Percentile in Subject Area achieves the highest correlation with F1000 ratings; we can assert that for further three other indicators (Times Cited, 2nd Generation Citations, and Category Actual/Expected Citations) the “true” correlation with the ratings reaches at least a medium effect size.     

Altmetrics of Papers From Scientific Periphery Reflect Global Trends: A Case Study of Publications by Zagreb University School of Medicine

This study aims to investigate the altmetric activity of papers published by the University of Zagreb School of Medicine in internationally visible journals and to identify differences in altmetric activity between the papers published in international and local journals and between those published in English and Croatian. We also investigated changes in altmetric activity over time and the characteristics of papers with the highest Twitter and Mendeley activity.The sample included 390 papers collected from the bibliographic database Scopus. Their altmetric and citation activities were measured at three time points: in July 2014, 2015, and 2016.The findings generally correspond to those observed in the large-scale studies of medical papers. Papers in renowned journals, and papers reporting clinical guidelines and multicentric studies had the most intense altmetric activity. In contrast, papers published in local, Croatian journals showed minimal altmetric activity, especially the papers published in Croatian. These results indicate that the local publishing community has not yet recognised social media as a tool for promoting research and that non-English language publications have minimal chances to receive attention, even in social media.The evaluative potential of altmetric indicators has to be further explored in a broader context.     

科学评价视角下F1000、Mendeley与传统文献计量指标的比较

为了探讨同行评议、影响计量学以及传统文献计量指标在科学评价中的有效性,本文选取F1000、Mendeley以及Web of Science、Google Scholar数据库,采用SPSS 19.0软件,将心理学与生态学的1,3篇论文的同行评议结果即F1000因子、Mendeley阅读统计、期刊影响因子,以及Web of Science、Google Scholar数据库中被引频次进行相关分析。结果表明:同行评议结果、传统引文分析指标以及以Mendeley为代表的影响计量指标具有低度正相关性,这意味着上述指标在科学评价中审视视角的不同以及数字时代科学评价的多维构成;心理学筛选数据中F1000因子与期刊影响因子相关度几近为0,这一结论进一步证实了期刊影响因子与单篇论文影响力的严重背离;生态学与心理学指标相关分析结果的不同折射出科学评价中自然科学、社会科学的差异。图3。表4。参考文献10。     

Algorithmically generated subject categories based on citation relations: An empirical micro study using papers on overall water splitting

One important reason for the use of field categorization in bibliometrics is the necessity to make citation impact of papers published in different scientific fields comparable with each other. Raw citations are normalized by using field-categorization schemes to achieve comparable citation scores. There are different approaches to field categorization available. They can be broadly classified as intellectual and algorithmic approaches. A paper-based algorithmically constructed classification system (ACCS) was proposed which is based on citation relations. Using a few ACCS field-specific clusters, we investigate the discriminatory power of the ACCS. The micro study focusses on the topic ‘overall water splitting’ and related topics. The first part of the study investigates intellectually whether the ACCS is able to identify papers on overall water splitting reliably and validly. Next, we compare the ACCS with (1) a paper-based intellectual (INSPEC) classification and (2) a journal-based intellectual classification (Web of Science, WoS, subject categories). In the last part of our case study, we compare the average number of citations in selected ACCS clusters (on overall water splitting and related topics) with the average citation count of publications in WoS subject categories related to these clusters. The results of this micro study question the discriminatory power of the ACCS. We recommend larger follow-up studies on broad datasets.     

医学F1000因子与影响因子的比较研究

介绍了医学F1000因子的概念和计算公式,选取医学F1000网站部分文献,通过其F1000因子与其所在期刊影响因子的比较,发现医学F1000与影响因子在评价对象、评价方法、组织文献方法等方面的不同,建议在评价文献质量时应综合应用这两项指标。     

不同类型论文层面计量指标间的相关性研究

对补充计量学的研究背景进行梳理,总结出补充计量学的5个特点,并对论文层面计量指标体系和当前常用的补充计量工具进行介绍。为探究综合性指标Altmetric score与引用指标之间的关系,分别对PLOS和F1000网站中提供的论文数据和指标数据进行收集,检测两个指标之间的相关性,并对比两组实验结果。结果表明,Altmetric score与引用指标存在正相关关系,且在评价论文时具有一定的一致性。     

A Bibliometric Analysis of High Impact Research in the Middle East Using Champion Works

Increasing political and financial support for scientific research in the Middle East requires academic and research communities in the region to demonstrate the visibility and impact of their scientific output. However, for countries with smaller scientific communities or lack of detailed information on their scientific production, the use of common metrics of scientific impact (e.g., number of papers, impact factor, h-index, etc.) may fail to reveal their true ability to produce high quality research, and thus guarantee the wanted societal support. In such cases, identifying and highlighting outstanding papers produced by national institutions or scientists may be another way to demonstrate scientific capacity and impact. In this context, this work aims to provide an overview of champion works (papers that have received over 1,000 citations) produced by Middle East countries. This analysis focuses on science, medicine, and technology papers featured in the Science Citation Index Expanded of Web of Science. The authors identified 213 champion works authored by Middle East scientists published since the 1970s. Israel is currently the leading nation in the Middle East in terms of published champion works, but at least one such work was identified for the majority of countries in the region. Middle East champion works were published on a diverse range of subject categories and often featured in the top journals worldwide (e.g., Science, Nature, etc.). The top institutions in the Middle East authoring champion works and their leading collaborating countries worldwide are listed, and the role of international scientific collaborations in achieving these highly cited papers is highlighted.     

基于学术迹的湖南省高校学科学术影响力评价研究--以数学学科为例

[目的/意义]在“破五唯”的评价背景下,客观、合理的学术影响力评价指标对于高校学科建设、科研能力、科研质量的评估具有重要作用与意义。[方法/过程]本文选取学术迹指标,以湖南省10所高校数学学科2010—2019年的学术论文为实证样本,计算学术迹并结合h指数、10年影响因子等传统指标,从宏观角度评价学科学术影响力。接着对这10所高校的学术迹、学术矩阵进行分析,从微观角度探究影响学科学术影响力的具体因素。[结果/结论]结果表明:相较于传统指标而言,本文的学术影响力评价更加客观合理,具备较好的学科评价效果。在学术矩阵分析中,h尾以及e域的论文数量与比例最为关键,这能为高校学科建设及评估提供新的思路。     

Paper-patent citation linkages as early signs for predicting delayed recognized knowledge: Macro and micro evidence

In this study, we investigate the extent to which patent citations to papers can serve as early signs for predicting delayed recognized knowledge in science using a comparative study with a control group, i.e., instant recognition papers. We identify the two opposite groups of papers by the Bcp measure, a parameter-free index for identifying papers which were recognized with delay. We provide a macro (Science/Nature papers dataset) and micro (a case chosen from the dataset) evidence on paper-patent citation linkages as early signs for predicting delayed recognized knowledge in science. It appears that papers with delayed recognition show a stronger and longer technical impact than instant recognition papers. We provide indication that in the more recent years papers with delayed recognition are awakened more often and earlier by a patent rather than by a scientific paper (also called “prince”). We also found that patent citations seem to play an important role to avoid instant recognition papers to level off or to become a so called “flash in the pan”, i.e., instant recognition. It also appears that the sleeping beauties may firstly encounter negative citations and then patent citations and finally get widely recognized. In contrast to the two focused fields (biology and chemistry) for instant recognition papers, delayed recognition papers are rather evenly distributed in biology, chemistry, psychology, geology, materials science, and physics. We discovered several pairs of “science sleeping”-“technology inducing”, such as “biology-biotechnology/pharmaceuticals”, “chemistry-chemical engineering”, as well as some trans-fields science-technology interactions, such as “psychology - computer technology/control technology/audio-visual technology”, “physics - computer technology”, and “mathematics-computer technology”. We propose in further research to discover the potential ahead of time and transformative research by using citation delay analysis, patent & NPL analysis, and citation context analysis.     

An Introduction to Altmetrics

Altmetrics, or alternative metrics, are forging a new way to capture the impact of not only articles, but also scholarly or research “products” by tracking them when they are mentioned online, such as in blogs or social media platforms. While altmetrics have a lot of potential, there are also some limitations preventing their full acceptance alongside traditional citation metrics. This column will explain the basics of altmetrics and altmetric tools, discuss some of the ways they can be used in libraries, and explore some possible concerns with this new metric. A list of resources for additional information is also included.     

The citation disadvantage of clinical research

Biomedical research encompasses diverse types of activities, from basic science (“bench”) to clinical medicine (“bedside”) to bench-to-bedside translational research. It, however, remains unclear whether different types of research receive citations at varying rates. Here we aim to answer this question by using a newly proposed paper-level indicator that quantifies the extent to which a paper is basic science or clinical medicine. Applying this measure to 5 million biomedical papers, we find a systematic citation disadvantage of clinical oriented papers; they tend to garner far fewer citations and are less likely to be hit works than papers oriented towards basic science. At the same time, clinical research has a higher variance in its citation. We also find that the citation difference between basic and clinical research decreases, yet still persists, if longer citation-window is used. Given the increasing adoption of short-term, citation-based bibliometric indicators in funding decisions, the under-cited effect of clinical research may provide disincentives for bio-researchers to venture into the translation of basic scientific discoveries into clinical applications, thus providing explanations of reasons behind the existence of the gap between basic and clinical research that is commented as “valley of death” and the commentary of “extinction” risk of translational researchers. Our work may provide insights to policy-makers on how to evaluate different types of biomedical research.     

时序视域下图情领域重要国家的学术贡献研究

[目的/意义]揭示2002年以来图情领域重要国家在学术论文产出的总量、引证、研究主题、学术合作等维度上的学术贡献与特征演化,为图情领域的学科建设及相关研究者追踪研究主题与前沿提供参考。[方法/过程]以近15年WoS数据库收录的图情领域86种核心期刊为研究对象,应用文献计量学、科学知识图谱、计量模型等研究方法,运用TDA、Ucinet、VOSviewer等数据分析工具,从学术论文的总量特征、引证特征、主题特征三个角度对图情领域重要国家的学术贡献进行深入分析,并构建计量模型考察空间距离及学术能力对重要国家之间学术合作的影响机制。[结果/结论]美国在图情领域的科研实力强劲,中国的学术贡献增速最快,荷兰、芬兰、比利时三国则在归一化被引比率指标上表现突出;研究主题主要集中在图书馆、信息检索、互联网、知识管理、社交网络、计量学、用户研究、研究方法8个主题范畴,英国、西班牙、中国、美国的研究主题最为广泛;空间距离与学术能力都对国家间的学术合作具有显著影响。     

高被引论文与“睡美人”论文引用曲线及影响因素研究

[目的/意义]通过对潜在“睡美人”论文的引用分布分析,提炼其特征,以期为“睡美人”论文的预判研究提供思路。[方法/过程]采用引用曲线这一更为直观的反映论文引用分布的方法,以“天文学和天体物理”这一领域为例,构建其10的高被引论文、“睡美人”论文的10-20年被引用数据并进行引文分布的对比分析。[结果/结论]研究发现两类文献的引用曲线模式及特点——高被引论文的持续增长型、显峰型、双峰型、振荡型,“睡美人”论文的持续增长型、显峰型、双峰型、振荡型、稳定型等被引用曲线模式;针对施引文献、研究主题演化方向探讨了各模式引用曲线形成的相关因素,发现两类文献达到引用高峰的时间存在差异。     

Does the public discuss other topics on climate change than researchers? A comparison of explorative networks based on author keywords and hashtags

Twitter accounts have already been used in many scientometric studies, but the meaningfulness of the data for societal impact measurements in research evaluation has been questioned. Earlier research focused on social media counts and neglected the interactive nature of the data. We explore a new network approach based on Twitter data in which we compare author keywords to hashtags as indicators of topics. We analyze the topics of tweeted publications and compare them with the topics of all publications (tweeted and not tweeted). Our exploratory study is based on a comprehensive publication set of climate change research. We are interested in whether Twitter data are able to reveal topics of public discussions which can be separated from research-focused topics. We find that the most tweeted topics regarding climate change research focus on the consequences of climate change for humans. Twitter users are interested in climate change publications which forecast effects of a changing climate on the environment and to adaptation, mitigation and management issues rather than in the methodology of climate-change research and causes of climate change. Our results indicate that publications using scientific jargon are less likely to be tweeted than publications using more general keywords. Twitter networks seem to be able to visualize public discussions about specific topics.     

基于F1000与WoS的同行评议与文献计量相关性研究

为比较同行评议与文献计量方法在科学评价中的有效性及相关性,选取F1000以及Web of Science数据库,采用SPSS16.0软件,将近2000篇论文的F1000因子与Web of Science数据库中指标进行相关性比较。结果显示,F1000因子与统计区间内的被引频次呈显著正相关,同时一些F1000因子很高的论文并没有高频被引,反之亦然。结论指出:从统计学的视角,文献计量指标与同行评议结果具有正向相关性,但是无论是同行评议还是文献计量,单独作为科学评价标准都会有失偏颇,以引文分析为代表的定量指标与同行评议方法的结合将是未来科学评价的主流。     

“十三五”图情档学科进展:中外期刊论文研究热点与趋势解析

[目的/意义]图书馆学、情报学及档案学(简称“图情档学”)是当代培养新型信息知识人才的重要学科,本文揭示“十三五”期间图情档学科中外期刊论文的研究热点及趋势,以期为我国图情档学科的未来研究提供些许思路,有助于推动图情档学科的发展,提升学术话语权。[方法/过程]选取国内的CNKI和国外的Web of Science数据库为数据来源,以2016-2020年图情档的核心期刊论文为研究样本,运用Citespace、VOSviewer、Gephi等可视化软件,从发文量与被引量、关键词共现及时区分布揭示中外文期刊的研究热点以及发展趋势。[结果/结论]研究发现,“十三五”期间,中文期刊论文的发文量呈逐年略递减趋势,外文期刊论文发文量呈现上升趋势,学科领域的研究逐渐面向国际化;中外期刊论文研究热点呈现研究主题多元化、“图书馆”相关主题研究逐步深化、论文主题紧跟本学科的研究热点与动向,但各有侧重等特点;整体呈现出对传统研究领域的坚守与深化,与其他学科的交叉融合发展,向社会公众领域的拓展的研究趋势。     

Who shares health and medical scholarly articles on Facebook?

Over a million journal articles had been shared on public Facebook pages by 2017, but little is known about who is sharing (posting links to) these papers and whether mention counts could be an impact indicator. This study classified users who had posted about 749 links on Facebook before October 2017 mentioning 500 medical and health‐related research articles, obtained using altmetric.com data. Most accounts (68%) belonged to groups, including online communities, journals, academic organizations, and societies. Of individual profiles, academics accounted for only 4%, but the largest group were health care professionals (16%). More than half (58%) of all Facebook accounts examined were not academic. The non‐academic dominance suggests that public Facebook posts linking to health‐related articles are mostly used to facilitate scientific knowledge flow between non‐academic professionals and the public. Therefore, Facebook mention counts may be a combined academic and non‐academic attention indicator in the health and medical domains.     

影响因子分数平均值:一个评价学术论文质量的新指标

介绍影响因子分数平均值的概念,并用2004年SCI收录的我国大学、科研机构和医疗机构的科技论文数据进行分析讨论.认为影响因子分数平均值比较适于评价不同学科间的研究工作,是一个新的评价学术论文质量的较好的量化指标.