The validation of (advanced) bibliometric indicators through peer assessments: A comparative study using data from InCites and F1000

The data of F1000 and InCites provide us with the unique opportunity to investigate the relationship between peers’ ratings and bibliometric metrics on a broad and comprehensive data set with high-quality ratings. F1000 is a post-publication peer review system of the biomedical literature. The comparison of metrics with peer evaluation has been widely acknowledged as a way of validating metrics. Based on the seven indicators offered by InCites, we analyzed the validity of raw citation counts (Times Cited, 2nd Generation Citations, and 2nd Generation Citations per Citing Document), normalized indicators (Journal Actual/Expected Citations, Category Actual/Expected Citations, and Percentile in Subject Area), and a journal based indicator (Journal Impact Factor). The data set consists of 125 papers published in 2008 and belonging to the subject category cell biology or immunology. As the results show, Percentile in Subject Area achieves the highest correlation with F1000 ratings; we can assert that for further three other indicators (Times Cited, 2nd Generation Citations, and Category Actual/Expected Citations) the “true” correlation with the ratings reaches at least a medium effect size.     

Researchers’ Mobility,Productivity and Impact: Case of Top Producing Authors in Seven Disciplines

The purpose of this study was to examine whether scientific mobility, either between countries or between affiliations has an effect on researchers’ productivity and impact. In order to investigate this issue, we examined the relationships between the number of institutional affiliations and countries of the top 100 authors in seven disciplines. The selected authors’ profiles contained the number of affiliations and countries each author is assigned. We studied the number of affiliations and countries and compared them to three bibliometric indicators: the number of publications in international, peer-reviewed journals, h-index and Field Weighted Citations Impact. Our findings show that although there are differences in the relationship between mobility, productivity and impact between disciplines, mobility between at least two affiliations has an overall positive effect on both output and impact while mobility between countries does not. Therefore, in most disciplines positive impact and productivity effects are tracked in affiliation mobility within a single country.     

外文学术电子图书学科影响力比较研究与思考

[目的/意义] 比较分析不同学科的外文学术电子图书影响力差异,丰富电子图书评价方法,为完善电子图书分类分学科的科学评价体系提供有益参考。[方法/过程] 采用Bookmetrix,以经管类、教育类的学术电子图书为研究对象,对其传统引文指标与Altmetrics指标（Mendeley读者数、关注量、下载量）、书评量的相关性与一致性定量分析,比较两学科外文电子图书各指标之间的差异并进行非参数检验。[结果/结论] 研究发现：被引量、读者数、下载量等具有较高的指标覆盖率;经K-S Z独立双样本检验,经管类和教育类电子图书的被引量、下载量存在显著差异,关注量、读者数、书评量无显著差异（p=0.05）;指标相关性具有学科差异性,被引量与Mendeley读者数的相关性,经管类图书高于教育类图书;被引量测度的是学术电子图书的学术影响力,使用数据（下载量等）与补充计量学数据较多反映图书的社会影响力。评价中文学术电子图书应将多源异构数据处理转化,构建多指标综合评价体系,将定性与定量方法相融合,使评价更全面、科学。     

Author-level metrics in the new academic profile platforms: The online behaviour of the Bibliometrics community

The new web-based academic communication platforms do not only enable researchers to better advertise their academic outputs, making them more visible than ever before, but they also provide a wide supply of metrics to help authors better understand the impact their work is making. This study has three objectives: a) to analyse the uptake of some of the most popular platforms (Google Scholar Citations, ResearcherID, ResearchGate, Mendeley and Twitter) by a specific scientific community (bibliometrics, scientometrics, informetrics, webometrics, and altmetrics); b) to compare the metrics available from each platform; and c) to determine the meaning of all these new metrics. To do this, the data available in these platforms about a sample of 811 authors (researchers in bibliometrics for whom a public profile Google Scholar Citations was found) were extracted. A total of 31 metrics were analysed. The results show that a high number of the analysed researchers only had a profile in Google Scholar Citations (159), or only in Google Scholar Citations and ResearchGate (142). Lastly, we find two kinds of metrics of online impact. First, metrics related to connectivity (followers), and second, all metrics associated to academic impact. This second group can further be divided into usage metrics (reads, views), and citation metrics. The results suggest that Google Scholar Citations is the source that provides more comprehensive citation-related data, whereas Twitter stands out in connectivity-related metrics.     

Aggregating different paper quality measures with a generalized h-index

The process of assessing individual authors should rely upon a proper aggregation of reliable and valid papers’ quality metrics. Citations are merely one possible way to measure appreciation of publications. In this study we propose some new, SJR- and SNIP-based indicators, which not only take into account the broadly conceived popularity of a paper (manifested by the number of citations), but also other factors like its potential, or the quality of papers that cite a given publication. We explore the relation and correlation between different metrics and study how they affect the values of a real-valued generalized h-index calculated for 11 prominent scientometricians. We note that the h-index is a very unstable impact function, highly sensitive for applying input elements’ scaling. Our analysis is not only of theoretical significance: data scaling is often performed to normalize citations across disciplines. Uncontrolled application of this operation may lead to unfair and biased (toward some groups) decisions. This puts the validity of authors assessment and ranking using the h-index into question. Obviously, a good impact function to be used in practice should not be as much sensitive to changing input data as the analyzed one.     

A systematic empirical comparison of different approaches for normalizing citation impact indicators

We address the question how citation-based bibliometric indicators can best be normalized to ensure fair comparisons between publications from different scientific fields and different years. In a systematic large-scale empirical analysis, we compare a traditional normalization approach based on a field classification system with three source normalization approaches. We pay special attention to the selection of the publications included in the analysis. Publications in national scientific journals, popular scientific magazines, and trade magazines are not included. Unlike earlier studies, we use algorithmically constructed classification systems to evaluate the different normalization approaches. Our analysis shows that a source normalization approach based on the recently introduced idea of fractional citation counting does not perform well. Two other source normalization approaches generally outperform the classification-system-based normalization approach that we study. Our analysis therefore offers considerable support for the use of source-normalized bibliometric indicators.     

Proposal of composed altmetric indicators based on prevalence and impact dimensions

The aim of this study is to introduce two groups of impact indicators, Weighted Altmetric Impact (WAI) and Inverse Altmetric Impact (IAI). WAI is based in weights from the contributions of each metric to different components or impact dimensions (Principal Component Analysis). IAI is calculated according to the inverse prevalence of each metric in different impact dimensions (TF/IDF). These indicators were tested against 29,500 articles, using metrics from Altmetric.com, PlumX and CED. Altmetric Attention Score (AAScore) was also obtained to compare the resulting scores. Several statistical analyses were applied to value the advantages and limitations of these indicators. Frequency distributions showed that each group of metrics (Scientific Impact, Media Impact and Usage Impact) follows power law trends although with particular patterns. Correlation matrices have depicted associations between metrics and indicators. Multidimensional scaling (MDS) has plotted these interactions to visualize distances between indicators and metrics in each dimension. The 2018 Altmetric Top 100 was used to distinguish differences between rankings from AAScore and the proposed indicators. The paper concludes that the theoretical assumptions of dimensions and prevalence are suitable criteria to design transparent and reproducible impact indicators.     

Career Levels and Their Effect on Scholarly Output and Impact of Women Scientists

In this article we examined the scholarly output and impact of 81 women scientist at the Graduate School of Biomedical Sciences at Mount Sinai. The group was divided into three career level sub-groups based on the first year of publication of each scientist. We examined the number of publications, citations, readership and social media attention per each group. Our findings show that although senior faculty have more publications and overall citations, junior faculty receive more citations and more readership per paper. We also found that different career level faculty members receive different social media mentions. Mid-career faculty see more tweets that mention their research while senior faculty get more likes, shares and clicks via Facebook.     

The use of percentiles and percentile rank classes in the analysis of bibliometric data: Opportunities and limits

Percentiles have been established in bibliometrics as an important alternative to mean-based indicators for obtaining a normalized citation impact of publications. Percentiles have a number of advantages over standard bibliometric indicators used frequently: for example, their calculation is not based on the arithmetic mean which should not be used for skewed bibliometric data. This study describes the opportunities and limits and the advantages and disadvantages of using percentiles in bibliometrics. We also address problems in the calculation of percentiles and percentile rank classes for which there is not (yet) a satisfactory solution. It will be hard to compare the results of different percentile-based studies with each other unless it is clear that the studies were done with the same choices for percentile calculation and rank assignment.     

Do altmetrics assess societal impact in a comparable way to case studies? An empirical test of the convergent validity of altmetrics based on data from the UK research excellence framework (REF)

Altmetrics have been proposed as a way to assess the societal impact of research. Although altmetrics are already in use as impact or attention metrics in different contexts, it is still not clear whether they really capture or reflect societal impact. This study is based on altmetrics, citation counts, research output and case study data from the UK Research Excellence Framework (REF), and peers’ REF assessments of research output and societal impact. We investigated the convergent validity of altmetrics by using two REF datasets: publications submitted as research output (PRO) to the REF and publications referenced in case studies (PCS). Case studies, which are intended to demonstrate societal impact, should cite the most relevant research papers. We used the MHq’ indicator for assessing impact – an indicator which has been introduced for count data with many zeros. The results of the first part of the analysis show that news media as well as mentions on Facebook, in blogs, in Wikipedia, and in policy-related documents have higher MHq’ values for PCS than for PRO. Thus, the altmetric indicators seem to have convergent validity for these data. In the second part of the analysis, altmetrics have been correlated with REF reviewers’ average scores on PCS. The negative or close to zero correlations question the convergent validity of altmetrics in that context. We suggest that they may capture a different aspect of societal impact (which can be called unknown attention) to that seen by reviewers (who are interested in the causal link between research and action in society).     

Russian scientific publications 2005–2009 in the science citation index,scopus, and chemical abstracts databases

This paper presents the results of a bibliometric study of the scientific publications that are affiliated with Russia and included in multidisciplinary databases, such as Science Citation Index and Scopus, as well as in the specialized Chemical Abstracts database, in the period from 2005 to 2009. Major bibliometric indicators, including citation indexes, are explored. It is shown that the Scopus database contains most of the Russian publications in the period from 2005 to 2009. While a gradual decrease in the number of peer-reviewed Russian-language journals constitutes a general trend for all three databases, the Chemical Abstracts database leads in terms of the coverage of these journals. It is found that despite the large number of translated versions of journals, Russian publications are still significantly late in their coverage by foreign databases, which has a significant impact on their citation levels.     

新媒体时代的博客传播与图书出版研究

作为一种新兴的网络传播方式,博客对传统图书出版产生了重要影响.本文探讨如何保持博客传播的鲜活性和传统出版的精英性,达到双方共赢的局面.     

基于文献计量学的我国著名大学图书馆学术水平比较

以CNKI全文数据库、引文数据库和CSSCI索引数据库为来源数据库,选择"985工程"建设的39所国内著名大学图书馆为样本,分析比较这些大学图书馆2000-2009年间发表学术论文的一系列文献计量学指标。这些文献计量学指标包括:论文总数、总被引频次、篇均被引次数、h指数等。在此基础上,对国内大学图书馆提高学术水平提出了若干针对性建议。     

我国林业科学类期刊定量分析

以《中国学术期刊综合引证报告》(CAJCCR)为依据,以我国21种林业科学类核心期刊为研究对象,用文献计量学方法对学报学术影响力的各项计量指标进行对比分析与研究。从而了解21种林业科学类核心期刊载文所反映的学科地位、学术水平及期刊质量。追踪21种林业科学类核心期刊的总被引频次、影响因子和h指数等指标,客观、综合、全面评价21种林业科学类核心期刊的整体情况。     

The h-index: Advantages,limitations and its relation with other bibliometric indicators at the micro level

The relationship of the h-index with other bibliometric indicators at the micro level is analysed for Spanish CSIC scientists in Natural Resources, using publications downloaded from the Web of Science (1994–2004). Different activity and impact indicators were obtained to describe the research performance of scientists in different dimensions, being the h-index located through factor analysis in a quantitative dimension highly correlated with the absolute number of publications and citations. The need to include the remaining dimensions in the analysis of research performance of scientists and the risks of relying only on the h-index are stressed. The hypothesis that the achievement of some highly visible but intermediate-productive authors might be underestimated when compared with other scientists by means of the h-index is tested.     

Regularity in the research output of individual scientists: An empirical analysis by recent bibliometric tools

This paper proposes an empirical analysis of several scientists based on their time regularity, defined as the ability of generating an active and stable research output over time, in terms of both quantity/publications and impact/citations. In particular, we empirically analyse three recent bibliometric tools to perform qualitative/quantitative evaluations under the new perspective of regularity. These tools are respectively (1) the PY/CY diagram, (2) the publication/citation Ferrers diagram and triad indicators, and (3) a year-by-year comparison of the scientists’ output (Borda's ranking). Results of the regularity analysis are then compared with those obtained under the classical perspective of overall production.The proposed evaluation tools can be applied to competitive examinations for research position/promotion, as complementary instruments to the commonly adopted bibliometric techniques.     

Open Access Advantages as a Function of the Discipline: Mixed-methods Study

PurposeThis mixed-methods study integrates bibliometric and altmetric investigation with a qualitative method in order to assess the prevalence and societal-impact of Open-Access (OA) publications, and to reveal the considerations behind researchers' decision to publish articles in closed and open-access.Design/methodology/approachThe bibliometric-altmetric study analyzed 584 OA and closed publications published between 2014 and 2019 by 40 Israeli researchers: 20 from STEM (Science, Technology, Engineering, Math) and 20 from SSH (Social Sciences and Humanities) discipline. We used a multistage cluster sampling method to select a representative sample for the STEM disciplines group (engineering, computer science, biology, mathematics, and physics), and for the SSH disciplines group (sociology, economics, psychology, political science, and history). Required data were extracted from Scopus and Unpaywall databases, and the PlumX-platform. Among the 40 researchers who were selected for the bibliometric-altmetric study, 20 researchers agreed to be interviewed for this study.FindingsComparing bibliometrics and altmetrics for the general publications did not reveal any significant differences between OA and closed publications. These were found only when comparing OA and closed publications across disciplines. STEM-researchers published 59 % of their publications in OA, compared to just 29 % among those in SSH, and they received significantly more bibliometric and altmetric citations from SSH OA publications and from their own closed-access publications. The altmetrics findings indicate that researchers are well acquainted and active in social media. However, according to the interviewees, there is no academic contribution for sharing research findings on social-media; it is viewed as a “public-service”. Researchers' primary consideration for publishing in closed or OA was the journal impact-factor.Research limitations/implicationsOur findings contribute to the increasing body of research that addresses OA citations and societal-impact advantages. The findings suggest the need to adopt an OA-policy after a thorough assessment of the consequences for SSH disciplines.     

学科评价之文献计量指标分析

立足学科评价,重点在于对文献计量评价指标的研究。筛选国内外当前用于学科评价的文献计量指标,给出了一套科学常用的学科评价之文献计量指标参考集。设计指标分析过程和指标集成结构图,对指标参考集中指标进行功能分析和识别标志分析,试图发掘文献计量评价指标的内在特征和规律。     

文献计量指标的GIS空间展现

首先对文献计量指标的信息源与统计对象进行分析,包括投入、产出和效率三大类型以及不同数量研究人员的研究单元。然后对可GIS空间展现的文献计量指标进行综述,主要汇总一些常规的指标、h指标及其衍生指标、基于引用的知识流指标等。在此基础上,从空间展现模式和平台两个方面分析文献计量指标GIS空间展现的可行性。在展现模式方面,分析文献计量指标空间数据制备的过程,通过点、线、面3种方式进行展示,并进一步拓展空间分析的功能。在展示平台方面,从商用软件、开源软件和WebGIS 3个方面讨论平台的适用性。