Research assessment by percentile-based double rank analysis

In the double rank analysis of research publications, the local rank position of a country or institution publication is expressed as a function of the world rank position. Excluding some highly or lowly cited publications, the double rank plot fits well with a power law, which can be explained because citations for local and world publications follow lognormal distributions. We report here that the distribution of the number of country or institution publications in world percentiles is a double rank distribution that can be fitted to a power law. Only the data points in high percentiles deviate from it when the local and world μ parameters of the lognormal distributions are very different. The likelihood of publishing very highly cited papers can be calculated from the power law that can be fitted either to the upper tail of the citation distribution or to the percentile-based double rank distribution. The great advantage of the latter method is that it has universal application, because it is based on all publications and not just on highly cited publications. Furthermore, this method extends the application of the well-established percentile approach to very low percentiles where breakthroughs are reported but paper counts cannot be performed.     

Distributions for cited articles from individual subjects and years

The citations to a set of academic articles are typically unevenly shared, with many articles attracting few citations and few attracting many. It is important to know more precisely how citations are distributed in order to help statistical analyses of citations, especially for sets of articles from a single discipline and a small range of years, as normally used for research evaluation. This article fits discrete versions of the power law, the lognormal distribution and the hooked power law to 20 different Scopus categories, using citations to articles published in 2004 and ignoring uncited articles. The results show that, despite its popularity, the power law is not a suitable model for collections of articles from a single subject and year, even for the purpose of estimating the slope of the tail of the citation data. Both the hooked power law and the lognormal distributions fit best for some subjects but neither is a universal optimal choice and parameter estimates for both seem to be unreliable. Hence only the hooked power law and discrete lognormal distributions should be considered for subject-and-year-based citation analysis in future and parameter estimates should always be interpreted cautiously.     

Ranking national research systems by citation indicators. A comparative analysis using whole and fractionalised counting methods

This paper presents an empirical analysis of two different methodologies for calculating national citation indicators: whole counts and fractionalised counts. The aim of our study is to investigate the effect on relative citation indicators when citations to documents are fractionalised among the authoring countries. We have performed two analyses: a time series analysis of one country and a cross-sectional analysis of 23 countries. The results show that all countries’ relative citation indicators are lower when fractionalised counting is used. Further, the difference between whole and fractionalised counts is generally greatest for the countries with the highest proportion of internationally co-authored articles. In our view there are strong arguments in favour of using fractionalised counts to calculate relative citation indexes at the national level, rather than using whole counts, which is the most common practice today.     

The lognormal distribution explains the remarkable pattern documented by characteristic scores and scales in scientometrics

Characteristic scores and scales (CSS) – a well-established scientometric tool for the study of citation counts – have been used to document a striking phenomenon that characterizes citation distributions at high levels of aggregation: irrespective of scientific field and citation window empirical studies find a persistent pattern whereby about 70% of scientific papers belong to the class of poorly cited papers, about 21% belong to the class of fairly cited papers, 6% to that of remarkably cited papers and 3% to the class of outstandingly cited papers. This article aims to advance the understanding of this remarkable result by examining it in the context of the lognormal distribution, a popular model used to describe citation counts across scientific fields. The article shows that the application of the CSS method to lognormal distributions provides a very good fit to the 70–21–6–3% empirical pattern provided these distributions are characterized by a standard deviation parameter in the range of about 0.8–1.3. The CSS pattern is essentially explainable as an epiphenomenon of the lognormal functional form and, more generally, as a consequence of the skewness of science which is manifest in heavy-tailed citation distributions.     

Rooted citation graphs density metrics for research papers influence evaluation

Traditionally, citation count has served as the main evaluation measure for a paper's importance and influence. In turn, many evaluations of authors, institutions and journals are based on aggregations upon papers (e.g. h-index). In this work, we explore measures defined on the citation graph that offer a more intuitive insight into the impact of a paper than the superficial count of citations. Our main argument is focused on the identification of influence as an expression of the citation density in the subgraph of citations built for each paper. We propose two measures that capitalize on the notion of density providing researchers alternative evaluations of their work. While the general idea of impact for a paper can be viewed as how many researchers have shown interest to a piece of work, the proposed measures are based on the hypothesis that a piece of work may have influenced some papers even if they do not contain references to that piece of work. The proposed measures are also extended to researchers and journals.     

Journal topic citation potential and between-field comparisons: The topic normalized impact factor

The journal impact factor is not comparable among fields of science and social science because of systematic differences in publication and citation behavior across disciplines. In this work, a source normalization of the journal impact factor is proposed. We use the aggregate impact factor of the citing journals as a measure of the citation potential in the journal topic, and we employ this citation potential in the normalization of the journal impact factor to make it comparable between scientific fields. An empirical application comparing some impact indicators with our topic normalized impact factor in a set of 224 journals from four different fields shows that our normalization, using the citation potential in the journal topic, reduces the between-group variance with respect to the within-group variance in a higher proportion than the rest of indicators analyzed. The effect of journal self-citations over the normalization process is also studied.     

Predicting publication long-term impact through a combination of early citations and journal impact factor

The ability to predict the long-term impact of a scientific article soon after its publication is of great value towards accurate assessment of research performance. In this work we test the hypothesis that good predictions of long-term citation counts can be obtained through a combination of a publication's early citations and the impact factor of the hosting journal. The test is performed on a corpus of 123,128 WoS publications authored by Italian scientists, using linear regression models. The average accuracy of the prediction is good for citation time windows above two years, decreases for lowly-cited publications, and varies across disciplines. As expected, the role of the impact factor in the combination becomes negligible after only two years from publication.     

Combining mapping and citation network analysis for a better understanding of the scientific development: The case of the absorptive capacity field

The general aim of this paper is to show the results of a study in which we combined bibliometric mapping and citation network analysis to investigate the process of creation and transfer of knowledge through scientific publications. The novelty of this approach is the combination of both methods. In this case we analyzed the citations to a very influential paper published in 1990 that contains, for the first time, the term Absorptive Capacity. A bibliometric map identified the terms and the theories associated with the term while two techniques from the citation network analysis recognized the main papers during 15 years. As a result we identified the articles that influenced the research for some time and linked them into a research tradition that can be considered the backbone of the “Absorptive Capacity Field”.     

On the Shapley value and its application to the Italian VQR research assessment exercise

Research assessment exercises have now become common evaluation tools in a number of countries. These exercises have the goal of guiding merit-based public funds allocation, stimulating improvement of research productivity through competition and assessing the impact of adopted research support policies. One case in point is Italy's most recent research assessment effort, VQR 2011–2014 (Research Quality Evaluation), which, in addition to research institutions, also evaluated university departments, and individuals in some cases (i.e., recently hired research staff and members of PhD committees). However, the way an institution's score was divided, according to VQR rules, between its constituent departments or its staff members does not enjoy many desirable properties well known from coalitional game theory (e.g., budget balance, fairness, marginality). We propose, instead, an alternative score division rule that is based on the notion of Shapley value, a well known solution concept in coalitional game theory, which enjoys the desirable properties mentioned above. For a significant test case (namely, Sapienza University of Rome, the largest university in Italy), we present a detailed comparison of the scores obtained, for substructures and individuals, by applying the official VQR rules, with those resulting from Shapley value computations. We show that there are significant differences in the resulting scores, making room for improvements in the allocation rules used in research assessment exercises.     

引文分析领域研究热点前沿与高频作者的二维时空分析

以国内引文分析领域的3401篇论文作为样本,统计其30年间的年代分布状况;分析该领域的高频作者;借助CiteSpaceⅡ突变检测算法检测到引文分析领域的研究热点及前沿术语。运用CiteSpaceⅡ提供的复杂混合网络的可视化方法,将研究热点、研究前沿与作者建立空间上和时间上的对应关系,从空间和时间两个维度揭示我国引文分析领域研究热点的作者分布,研究前沿的作者分布以及引领引文分析研究前沿的关键作者。     

The distribution of references across texts: Some implications for citation analysis

In citation network analysis, complex behavior is reduced to a simple edge, namely, node A cites node B. The implicit assumption is that A is giving credit to, or acknowledging, B. It is also the case that the contributions of all citations are treated equally, even though some citations appear multiply in a text and others appear only once. In this study, we apply text-mining algorithms to a relatively large dataset (866 information science articles containing 32,496 bibliographic references) to demonstrate the differential contributions made by references. We (1) look at the placement of citations across the different sections of a journal article, and (2) identify highly cited works using two different counting methods (CountOne and CountX). We find that (1) the most highly cited works appear in the Introduction and Literature Review sections of citing papers, and (2) the citation rankings produced by CountOne and CountX differ. That is to say, counting the number of times a bibliographic reference is cited in a paper rather than treating all references the same no matter how many times they are invoked in the citing article reveals the differential contributions made by the cited works to the citing paper.     

Regularity in the research output of individual scientists: An empirical analysis by recent bibliometric tools

This paper proposes an empirical analysis of several scientists based on their time regularity, defined as the ability of generating an active and stable research output over time, in terms of both quantity/publications and impact/citations. In particular, we empirically analyse three recent bibliometric tools to perform qualitative/quantitative evaluations under the new perspective of regularity. These tools are respectively (1) the PY/CY diagram, (2) the publication/citation Ferrers diagram and triad indicators, and (3) a year-by-year comparison of the scientists’ output (Borda's ranking). Results of the regularity analysis are then compared with those obtained under the classical perspective of overall production.The proposed evaluation tools can be applied to competitive examinations for research position/promotion, as complementary instruments to the commonly adopted bibliometric techniques.     

Distributions of citations of papers of individual authors publishing in different scientific disciplines: Application of Langmuir-type function

The distribution of cumulative citations L and contributed citations L_f to individual multiauthored papers published by selected authors working in different scientific disciplines is analyzed and discussed using Langmuir-type function: y_n = y₀[1 − αKn/(1 + Kn)], where y_n denotes the total number of normalized cumulative citations l_n* and normalized contributed citations l_nf* received by individual papers of rank n, y₀ is the maximum value of y_n when n = 0, α ≥ 1 is an effectiveness parameter, and K is the Langmuir constant related to the dimensionless differential energy Q = ln(KN_c), with N_c as the number of papers receiving citations. Relationships between the values of the Langmuir constant K of the distribution function, the number N_c of papers of an individual author receiving citations and the effectiveness parameter α of this function, obtained from analysis of the data of rank-size distributions of the authors, are investigated. It was found that: (1) the quantity KN_c obtained from the real citation distribution of papers of various authors working in different disciplines is inversely proportional to (α − 1) with a proportional constant (KN_c)₀ < 1, (2) the relation KN_c = (KN_c)₀/(α − 1) also holds for the citation distribution of journals published in countries of two different groups, investigated earlier (Sangwal, K. (2013). Journal of Informetrics, 7, 487–504), and (3) deviations of the real citation distribution from curves predicted by the Langmuir-type function are associated with changing activity of sources of generation of items (citations).     

Information and misinformation in bibliometric time-trend analysis

A diachronous time-series of bibliometric data (using all data available) suggests rising normalised citation impact (nci) for Germany and other G7 nations, while China suffers a decline in later years of any series. This is shown to be a consequence of the time-series, which has led to an erroneous interpretation of real trajectories. A synchronous series (using fixed time windows) based on the final year suggests a lower trajectory while a diachronous series tracking the fate of a single publication year reveals that nci progressively falls for Germany and the USA whereas it climbs for China. This has implications for research policy and for the interpretation of changes in the competitive research environment in the presence of dynamic growth. By analogy, this may extend to institutional as well as national comparisons. It has implications for analytical methodology, supporting prior suggestions that recent papers should be omitted from citation analysis.     

Regression for citation data: An evaluation of different methods

Citations are increasingly used for research evaluations. It is therefore important to identify factors affecting citation scores that are unrelated to scholarly quality or usefulness so that these can be taken into account. Regression is the most powerful statistical technique to identify these factors and hence it is important to identify the best regression strategy for citation data. Citation counts tend to follow a discrete lognormal distribution and, in the absence of alternatives, have been investigated with negative binomial regression. Using simulated discrete lognormal data (continuous lognormal data rounded to the nearest integer) this article shows that a better strategy is to add one to the citations, take their log and then use the general linear (ordinary least squares) model for regression (e.g., multiple linear regression, ANOVA), or to use the generalised linear model without the log. Reasonable results can also be obtained if all the zero citations are discarded, the log is taken of the remaining citation counts and then the general linear model is used, or if the generalised linear model is used with the continuous lognormal distribution. Similar approaches are recommended for altmetric data, if it proves to be lognormally distributed.     

基于WebofScience的云计算研究的文献计量分析

论文运用webofScience精炼检索结果功能和文献计量工具Endnote,对数据库中收录的云计算研究的相关文献的发表时间、文献类型、发表地区、期刊来源、著者、主题以及引证情况进行统计分析,试图全面揭示国内外云计算研究的概貌,同时预测了云计算的未来发展方向。     

Content-based author co-citation analysis

Author co-citation analysis (ACA) has long been used as an effective method for identifying the intellectual structure of a research domain, but it relies on simple co-citation counting, which does not take the citation content into consideration. The present study proposes a new method for measuring the similarity between co-cited authors by considering author's citation content. We collected the full-text journal articles in the information science domain and extracted the citing sentences to calculate their similarity distances. We compared our method with traditional ACA and found out that our approach, while displaying a similar intellectual structure for the information science domain as the other baseline methods, also provides more details about the sub-disciplines in the domain than with traditional ACA.     

Is the new citation-rank approach P100′ in bibliometrics really new?

The percentile-based rating scale P100 describes the citation impact in terms of the distribution of unique citation values. This approach has recently been refined by considering also the frequency of papers with the same citation counts. Here I compare the resulting P100′ with P100 for an empirical dataset and a simple fictitious model dataset. It is shown that P100′ is not much different from standard percentile-based ratings in terms of citation frequencies. A new indicator P100″ is introduced.