A meta-evaluation of scientific research proposals: Different ways of comparing rejected to awarded applications

Combining different data sets with information on grant and fellowship applications submitted to two renowned funding agencies, we are able to compare their funding decisions (award and rejection) with scientometric performance indicators across two fields of science (life sciences and social sciences). The data sets involve 671 applications in social sciences and 668 applications in life sciences. In both fields, awarded applicants perform on average better than all rejected applicants. If only the most preeminent rejected applicants are considered in both fields, they score better than the awardees on citation impact. With regard to productivity we find differences between the fields. While the awardees in life sciences outperform on average the most preeminent rejected applicants, the situation is reversed in social sciences.     

Gatekeepers of science—Effects of external reviewers’ attributes on the assessments of fellowship applications

AimThe scientific norm of universalism prescribes that external reviewers recommend the allocation of awards to young scientists solely on the basis of their scientific achievement. Since the evaluation of grants utilizes scientists with different personal attributes, it is natural to ask whether the norm of universalism reflects the actual evaluation practice.Subjects and methodsWe investigated the influence of three attributes of external reviewers on their ratings in the selection procedure followed by the Boehringer Ingelheim Fonds (B.I.F.) for awarding long-term fellowships to doctoral and post-doctoral researchers in biomedicine: (i) number of applications assessed in the past for the B.I.F. (reviewers’ evaluation experience), (ii) the reviewers’ country of residence and (iii) the reviewers’ gender. To analyze the reviewers’ ratings (1: award; 2: maybe award; 3: no award) in an ordinal regression model (ORM) the following were considered in addition to the three attributes: (i) the scientific achievements of the fellowship applicants, (ii) interaction effects between reviewers’ and applicants’ attributes and (iii) judgmental tendencies of reviewers.ResultsThe results of the model estimations show no significant effect of the reviewers’ attributes on the evaluation of B.I.F. fellowship applications. The ratings of the external reviewers are mainly determined by the applicants’ scientific achievement prior to application.ConclusionsThe results suggest that the external reviewers of the B.I.F. indeed achieved the foundation's goal of recommending applicants with higher scientific achievement for fellowships and of recommending those with lower scientific achievement for rejection.     

Monotonicity and the Hirsch index

The Hirsch index is a number that synthesizes a researcher's output. It is the maximum number h such that the researcher has h papers with at least h citations each. Woeginger [Woeginger, G. J. (2008a). An axiomatic characterization of the Hirsch-index. Mathematical Social Sciences, 56(2), 224–232; Woeginger, G. J. (2008b). A symmetry axiom for scientific impact indices. Journal of Informetrics, 2(3), 298–303] characterizes the Hirsch index when indices are assumed to be integer-valued. In this note, the Hirsch index is characterized, when indices are allowed to be real-valued, by adding to Woeginger's monotonicity two axioms in a way related to the concept of monotonicity.     

A new family of old Hirsch index variants

The Hirsch index h and the g index proposed by Egghe as well as the f index and the t index proposed by Tol are shown to be special cases of a family of Hirsch index variants, based on the generalized mean with exponent p. Inequalities between the different indices are derived from the generalized mean inequality. The graphical determination of the indices is shown for one example.     

An alternative axiomatization of the Hirsch index

We axiomatize the well-known Hirsch index (h-index), which evaluates researcher productivity and impact on a field, and formalize a new axiom called head-independence. Under head-independence, a decrease, to some extent, in the number of citations of “frequently cited papers” has no effect on the index. Together with symmetry and axiom D, head-independence uniquely characterizes the h-index on a certain domain of indices. Some relationships between our axiomatization and those in the literature are also investigated.     

On the h-index,the size of the Hirsch core and Jin's A-index

Hirsch's h-index seeks to give a single number that in some sense summarizes an author's research output and its impact. Essentially, the h-index seeks to identify the most productive core of an author's output in terms of most received citations. This most productive set we refer to as the Hirsch core, or h-core. Jin's A-index relates to the average impact, as measured by the average number of citations, of this “most productive” core. In this paper, we investigate both the total productivity of the Hirsch core – what we term the size of the h-core – and the A-index using a previously proposed stochastic model for the publication/citation process, emphasising the importance of the dynamic, or time-dependent, nature of these measures. We also look at the inter-relationships between these measures. Numerical investigations suggest that the A-index is a linear function of time and of the h-index, while the size of the Hirsch core has an approximate square-law relationship with time, and hence also with the A-index and the h-index.     

The Hirsch spectrum: A novel tool for analyzing scientific journals

This paper introduces the Hirsch spectrum (h-spectrum) for analyzing the academic reputation of a scientific journal. h-Spectrum is a novel tool based on the Hirsch (h) index. It is easy to construct: considering a specific journal in a specific interval of time, h-spectrum is defined as the distribution representing the h-indexes associated to the authors of the journal articles. This tool allows defining a reference profile of the typical author of a journal, compare different journals within the same scientific field, and provide a rough indication of prestige/reputation of a journal in the scientific community. h-Spectrum can be associated to every journal. Ten specific journals in the Quality Engineering/Quality Management field are analyzed so as to preliminarily investigate the h-spectrum characteristics.     

The h-index: A case of the tail wagging the dog?

From the way that it was initially defined (Hirsch, 2005), the h-index naturally encourages focus on the most highly cited publications of an author and this in turn has led to (predominantly) a rank-based approach to its investigation. However, Hirsch (2005) and Burrell (2007a) both adopted a frequency-based approach leading to general conjectures regarding the relationship between the h-index and the author's publication and citation rates as well as his/her career length. Here we apply the distributional results of Burrell, 2007a, Burrell, 2013b to three published data sets to show that a good estimate of the h-index can often be obtained knowing only the number of publications and the number of citations. (Exceptions can occur when an author has one or more “outliers” in the upper tail of the citation distribution.) In other words, maybe the main body of the distribution determines the h-index, not the wild wagging of the tail. Furthermore, the simple geometric distribution turns out to be the key.     

Modifying h-index by allocating credit of multi-authored papers whose author names rank based on contribution

In the present work we introduce a modification of the h-index for multi-authored papers with contribution based author name ranking. The modified h-index is denoted by h_mc-index. It employs the framework of the h_m-index, which in turn is a straightforward modification of the Hirsch index, proposed by Schreiber. To retain the merit of requiring no additional rearrangement of papers in the h_m-index and in order to overcome its shortage of benefiting secondary authors at the expense of primary authors, h_mc-index uses combined credit allocation (CCA) to replace fractionalized counting in the h_m-index. The h_m-index is a special form of h_mc-index and fits for papers with equally important authors or alphabetically ordered authorship. There is a possibility of an author of lower contribution to the whole scientific community obtaining a higher h_mc-index. Rational h_mc-index, denoted by h_mcr-index, can avoid it. A fictitious example as a model case and two empirical cases are analyzed. The correlations of the h_mcr-index with the h-index and its several variants considering multiple co-authorship are inspected with 30 researchers’ citation data. The results show that the h_mcr-index is more reasonable for authors with different contributions. A researcher playing more important roles in significant work will obtain higher h_mcr-index.     

Axiomatics for the Hirsch index and the Egghe index

The Hirsch index and the Egghe index are both numbers that synthesize a researcher's output. The h-index associated with researcher r is the maximum number h such that r has h papers with at least h citations each. The g-index is the maximum number g of papers by r such that the average number of citations of the g papers is at least g. Both indices are characterized in terms of four axioms. One identifies outputs deserving index at most one. A second one establishes a strong monotonicity condition. A third one requires the index to satisfy a property of subadditivity. The last one consists of a monotonicity condition, for the h-index, and an aggregate monotonicity condition, for the g-index.     

The influence of missing publications on the Hirsch index

We show that usually the influence on the Hirsch index of missing highly cited articles is much smaller than the number of missing articles. This statement is shown by a combinatorial argument. We further show, by using a continuous power law model, that the influence of missing articles is largest when the total number of publications is small, and non-existing when the number of publications is very large. The same conclusion can be drawn for missing citations. Hence, the h-index is resilient to missing articles and to missing citations.     

Total influence and mainstream measures for scientific researchers

We propose two new indices that are able to measure a scientific researcher's overall influence and the level of his/her works’ association with the mainstream research subjects within a scientific field. These two new measures – the total influence index and the mainstream index – differ from traditional performance measures such as the simple citation count and the h-index in that they take into account the indirect influence of an author's work. Indirect influence describes a scientific publication's impact upon subsequent works that do not reference it directly. The two measures capture indirect influence information from the knowledge emanating paths embedded in the citation network of a target scientific field. We take the Hirsch index, data envelopment analysis, and lithium iron phosphate battery technology field to examine the characteristics of these two measures. The results show that the total influence index favors earlier researchers and successfully highlights those researchers who have made crucial contributions to the target scientific field. The mainstream index, in addition to underlining total influence, also spotlights active researchers who enter into a scientific field in a later development stage. In summary, these two new measures are valuable complements to traditional scientific performance measures.     

Characteristic scores and scales based on h-type indices

Based on the rank-order citation distribution of e.g. a researcher, one can define certain points on this distribution, hereby summarizing the citation performance of this researcher. Previous work of Glänzel and Schubert defined these so-called “characteristic scores and scales” (CSS), based on average citation data of samples of this ranked publication–citation list.In this paper we will define another version of CSS, based on diverse h-type indices such as the h-index, the g-index, the Kosmulski's h⁽²⁾-index and the g-variant of it, the g⁽²⁾-index.Mathematical properties of these new CSS are proved in a Lotkaian framework. These CSS also provide an improvement of the single h-type indices in the sense that they give h-type index values for different parts of the ranked publication–citation list.     

A mathematical characterization of the Hirsch-index by means of minimal increments

The minimum configuration to have a h-index equal to h is h papers each having h citations, hence h² citations in total. To increase the h-index to h + 1 we minimally need (h + 1)² citations, an increment of I₁(h) = 2h + 1. The latter number increases with 2 per unit increase of h. This increment of the second order is denoted I₂(h) = 2.If we define I₁ and I₂ for a general Hirsch configuration (say n papers each having f(n) citations) we calculate I₁(f) and I₂(f) similarly as for the h-index. We characterize all functions f for which I₂(f) = 2 and show that this can be obtained for functions f(n) different from the h-index. We show that f(n) = n (i.e. the h-index) if and only if I₂(f) = 2, f(1) = 1 and f(2) = 2.We give a similar characterization for the threshold index (where n papers have a constant number C of citations). Here we deal with second order increments I₂(f) = 0.     

A case study of the modified g index: Counting multi-author publications fractionally

The citation records of 26 physicists are analyzed in order to determine the modified g index g_m which takes multiple coauthorship into account by fractionalized counting of the publications. The results are compared with the original g index as well as with the h index and the respective modified h index h_m. Although the correlations between these indices are relatively strong, the arrangement of the datasets is significantly different in detail depending on whether they are put into order according to the values of either the original or the modified indices.     

Would it be possible to increase the Hirsch-index, π-index or CDS-index by increasing the number of publications or citations only by unity?

The aim of the study is to explore the effects of the increase in the number of publications or citations on several impact indicators by a single journal paper or citation. The possible change of the h-index, A-index, R-index, π-index, π-rate, Journal Paper Citedness (JPC), and Citation Distribution Score (CDS) is followed by models. Particular attention is given to the increase of the indices by a single plus citation. The results obtained by the “successively built-up indicator” model show that with increasing number of citations or self-citations the indices may increase substantially.