Are NIH-funded publications fulfilling the proposed research? An examination of concept-matchedness between NIH research grants and their supported publications

The conceptual connections between scientific grants and publications are important, yet often overlooked in quantitative studies of science. An analysis of such connections could offer important insights into how science is conducted by individual researchers and research teams under the social and economic conditions of science. This study aims to offer the first piece of evidence towards this endeavor by analyzing the ratio of keyword matchedness between accepted NIH research grants from 2008 to 2015 and their funded publications. By applying linear regression method, we identified and examined three identified predictors of the outcome: 1) the funding rate of an NIH research program in a specific year, 2) the year difference between grant and publication, and 3) the funding size of a grant. Our findings suggest that these three factors contribute to the outcome in different capacities. Moreover, all of them may have different performances in individual funding programs, which highlights the importance of understanding the differences among individual funding mechanisms.     

Three-feature model to reproduce the topology of citation networks and the effects from authors’ visibility on their h-index

Various factors are believed to govern the selection of references in citation networks, but a precise, quantitative determination of their importance has remained elusive. In this paper, we show that three factors can account for the referencing pattern of citation networks for two topics, namely “graphenes” and “complex networks”, thus allowing one to reproduce the topological features of the networks built with papers being the nodes and the edges established by citations. The most relevant factor was content similarity, while the other two – in-degree (i.e. citation counts) and age of publication – had varying importance depending on the topic studied. This dependence indicates that additional factors could play a role. Indeed, by intuition one should expect the reputation (or visibility) of authors and/or institutions to affect the referencing pattern, and this is only indirectly considered via the in-degree that should correlate with such reputation. Because information on reputation is not readily available, we simulated its effect on artificial citation networks considering two communities with distinct fitness (visibility) parameters. One community was assumed to have twice the fitness value of the other, which amounts to a double probability for a paper being cited. While the h-index for authors in the community with larger fitness evolved with time with slightly higher values than for the control network (no fitness considered), a drastic effect was noted for the community with smaller fitness.     

An investigation of the relationship between scientists’ mobility to/from China and their research performance

A large number of overseas elites were brought back to China by the policy in the past decade. However, name disambiguation defied investigations on the relationship between their mobility and research performance. By taking advantage of the ORCID website and applying causal inference strategies, we investigated 2489 China-connected scientists’ academic performance in the Web of Science database in terms of their job mobility, including 1388 scientists who moved to China the treatment group, and 1101 scientists with a possibility to move to China the control group. The results show that first, scientists moving to China have a new growth pattern where both their productivity and the rates of being corresponding authors in publications grew more rapidly than before; however, they made fewer contributions to the four top journals, Nature, Science, Cell, and PNAS. Second, the research performance of the scientists is affected by the time of mobility towards China, the countries from which they moved, and the disciplines of their publications. Last, China now maintains symmetrical inflow-outflow patterns with most countries, especially developed countries in Europe and North America, with only a few exceptions (e.g., Pakistan).     

A sensitivity analysis of research institutions’ productivity rankings to the time of citation observation

One of the critical issues in bibliometric research assessments is the time required to achieve maturity in citations. Citation counts can be considered a reliable proxy of the real impact of a work only if they are observed after sufficient time has passed from publication date. In the present work the authors investigate the effect of varying the time of citation observation on accuracy of productivity rankings for research institutions. Research productivity measures are calculated for all Italian universities active in the hard sciences in the 2001–2003 period, by individual field and discipline, with the time of the citation observation varying from 2004 to 2008. The objective is to support policy-makers in choosing a citation window that optimizes the tradeoff between accuracy of rankings and timeliness of the exercise.     

Scientists’ genders and international academic collaboration: An empirical study of Chinese universities and research institutes

In the previous literature, no clear conclusions have been reached about the effect of gender differences on research performance (RP) in science, as measured by publication productivity, number of citations, and academic awards. Meanwhile, a gap also exists in the research regarding gender differences in international academic collaboration. To complement the existing literature, this study investigated the achievements of scientists engaged in international academic collaboration, which places heavy demands on language and communication skills and in which female scientists appear to have more advantages than male scientists. We investigated the effect of international collaboration carried out by chemists from China’s Project 985 universities and the Chinese Academy of Sciences and compared the extent to which the international collaboration improved female and male scientists’ academic performance. The results indicated that, compared to male scientists, female scientists performed better and significantly improved their academic performance through international collaboration. This conclusion was valid for different periods throughout chemists’ academic careers. The policy implications are discussed at the end of this study.     

The interdisciplinarity of research data: How widely is shared research data reused in the STEM fields?

The aim of this study was to examine the interdisciplinarity of research data in the science, technology, engineering, and mathematics (STEM) fields. The findings revealed that interdisciplinarity was not distributed evenly across journals serving the STEM fields. Based on the diversity of the references as measured by the Gini coefficient index, the mathematical sciences showed the greatest inequality, followed by astronomy/physics, the earth sciences, the biological sciences, and technology. Based on the number of Essential Science Indicator (ESI) fields, the biological sciences showed the greatest variety, followed by the earth sciences, technology, the mathematical sciences, chemistry, and computing, while engineering showed no variety. Lastly, based on the Leydesdorff interdisciplinarity formula outcomes, the earth sciences showed the greatest diversity, but earth sciences articles were cited in articles in fewer fields than biological sciences articles. This study contributes to the study of interdisciplinary data citation for data sharing and reuse in STEM fields with respect to the measurement of the balance, variety, and diversity of research data.     

Would it be possible to increase the Hirsch-index, π-index or CDS-index by increasing the number of publications or citations only by unity?

The aim of the study is to explore the effects of the increase in the number of publications or citations on several impact indicators by a single journal paper or citation. The possible change of the h-index, A-index, R-index, π-index, π-rate, Journal Paper Citedness (JPC), and Citation Distribution Score (CDS) is followed by models. Particular attention is given to the increase of the indices by a single plus citation. The results obtained by the “successively built-up indicator” model show that with increasing number of citations or self-citations the indices may increase substantially.     

How important is software to library and information science research? A content analysis of full-text publications

We investigate the contributions of scientific software to library and information science (LIS) research using a sample of 572 English language articles published in 13 journals in 2008, 2011, 2014, and 2017. In particular, we examine the use and citation of software freely available for academic use in the LIS literature; we also explore the extent to which researchers follow software citation instructions provided by software developers. Twenty-seven percent of the LIS journal articles in our sample explicitly mention and use software. Yet although LIS researchers are becoming increasingly reliant on software that is freely available for academic use, many still fail to include formal citations of such software in their publications. We also find that a substantial proportion of researchers, when documenting software use, do not cite the software in the manner recommended by its developers.     

Chinese and non‐Chinese scholars’ contributions to communication research on greater China, 1978–98

This content‐analytical study investigates, over a two‐decade period, the contribution of ethnic and non‐ethnic scholars to communication research on Greater China. Although its findings show a significant increase in research publications on China over the years by both Chinese and non‐Chinese scholars, Chinese scholars are much more productive than non‐Chinese scholars. In addition, mixed ethnicity scholars have published more comparative studies than single ethnicity scholars. Telecommunications and advertising are the two fastest growing areas in communication research on China.     

Introducing recalibrated academic performance indicators in the evaluation of individuals’ research performance: A case study from Eastern Europe

In Hungary, the highest and most prestigious scientific qualification is considered to be the Doctor of Science (DSc) title being awarded by the Hungarian Academy of Sciences. The academic performance indicators of the DSc title are of high importance in the evaluation of individuals’ research performance not only when a researcher applies for obtaining a DSc title, but also during promotions and appointments at universities, and in the case of the evaluation of applications for scientific titles and degrees, and the assessment of applications for funding. In the Section of Earth Sciences encompassing nine related disciplines, rather than carrying out a straightforward bibliometric analysis, the performance indicators were designed as a result of a consensual agreement between leading academicians, each of whom represented a particular discipline. Therefore, the minimum values of the indicators, required to be fulfilled if one is applying for a DSc title, do not adequately reflect the actual discipline-specific performance of researchers. This problem may generate tension between researchers during the evaluation process. The main goal of this paper is to recalibrate the minimum values of four major performance indicators by taking the actual discipline-specific distance ratios into account. In addition, each minimum value will be defined by employing integer and fractional counting methods as well. The research outcome of this study can provide impetus for the Section of Earth Sciences (and eventually other sections of the Hungarian Academy of Sciences) to optimize the minimum values of the DSc title performance indicators by taking the specifics of each discipline into account. Because academic performance indicators are also employed in other Eastern European countries in the evaluation of individuals’ research performance, the methods used in that paper can be placed into a wider geographical context.     

The problems associated with the evaluation of world-class competitiveness of Russia’s science, as illustrated by clinical medicine

The main vectors for reforming Russia??s scientific-technological and innovative spheres, which are reflected in program documents that were adopted in late 2011 and early 2012, are discussed in the article. The authors propose to formalize the concepts that were used in these documents, such as world-class research competence and world-leading science and technology sectors. Using the Field Normalized Citation Score (NCSf) as a bibliometric indicator for the analysis of different research areas of Russia??s clinical medicine, the authors show the degree of variance between individual research areas within the same national subject field in terms of their correspondence to world-class excellence. The authors also emphasize the necessity of developing Russia??s national methodology for auditing Russian science, which will take its Russian-language segment into account where the world??s recognized methodologies are not always adequately applicable.     

A ‘powerful,creative history’: the reticence of women architects to donate their professional records to archival repositories

The article reports the findings from research investigating the impact of international standards for records management. This is a significant issue for the recordkeeping community given the significant resourcing devoted to the development of international standards. Previous attempts to measure impact have met with little success, so this research approached the problem area from a new angle, namely the indirect influence of standards on practice via their use in education and training. Data was collected from a global survey of educators and trainers. Findings demonstrate that the standards are being used extensively at undergraduate and postgraduate level as well as in vocational training. However, a number of barriers to use are identified. The research provides a strong foundation for further investigation of impact and use of the international standards.     

The land of confusion?: High school students and their use of the World Wide Web for research

The use of the World Wide Web by high school students to complete school assignments was studied by interviewing high school students. The author expected to find that students were using the World Wide Web as their primary source of information and that the students had difficulty in evaluating whether information on a web site was scholarly. The author considered Perry's Scheme of Student Development and the unique nature of the World Wide Web to come to these assumptions, which were only partially validated by this study. As is typical in a qualitative study, the sample was small, but some themes that could help provide direction to further research emerged. It was discovered that the students in the study used a good variety of resources, including libraries and the World Wide Web, to find information for school assignments. However, the students were weak at determining the quality of the information that they found on a web site. Students did poorly at evaluating web site information in the absence of gatekeepers.     

Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics

Today, it is not clear how the impact of research on other areas of society than science should be measured. While peer review and bibliometrics have become standard methods for measuring the impact of research in science, there is not yet an accepted framework within which to measure societal impact. Alternative metrics (called altmetrics to distinguish them from bibliometrics) are considered an interesting option for assessing the societal impact of research, as they offer new ways to measure (public) engagement with research output. Altmetrics is a term to describe web-based metrics for the impact of publications and other scholarly material by using data from social media platforms (e.g. Twitter or Mendeley). This overview of studies explores the potential of altmetrics for measuring societal impact. It deals with the definition and classification of altmetrics. Furthermore, their benefits and disadvantages for measuring impact are discussed.     

An empirical investigation of the tribes and their territories: Are research specialisms rural and urban?

We propose an operationalization of the rural and urban analogy introduced in Becher and Trowler (2001). According to them, a specialism is rural if it is organized into many, smaller topics of research, with higher author mobility among them, lower rate of collaboration and productivity, lower competition for resources and citation recognitions compared to an urban specialism. It is assumed that most humanities specialisms are rural while science specialisms are in general urban: we set to test this hypothesis empirically. We first propose an operationalization of the theory in most of its quantifiable aspects. We then consider specialisms from history, literature, computer science, biology, astronomy. Our results show that specialisms in the humanities present a sensibly lower citation and textual connectivity, in agreement with their organization into more, smaller topics per specialism, as suggested by the analogy. We argue that the intellectual organization of rural specialisms might indeed be qualitative different from urban ones, discouraging the straightforward application of citation-based indicators commonly applied to urban specialisms without a dedicated re-design in acknowledgement of these differences.     

A sensitivity analysis of researchers’ productivity rankings to the time of citation observation

In this work we investigate the sensitivity of individual researchers’ productivity rankings to the time of citation observation. The analysis is based on observation of research products for the 2001–2003 triennium for all research staff of Italian universities in the hard sciences, with the year of citation observation varying from 2004 to 2008. The 2008 rankings list is assumed the most accurate, as citations have had the longest time to accumulate and thus represent the best possible proxy of impact. By comparing the rankings lists from each year against the 2008 benchmark we provide policy-makers and research organization managers a measure of trade-off between timeliness of evaluation execution and accuracy of performance rankings. The results show that with variation in the evaluation citation window there are variable rates of inaccuracy across the disciplines of researchers. The inaccuracy results negligible for Physics, Biology and Medicine.     

Using a Targeted Rubric to Deepen Direct Assessment of College Students’ Abilities to Evaluate the Credibility of Sources

At the Sonoma State University Library, a rubric was designed specifically for deepening the assessment of freshmen students’ ability to evaluate the credibility of sources. The course-level rubric was developed after program-level assessments proved too broad to uncover specific challenges in this particular skill. As a direct assessment tool, the rubric allows teaching faculty and librarians to more clearly identify and address the smaller steps required for a student to successfully learn to evaluate a source. The rubric and the results of using it as an assessment tool with a freshmen-level, general education oral communication/critical thinking course are discussed.