Beyond citation analysis: a model for assessment of research impact

Question:

Is there a means of assessing research impact beyond citation analysis?

Setting:

The case study took place at the Washington University School of Medicine Becker Medical Library.

Method:

This case study analyzed the research study process to identify indicators beyond citation count that demonstrate research impact.

Main Results:

The authors discovered a number of indicators that can be documented for assessment of research impact, as well as resources to locate evidence of impact. As a result of the project, the authors developed a model for assessment of research impact, the Becker Medical Library Model for Assessment of Research.

Conclusion:

Assessment of research impact using traditional citation analysis alone is not a sufficient tool for assessing the impact of research findings, and it is not predictive of subsequent clinical applications resulting in meaningful health outcomes. The Becker Model can be used by both researchers and librarians to document research impact to supplement citation analysis.     

Stabilisation operationalised: Using time series analysis to understand the dynamics of research collaboration

The aim of the paper is to investigate the use of online data and time series analysis, in order to study the dynamics of new types of research collaboration in a systematic way. Two international research teams were studied for more than 3 years, and quantitative data about their internet use together with observation of their collaboration patterns were gathered. Time series analysis (ARIMA modelling) was performed on their use of internet, and specific types of models related to specific ways of conducting research at a distance. The paper proposes the use of online data and ARIMA models to identify the stabilisation of a complex system, such as a research team, and investigate everyday research practices.     

Utilizing citation network structure to predict paper citation counts: A Deep learning approach

With the advancement of science and technology, the number of academic papers published each year has increased almost exponentially. While a large number of research papers highlight the prosperity of science and technology, they also give rise to some problems. As we know, academic papers are the most intuitive embodiment of the research results of scholars, which can reflect the level of researchers. It is also the standard for evaluation and decision-making of them, such as promotion and allocation of funds. Therefore, how to measure the quality of an academic paper is very critical. The most common standard for measuring the quality of academic papers is the number of citation counts of them, as this indicator is widely used in the evaluation of scientific publications. It also serves as the basis for many other indicators (such as the h-index). Therefore, it is very important to be able to accurately predict the citation counts of academic papers. To improve the effective of citation counts prediction, we try to solve the citation counts prediction problem from the perspective of information cascade prediction and take advantage of deep learning techniques. Thus, we propose an end-to-end deep learning framework (DeepCCP), consisting of graph structure representation and recurrent neural network modules. DeepCCP directly uses the citation network formed in the early stage of the paper as the input, and outputs the citation counts of the corresponding paper after a period of time. It only exploits the structure and temporal information of the citation network, and does not require other additional information. According to experiments on two real academic citation datasets, DeepCCP is shown superior to the state-of-the-art methods in terms of the accuracy of citation count prediction.     

Dissemination and impact of U.S. department of education's library research and demonstration projects: A citation analysis

Between 1965 and 1980, the Library Research and Demonstration Branch within the Department of Education awarded over $25 million to 312 projects. By tracing the citations in Social Sciences Citation Index from a random sampling of 52% of these projects, this study has attempted to assess the dissemination and impact of the projects in the professional literature.Approximately half of the projects were not cited in SSCI. The citations tended to be clustered among a small number of library-related serials. A small number of funded projects accounted for a large number of the citations. The most cited projects cost only one-fifth as much as the most expensive studies, yet were cited nearly five times as often.     

Combining mapping and citation network analysis for a better understanding of the scientific development: The case of the absorptive capacity field

The general aim of this paper is to show the results of a study in which we combined bibliometric mapping and citation network analysis to investigate the process of creation and transfer of knowledge through scientific publications. The novelty of this approach is the combination of both methods. In this case we analyzed the citations to a very influential paper published in 1990 that contains, for the first time, the term Absorptive Capacity. A bibliometric map identified the terms and the theories associated with the term while two techniques from the citation network analysis recognized the main papers during 15 years. As a result we identified the articles that influenced the research for some time and linked them into a research tradition that can be considered the backbone of the “Absorptive Capacity Field”.     

Non-linearity between referencing behavior and citation impact: A large-scale,discipline-level analysis

References formulate the foundation and endorsements of scientific research. Using articles published in 2005 covered by Microsoft Academic Graph, the current paper defines and calculates five indicators of references, i.e., the number of references, the number of citations of references, the age of references, the number of nodes in a reference cascade (a multi-generation reference network), and the density of bibliographic coupling networks by references, and investigates their relations with the citation impact of the focal publication. A non-linear relationship is shown in all the five indicators; specifically, we observe two types of patterns, namely inverted-L and -U relations, both presenting the existence of critical points. We further explore the discipline-level differences of such a relationship and how it relates to the characteristics of the discipline itself. Among all five indicators, the effect of disciplinary academic “environments” is universally identified. We believe that the current paper provides insightful views to the discussion regarding the significance of a reference list.     

The landmark citation method: analysis of a citation pattern as a collection assessment method.

The landmark citation method is a new collection assessment method based on the citation record of a single landmark article. This citation record is developed by identifying sources which cite the landmark article. A bibliography, extracted from the citation record, is then used to complete an assessment of the collection. This method was developed and used to assess the biotechnology collection of the National Library of Medicine. The information gained from this study, in addition to demonstrating the technique, also provided insight into the evolution of the biotechnology literature.     

Faculty citations: An approach to assessing the impact of diminishing resources on scientific research

In an effort to assess the impact of diminished library resources on scientific research, the Georgia Institute of Technology library collection was evaluated by analyzing the sources cited in the publications of faculty in the applied and physical sciences for 1989. This study indicates that even with substantial periodical cancellations, the periodical collection supports the majority of faculty needs since they rely primarily on core journals in their disciplines. Other materials cited less frequently in published faculty research — monographs, proceedings, technical reports, other serials, and government documents — are examined. The limitations of this method, its institutionally specific results, and the need to learn more about the information-seeking behaviors of research science faculty are also discussed.     

A comparative study of cross-domain research output and citations: Research impact cubes and binary citation frequencies

Across the various scientific domains, significant differences occur with respect to research publishing formats, frequencies and citing practices, the nature and organisation of research and the number and impact of a given domain's academic journals. Consequently, differences occur in the citations and h-indices of the researchers. This paper attempts to identify cross-domain differences using quantitative and qualitative measures. The study focuses on the relationships among citations, most-cited papers and h-indices across domains and for research group sizes. The analysis is based on the research output of approximately 10,000 researchers in Slovenia, of which we focus on 6536 researchers working in 284 research group programmes in 2008–2012.As comparative measures of cross-domain research output, we propose the research impact cube (RIC) representation and the analysis of most-cited papers, highest impact factors and citation distribution graphs (Lorenz curves). The analysis of Lotka's model resulted in the proposal of a binary citation frequencies (BCF) distribution model that describes well publishing frequencies. The results may be used as a model to measure, compare and evaluate fields of science on the global, national and research community level to streamline research policies and evaluate progress over a definite time period.     

Skewed citation distributions and bias factors: Solutions to two core problems with the journal impact factor

The journal impact factor (JIF) proposed by Garfield in the year 1955 is one of the most prominent and common measures of the prestige, position, and importance of a scientific journal. The JIF may profit from its comprehensibility, robustness, methodological reproducibility, simplicity, and rapid availability, but it is at the expense of serious technical and methodological flaws. The paper discusses two core problems with the JIF: first, citations of documents are generally not normally distributed, and, furthermore, the distribution is affected by outliers, which has serious consequences for the use of the mean value in the JIF calculation. Second, the JIF is affected by bias factors that have nothing to do with the prestige or quality of a journal (e.g., document type). For solving these two problems, we suggest using McCall's area transformation and the Rubin Causal Model. Citation data for documents of all journals in the ISI Subject Category “Psychology, Mathematical” (Journal Citation Report) are used to illustrate the proposal.     

Leadership competencies and the importance of research methods and statistical analysis in decision making and research and publication: A study of citation patterns

The roles of academic librarians and administrators include the use of statistical analysis and general analytical abilities in their decision-making processes, as well as in their roles as researchers, in developing research collections, and in providing instruction and reference assistance for students and other researchers. The research presented in this article addresses the original research published in frequently cited library and information science (LIS) journals to consider the extent to which academic librarians and administrators conduct and publish original research and to evaluate the range of research methodologies used and the level of collaboration among academic librarians, LIS faculty members, and others. The research results extend the prior self-report research in this area and indicate that academic librarians, administrators, and LIS faculty are authors of most of the published research in highly ranked journals. The level of collaboration among those in different types of roles is limited, however. In addition, most of the research articles have been authored or coauthored by those in research university libraries.     

Sharing the value and impact of outreach: Taking a multifaceted approach to outreach assessment

Although outreach is a common activity in academic libraries, little has been written about strategies for assessing library outreach efforts. Assessing outreach efforts is important in order to measure the success of the outreach activity, identify areas for iterative improvement, and demonstrate the value of the outreach activity to stakeholders. This article is a case study describing the multifaceted strategies employed to assess a major outreach event, Texas A&M University Libraries annual Open House event. It details demonstrating value and programmatic improvement as the articulated goals for outreach assessment as well as the specific strategies used and the insights gleaned from each assessment strategy.     

A citation analysis of the quantitative/qualitative methods debate's reflection in sociology research: Implications for library collection development

This study examines how the social sciences' debate between qualitative and quantitative methods is reflected in the citation patterns of sociology journal articles. Citation analysis revealed that quantitative articles were more likely to cite journal articles than monographs, while qualitative articles were more likely to cite monographs than journals. Quantitative articles cited other articles from their own quantitative-dominated journals but virtually excluded citations to articles from qualitative journals, while qualitative articles cited articles from the quantitative-dominated journals as well as their own qualitative-specialized journals. Discussion and conclusions include this study's implications for library collection development.     

Overlaps of multiple database retrieval and citation tracking in dementia care research: a methodological study

Objective:We aimed to determine overlaps and optimal combination of multiple database retrieval and citation tracking for evidence synthesis, based on a previously conducted scoping review on facilitators and barriers to implementing nurse-led interventions in dementia care.Methods:In our 2019 scoping review, we performed a comprehensive literature search in eight databases (CENTRAL, CINAHL, Embase, Emcare, MEDLINE, Ovid Nursing Database, PsycINFO, and Web of Science Core Collection) and used citation tracking. We retrospectively analyzed the coverage and overlap of 10,527 retrieved studies published between 2015 and 2019. To analyze database overlap, we used cross tables and multiple correspondence analysis (MCA).Results:Of the retrieved studies, 6,944 were duplicates and 3,583 were unique references. Using our search strategies, considerable overlaps can be found in some databases, such as between MEDLINE and Web of Science Core Collection or between CINAHL, Emcare, and PsycINFO. Searching MEDLINE, CINAHL, and Web of Science Core Collection and using citation tracking were necessary to retrieve all included studies of our scoping review.Conclusions:Our results can contribute to enhancing future search practice related to database selection in dementia care research. However, due to limited generalizability, researchers and librarians should carefully choose databases based on the research question. More research on optimal database retrieval in dementia care research is required for the development of methodological standards.     

Do open access articles have greater citation impact?: A critical review of the literature

The last few years have seen the emergence of several open access options in scholarly communication which can broadly be grouped into two areas referred to as ‘gold’ and ‘green’ open access (OA). In this article we review the literature examining the relationship between OA status and citation counts of scholarly articles. Early studies showed a correlation between the free online availability or OA status of articles and higher citation counts, and implied causality without due consideration of potential confounding factors. More recent investigations have dissected the nature of the relationship between article OA status and citations. Three non-exclusive postulates have been proposed to account for the observed citation differences between OA and non-OA articles: an open access postulate, a selection bias postulate, and an early view postulate. The most rigorous study to date (in condensed matter physics) showed that, after controlling for the early view postulate, the remaining difference in citation counts between OA and non-OA articles is explained by the selection bias postulate. No evidence was found to support the OA postulate per se; i.e. article OA status alone has little or no effect on citations. Further studies using a similarly rigorous approach are required to determine the generality of this finding.     

Facilitating access to consumer health information: a collaborative approach employing applied research

Responsibility for the provision of consumer health information typically lies within the public library domain or the health sciences library arena. In many communities, both types of libraries provide support for consumer health information services, often with varying levels of expertise. Public library employees may or may not be trained in the use of health and biomedical information resources. Conversely, health sciences library staff may or may not be familiar with information needs of the lay public within their community. To offset these potential differences, a model was developed to assess the health information needs of community members and to provide training for public library staff regarding use of relevant electronic consumer health information resources.