Beyond citation analysis: a model for assessment of research impact

Question:

Is there a means of assessing research impact beyond citation analysis?

Setting:

The case study took place at the Washington University School of Medicine Becker Medical Library.

Method:

This case study analyzed the research study process to identify indicators beyond citation count that demonstrate research impact.

Main Results:

The authors discovered a number of indicators that can be documented for assessment of research impact, as well as resources to locate evidence of impact. As a result of the project, the authors developed a model for assessment of research impact, the Becker Medical Library Model for Assessment of Research.

Conclusion:

Assessment of research impact using traditional citation analysis alone is not a sufficient tool for assessing the impact of research findings, and it is not predictive of subsequent clinical applications resulting in meaningful health outcomes. The Becker Model can be used by both researchers and librarians to document research impact to supplement citation analysis.     

The college textbook as a tool for collection evaluation,analysis, and retrospective collection development

The search for effective methods of predicting book and journal use has been a central concern of acquisitions librarians. This paper suggests the application of citation analysis to the college textbook as a means of evaluating, analyzing, and developing book collections in small and medium sized academic libraries. Arguments for the relevance of textbook citations are presented, and a study of the references found in 25 texts in five separate disciplines indicates that the titles cited go far beyond basic checklists in number and depth and that many of these titles will not be included in smaller academic library collections developed according to traditional methods.     

Patent citation spectroscopy (PCS): Online retrieval of landmark patents based on an algorithmic approach

One essential component in the construction of patent landscapes in biomedical research and development (R&D) is identifying the most seminal patents. Hitherto, the identification of seminal patents required subject matter experts within biomedical areas. In this article, we report an analytical method and tool, Patent Citation Spectroscopy (PCS), for the online identification of landmark patents in user-specified areas of biomedical innovation. Using USPTO data, PCS mines the cited references within large sets of patents at the internet and provides an estimate of the historically most impactful prior work. We show the efficacy of PCS in three case studies of biomedical innovation with clinical relevance: (1) RNA interference (RNAi), (2) cholesterol and (3) cloning. PCS mined and analyzed cited references related to patents on RNA interference and correctly identified the foundational patent of this technology, as independently reported by subject matter experts on RNAi intellectual property. Secondly, we apply PCS to a broad set of patents dealing with cholesterol – a case study chosen to reflect a more general, as opposed to expert, patent search query. PCS mined through cited references and identified the seminal patent as that for Lipitor, the groundbreaking medication for treating high cholesterol as well as the pair of patents underlying Repatha. The final case study, cloning, highlights some of the advantages conferred by the PCS methodology in identifying seminal patents. These cases suggest that PCS provides a useful method for identifying seminal patents in areas of biomedical innovation and therapeutics. The interactive tool is free-to-use at: http://www.leydesdorff.net/comins/pcs/index.html.     

A citation analysis of the quantitative/qualitative methods debate's reflection in sociology research: Implications for library collection development

This study examines how the social sciences' debate between qualitative and quantitative methods is reflected in the citation patterns of sociology journal articles. Citation analysis revealed that quantitative articles were more likely to cite journal articles than monographs, while qualitative articles were more likely to cite monographs than journals. Quantitative articles cited other articles from their own quantitative-dominated journals but virtually excluded citations to articles from qualitative journals, while qualitative articles cited articles from the quantitative-dominated journals as well as their own qualitative-specialized journals. Discussion and conclusions include this study's implications for library collection development.     

基于稿件引文内容分析的同行评审专家遴选方法

同行评审意见是期刊编辑用稿决策的重要依据,同行评审专家的来源和遴选与评审结果的质量和公平性密切相关。编辑为稿件遴选出评审意愿、专业、能力匹配,且时间充裕的审稿专家是比较费时、费力的。本文基于投稿论文的参考文献和引用的内容、动机及情感分析,提出一种同行评审专家的遴选方法。该方法有助于“小同行”专家的发现,降低不同学术范式导致的同行评议误判率,为编辑的稿件送审工作提供参考。     

The distribution of references across texts: Some implications for citation analysis

In citation network analysis, complex behavior is reduced to a simple edge, namely, node A cites node B. The implicit assumption is that A is giving credit to, or acknowledging, B. It is also the case that the contributions of all citations are treated equally, even though some citations appear multiply in a text and others appear only once. In this study, we apply text-mining algorithms to a relatively large dataset (866 information science articles containing 32,496 bibliographic references) to demonstrate the differential contributions made by references. We (1) look at the placement of citations across the different sections of a journal article, and (2) identify highly cited works using two different counting methods (CountOne and CountX). We find that (1) the most highly cited works appear in the Introduction and Literature Review sections of citing papers, and (2) the citation rankings produced by CountOne and CountX differ. That is to say, counting the number of times a bibliographic reference is cited in a paper rather than treating all references the same no matter how many times they are invoked in the citing article reveals the differential contributions made by the cited works to the citing paper.     

The Citation Cloud of a biomedical article: a free,public, web-based tool enabling citation analysis

Background:An article''s citations are useful for finding related articles that may not be readily found by keyword searches or textual similarity. Citation analysis is also important for analyzing scientific innovation and the structure of the biomedical literature. We wanted to facilitate citation analysis for the broad community by providing a user-friendly interface for accessing and analyzing citation data for biomedical articles.Case Presentation:We seeded the Citation Cloud dataset with over 465 million open access citations culled from six different sources: PubMed Central, Microsoft Academic Graph, ArnetMiner, Semantic Scholar, Open Citations, and the NIH iCite dataset. We implemented a free, public extension to PubMed that allows any user to visualize and analyze the entire citation cloud around any paper of interest A: the set of articles cited by A, those which cite A, those which are co-cited with A, and those which are bibliographically coupled to A.Conclusions:Citation Cloud greatly enables the study of citations by the scientific community, including relatively advanced analyses (co-citations and bibliographic coupling) that cannot be undertaken using other available tools. The tool can be accessed by running any PubMed query on the Anne O''Tate value-added search interface and clicking on the Citations button next to any retrieved article.     

LibStatCAT: a library statistical collection assessment tool for individual libraries & cooperative collection development

The following presentation to the Acquisitions Institute at Timberline Lodge focuses on LibStatCAT, a prototype collection assessment tool under development at Portland State University. Our presentation explores the current uses of LibStatCAT, as well as future developments of LibStatCAT as a regional library data farm for comprehensive assessment of library collections and services for individual libraries and cooperative collection development. We share a variety of other collection assessment tools in use at Portland State University Library.     

A business citation analysis: A local follow-up study

While there is no official style for business writing, the most common citation style in business research has not been determined previously. The purpose of this study is to evaluate the top scholarly business journals to determine the most commonly used citation style in business academic research. The author used the list of 452 top business journals included in the Social Sciences Citation Index (SSCI) Journal Citation Report. The results show that 69% of business journals use a publisher-dictated style, whereas of the “traditional” citation styles, 11% use the American Psychological Association style, 10% use Harvard style, 8% use Chicago style, and none use Modern Language Association style.     

Characterizing highly cited method and non-method papers using citation contexts: The role of uncertainty

The top 1000 biomedical papers by number of citations are classified by method, type of method and non-methods by examination of citation contexts. Supervised machine learning is applied to the context data for a training sample of papers which is then used to classify the full list, revealing that words indicating utility are most important for the classification of methods. Further word analysis is carried out using corpus linguistics to uncover context words that characterize non-methods. Hedging words are found to play an important role for non-methods, and several are selected for further analysis with logistic regression. Other variables in the regression are a consensus variable based on the similarity of contexts for a paper and another variable based on whether citations come from “methods” sections of citing papers. Accuracy of predictions from logistic regression is comparable to machine learning. The results are interpreted in terms of the perceived certainty or uncertainty of the underlying knowledge, that is, methods and their outputs have higher certainty, and non-methods higher uncertainty. Evidence is found that hedging is inversely related to citation frequency. Implications of this work for the study of the development of science and the role of methods and tools in biomedical research are discussed.     

The effect of citation analysis on query expansion for patent retrieval

Patent prior art search is a type of search in the patent domain where documents are searched for that describe the work previously carried out related to a patent application. The goal of this search is to check whether the idea in the patent application is novel. Vocabulary mismatch is one of the main problems of patent retrieval which results in low retrievability of similar documents for a given patent application. In this paper we show how the term distribution of the cited documents in an initially retrieved ranked list can be used to address the vocabulary mismatch. We propose a method for query modeling estimation which utilizes the citation links in a pseudo relevance feedback set. We first build a topic dependent citation graph, starting from the initially retrieved set of feedback documents and utilizing citation links of feedback documents to expand the set. We identify the important documents in the topic dependent citation graph using a citation analysis measure. We then use the term distribution of the documents in the citation graph to estimate a query model by identifying the distinguishing terms and their respective weights. We then use these terms to expand our original query. We use CLEF-IP 2011 collection to evaluate the effectiveness of our query modeling approach for prior art search. We also study the influence of different parameters on the performance of the proposed method. The experimental results demonstrate that the proposed approach significantly improves the recall over a state-of-the-art baseline which uses the link-based structure of the citation graph but not the term distribution of the cited documents.     

引文分析在图书馆期刊馆藏资源建设中的作用 --济南大学应用化学专业硕士论文引文分析研究

论文时济南大学应用化学专业2001～2004硕士研究生学位论文引文的数量、语种、类型、年代及该专业的文献半衰期进行了统计分析，时引文量最大的外文期刊进行排序并考察其在本校的收藏情况，从而掌握本校的期刊保障率，为馆藏期刊资源建设提供了参考依据。     

The use of citation databases for the assessment of the research activities of organizations in Belarus

This paper reviews the use of publication activity indicators for the assessment of the qualifications of scientific and teaching staff. A new section entitled The Publication Activities of Scientists in Belarus has been created on the website of the Central Science Library of the National Academy of Sciences of Belarus. This section includes a list of periodicals that are available for the publication of scientific results in key areas of research (based on the Web of Knowledge) and a listof Belarusian organizations that are ranked by the h-index and the number of references to their papers according to Scopus. This section also provides the editors of scientific journals with recommendations on the inclusion of citation data into global databases and other materials for researchers.