How to detect indications of potential sources of bias in peer review: A generalized latent variable modeling approach exemplified by a gender study

The universalism norm of the ethos of science requires that contributions to science are not excluded because of the contributors’ gender, nationality, social status, or other irrelevant criteria. Here, a generalized latent variable modeling approach is presented that grant program managers at a funding organization can use in order to obtain indications of potential sources of bias in their peer review process (such as the applicants’ gender). To implement the method, the data required are the number of approved and number of rejected applicants for grants among different groups (for example, women and men or natural and social scientists). Using the generalized latent variable modeling approach indications of potential sources of bias can be examined not only for grant peer review but also for journal peer review.     

National Library of Medicine resource grants: application and review.

National Library of Medicine resource grants provide assistance in developing information services or activities, which are then made available to others. The resource grant program was redefined in 1989 to establish information access grants and information systems grants. The preparation of a resource grant proposal is discussed, with examples included for some sections. All applicants must use the PHS 398 application form, which is geared to research grants. The review process and reapplication are described. Problems with National Institutes of Health grant proposals that have been reported in the literature are discussed.     

Are NIH-funded publications fulfilling the proposed research? An examination of concept-matchedness between NIH research grants and their supported publications

The conceptual connections between scientific grants and publications are important, yet often overlooked in quantitative studies of science. An analysis of such connections could offer important insights into how science is conducted by individual researchers and research teams under the social and economic conditions of science. This study aims to offer the first piece of evidence towards this endeavor by analyzing the ratio of keyword matchedness between accepted NIH research grants from 2008 to 2015 and their funded publications. By applying linear regression method, we identified and examined three identified predictors of the outcome: 1) the funding rate of an NIH research program in a specific year, 2) the year difference between grant and publication, and 3) the funding size of a grant. Our findings suggest that these three factors contribute to the outcome in different capacities. Moreover, all of them may have different performances in individual funding programs, which highlights the importance of understanding the differences among individual funding mechanisms.     

Latent Markov modeling applied to grant peer review

In the grant peer review process we can distinguish various evaluation stages in which assessors judge applications on a rating scale. Research on the grant peer review process that considers its multi-stage character scarcely exists. In this study we analyze 1954 applications for doctoral and post-doctoral fellowships from the Boehringer Ingelheim Fonds (B.I.F.), which are evaluated in three stages (first: evaluation by an external reviewer; second: internal evaluation by a staff member; third: final decision by the B.I.F. Board of Trustees). The results of a latent Markov model (in combination with latent class analysis) show that a fellowship application has a chance of approval only if it is recommended for support already in the first evaluation stage, that is, if the external reviewer's evaluation is positive. Based on these results, a form of triage or pre-screening of applications seems desirable.     

Grants for Processing: A Case Study in Grant Writing and Project Management at the Brooklyn College Archives

This article describes the process of finding the right grant to meet the needs of a processing project, the application and peer review process, and the experience of administering the grant received from the National Historical Publications and Records Commission (NHPRC). The project is the processing of the YWCA of Brooklyn Collection, housed at Brooklyn College. The records have significant research potential in a variety of fields including women’s studies and sociology. While generally smooth, the administration of the grant has required the replacement of a departing Project Archivist, and the use of interns for some conservation work.     

Training for Librarians: An Analysis of the Institute of Museum and Library Services Laura Bush 21st Century Librarian Program

Abstract

The Institute of Museum and Library Services (IMLS), a major funding agency for museums and libraries in the United States, offers a variety of grant programs. The Laura Bush 21st Century Librarian Program (LB21) is one of the IMLS programs that focuses on training. Through analyzing awarded grants from 2003 to 2018, this project investigated the grant award distribution among different organizations and analyzed trends over time. Results indicated that researchers from higher education institutions received most of the grant money, and were relevantly concentrated among these institutions. Results for higher education grants in recent years also showed that collaborative projects among different institutions generally received large grants. Some recurring themes for recently awarded grants were also discussed. Findings from this project provide an overview of LB21 grant distribution and themes, and helps librarians and administrators identify training needs, and inspiring new grant ideas.     

Creating a departmental culture for communication grants

Extramural funding to support communication research influences and is influenced by the culture of our discipline, universities, and departments. Both the National Communication Association and the International Communication Association have established closer ties to funding agencies and encouraged members' grant writing pursuits through mentorship and convention programs, newsletter columns, and publicity. Universities vary in how much support they have available as infrastructure to assist in the submission and administration of grants. Some institutions facilitate extramural funding endeavors through mentoring programs, university-wide research centers, sharing of indirect costs, and space. Department cultures, too, vary in the resources available to support grants. Departments in many ways have the most direct effects on grant-supporting cultures. In addition to activities indicated for the discipline and university, departments can build a grant culture by recognizing grant writing in their personnel practices, creating systems for rewarding and supporting grant activities, and having policies in place for the many related issues (e.g., release time, indirect cost sharing, research assistants). Finally, the interpersonal culture, whether colleagues recognize and support grants, plays a major role in individual pursuit of, and satisfaction with, seeking extramural funding to support communication research.     

Creating a departmental culture for communication grants

Extramural funding to support communication research influences and is influenced by the culture of our discipline, universities, and departments. Both the National Communication Association and the International Communication Association have established closer ties to funding agencies and encouraged members' grant writing pursuits through mentorship and convention programs, newsletter columns, and publicity. Universities vary in how much support they have available as infrastructure to assist in the submission and administration of grants. Some institutions facilitate extramural funding endeavors through mentoring programs, university-wide research centers, sharing of indirect costs, and space. Department cultures, too, vary in the resources available to support grants. Departments in many ways have the most direct effects on grant-supporting cultures. In addition to activities indicated for the discipline and university, departments can build a grant culture by recognizing grant writing in their personnel practices, creating systems for rewarding and supporting grant activities, and having policies in place for the many related issues (e.g., release time, indirect cost sharing, research assistants). Finally, the interpersonal culture, whether colleagues recognize and support grants, plays a major role in individual pursuit of, and satisfaction with, seeking extramural funding to support communication research.     

Data exchange standards to support and acknowledge peer‐review activity

• Peer review is used to evaluate research, including publications, scientific awards, and grant proposals, and there is a continuum of at least six approaches to review from completely closed, double‐blind review to fully‐open and citable peer review.
• It is getting harder to find suitable experts to serve as reviewers so publishers and others are experimenting with methods to incentivize researcher participation, with a growing interest in enabling citation of peer‐review activity as a component.
• A Working Group on Peer Review Service, facilitated by CASRAI, was created to develop a data model and citation standard for peer‐review activity that can be used to support both existing and new review models.
• Standardized citation structures for reviews can enable the inclusion of peer‐review activity in personal recognition and evaluation, as well the ability to refer to reviews as part of the scholarly literature.

     

中国学者对学术论文公开同行评议的接受度研究

[目的/意义]同行评议作为一种评审制度一直受到"主观"而不够"客观"的批评。公开同行评议可以在一定程度上缓解这个问题。学者对公开同行评议的接受度如何是学术期刊实施该制度首要考虑的问题。[方法/过程]首先通过文献调研对学术论文公开同行评议的概念、相比传统同行评议的优势和不足进行论述,接着就公开评审流程中的公开内容对来自中国各个学科及研究领域的研究人员进行问卷调查,获得中国学者对学术论文公开同行评议的接受度数据,并对中国学者对论文开放同行评议的接受度进行分析。[结果/结论]问卷调查对象来自不同的学科领域,其中100%有发文经历,70%以上具有审稿经历,40%以上曾为国际期刊审过稿。调查结果表明,半数（占50.33%）中国学者对学术论文公开评审是接受的,在学术论文评审的不同阶段,中国学者的接受度不同。经过非参数统计检验,不同学科同行评议者接受度有所差异;是否具有国际期刊审稿经验的同行评议专家接受度差异不明显。论文相关分析数据可为中文学术期刊实施公开同行评议制度提供支持。     

Peer review: still king in the digital age

The article presents one of the main findings of an international study of 4,000 academic researchers that examined how trustworthiness is determined in the digital environment when it comes to scholarly reading, citing, and publishing. The study shows that peer review is still the most trustworthy characteristic of all. There is, though, a common perception that open access journals are not peer reviewed or do not have proper peer‐review systems. Researchers appear to have moved inexorably from a print‐based system to a digital system, but it has not significantly changed the way they decide what to trust. They do not trust social media. Only a minority – although significantly mostly young and early career researchers – thought that social media are anything other than more appropriate to personal interactions and peripheral to their professional/academic lives. There are other significant differences, according to the age of the researcher. Thus, in regard to choosing an outlet for publication of their work, young researchers are much less concerned with the fact that it is peer reviewed.     

选择合适审稿人提高刊物学术质量--《武汉理工大学学报-材料科学版》(英文版)遴选审稿人的体会

根据<武汉理工大学学报-材料科学版>(英文版)的审稿程序及专家审稿过程中存在的问题,提出了提高审稿费,加大刊物宣传力度,多渠道选择审稿人并对他们进行动态调整,利用现代化通信工具送审,必要时请作者推荐相关审稿人等措施来解决这些问题.通过这些方法,编辑能找到合适的审稿人,保证审稿的质量与时效性,为提高刊物的学术质量打下了坚实基础.     

What makes or breaks competitive research proposals? A mixed-methods analysis of research grant evaluation reports

The evaluation of grant proposals is an essential aspect of competitive research funding. Funding bodies and agencies rely in many instances on external peer reviewers for grant assessment. Most of the research available is about quantitative aspects of this assessment, and there is little evidence from qualitative studies. We used a combination of machine learning and qualitative analysis methods to analyse the reviewers' comments in evaluation reports from 3667 grant applications to the Initial Training Networks (ITN) of the Marie Curie Actions under the Seventh Framework Programme (FP7). Our results show that the reviewers' comments for each evaluation criterion were aligned with the Action's prespecified criteria and that the evaluation outcome was more influenced by the proposals’ weaknesses than by their strengths.     

The evolution of Web‐based peer‐review systems

Web‐based peer‐review systems are ubiquitous within scholarly publishing today, providing valuable efficiencies for authors, editors, and referees. These systems are the result of a general evolution from paper‐based workflows to electronic processes that began in the 1970s. DOS‐based systems paved the way for Windows desktop systems and, in the mid‐1990s, Web‐based peer review. Governmental, academic, and commercial stakeholders all participated in advancing the state of peer review by experimenting with different technologies, workflows, and features. These experiments have coalesced into a new steady state in which Web‐based peer‐review systems are the norm, and in which continued evolution tends to focus on incremental improvements to traditional workflow.     

Understanding the peer review endeavor in scientific publishing

Peer review plays an essential role in the scholarly publishing life cycle. Using the verified peer review records of reviewers who use the Publons, we employed review length as a potential indicator of the effort researchers spend on peer review. We then examined the associations between various factors and review length. Special focus was placed on estimating the relationships between non-academic (economic and sociological aspects) factors and review length. Our results show that gender, country-level cultural backgrounds, and country-level economic backgrounds were significantly associated with review length. In addition, there are significant associations of disciplines (humanities & social sciences or hard sciences), English proficiency, publications, and verified reviews with review length.     

Does grant funding foster research impact? Evidence from France

Policymakers are interested in assessing the effectiveness of the competitive grant funding model in producing impactful research. In the French context, we compare the impact of scientific articles supported by competitive grants with the impact of articles not supported by grants using a probabilistic matching procedure. We rely on publication acknowledgments to retrieve funding information and on citation data to assess the articles’ impact. We find that articles supported by competitive grants receive more citations than articles not supported by grants in the long run, while the difference is not significant in the short run. We find heterogeneity across fields.     

同行评审的新趋势

为避免传统同行评审形式出现的诸多弊端,学术出版界不断探索新的同行评议方法,出现了诸多的创新。本文主要对当前正在尝试的新兴同行评审方法进行调查,并介绍相关经验。这些创新方法包括开放性同行评审、非选择性同行评审、开放预审稿件的公众评审、便携式同行评审、反弹式同行评审、背书式同行评审。调查结果显示,同行评审实践发生了巨大的变化,与互联网革命和开放获取出版紧密相关,科学出版商为作者提供了更多的选择,新的同行评议形式为学术交流注入了新的活力。     

国外期刊论文同行评议创新态势述评

[目的/意义]综述国外期刊论文同行评议的创新实践,以期为国内学术出版提供参考.[方法/过程]通过对国外期刊论文同行评议的创新实践加以述评,分析其优势和挑战,总结创新实践的整体趋势.[结果/结论]预印本和发表后同行评议反映了人们对同行评议更快捷的期待,非选择性同行评议和注册报告反映了人们对同行评议更客观的期待,开放同行评...     

对专家审稿的分析和思考

为了研究专家审稿的规律和特征,对2008年度《哈尔滨工业大学学报》96篇论文的195次送审情况进行统计分析。结果表明:副教授的审回率和A类意见率都高于教授;40岁以下专家审回率和A类意见率最高,40～44岁的专家次之,45～54岁的专家审回率和A类意见率低,65岁以上专家审回率不高,但是一旦审稿就会比较认真,而且返回迅速;学术氛围浓厚的著名学府的专家在审稿方面更有责任感和奉献精神;工作繁忙是专家不审稿和审稿不认真的一个主要原因。建议高校学报大胆选用青年审稿专家,不断优化审稿专家数据库,与审稿专家建立良好的合作关系。