Reviewer assignment algorithms for peer review automation: A survey

Assigning paper to suitable reviewers is of great significance to ensure the accuracy and fairness of peer review results. In the past three decades, many researchers have made a wealth of achievements on the reviewer assignment problem (RAP). In this survey, we provide a comprehensive review of the primary research achievements on reviewer assignment algorithm from 1992 to 2022. Specially, this survey first discusses the background and necessity of automatic reviewer assignment, and then systematically summarize the existing research work from three aspects, i.e., construction of candidate reviewer database, computation of matching degree between reviewers and papers, and reviewer assignment optimization algorithm, with objective comments on the advantages and disadvantages of the current algorithms. Afterwards, the evaluation metrics and datasets of reviewer assignment algorithm are summarized. To conclude, we prospect the potential research directions of RAP. Since there are few comprehensive survey papers on reviewer assignment algorithm in the past ten years, this survey can serve as a valuable reference for the related researchers and peer review organizers.     

科技期刊审稿人科研能力的调查与分析——以《护理学报》为例

【目的】 分析《护理学报》审稿人科研能力现状,探讨提升其科研能力的策略,以提高护理学科技期刊审稿人的审稿质量。【方法】 采用一般资料问卷和科研能力自评量表网络发放方式对《护理学报》审稿人进行调查。【结果】 《护理学报》审稿人科研能力总分为(95.36±17.39)分,得分指标为79.47%,处于高水平组。科研能力量表6个维度得分指标由高到低依次为论文写作能力、问题发现能力、科研实践能力、文献查阅能力、科研设计能力和资料处理能力。【结论】 护理学科技期刊审稿人科研能力总体处于较高水平,其中论文写作能力和问题发现能力较强,科研设计能力和资料处理能力相对较差。需采取策略提升其科研能力水平,加强相关知识学习,以提升其审稿质量。     

对高校学报审稿工作的思考

本文分析了高校学报审稿工作的现状,明确了各环节的审稿主体及其主要职责,提出了选择审稿人的原则和审稿工作应采取的措施。     

Exploring Goodreads reviews for book impact assessment

Online book reviews reflect readers’ attitudes and opinions and serve as a data source for book impact assessment. Most research has only focused on the number of ratings and reviews to assess the impact of books. However, it is necessary to more thoroughly explore online book reviews, to analyze the viewpoints and sentiments expressed in them and the identity and motivation of the reviewers in order to evaluate the value of different types of book reviews. In this study, we collect Goodreads reviews of books indexed by the Book Citation Index and consider them according to the following three aspects: the popularity of highly cited books in Goodreads, the influence of reviewer roles (of author, librarian, and ordinary user) on book reviews, and the emotions and opinions behind reviewers’ ratings. Results consider the number of books reviewed in different disciplines, the variations in ratings of highly cited and non-highly cited books, differences in book reviews given by the reviewer roles, and the way reviewers express their sentiments about the books. The study concludes that if online reviews are to be used as indicators of book impact assessment, key considerations should include the subject discipline, the reviewer's role, and the sentiment polarity.     

防范学术不端论文出版的立体策略

【目的】 为了更有效地防范学术不端论文的出版,提出防范学术不端行为的立体策略。【方法】 利用逻辑推理法、举例法及综合分析法,分析学术期刊中存在的学术不端行为,从作者、责任编辑、审稿专家、主编四个维度分析不端文献出版的原因。【结果】 构建了立体防范学术不端论文出版的策略：初审前把关作者提供的基本信息;审稿时严格把关学术不端行为;加强主编监督机制;出版后建立问责制度;重视总结经验教训。【结论】 立体防范策略能够更全面地防范学术不端论文的出版。     

王韬：中国历史上首位报刊政论家——中国新闻社会史的视角

著名报人王韬一生做了三件大事：上书太平军,游历英法,主持《循环日报》——一张中国人自办成功的最早中文日报。他是其报言论的灵魂,风行一时。王韬形成了自己的办报理论与新闻思想,是中国近代报刊思想的奠基人。     

An automated conflict of interest based greedy approach for conference paper assignment system

Reviewer Assignment Problem (RAP) is a crucial problem for the conference due to time constraints and inadequate availability of expert reviewers. A fair evaluation of paper is key to an author's success, paper quality, conference reputation, and productive usage of funds. Recent studies reflect on the issue of reviewer bias in bids favoring authors belonging to the top institution and higher authority. Existing Conference Management Systems (CMS) are solely dependent upon self-declared Conflict of Interest (CoI) made by the authors, and reviewers. In literature, existing studies considers topic similarity, potential CoI, and reviewer's workload as trivial factors for ensuring review quality. Other factors include the diversity and authority of a reviewer. Past studies propose several theoretical optimization models. In this paper, we first individually model the factors using the best possible strategy in a constrained-based optimization framework. We tried to propose a completely novel framework that can be practically implemented to improve upon the performance of existing CMS. We map the RAP to an equilibrium multi-job assignment problem. Moreover, we propose a meta-heuristic greedy solution to solve it using weighted matrix factorization. We re-define an assignment quality metric required to validate such assignments. A real conference assignment data set collected from EasyChair is used for a comparative study. The TPMS is used as a baseline because it also uses similar factors, and due to its integration with widely used Microsoft CMS. The results show that the mean assignment quality of the proposed method is superior to other benchmark RAP systems.     

同行评议专家工作业绩测评及其模型研究

本文在对国家自然科学基金同行评议运行状况分析的基础上,介绍了同行评议工作业绩评估对于项目评议的重要性及其研究现状,初步给出了基于项目评议结果的同行评议工作测评指标和模型,并对其存在的不足以及下一步的研究内容进行了分析。     

学术期刊审稿专家研究领域与稿件匹配度的优化

【目的】 根据待审稿件的学科方向与内容,为其选择匹配度较高的审稿专家。【方法】 从文献数据库中选择与待审稿件学科方向相同的文献,优选其中合适的作者作为审稿专家,并依次通过采编系统的审稿专家库、国家和国家各部委基金项目数据库、专家所在单位官方网站验证审稿专家的各项信息。【结果】 选择与待审稿件学科方向相同的文献作者作为审稿专家,审稿结果更为科学合理,审稿效率得到很大提高。【结论】 选择审稿专家不能仅仅依据其一级学科方向从审稿专家库中简单选择,还需要从文献数据库、基金项目数据库和专家所在单位官方网站三个维度验证专家所在研究方向与待审稿件学科方向的匹配度,满足条件才能作为审稿专家。     

Multi-Aspect Review-Team Assignment using Latent Research Areas

Reviewer assignment is an important task in many research-related activities, such as conference organization and grant-proposal adjudication. The goal is to assign each submitted artifact to a set of reviewers who can thoroughly evaluate all aspects of the artifact’s content, while, at the same time, balancing the workload of the reviewers. In this paper, we focus on textual artifacts such as conference papers, where both (aspects of) the submitted papers and (expertise areas of) the reviewers can be described with terms and/or topics extracted from the text. We propose a method for automatically assigning a team of reviewers to each submitted paper, based on the clusters of the reviewers’ publications as latent research areas. Our method extends the definition of the relevance score between reviewers and papers using the latent research areas information to find a team of reviewers for each paper, such that each individual reviewer and the team as a whole cover as many paper aspects as possible. To solve the constrained problem where each reviewer has a limited reviewing capacity, we utilize a greedy algorithm that starts with a group of reviewers for each paper and iteratively evolves it to improve the coverage of the papers’ topics by the reviewers’ expertise. We experimentally demonstrate that our method outperforms state-of-the-art approaches w.r.t several standard quality measures.