Identifying top news using crowdsourcing

The influential Text REtrieval Conference (TREC) retrieval conference has always relied upon specialist assessors or occasionally participating groups to create relevance judgements for the tracks that it runs. Recently however, crowdsourcing has been championed as a cheap, fast and effective alternative to traditional TREC-like assessments. In 2010, TREC tracks experimented with crowdsourcing for the very first time. In this paper, we report our successful experience in creating relevance assessments for the TREC Blog track 2010 top news stories task using crowdsourcing. In particular, we crowdsourced both real-time newsworthiness assessments for news stories as well as traditional relevance assessments for blog posts. We conclude that crowdsourcing not only appears to be a feasible, but also cheap and fast means to generate relevance assessments. Furthermore, we detail our experiences running the crowdsourced evaluation of the TREC Blog track, discuss the lessons learned, and provide best practices.     

Implementing crowdsourcing-based relevance experimentation: an industrial perspective

Crowdsourcing has emerged as a viable platform for conducting different types of relevance evaluation. The main reason behind this trend is that it makes possible to conduct experiments extremely fast, with good results at a low cost. However, like in any experiment, there are several implementation details that would make an experiment work or fail. To gather useful results, clear instructions, user interface guidelines, content quality, inter-rater agreement metrics, work quality, and worker feedback are important characteristics of a successful crowdsourcing experiment. Furthermore, designing and implementing experiments that require thousands or millions of labels is different than conducting small scale research investigations. In this paper we outline a framework for conducting continuous crowdsourcing experiments, emphasizing aspects that should be of importance for all sorts of tasks. We illustrate the value of characteristics that can impact the overall outcome using examples based on TREC, INEX, and Wikipedia data sets.     

Crowdsourcing—an Introduction: From Public Goods to Public Good

“Crowdsourcing” is a practice that combines the concepts of “the crowd” and “outsourcing.” Introducing two articles on crowdsourcing in this issue, Nancy Proctor argues that—although we associate crowdsourcing with Web 2.0 and the social media revolution—its origins stretch back to the nineteenth century. Crowdsourcing is examined for its usefulness in creating radical new relationships between museum constituents, users, and institutions—putting the “wisdom of the crowd” in dialogue rather than in competition with formal institutional knowledge.     

The effect of crowdsourcing on organizational learning: Evidence from local governments

Modern local governments must learn in order to deal effectively with the growing pressures of a rapidly changing environment. Crowdsourcing has entered the stage of popularization and wide adoption as a part of local government. Based on organizational learning perspective, this study develops a multi-level model of learning from crowdsourcing and a joint relationship between crowdsourcing and organizational learning in local governments. This paper adopts a multi-method research approach: (1) the model was tested with partial least squares structural equation modelling (PLS-SEM) using a survey of 205 local governments units in Poland; (2) in-depth semi-structured interviews conducted with 18 top managers from 18 local governments units in Poland. The empirical results show that: (1) crowdsourcing has significantly positive affects on individual, group, organizational levels of learning, feed-forward, and feed-back flows; (2) type of crowdsourcing, such wisdom crowd, crowd creation, and crowd voting are linked with organizational learning. Moreover, one type of crowdsourcing, such crowdfunding is not related with organizational learning of local governments. These findings contribute to both academic research and practical implications, advancing our understandings of relationship between crowdsourcing and organizational learning in local governments.     

Digital Cultural Heritage and the Crowd

Libraries, archives, and museums have a long history of collaboration with members of the public. There is already considerable interest in extending this relationship, inviting members of the public, often referred to as “the crowd,” to tag and classify, transcribe, organize, and otherwise add value to digital cultural heritage collection content. In this essay, current discussions of crowdsourcing are connected with the mission and values of cultural heritage organizations and a framework is offered for thinking about distinct components of different kinds of projects that have been lumped together.     

公共文化服务体系下的农民工阅读思考

提出在公共文化服务体系建设中应加强对农民工群体阅读问题的重视,通过对影响农民工阅读现实因素的分析,阐述了公共图书馆服务农民工的几点思考和愿景。     

众包与众包馆员制度:助推图书馆服务转型

首先介绍了众包的缘起与发展情况,分析了众包的内涵,例举了国外图书馆界的众多众包案例。在此基础之上,提出众包馆员制度,并与图书馆员2.0进行比较,提出众包馆员实施众包项目的关键步骤:选择众包项目,进行任务分解,制定激励机制和评审机制,选择发布平台,引导与监管众包过程,进行知识管理、整合及利用。最后,阐述了众包馆员的重要作用。     

Unpacking organizational awareness: scale development and empirical examinations in the context of distributed knowledge sharing

Organizational awareness has been viewed as a critical factor in facilitating collaboration and knowledge sharing, particularly among dispersed workers. Drawing on the findings of three studies, we disentangle the complex nature of organizational awareness, with a focus on its role in distributed knowledge sharing. First, we developed and validated an organizational awareness scale to investigate organizational awareness as a multidimensional construct that consists of availability awareness, task awareness, and social awareness. Second, employing the scale, we examined the relationships among organizational awareness, the use of enterprise social media (ESM), and knowledge acquisition in a global organization. The results of structural equation modeling demonstrated that ESM use was positively linked to distributed workers’ task awareness, which subsequently enhanced knowledge acquisition. Lastly, interview findings revealed how task awareness was cultivated by ESM use. Synthesizing the findings, we offer detailed accounts of organizational awareness and its relationships with dispersed workers’ knowledge sharing practice.     

科研众包视角下公众科学项目刍议:概念解析、模式探索及学科机遇

过去十年,公众科学已经从一个新兴概念逐渐演化成互联网环境下群体协作的利器。然而,由于公众科学项目发起主体的多元性、参与方的广泛性和异质性、实施过程的复杂性和动态性,其运行机制和管理方式还有很多难点。科研众包理念是成功开展公众科学项目的前提和重点,因此可以作为公众科学的理论视角去进行概念解析,而对于科研众包类型的解构也将有助于公众科学项目的模式设计。本研究认为基于科研众包类型和公众科学开展环境两个维度,可以有效构建公众科学项目的业务模式并进行业务划分;以“机构观”的思想去驱动、管理并维系公众科学项目的发展,加入第三方组织机构这一实体可以重构公众科学项目的运作模式。而立足于图书情报学科,可以对数字人文类公众科学项目进行深入探索,尤其可以从特色馆藏利用、数字人文平台构建、科研数据管护以及用户激励和培训等方面开展后续工作。图4。表1。参考文献45。     

Crowdsourcing interactions: using crowdsourcing for evaluating interactive information retrieval systems

In the field of information retrieval (IR), researchers and practitioners are often faced with a demand for valid approaches to evaluate the performance of retrieval systems. The Cranfield experiment paradigm has been dominant for the in-vitro evaluation of IR systems. Alternative to this paradigm, laboratory-based user studies have been widely used to evaluate interactive information retrieval (IIR) systems, and at the same time investigate users’ information searching behaviours. Major drawbacks of laboratory-based user studies for evaluating IIR systems include the high monetary and temporal costs involved in setting up and running those experiments, the lack of heterogeneity amongst the user population and the limited scale of the experiments, which usually involve a relatively restricted set of users. In this paper, we propose an alternative experimental methodology to laboratory-based user studies. Our novel experimental methodology uses a crowdsourcing platform as a means of engaging study participants. Through crowdsourcing, our experimental methodology can capture user interactions and searching behaviours at a lower cost, with more data, and within a shorter period than traditional laboratory-based user studies, and therefore can be used to assess the performances of IIR systems. In this article, we show the characteristic differences of our approach with respect to traditional IIR experimental and evaluation procedures. We also perform a use case study comparing crowdsourcing-based evaluation with laboratory-based evaluation of IIR systems, which can serve as a tutorial for setting up crowdsourcing-based IIR evaluations.     

双语句对翻译众包辅助平台设计与实现

本文提出了一种采用众包工作模式的科技领域日汉机器翻译辅助平台的构建方法。在充分调研众包生产模式、质量控制等研究和实践的基础上,设计了集用户管理、团队管理、语料管理、机器辅助翻译、术语辅助翻译等功能为一体的双语句对生产平台,针对不同角色和不同技术类别分别构建出相应的功能模块。该平台在众包工作模式的基础上结合了多源信息辅助译者完成翻译,翻译效率高且翻译成本低,平台的开发和运行为科技领域实用型日汉机器翻译系统的建设提供了有力支撑。     

Fear of Isolation and Perceived Affordances: The Spiral of Silence on Social Networking Sites Regarding Police Discrimination

The spiral of silence (SoS) framework elaborates the factors that determine whether individuals are willing to express their opinions in public. Although previous scholarship has examined differences in between face-to-face and computer-mediated communication, research studies have rarely tested how perceived affordances of the channel influence whether individuals express opinions or self-censor. In this study (N = 399), we examine several propositions of SoS within the context of discussing police discrimination on Facebook. To extend the theory’s relevance to social networking sites, we examined how users’ perceptions of network association, social presence, anonymity, and persistence related to opinion expression. Findings indicate support for some of the theory’s original tenets, as well as the role of multiple perceived affordances in determining whether people will express an opinion to their online social network. We discuss the implications for measuring and understanding political expression and silencing on social media as well as offline.     

Crowdsourcing for information retrieval: introduction to the special issue

This introduction to the special issue summarizes and contextualizes six novel research contributions at the intersection of information retrieval (IR) and crowdsourcing (also overlapping crowdsourcing’s closely-related sibling, human computation). Several of the papers included in this special issue represent deeper investigations into research topics for which earlier stages of the authors’ research were disseminated at crowdsourcing workshops at SIGIR and WSDM conferences, as well as at the NIST TREC conference. Since the first proposed use of crowdsourcing for IR in 2008, interest in this area has quickly accelerated and led to three workshops, an ongoing NIST TREC track, and a great variety of published papers, talks, and tutorials. We briefly summarize the area in order to help situate the contributions appearing in this special issue. We also discuss some broader current trends and issues in crowdsourcing which bear upon its use in IR and other fields.     

Crowdsourcing: A Platform for Crowd Engagement in the Publishing Industry

Publishers will usually make decisions based on their experience and knowledge in book publishing. However, there are risks of losses as markets can be unpredictable. Now, with the availability of various online social platforms, “crowdsourcing” is being used by publishers and authors to gather new ideas and innovations. This article will analyze how the crowdsourcing platform is used by industry players to attract active participation from the public, especially in the title development process. Publishers or authors also need to understand which platform is appropriate and how to use the platforms as a marketing communication tool. The netnography approach will be used to gather and analyze the data related to the specific subject area of the study, including literature review and online observation. Based on our observations, platforms such Wattpad and Ilham Karangkraf are popular among Malaysian authors and publishers, and had been used to get crowd engagement as well as to communicate with readers. With it, sometimes a work that has many readers or followers will be published.     

面向突发公共卫生事件的公众科学应用探索及 平台体系构建

[目的/意义]突发公共卫生事件的复杂性、交互性、专业性及影响力的日益增强,给传统的应急管理系统带来了巨大挑战。本文旨在基于大众力量和群体智慧的公众科学模式,为突发公共卫生事件的应急管理提供一种创新的、多层次的视角。[方法/过程]研究以新冠疫情的暴发为背景,在分析梳理公众科学及危机众包的研究和实践现状的基础上,重新审视了群策群力在应急管理体系中的重要性和必要性,提出构建面向突发公共卫生事件的公众科学平台的现实任务。[结果/结论]研究指出了目前公众科学介入突发公共事件应急管理的三个症结:重响应,轻防护;重参与,轻合作;重贡献,轻共创。针对该现状,文章首先明确了公共卫生类公众科学的内涵定位,其次从"造"与"用"、基本框架和能动性三个角度构建了面向突发公共卫生事件公众科学平台的体系。研究为完善国家公共卫生应急管理体系,提升社会防控治理能力,增强公众健康素养提供参考。     

State-of-the-art in biomedical literature retrieval for clinical cases: a survey of the TREC 2014 CDS track

Providing access to relevant biomedical literature in a clinical setting has the potential to bridge a critical gap in evidence-based medicine. Here, our goal is specifically to provide relevant articles to clinicians to improve their decision-making in diagnosing, treating, and testing patients. To this end, the TREC 2014 Clinical Decision Support Track evaluated a system’s ability to retrieve relevant articles in one of three categories (Diagnosis, Treatment, Test) using an idealized form of a patient medical record. Over 100 submissions from over 25 participants were evaluated on 30 topics, resulting in over 37k relevance judgments. In this article, we provide an overview of the task, a survey of the information retrieval methods employed by the participants, an analysis of the results, and a discussion on the future directions for this challenging yet important task.     

Low cost organization of a mental retardation facility staff library

A mental retardation facility staff library of 700 volumes required complete organization. Two restraining factors were lack of personnel and limited funds. A plan was developed to classify titles according to Library of Congress, with the subject heading catalog predominantly based on Medical Subject Headings (MeSH). A cataloging service was located which produced custom card sets and book labels from CATLINE and MARC computer databases. Where necessary, MARC records were edited to alter LC subject headings to MeSH terminology. Organization was accomplished at an average cost per title of less than $1.50.     

历史档案数字化著录存在的问题与对策研究

著录是档案数字化的关键步骤,也是决定数字化档案利用效果的重要环节。从国内历史档案数字化实践来看,著录工作耗时较多,成为影响数字化工作进度的瓶颈;历史档案著录工作有分阶段完成的趋势,著录方式有内包和外包。历史档案著录客体存在量大、整理基础差、文字识别难等问题,著录主体存在专业人才缺乏的问题,著录标准存在内容不全和不统一的问题,工作环境存在影响健康的问题。为提高历史档案著录工作水平,需要从国家战略的高度制订规划,加大投入;同时加强人才培养,增加人才储备;完善标准,提高著录精度;最后要通过优化流程和技术创新,通过众包的方式,充分利用社会计算资源,降低著录成本,提高著录效率。     

Socially Responsive Design and Evaluation of a Workers’ Compensation Thesaurus for a Community Organization with Selective Application of Cognitive Work Analysis: A Case Study

This article presents a case study of the evaluation of social responsiveness and relevance of terminology used in a specialized thesaurus constructed for a community legal clinic library. The thesaurus is intended to assist in meeting information discovery and educational needs of a small organization that advocates on behalf of injured workers for legal and social justice within Ontario's workers’ compensation system. The authors include an overview of the thesaurus project and the historical context of workers’ compensation. They discuss the use of Cognitive Work Analysis as an evaluation methodology particularly appropriate to both the material and the clinic's culture of collaboration, with examples of its application in practice and some lessons learned.     

Mining subtopics from text fragments for a web query

Web search queries are often ambiguous or faceted, and the task of identifying the major underlying senses and facets of queries has received much attention in recent years. We refer to this task as query subtopic mining. In this paper, we propose to use surrounding text of query terms in top retrieved documents to mine subtopics and rank them. We first extract text fragments containing query terms from different parts of documents. Then we group similar text fragments into clusters and generate a readable subtopic for each cluster. Based on the cluster and the language model trained from a query log, we calculate three features and combine them into a relevance score for each subtopic. Subtopics are finally ranked by balancing relevance and novelty. Our evaluation experiments with the NTCIR-9 INTENT Chinese Subtopic Mining test collection show that our method significantly outperforms a query log based method proposed by Radlinski et al. (2010) and a search result clustering based method proposed by Zeng et al. (2004) in terms of precision, I-rec, D-nDCG and D#-nDCG, the official evaluation metrics used at the NTCIR-9 INTENT task. Moreover, our generated subtopics are significantly more readable than those generated by the search result clustering method.