TREC发展历程及现状分析

依据TREC会议集对历年参与团队与项目进行了统计,重点介绍了中国的TREC历程、TREC-16新推出的Million Query Track,指明了TREC三个未来关注焦点：非正式交流信息、特定学科领域以及用户交互。认为国内研究者应更加关注TREC以及中文语料库的建设。     

INEX与TREC的比较研究

INEX与TREC是检索领域的两大检索系统评价平台,在检索技术发展迅速的今天依然保持强大生命力,在当今检索技术评价领域起着十分重要的作用。本篇文章通过对INEX与TREC的研究目标以及平台的构成要素包括三个方面:测试集、检索问题的构造、相关性评估的比较,找出INEX相对于TREC评测平台的创新及不同点,以便更加深入和全面地了解INEX的评测方法。     

文本信息检索技术进展和性能评价框架

本文介绍TREC 评价信息检索系统的动向及其在推动研制新型检索系统中所起的作用, 并介绍新型检索系统的模式和特征及国外商品化全文文本检索系统性能评测指标。文中探讨了文本信息检索的性质评价标准问题, 并提出一个中文文本信息检索的系统评价框架。     

LIBQUAL+TM-图书馆服务质量评价方法新进展

LibQUAL+~(TM)是美国研究图书馆协会正在实施的试验项目,试图对SERVQUAL进行改造,从而建立适合于图书馆的服务质量评价体系。本文对LibQUAL+~(TM)的产生背景、发展历史、评价项目、作用等方面进行了较为详细的介绍。     

网络信息检索的检全率、检准率影响因素研究

主要介绍信息检索系统评价的两个常用指标--检全率、检准率,从信息源、搜索引擎检索机制、用户与系统的交互过程3方面分析网络环境下影响它们的因素,并对如何克服不良因素,提高检索质量提出一些建议。     

智能变电站试点项目评价指标体系研究

智能变电站试点项目评价是评估智能电网第一阶段（规划试点阶段）建设情况的重要环节和步骤。本文在充分考虑项目技术性、经济性以及实用性等方面内容的情况下,参照世界银行及政府部门普遍采用的设计准则——SMART准则,来构建智能变电站试点项目评价指标体系框架。     

国外数字图书馆评价实践综述

从基于计算机领域信息系统评价的数字图书馆项目评价、基于数字资源与服务使用计量的数字图书馆评价、基于服务评价的数字图书馆评价三方面梳理和总结国外已有的数字图书馆评价实践,并从指标特征、指标内容、评价方法等方面分别对上述实践的特点做简要分析,以期对我国数字图书馆评价的实践与理论研究提供借鉴。     

国家重点实验室网站科技资源信息开放共享评价

从科技发展、科技管理与国家政策制定等需求出发,阐述科技资源信息开放对于国家重点实验室和社会 的重要意义,介绍评价原则和评价指标体系,并从总体情况、信息公开程度、网站性能、互动交互能力4 个方面介绍评 价结果,得出国家重点实验室科技资源信息的开放水平尚处于初级阶段等结论,提出加强开放共享文化建设、将科技 资源信息开放作为考核的内容之一、建立开放标准规范等建议。     

欧盟长期保存项目Planets的体系与特点

简要介绍Planets项目的基本概况,详细描述其在保存计划、内容特征化、保存行为、互操作框架、试验平台等方面的技术实施。Planets项目能够提供长期保存过程中所需的多种工具和服务,推进数字资源长期保存的发展,其很多方面值得借鉴。     

基于TREC目标优化的过滤阈值调整算法

检出阈值的优化调整是自适应信息过滤的重点和难点之一。分析现有的阈值调整方法中普遍存在的问题,以TREC效用指标为目标函数,对阈值调整方法中的极大似然估计法和局部优化法进行比较分析,提出基于TREC目标优化的全局极大似然估计法与局部效用指标优化相结合的自适应过滤阈值调整算法。实验结果表明该方法能有效地提高信息过滤系统的性能。     

Identifying top news using crowdsourcing

The influential Text REtrieval Conference (TREC) retrieval conference has always relied upon specialist assessors or occasionally participating groups to create relevance judgements for the tracks that it runs. Recently however, crowdsourcing has been championed as a cheap, fast and effective alternative to traditional TREC-like assessments. In 2010, TREC tracks experimented with crowdsourcing for the very first time. In this paper, we report our successful experience in creating relevance assessments for the TREC Blog track 2010 top news stories task using crowdsourcing. In particular, we crowdsourced both real-time newsworthiness assessments for news stories as well as traditional relevance assessments for blog posts. We conclude that crowdsourcing not only appears to be a feasible, but also cheap and fast means to generate relevance assessments. Furthermore, we detail our experiences running the crowdsourced evaluation of the TREC Blog track, discuss the lessons learned, and provide best practices.     

Noun phrases in interactive query expansion and document ranking

The paper presents several techniques for selecting noun phrases for interactive query expansion following pseudo-relevance feedback and a new phrase-based document ranking method. A combined syntactico-statistical method was used for the selection of phrases for query expansion. Several statistical measures of phrase selection were evaluated. Experiments were also conducted studying the effectiveness of noun phrases in document ranking. One of the major problems in phrase-based document retrieval is weighting of overlapping and non-contiguous word sequences in documents. The paper presents a new method of phrase weighting, which addressed this problem, and its evaluation on the TREC dataset.     

Session search modeling by partially observable Markov decision process

Session search, the task of document retrieval for a series of queries in a session, has been receiving increasing attention from the information retrieval research community. Session search exhibits the properties of rich user-system interactions and temporal dependency. These properties lead to our proposal of using partially observable Markov decision process to model session search. On the basis of a design choice schema for states, actions and rewards, we evaluate different combinations of these choices over the TREC 2012 and 2013 session track datasets. According to the experimental results, practical design recommendations for using PODMP in session search are discussed.     

Scaling Up the TREC Collection

Due to the popularity of Web search engines, a large proportion of real text retrieval queries are now processed over collections measured in tens or hundreds of gigabytes. A new Very Large test Collection (VLC) has been created to support qualification, measurement and comparison of systems operating at this level and to permit the study of the properties of very large collections. The VLC is an extension of the well-known TREC collection and has been distributed under the same conditions. A simple set of efficiency and effectiveness measures have been defined to encourage comparability of reporting. The 20 gigabyte first-edition of the VLC and a representative 10% sample have been used in a special interest track of the 1997 Text Retrieval Conference (TREC-6). The unaffordable cost of obtaining complete relevance assessments over collections of this scale is avoided by concentrating on early precision and relying on the core TREC collection to support detailed effectiveness studies. Results obtained by TREC-6 VLC track participants are presented here. All groups observed a significant increase in early precision as collection size increased. Explanatory hypotheses are advanced for future empirical testing. A 100 gigabyte second edition VLC (VLC2) has recently been compiled and distributed for use in TREC-7 in 1998.     

Mining subtopics from different aspects for diversifying search results

User queries to the Web tend to have more than one interpretation due to their ambiguity and other characteristics. How to diversify the ranking results to meet users’ various potential information needs has attracted considerable attention recently. This paper is aimed at mining the subtopics of a query either indirectly from the returned results of retrieval systems or directly from the query itself to diversify the search results. For the indirect subtopic mining approach, clustering the retrieval results and summarizing the content of clusters is investigated. In addition, labeling topic categories and concept tags on each returned document is explored. For the direct subtopic mining approach, several external resources, such as Wikipedia, Open Directory Project, search query logs, and the related search services of search engines, are consulted. Furthermore, we propose a diversified retrieval model to rank documents with respect to the mined subtopics for balancing relevance and diversity. Experiments are conducted on the ClueWeb09 dataset with the topics of the TREC09 and TREC10 Web Track diversity tasks. Experimental results show that the proposed subtopic-based diversification algorithm significantly outperforms the state-of-the-art models in the TREC09 and TREC10 Web Track diversity tasks. The best performance our proposed algorithm achieves is α-nDCG@5 0.307, IA-P@5 0.121, and α#-nDCG@5 0.214 on the TREC09, as well as α-nDCG@10 0.421, IA-P@10 0.201, and α#-nDCG@10 0.311 on the TREC10. The results conclude that the subtopic mining technique with the up-to-date users’ search query logs is the most effective way to generate the subtopics of a query, and the proposed subtopic-based diversification algorithm can select the documents covering various subtopics.     

Crowdsourcing for information retrieval: introduction to the special issue

This introduction to the special issue summarizes and contextualizes six novel research contributions at the intersection of information retrieval (IR) and crowdsourcing (also overlapping crowdsourcing’s closely-related sibling, human computation). Several of the papers included in this special issue represent deeper investigations into research topics for which earlier stages of the authors’ research were disseminated at crowdsourcing workshops at SIGIR and WSDM conferences, as well as at the NIST TREC conference. Since the first proposed use of crowdsourcing for IR in 2008, interest in this area has quickly accelerated and led to three workshops, an ongoing NIST TREC track, and a great variety of published papers, talks, and tutorials. We briefly summarize the area in order to help situate the contributions appearing in this special issue. We also discuss some broader current trends and issues in crowdsourcing which bear upon its use in IR and other fields.     

Tasks, topics and relevance judging for the TREC Genomics Track: five years of experience evaluating biomedical text information retrieval systems

With the help of a team of expert biologist judges, the TREC Genomics track has generated four large sets of “gold standard” test collections, comprised of over a hundred unique topics, two kinds of ad hoc retrieval tasks, and their corresponding relevance judgments. Over the years of the track, increasingly complex tasks necessitated the creation of judging tools and training guidelines to accommodate teams of part-time short-term workers from a variety of specialized biological scientific backgrounds, and to address consistency and reproducibility of the assessment process. Important lessons were learned about factors that influenced the utility of the test collections including topic design, annotations provided by judges, methods used for identifying and training judges, and providing a central moderator “meta-judge”.     

Replicating Web Structure in Small-Scale Test Collections

Linkage analysis as an aid to web search has been assumed to be of significant benefit and we know that it is being implemented by many major Search Engines. Why then have few TREC participants been able to scientifically prove the benefits of linkage analysis in recent years? In this paper we put forward reasons why many disappointing results have been found in TREC experiments and we identify the linkage density requirements of a dataset to faithfully support experiments into linkage-based retrieval by examining the linkage structure of the WWW. Based on these requirements we report on methodologies for synthesising such a test collection.     

基于文档权重归并法的企业专家检索*

针对企业专家的专长识别与检索问题,采用文档权重归并法,利用TREC W3C数据集实现企业内的专家检索,并与专家档案法进行了比较。研究结果表明同样采用BM25模型,采用文档权重归并法具有稳定的优势。