与自然语言查询表述相关的词语分析

考察在信息检索过程中用户以自然语言表述的查询语句中的词语使用情况。以一个信息需求描述颗粒度不等的查询表述语料库为素材,辅以汉语通用语料作为对照,通过词频以及词语的文本覆盖率等统计数据,按照是否需要在目标文本中直接或以其他形式出现,将查询表述语句中的词语区分为对汉语文本处理具有普遍意义的通用停用词、服务于信息检索表述用的专用停用词和与特定需求相关的信息内容词语。区分词语使用的不同性质,能为信息系统前端的自然语言查询处理增加一道剥离工序,防止将整个查询语句的分词结果全部作为检索项所造成的效率和准确率的退化。     

Using Statistical Term Similarity for Sense Disambiguation in Cross-Language Information Retrieval

With the increasing availability of machine-readable bilingual dictionaries, dictionary-based automatic query translation has become a viable approach to Cross-Language Information Retrieval (CLIR). In this approach, resolving term ambiguity is a crucial step. We propose a sense disambiguation technique based on a term-similarity measure for selecting the right translation sense of a query term. In addition, we apply a query expansion technique which is also based on the term similarity measure to improve the effectiveness of the translation queries. The results of our Indonesian to English and English to Indonesian CLIR experiments demonstrate the effectiveness of the sense disambiguation technique. As for the query expansion technique, it is shown to be effective as long as the term ambiguity in the queries has been resolved. In the effort to solve the term ambiguity problem, we discovered that differences in the pattern of word-formation between the two languages render query translations from one language to the other difficult.     

一种用于主题提取的非线性加权方法

主题提取是文本处理的一项重要工作。本文首先分析了主题抽取中加权方法形成时的一些定量问题,然后提出了主题相关词一种非线性加权处理方法,对比实验结果显示它不仅是一种比较稳健的方法,而且能在一定程度上提高主题提取的正确率。     

Exploiting the Similarity of Non-Matching Terms at Retrieval Time

In classic Information Retrieval systems a relevant document will not be retrieved in response to a query if the document and query representations do not share at least one term. This problem, known as term mismatch, has been recognised for a long time by the Information Retrieval community and a number of possible solutions have been proposed. Here I present a preliminary investigation into a new class of retrieval models that attempt to solve the term mismatch problem by exploiting complete or partial knowledge of term similarity in the term space. The use of term similarity enables to enhance classic retrieval models by taking into account non-matching terms. The theoretical advantages and drawbacks of these models are presented and compared with other models tackling the same problem. A preliminary experimental investigation into the performance gain achieved by exploiting term similarity with the proposed models is presented and discussed.     

面向信息检索的排除词识别研究

针对信息检索中存在的词语排除关系问题，给出排除词的定义并说明排除词在信息检索中的作用。指出排除词实质上是最大准交集型歧义切分字段的伪歧义切分所导致的，描述排除词的识别方法，并给出识别的结果，并在实际的信息检索平台上对排除词词库进行应用测评。     

Incremental Relevance Feedback in Japanese Text Retrieval

The application of relevance feedback techniques has been shown to improve retrieval performance for a number of information retrieval tasks. This paper explores incremental relevance feedback for ad hoc Japanese text retrieval; examining, separately and in combination, the utility of term reweighting and query expansion using a probabilistic retrieval model. Retrieval performance is evaluated in terms of standard precision-recall measures, and also using number-to-view graphs. Experimental results, on the standard BMIR-J2 Japanese language retrieval collection, show that both term reweighting and query expansion improve retrieval performance. This is reflected in improvements in both precision and recall, but also a reduction in the average number of documents which must be viewed to find a selected number of relevant items. In particular, using a simple simulation of user searching, incremental application of relevance information is shown to lead to progressively improved retrieval performance and an overall reduction in the number of documents that a user must view to find relevant ones.     

汉语文本特征词的抽取方法

本文从自动文摘的需求出发 ,探讨特征词自动抽取的方法和技术 ,设计并实现了两种不同的特征词自动抽取算法。这些方法对文本的自动分类和全文检索也有一定的借鉴意义。     

中文全文检索技术的研究及实现

本文设计了一个中文全文检索系统 ,在单汉字全文数据库的基础之上进行了全文检索的算法研究 ,提出了针对特定检索策略的计算公式。同时还对检索结果集的排序问题进行了讨论 ,并采用用户反馈信息量 ,使最后检出的结果在应用中不断得到优化     

基于关键词共现分析的检索结果聚类研究

随着互联网规模的急剧扩张,提升信息检索的效用变得相当困难.本文首先通过特定算法提取每篇文档的关键词,然后运用统计方法计量不同文档的共现关键词并形成相应的共现关键词标签矩阵,最后利用层次聚类算法对共现关键词标签进行聚类并形成相应的层次标签树来构造文档聚类束.该方法可以对源搜索引擎返回的结果进行有效的分类,使用户在更高主题层次上查看检索词的相关信息,准确地找到感兴趣的信息.通过与Lingo算法的比较,显示本文算法所得的标签更具可读性和概括性,同时F-measure评价指标也表明本算法在文本聚类的质量上有了一定的提升.     

一种面向中文信息检索的汉语自动分词方法

阐述信息检索对汉语分词技术的要求，分析中文信息检索与汉语分词技术结合过程中有待解决的关键问题，并重点针对这些要求及关键问题提出一种面向中文信息检索的汉语自动分词方法。     

A Fusion Approach to XML Structured Document Retrieval

In this paper we evaluate the application of data fusion or meta-search methods, combining different algorithms and XML elements, to content-oriented retrieval of XML structured data. The primary approach is the combination of a probabilistic methods using Logistic regression and the Okapi BM-25 algorithm for estimation of document relevance or XML element relevance, in conjunction with Boolean approaches for some query elements. In the evaluation we use the INEX XML test collection to examine the relative performance of individual algorithms and elements and compare these to the performance of the data fusion approaches.     

一种面向语义的信息检索方法

传统的信息检索技术忽视了语义对检索过程的影响,这是造成查准率不高的一个重要原因.论文提出了一种面向语义的信息检索方法,该方法强调使用基于知网的语义处理技术实现对用户查询需求和目标文档的语义标注,使用基于知网的词汇链技术实现对文档特征词汇的过滤.一方面可以实现语义级别的检索匹配,另一方面可以降低大量无关词对检索结果的干扰.论文描述了一个实现该方法的信息检索系统SOIRS,并且利用该系统与传统检索系统做了对比实验.实验结果表明面向语义的信息检索方法在查准率方面要明显优于传统信息检索方法.     

信息检索系统中的相关反馈技术

本文论述了布尔模型、向量空间模型以及概率模型中所采用的相关反馈技术,其中主要集中于检索词权值调整以及查询扩展等两项技术。作者还讨论了相关反馈技术对检索性能影响的评估方法,并提出了相关反馈在实际应用中需要解决的问题。     

词典类软件全文检索技术研究及实现

针对词典类软件的特点, 用四种不同技术实现支持全文检索的词典类软件, 包括单机版、企业版和网络版。分析各种技术的实现方法、技术特点和测试结果, 最终作以评价。     

On Event Spaces and Probabilistic Models in Information Retrieval

A basic notion of probability theory is the event space, on which the probability measure is defined. A probabilistic model needs an event space. However, some classes of events (which we may want to model probabilistically) exhibit structure which does not fit well into the traditional event space notion. A simple one-to-many example is discussed at length. The information retrieval case, involving queries, documents and relevance, is analysed. The event space issue makes for some difficulty in comparing different probabilistic models in IR.Revised version of a paper presented at the MF/IR Workshop, SIGIR 2002, Tampere, Finland, under the title On Bayesian models and event spaces in information retrieval.     

利用语音识别进行信息检索

本文先简要介绍语音识别技术的主要思想 ,然后着重讨论利用该技术进行网上文字、图像、语音和音频信息检索并针对目前受语音识别技术水平所限而带来的问题提出解决方法。     

Applying Machine Learning to Text Segmentation for Information Retrieval

We propose a self-supervised word segmentation technique for text segmentation in Chinese information retrieval. This method combines the advantages of traditional dictionary based, character based and mutual information based approaches, while overcoming many of their shortcomings. Experiments on TREC data show this method is promising. Our method is completely language independent and unsupervised, which provides a promising avenue for constructing accurate multi-lingual or cross-lingual information retrieval systems that are flexible and adaptive. We find that although the segmentation accuracy of self-supervised segmentation is not as high as some other segmentation methods, it is enough to give good retrieval performance. It is commonly believed that word segmentation accuracy is monotonically related to retrieval performance in Chinese information retrieval. However, for Chinese, we find that the relationship between segmentation and retrieval performance is in fact nonmonotonic; that is, at around 70% word segmentation accuracy an over-segmentation phenomenon begins to occur which leads to a reduction in information retrieval performance. We demonstrate this effect by presenting an empirical investigation of information retrieval on Chinese TREC data, using a wide variety of word segmentation algorithms with word segmentation accuracies ranging from 44% to 95%, including 70% word segmentation accuracy from our self-supervised word-segmentation approach. It appears that the main reason for the drop in retrieval performance is that correct compounds and collocations are preserved by accurate segmenters, while they are broken up by less accurate (but reasonable) segmenters, to a surprising advantage. This suggests that words themselves might be too broad a notion to conveniently capture the general semantic meaning of Chinese text. Our research suggests machine learning techniques can play an important role in building adaptable information retrieval systems and different evaluation standards for word segmentation should be given to different applications.     

基于查询词出现的相关度改进

对信息检索系统返回结果相关度的改进,一直是信息检索领域重要的研究内容。本文首先引入查询词出现信息的概念,随后给出了查询词出现权重的形式化表示,进而将其与BM25模型结合起来。对于查询词出现权重的计算,本文采用了两种方法,即线性加权方法和因数加权方法。我们通过在GOV2数据集上的实验发现,无论哪种方法,通过加入查询词出现权重,都可以有效的改进检索结果的相关度。实验显示,对于TREC 2005的查询,MAP值的改进达到15.78%,p@10的改进达到3468%。本文所描述的方法已经应用到TREC 2009的WebTrack中。     

关于创建我党民族问题文献的主题词检索体系之我见

针对专题文献馆藏--包括纸质、图片、音像、档案手稿、电子数据和实物文物等载体形式的中国共产党处理民族问题的珍贵文献,指出要从内容上把握其意义,将我党民族问题文献按照分析文献内容→确定主题,形成概念→选择主题词加以标识→转换成特定索引语言的步骤,创建一套检索系统,便于读者从主题词直接查找到相关文献。同时,也便于机读和电子查询。     

Using Corpus-Based Approaches in a System for Multilingual Information Retrieval

We present a system for multilingual information retrieval that allows users to formulate queries in their preferred language and retrieve relevant information from a collection containing documents in multiple languages. The system is based on a process of document level alignments, where documents of different languages are paired according to their similarity. The resulting mapping allows us to produce a multilingual comparable corpus. Such a corpus has multiple interesting applications. It allows us to build a data structure for query translation in cross-language information retrieval (CLIR). Moreover, we also perform pseudo relevance feedback on the alignments to improve our retrieval results. And finally, multiple retrieval runs can be merged into one unified result list. The resulting system is inexpensive, adaptable to domain-specific collections and new languages and has performed very well at the TREC-7 conference CLIR system comparison.