The TREC interactive track: an annotated bibliography

The study of interactive information retrieval (IR) has been a small but constant part of the Text REtrieval Conferences (TREC 1–8) from the beginning. The main arena for such work has been the Interactive Track (TREC 3–8). This report presents a bibliography of those efforts together with a summary of the evolving experimental framework within which the studies took place.     

Evaluating combinations of ranked lists and visualizations of inter-document similarity

We are interested in how ideas from document clustering can be used to improve the retrieval accuracy of ranked lists in interactive systems. In particular, we are interested in ways to evaluate the effectiveness of such systems to decide how they might best be constructed. In this study, we construct and evaluate systems that present the user with ranked lists and a visualization of inter-document similarities. We first carry out a user study to evaluate the clustering/ranked list combination on instance-oriented retrieval, the task of the TREC-6 Interactive Track. We find that although users generally prefer the combination, they are not able to use it to improve effectiveness. In the second half of this study, we develop and evaluate an approach that more directly combines the ranked list with information from inter-document similarities. Using the TREC collections and relevance judgments, we show that it is possible to realize substantial improvements in effectiveness by doing so, and that although users can use the combined information effectively, the system can provide hints that substantially improve on the user's solo effort. The resulting approach shares much in common with an interactive application of incremental relevance feedback. Throughout this study, we illustrate our work using two prototype systems constructed for these evaluations. The first, AspInQuery, is a classic information retrieval system augmented with a specialized tool for recording information about instances of relevance. The other system, Lighthouse, is a Web-based application that combines a ranked list with a portrayal of inter-document similarity. Lighthouse can work with collections such as TREC, as well as the results of Web search engines.     

Review and discussion: E-learning for academia and industry

This paper presents a high level review and discussion about e-learning and proposes the use of interactive learning as a recommended method for staff training in industry and academia. Interactive learning is focused on the integrated e-learning and face-to-face learning to ensure that the process of learning can stimulate learners' interests, report their progress and have tutors to provide their feedback and guide learners to the expected targets. Learning activities and varieties have been illustrated with discussion about how industry and academia can use interactive learning. Five successful examples of interactive learning to demonstrate the effectiveness of interactive learning. Positive impacts have been reported in RBS, SMEs using SAP, University of Cambridge, University of Greenwich and Leeds Beckett University to support the positive outcomes for learners and trainers. Future directions have been discussed, particularly the use of emerging services can enhance the learning experience and satisfaction for learners and trainers.     

Using clustering and classification approaches in interactive retrieval

Satisfying non-trivial information needs involves collecting information from multiple resources, and synthesizing an answer that organizes that information. Traditional recall/precision-oriented information retrieval focuses on just one phase of that process: how to efficiently and effectively identify documents likely to be relevant to a specific, focused query. The TREC Interactive Track has as its goal the location of documents that pertain to different instances of a query topic, with no reward for duplicated coverage of topic instances. This task is similar to the task of organizing answer components into a complete answer. Clustering and classification are two mechanisms for organizing documents into groups. In this paper, we present an ongoing series of experiments that test the feasibility and effectiveness of using clustering and classification as an aid to instance retrieval and, ultimately, answer construction. Our results show that users prefer such structured presentations of candidate result set to a list-based approach. Assessment of the structured organizations based on the subjective judgement of the experiment subjects suggests that the structured organization can be more effective; however, assessment based on objective judgements shows mixed results. These results indicate that a full determination of the success of the approach depends on assessing the quality of the final answers generated by users, rather than on performance during the intermediate stages of answer construction.     

Gene ontology annotation as text categorization: An empirical study

Gene ontology (GO) consists of three structured controlled vocabularies, i.e., GO domains, developed for describing attributes of gene products, and its annotation is crucial to provide a common gateway to access different model organism databases. This paper explores an effective application of text categorization methods to this highly practical problem in biology. As a first step, we attempt to tackle the automatic GO annotation task posed in the Text Retrieval Conference (TREC) 2004 Genomics Track. Given a pair of genes and an article reference where the genes appear, the task simulates assigning GO domain codes. We approach the problem with careful consideration of the specialized terminology and pay special attention to various forms of gene synonyms, so as to exhaustively locate the occurrences of the target gene. We extract the words around the spotted gene occurrences and used them to represent the gene for GO domain code annotation. We regard the task as a text categorization problem and adopt a variant of kNN with supervised term weighting schemes, making our method among the top-performing systems in the TREC official evaluation. Furthermore, we investigate different feature selection policies in conjunction with the treatment of terms associated with negative instances. Our experiments reveal that round-robin feature space allocation with eliminating negative terms substantially improves performance as GO terms become specific.     

Support for interactive document selection in cross-language information retrieval

As digital libraries grow to global scale, the provision of interactive access to content in many languages will become increasingly important. In systems that support query-based searching, the presence of multilingual content will affect both the search technology itself and the user interface components that support query formulation, document selection and query refinement. This article describes the interactions among these components and presents a practical way of evaluating the adequacy of the selection interface. A categorization-based model for the user's selection process is presented and an experimental methodology suitable for obtaining process centered results in this context is developed. The methodology is applied to assess the adequacy of a selection interface in which multiple candidate translations for a term can be simultaneously presented. The results indicate that the modeled selection process is somewhat less effective when users are presented with multi-translation glosses from Japanese to English rather than materials generated originally in English, but that users with access to the gloss translations substantially outperform a Naive Bayes classification algorithm.     

移动学习互动性的设计分析

本文通过对移动学习、协同学习与联通主义等概念的介绍,结合目前国内研究发展现状,对互动性设计的必要性、可行性和主要方向进行了阐述。3G/4G网络的发展及智能手机的普及为移动学习提供了更加良好的社交平台,联通主义的学习观也证明建立互动管道比管道中的内容更加重要。     

心理学在社会治理中的作用

推进国家治理体系和治理能力现代化是我国全面深化改革的一个总体目标。对此心理学的研究可以提供重要的科学支撑。一方面是因为心理学具有社会科学的学科性质,能够解释很多社会过程和社会问题的作用机制;另一方面,是因为社会治理理念所倡导的多元主体、协商民主、消解矛盾和精细化的思想,都与心理学有密切的联系,需要对人的心理和行为有深刻的理解。未来学界应更多致力于有关社会治理的心理学研究,而且该领域的研究应注重理论本土化和方法多元化,坚持多学科协同创新的定位,处理好基础研究和应用研究的关系,以及科学研究与人文关怀的关系。     

Seeking and implementing automated assistance during the search process

Searchers seldom make use of the advanced searching features that could improve the quality of the search process because they do not know these features exist, do not understand how to use them, or do not believe they are effective or efficient. Information retrieval systems offering automated assistance could greatly improve search effectiveness by suggesting or implementing assistance automatically. A critical issue in designing such systems is determining when the system should intervene in the search process. In this paper, we report the results of an empirical study analyzing when during the search process users seek automated searching assistance from the system and when they implement the assistance. We designed a fully functional, automated assistance application and conducted a study with 30 subjects interacting with the system. The study used a 2G TREC document collection and TREC topics. Approximately 50% of the subjects sought assistance, and over 80% of those implemented that assistance. Results from the evaluation indicate that users are willing to accept automated assistance during the search process, especially after viewing results and locating relevant documents. We discuss implications for interactive information retrieval system design and directions for future research.     

关于长江三角洲地区推进稻田退水零直排工程建设的建议

长江三角洲(以下简称"长三角")是我国重要商品粮基地,但近年来农业面源污染问题日益突出。文章结合实地调查、政策解析、工程设计、效益评估等方式,提出了关于推进我国长三角地区稻田退水零直排工程建设的建议,明确了稻田退水零直排工程建设的政策适应性,以及长三角地区实施稻田退水零直排的重要性,对稻田退水零直排工程建设的基本原则、设计要求、模式选择与投资成本、可预期的生态经济效益进行了全面分析,以期为我国面源污染防治与农业绿色优质发展提供决策支持。     

Topics of the nationwide phone-ins with Vladimir Putin and their role for public support and Russian economy

The addresses of national leaders can affect their public support and spur changes in the country's economy. To date, very few studies exist establishing these relationships, and no research has been done on the addresses from Vladimir Putin. In this paper we fill this knowledge gap by analysing the nationwide phone-ins of Putin, a special annual format where he addresses the public, and using structural topic modelling studying their topics over time. Furthermore, we relate these topics to public approval of the president and the government as well as to some Russian macroeconomic indicators such as inflation and budget expenditures. Based on our data containing 1938 responses and almost 250 thousand words, we identify 16 main topics covering areas from healthcare and education through economics to elections and legislation. We find that the topic of foreign affairs has gained in popularity over time the most (from around 4.5% at the beginning to more than 10% starting from 2014). Another topic, consistently gaining weight in the president's statements, is related to solving particular problems of the general public (from 8% to 12.5%) and is significantly correlated with subsequent decrease in the country's unemployment (Pearson's correlation coefficient -0.502). We also find that when the government's support is decreasing, Putin tends to discuss more socially significant topics (e.g., inflation, healthcare, Pearson's coef. around -0.5), while when the support is rising, he speaks more about foreign affairs (Pearson's coef. 0.773). Our study provides first evidence that Vladimir Putin may adapt the content of his phone-in meetings to gather public support and influence the country's economy.     

A domain-independent approach to finding related entities

We propose an approach to the retrieval of entities that have a specific relationship with the entity given in a query. Our research goal is to investigate whether related entity finding problem can be addressed by combining a measure of relatedness of candidate answer entities to the query, and likelihood that the candidate answer entity belongs to the target entity category specified in the query. An initial list of candidate entities, extracted from top ranked documents retrieved for the query, is refined using a number of statistical and linguistic methods. The proposed method extracts the category of the target entity from the query, identifies instances of this category as seed entities, and computes similarity between candidate and seed entities. The evaluation was conducted on the Related Entity Finding task of the Entity Track of TREC 2010, as well as the QA list questions from TREC 2005 and 2006. Evaluation results demonstrate that the proposed methods are effective in finding related entities.     

Improving patient record search: A meta-data based approach

The International Classification of Diseases (ICD) is a type of meta-data found in many Electronic Patient Records. Research to explore the utility of these codes in medical Information Retrieval (IR) applications is new, and many areas of investigation remain, including the question of how reliable the assignment of the codes has been. This paper proposes two uses of the ICD codes in two different contexts of search: Pseudo-Relevance Judgments (PRJ) and Pseudo-Relevance Feedback (PRF). We find that our approach to evaluate the TREC challenge runs using simulated relevance judgments has a positive correlation with the TREC official results, and our proposed technique for performing PRF based on the ICD codes significantly outperforms a traditional PRF approach. The results are found to be consistent over the two years of queries from the TREC medical test collection.     

TREC interactive with Cheshire II

This paper describes the UC Berkeley's participation in the TREC-6, 7 and 8 interactive track experiments. In these three studies 24 searchers (four in TREC-6, eight in TREC-7, and 12 in TREC-8) conducted a total of 160 searches, half on the Cheshire II system and the other half on the ZPRISE system. In TREC-7 and TREC-8 questionnaires were administered to gather information about basic demographic and searching experience, about each search, about each of the systems, and finally, about the user's perceptions of the systems. In this paper I will briefly describe the systems used in the study and how they differ in design goals and implementation. The results of the interactive track evaluations and the information derived from the questionnaires are then discussed and plans for further research are considered.     

Navigating information spaces: A case study of related article search in PubMed

The concept of an “information space” provides a powerful metaphor for guiding the design of interactive retrieval systems. We present a case study of related article search, a browsing tool designed to help users navigate the information space defined by results of the PubMed^® search engine. This feature leverages content-similarity links that tie MEDLINE^® citations together in a vast document network. We examine the effectiveness of related article search from two perspectives: a topological analysis of networks generated from information needs represented in the TREC 2005 genomics track and a query log analysis of real PubMed users. Together, data suggest that related article search is a useful feature and that browsing related articles has become an integral part of how users interact with PubMed.     

Mix and match: combining terms and operators for successful Web searches

This paper presents a detailed analysis of the structure and components of queries written by experimental participants in a study that manipulated two factors found to affect end-user information retrieval performance: training in Boolean logic and the type of search interface. As reported previously, we found that both Boolean training and the use of an assisted interface improved the participants' ability to find correct responses to information requests. Here, we examine the impact of these training and interface manipulations on the Boolean operators and search terms that comprise the submitted queries. Our analysis shows that both Boolean training and the use of an assisted interface improved the participants' ability to correctly utilize various operators. An unexpected finding is that this training also had a positive impact on term selection. The terms and, to a lesser extent, the operators comprising a query were important factors affecting the participants' performance in query tasks. Our findings demonstrate that even small training interventions can improve the users' search performance and highlight the need for additional information retrieval research into how search interfaces can provide superior support to today's untrained users of the Web.     

大学课堂教学交互方法研究

设计合理且反馈及时的课堂交互可以有效提高教学效率和教学效果,本文作者详细分析各种课堂交互方法的特点和适用场景,结合访学体会和任教课程实际情况进行深入思考,总结出适合理工科基础课的课堂交互方法设计原则,并结合当前无处不在的信息技术给出实际课堂交互操作方法,在“概率论与数理统计”和“数字电路”课程的课堂教学和实验课堂教学中进行实践。     

Improving Information Retrieval Performance on OCRed Text in the Absence of Clean Text Ground Truth

OCR errors in text harm information retrieval performance. Much research has been reported on modelling and correction of Optical Character Recognition (OCR) errors. Most of the prior work employ language dependent resources or training texts in studying the nature of errors. However, not much research has been reported that focuses on improving retrieval performance from erroneous text in the absence of training data. We propose a novel approach for detecting OCR errors and improving retrieval performance from the erroneous corpus in a situation where training samples are not available to model errors. In this paper we propose a method that automatically identifies erroneous term variants in the noisy corpus, which are used for query expansion, in the absence of clean text. We employ an effective combination of contextual information and string matching techniques. Our proposed approach automatically identifies the erroneous variants of query terms and consequently leads to improvement in retrieval performance through query expansion. Our proposed approach does not use any training data or any language specific resources like thesaurus for identification of error variants. It also does not expend any knowledge about the language except that the word delimiter is blank space. We have tested our approach on erroneous Bangla (Bengali in English) and Hindi FIRE collections, and also on TREC Legal IIT CDIP and TREC 5 Confusion track English corpora. Our proposed approach has achieved statistically significant improvements over the state-of-the-art baselines on most of the datasets.     

Federalism in the Regulation of Chemical Pollutants in Australia

In response to the growth of the environment movement and increasing citizen concern for the environment, the Commonwealth and the Australian states moved to set up government departments of environment, or Environment Protection Agencies, in the early 1970s. At first the issues which engaged the public were 'green' ones, involving land and marine degradation, biodiversity (a term not then in use), and activities such as logging, mining and the generation of hydro-electric power. These were soon joined on the public agenda, however, by concerns over 'brown' issues such as chemical pollution and the management of hazardous chemicals and chemical wastes, but these were less well understood by the general public than the more visible 'green' issues. At this time, also, Australian governments took significant steps to achieve nationally consistent regulation through the work of ministerial councils, notably the Australian Environment Council (AEC) and subsequently the Australian and New Zealand Environment and Conservation Council (ANZECC). A further wave of environmentalism in the late 1980s and early 1990s, coupled with further development of the federalism model for environmental regulation that flowed from high-level inter-government meetings that were formalised as the Council of Australian Governments (COAG), gave rise to the formation of the National Environment Protection Council (NEPC). This body, while not giving more power to the Commonwealth, achieves national harmonisation of regulations through slightly more coercive processes than are available to the older-style ministerial councils. These continue to exist, however, operating in parallel with NEPC. For chemicals coming into use, there are national schemes under which the Commonwealth, states and territories regulated industrial, agricultural and veterinary chemicals, therapeutic substances, and food additives. Most 'brown' issues, however, remain with the successor to ANZECC, the Environment Protection and Heritage Council, as part of which NEPC continues to develop national approaches. In both periods of rapid change, there was a coincidence of environmentalism and federalism, but international developments were also important--in the first phase the United Nations' first 'environment' meeting, held in 1972 in Stockholm, and in the second the developments flowing from the so-called 'Earth Summit' held in Rio de Janeiro in 1992. Although public environmentalism continues to be mainly concerned with 'green' issues, the 'brown' issues remain a focus for governments and some non-government organisations.