A usage study of retrieval modalities for video shot retrieval

As an information medium, video offers many possible retrieval and browsing modalities, far more than text, image or audio. Some of these, like searching the text of the spoken dialogue, are well developed, others like keyframe browsing tools are in their infancy, and others not yet technically achievable. For those modalities for browsing and retrieval which we cannot yet achieve we can only speculate as to how useful they will actually be, but we do not know for sure. In our work we have created a system to support multiple modalities for video browsing and retrieval including text search through the spoken dialogue, image matching against shot keyframes and object matching against segmented video objects. For the last of these, automatic segmentation and tracking of video objects is a computationally demanding problem which is not yet solved for generic natural video material, and when it is then it is expected to open up possibilities for user interaction with objects in video, including searching and browsing. In this paper we achieve object segmentation by working in a closed domain of animated cartoons. We describe an interactive user experiment on a medium-sized corpus of video where we were able to measure users’ use of video objects versus other modes of retrieval during multiple-iteration searching. Results of this experiment show that although object searching is used far less than text searching in the first iteration of a user’s search it is a popular and useful search type once an initial set of relevant shots have been found.     

基于内容的图像信息检索综述

基于内容的图像检索技术，即从大量的静止或活动视频图像库中检索包含目标物体的图像（或视频片段），在高度信息化的今天，已成为内容图像库中图像信息组织和管理不可缺少的技术，本文介绍了基于内容检索技术的进展，并对其主要方法如基于颜色、形状、纹理等静止图像检索技术以及视频检索技术进行了讨论。     

信息检索相关性及其发展策略研究

[目的/意义]旨在为信息检索相关性研究提供参考。[方法/过程]以CNKI为数据源,采用定性方法,从信息检索的历史脉络和研究学派进行梳理总结,分析信息检索的影响因素和发展趋势。[结果/结论]信息检索相关性是用户、系统的相关性的综合体,任何一方都不能脱离。相关性应该是以用户为关键,系统为基础,研究用户与检索系统的交互、认知以及真实需求的描述与反馈。随着信息检索相关性研究的深入,系统观与用户观将会相互交融,检索技术与用户需求将会协调统一,共同推进检索相关性的发展。     

The usefulness of multimedia surrogates for making relevance judgments about digital video objects

Large collections of digital video are increasingly accessible. The large volume and range of available video demands search tools that allow people to browse and query easily and to quickly make sense of the videos behind the result sets. This study focused on the usefulness of several multimedia surrogates, in terms of effectiveness, efficiency, and user satisfaction. Three surrogates were evaluated and compared: a storyboard, a 7-second segment, and a fast forward. Thirty-six experienced users of digital video conducted searches on each of four systems: three incorporated one of the surrogates each, and the fourth made all three surrogates available. Participants judged the relevance of at least 10 items for each search based on the surrogate(s) available, then re-judged the relevance of two of those items based on viewing the full video. Transaction logs and post-search and post-session questionnaires provided data on user interactions, including relevance judgments, and user perceptions. All of the surrogates provided a basis for accurate relevance judgments, though they varied (in expected ways) in terms of their efficiency. User perceptions favored the system with all three surrogates available, even though it took longer to use; they found it easier to learn and easier to use, and it gave them more confidence in their judgments. Based on these results, we can conclude that it's important for digital video collections to provide multiple surrogates, each providing a different view of the video.     

Supporting exploratory video retrieval tasks with grouping and recommendation

In this paper, we present ViGOR (Video Grouping, Organisation and Recommendation), an exploratory video retrieval system. Exploratory video retrieval tasks are hampered by the lack of semantics associated to video and the overwhelming amount of video items stored in these types of collections (e.g. YouTube, MSN video, etc.). In order to help facilitate these exploratory video search tasks we present a system that utilises two complementary approaches: the first a new search paradigm that allows the semantic grouping of videos and the second the exploitation of past usage history in order to provide video recommendations. We present two types of recommendation techniques adapted to the grouping search paradigm: the first is a global recommendation, which couples the multi-faceted nature of explorative video retrieval tasks with the current user need of information in order to provide recommendations, and second is a local recommendation, which exploits the organisational features of ViGOR in order to provide more localised recommendations based on a specific aspect of the user task. Two user evaluations were carried out in order to (1) validate the new search paradigm provided by ViGOR, characterised by the grouping functionalities and (2) evaluate the usefulness of the proposed recommendation approaches when integrated into ViGOR. The results of our evaluations show (1) that the grouping, organisational and recommendation functionalities can result in an improvement in the users’ search performance without adversely impacting their perceptions of the system and (2) that both recommendation approaches are relevant to the users at different stages of their search, showing the importance of using multi-faceted recommendations for video retrieval systems and also illustrating the many uses of collaborative recommendations for exploratory video search tasks.     

User relevance criteria choices and the information search process

Relevance judgments occur within an information search process, where time, context and situation can impact the judgments. The determination of relevance is dependent on a number of factors and variables which include the criteria used to determine relevance. The relevance judgment process and the criteria used to make those judgments are manifestations of the cognitive changes which occur during the information search process.Understanding why these relevance criteria choices are made, and how they vary over the information search process can provide important information about the dynamic relevance judgment process. This information can be used to guide the development of more adaptive information retrieval systems which respond to the cognitive changes of users during the information search process.The research data analyzed here was collected in two separate studies which examined a subject’s relevance judgment over an information search process. Statistical analysis was used to examine these results and determine if there were relationships between criteria selections, relevance judgments, and the subject’s progression through the information search process. Findings confirm and extend findings of previous studies, providing strong statistical evidence of an association between the information search process and the choices of relevance criteria by users, and identifying specific changes in the user preferences for specific criteria over the course of the information search process.     

A risk minimization framework for information retrieval

This paper presents a probabilistic information retrieval framework in which the retrieval problem is formally treated as a statistical decision problem. In this framework, queries and documents are modeled using statistical language models, user preferences are modeled through loss functions, and retrieval is cast as a risk minimization problem. We discuss how this framework can unify existing retrieval models and accommodate systematic development of new retrieval models. As an example of using the framework to model non-traditional retrieval problems, we derive retrieval models for subtopic retrieval, which is concerned with retrieving documents to cover many different subtopics of a general query topic. These new models differ from traditional retrieval models in that they relax the traditional assumption of independent relevance of documents.     

A study of awareness in multimedia search

Awareness of another’s activity is an important aspect of facilitating collaboration between users, enabling an “understanding of the activities of others” (Dourish & Bellotti, 1992). In this paper we investigate the role of awareness and its effect on search performance and behaviour in collaborative multimedia retrieval. We focus on the scenario where two users are searching at the same time on the same task, and via an interface, can see the activity of the other user. The main research question asks: does awareness of another searcher aid a user when carrying out a multimedia search session?To encourage awareness, an experimental study was designed where two users were asked to compete to find as many relevant video shots as possible under different awareness conditions. These were individual search (no awareness), Mutual awareness (where both users could see the other’s search screen), and unbalanced awareness (where one user is able to see the other’s screen, but not vice-versa). Twelve pairs of users were recruited, and the four worst performing TRECVID 2006 search topics were used as search tasks, under four different awareness conditions. We present the results of this study, followed by a discussion of the implications for multimedia information retrieval systems.     

A review of ranking approaches for semantic search on Web

With ever increasing information being available to the end users, search engines have become the most powerful tools for obtaining useful information scattered on the Web. However, it is very common that even most renowned search engines return result sets with not so useful pages to the user. Research on semantic search aims to improve traditional information search and retrieval methods where the basic relevance criteria rely primarily on the presence of query keywords within the returned pages. This work is an attempt to explore different relevancy ranking approaches based on semantics which are considered appropriate for the retrieval of relevant information. In this paper, various pilot projects and their corresponding outcomes have been investigated based on methodologies adopted and their most distinctive characteristics towards ranking. An overview of selected approaches and their comparison by means of the classification criteria has been presented. With the help of this comparison, some common concepts and outstanding features have been identified.     

相关性基础理论及其在检索建模中的作用研究

本文是对信息检索的一次理论研究。在总结了已有相关性研究的基础上,对信息检索模型之中的相关性因素做了系统梳理,认为现有信息检索模型中的系统相关性因素体现得不十分明显,同时用户相关性的因素没有很好地纳入系统相关性和系统设计研究之中。与相关性有关的概念是相似,它存在于文本空间之中。与相关相比,相似具有更好的数学特征。查询是相关判断的过程载体,它同时也是信息检索研究的瓶颈。寻找更为恰当的相关性的隐喻则需要跳出文本的藩篱,从更为深入的模式相关切入,探索更为复杂的相关性因素。     

语义对等网环境下基于节点知识地图的用户模型构建

利用用户兴趣可以有效地提高语义对等网环境下信息检索的效率,如何准确构建用户兴趣模型是关键。鉴于本地节点的信息资源可以有效反映用户兴趣,文章提出利用组织与管理本地节点资源的知识地图构建节点用户兴趣模型。主要思路是利用本体描述语言OWL描述本地知识实体及其关系,形成反映节点用户全局知识结构的知识地图,依据支持向量机分类原理从知识地图抽取出的兴趣特征训练集挖掘用户兴趣,最终形成用户兴趣模型并以兴趣描述文档的形式保存。     

SLTFNet: A spatial and language-temporal tensor fusion network for video moment retrieval

This paper focuses on temporal retrieval of activities in videos via sentence queries. Given a sentence query describing an activity, temporal moment retrieval aims at localizing the temporal segment within the video that best describes the textual query. This is a general yet challenging task as it requires the comprehending of both video and language. Existing research predominantly employ coarse frame-level features as the visual representation, obfuscating the specific details (e.g., the desired objects “girl”, “cup” and action “pour”) within the video which may provide critical cues for localizing the desired moment. In this paper, we propose a novel Spatial and Language-Temporal Tensor Fusion (SLTF) approach to resolve those issues. Specifically, the SLTF method first takes advantage of object-level local features and attends to the most relevant local features (e.g., the local features “girl”, “cup”) by spatial attention. Then we encode the sequence of the local features on consecutive frames by employing LSTM network, which can capture the motion information and interactions among these objects (e.g., the interaction “pour” involving these two objects). Meanwhile, language-temporal attention is utilized to emphasize the keywords based on moment context information. Thereafter, a tensor fusion network learns both the intra-modality and inter-modality dynamics, which can enhance the learning of moment-query representation. Therefore, our proposed two attention sub-networks can adaptively recognize the most relevant objects and interactions in the video, and simultaneously highlight the keywords in the query for retrieving the desired moment. Experimental results on three public benchmark datasets (obtained from TACOS, Charades-STA, and DiDeMo) show that the SLTF model significantly outperforms current state-of-the-art approaches, and demonstrate the benefits produced by new technologies incorporated into SLTF.     

基于上下文的Web即时信息检索

用户当前正在浏览的网页内容有助于说明用户的即时信息需求.在现有相关研究的基础上提出了一种基于上下文的Web即时信息检索方法,该方法允许用户从正在浏览的网页中选择一段文本作为原始检索条件,由检索系统从其上下文中提取一级扩展词和二级扩展词来形成新的检索条件进行检索,并将检索结果按相似度从大到小的顺序呈现给用户.     

Adaptive relevance feedback method of extended Boolean model using hierarchical clustering techniques

The relevance feedback process uses information obtained from a user about a set of initially retrieved documents to improve subsequent search formulations and retrieval performance. In extended Boolean models, the relevance feedback implies not only that new query terms must be identified and re-weighted, but also that the terms must be connected with Boolean And/Or operators properly. Salton et al. proposed a relevance feedback method, called DNF (disjunctive normal form) method, for a well established extended Boolean model. However, this method mainly focuses on generating Boolean queries but does not concern about re-weighting query terms. Also, this method has some problems in generating reformulated Boolean queries. In this study, we investigate the problems of the DNF method and propose a relevance feedback method using hierarchical clustering techniques to solve those problems. We also propose a neural network model in which the term weights used in extended Boolean queries can be adjusted by the users’ relevance feedbacks.     

A split-list approach for relevance feedback in information retrieval

In this paper we present a new algorithm for relevance feedback (RF) in information retrieval. Unlike conventional RF algorithms which use the top ranked documents for feedback, our proposed algorithm is a kind of active feedback algorithm which actively chooses documents for the user to judge. The objectives are (a) to increase the number of judged relevant documents and (b) to increase the diversity of judged documents during the RF process. The algorithm uses document-contexts by splitting the retrieval list into sub-lists according to the query term patterns that exist in the top ranked documents. Query term patterns include a single query term, a pair of query terms that occur in a phrase and query terms that occur in proximity. The algorithm is an iterative algorithm which takes one document for feedback in each of the iterations. We experiment with the algorithm using the TREC-6, -7, -8, -2005 and GOV2 data collections and we simulate user feedback using the TREC relevance judgements. From the experimental results, we show that our proposed split-list algorithm is better than the conventional RF algorithm and that our algorithm is more reliable than a similar algorithm using maximal marginal relevance.     

Automatic term class construction using relevance—A summary of work in automatic pseudoclassification

Term classifications and thesauri can be used for many purposes in automatic information retrieval. Normally a thesaurus is generated manually by subject experts: alternatively, the associations between the terms can be obtained automatically by using the occurrence characteristics of the terms across the documents of a collection. A third possibility consists in taking into account user relevance assessments of certain documents with respect to certain queries in order to build term classes designed to retrieve the relevant documents and simultaneously to reject the nonrelevant documents. This last strategy, known as pseudoclassification, produces a user-dependent term classification.A number of pseudoclassification studies are summarized in the present report, and conclusions are reached concerning the effectiveness and feasibility of constructing term classifications based on human relevance assessments.     

一种新的基于SVM相关反馈的图像检索算法

提出了一种结合颜色和形状特征的图像检索方法,针对传统基于内容图像检索不能很好满足用户需求的问题,提出了一种基于支持向量机（SVM）的相关反馈算法来捕捉用户的检索意图。实验结果证明,算法能发挥用户在检索过程中的作用,具有较好检索性能。     

Evaluating combinations of ranked lists and visualizations of inter-document similarity

We are interested in how ideas from document clustering can be used to improve the retrieval accuracy of ranked lists in interactive systems. In particular, we are interested in ways to evaluate the effectiveness of such systems to decide how they might best be constructed. In this study, we construct and evaluate systems that present the user with ranked lists and a visualization of inter-document similarities. We first carry out a user study to evaluate the clustering/ranked list combination on instance-oriented retrieval, the task of the TREC-6 Interactive Track. We find that although users generally prefer the combination, they are not able to use it to improve effectiveness. In the second half of this study, we develop and evaluate an approach that more directly combines the ranked list with information from inter-document similarities. Using the TREC collections and relevance judgments, we show that it is possible to realize substantial improvements in effectiveness by doing so, and that although users can use the combined information effectively, the system can provide hints that substantially improve on the user's solo effort. The resulting approach shares much in common with an interactive application of incremental relevance feedback. Throughout this study, we illustrate our work using two prototype systems constructed for these evaluations. The first, AspInQuery, is a classic information retrieval system augmented with a specialized tool for recording information about instances of relevance. The other system, Lighthouse, is a Web-based application that combines a ranked list with a portrayal of inter-document similarity. Lighthouse can work with collections such as TREC, as well as the results of Web search engines.     

A retrospective study of a hybrid document-context based retrieval model

This paper describes our novel retrieval model that is based on contexts of query terms in documents (i.e., document contexts). Our model is novel because it explicitly takes into account of the document contexts instead of implicitly using the document contexts to find query expansion terms. Our model is based on simulating a user making relevance decisions, and it is a hybrid of various existing effective models and techniques. It estimates the relevance decision preference of a document context as the log-odds and uses smoothing techniques as found in language models to solve the problem of zero probabilities. It combines these estimated preferences of document contexts using different types of aggregation operators that comply with different relevance decision principles (e.g., aggregate relevance principle). Our model is evaluated using retrospective experiments (i.e., with full relevance information), because such experiments can (a) reveal the potential of our model, (b) isolate the problems of the model from those of the parameter estimation, (c) provide information about the major factors affecting the retrieval effectiveness of the model, and (d) show that whether the model obeys the probability ranking principle. Our model is promising as its mean average precision is 60–80% in our experiments using different TREC ad hoc English collections and the NTCIR-5 ad hoc Chinese collection. Our experiments showed that (a) the operators that are consistent with aggregate relevance principle were effective in combining the estimated preferences, and (b) that estimating probabilities using the contexts in the relevant documents can produce better retrieval effectiveness than using the entire relevant documents.