Crowdsourcing interactions: using crowdsourcing for evaluating interactive information retrieval systems

In the field of information retrieval (IR), researchers and practitioners are often faced with a demand for valid approaches to evaluate the performance of retrieval systems. The Cranfield experiment paradigm has been dominant for the in-vitro evaluation of IR systems. Alternative to this paradigm, laboratory-based user studies have been widely used to evaluate interactive information retrieval (IIR) systems, and at the same time investigate users’ information searching behaviours. Major drawbacks of laboratory-based user studies for evaluating IIR systems include the high monetary and temporal costs involved in setting up and running those experiments, the lack of heterogeneity amongst the user population and the limited scale of the experiments, which usually involve a relatively restricted set of users. In this paper, we propose an alternative experimental methodology to laboratory-based user studies. Our novel experimental methodology uses a crowdsourcing platform as a means of engaging study participants. Through crowdsourcing, our experimental methodology can capture user interactions and searching behaviours at a lower cost, with more data, and within a shorter period than traditional laboratory-based user studies, and therefore can be used to assess the performances of IIR systems. In this article, we show the characteristic differences of our approach with respect to traditional IIR experimental and evaluation procedures. We also perform a use case study comparing crowdsourcing-based evaluation with laboratory-based evaluation of IIR systems, which can serve as a tutorial for setting up crowdsourcing-based IIR evaluations.     

Evaluation of retrieval effectiveness with incomplete relevance data: Theoretical and experimental comparison of three measures

This paper investigates two relatively new measures of retrieval effectiveness in relation to the problem of incomplete relevance data. The measures, Bpref and RankEff, which do not take into account documents that have not been relevance judged, are compared theoretically and experimentally. The experimental comparisons involve a third measure, the well-known mean uninterpolated average precision. The results indicate that RankEff is the most stable of the three measures when the amount of relevance data is reduced, with respect to system ranking and absolute values. In addition, RankEff has the lowest error-rate.     

Evaluating epistemic uncertainty under incomplete assessments

The thesis of this study is to propose an extended methodology for laboratory based Information Retrieval evaluation under incomplete relevance assessments. This new methodology aims to identify potential uncertainty during system comparison that may result from incompleteness. The adoption of this methodology is advantageous, because the detection of epistemic uncertainty – the amount of knowledge (or ignorance) we have about the estimate of a system’s performance – during the evaluation process can guide and direct researchers when evaluating new systems over existing and future test collections. Across a series of experiments we demonstrate how this methodology can lead towards a finer grained analysis of systems. In particular, we show through experimentation how the current practice in Information Retrieval evaluation of using a measurement depth larger than the pooling depth increases uncertainty during system comparison.     

Linked Data Annotation Without the Pointy Brackets: Introducing Recogito 2

Recogito 2 is an open source annotation tool currently under development by Pelagios, an international initiative aimed at facilitating better linkages between online resources documenting the past. With Recogito 2, we aim to provide an environment for efficient semantic annotation—i.e., the task of enriching content with references to controlled vocabularies—in order to facilitate links between online data. At the same time, we address a perceived gap in the performance of existing tools, by emphasizing the development of mechanisms for manual intervention and editorial control that support the curation of quality data. While Recogito 2 provides an online workspace for general-purpose document annotation, it is particularly well-suited for geo-annotation, in other words annotating documents with references to gazetteers, and supports the annotation of both texts and images (i.e., digitized maps). Already available for testing at http://recogito.pelagios.org, its formal release to the public occurred in December 2016.     

The potential of Supplemental Instruction in engineering education – helping new students to adjust to and succeed in University studies

Supplemental Instruction (SI) is a programme that is attached to difficult courses with the objective of increasing student performance and retention. However, an SI programme also has the potential to increase overall student performance and retention during the first critical year if applied to introductory courses. In this study the latter objective is investigated in an engineering educational environment. The study shows that an SI programme attached to difficult first semester courses for new engineering students has substantial positive effects on both first-year student performance and retention. Both male and female students appear to benefit from attending SI to the same extent. Some potential reasons for these improved first-year student performances are that attendance at SI sessions appears to lead to improved self-confidence, a broader network of study partners, improved study strategies and problem-solving skills and an increased ability to critically review material and work with others.     

The creative use of companion values in environmental education and education for sustainable development: exploring the educative moment

Our paper addresses the emergence and evolution of values in educational settings. It builds upon and extends earlier work on companion meanings to develop a theory of the creative use of companion values and meanings in education. The recognition of companion values in educational practices highlight epistemological, ethical, and aesthetic transactions that occur in ways we characterize as ‘other than modern.’ Introducing the idea of educative moments allows us to identify situations where value spheres interpenetrate and interrogate each other in the meaning-making of students and teachers. These moments occur when students suddenly experience companion meanings and values such that teacher and student must deliberate together rather than the teacher dictating some dominating epistemological, ethical, or aesthetic value. This way, it is possible to accommodate critical and creative reflection in education where new values can emerge or evolve. We illustrate the theory by empirical examples from classroom conversations.     

Historical Views of Film

Ted Sennett's Warner Brothers Presents: The Most Exciting Years--From "The Jazz Singer" to "White Heat" (New Rochelle, N.Y.: Arlington House, 1971---$11.95)

Leif Furhammar and Folke Isaksson's Politics and Film (New York: Praeger, 1971---$12.50)

Peter Cowie (ed.) The Concise History of the Cinema: Volume I, Before 1940. (Cranbury, N.J.: A.S. Barnes, 1971---$3.50, paper)

Roy Armes' The French Cinema since 1946: Volume I, The Great Tradition (Cranbury, N. J.: A.S. Barnes, 1970---$2.95, paper)

Gordon Gow's Hollywood in the Fifties (Cranbury, N.J.: A.S. Barnes, 1971---$2.95, paper)

Edward Wagenknecht's The Movies in the Age of Innocence (New York: Ballentine, $2.95, paper)

Thorold Dickenson's A Discovery of Cinema (New York: Oxford University Press, 1971---$8.50 with a paperback edition available)

Rachael Low's classic History of the British Film (London: Allen and Unwin, 1971---price not known)

Bob Thomas' The Heart of Hollywood: A 50-year Pictorial History of the Film Capital and the Famed Motion Picture and Television Fund (Los Angeles: Price/Stern/Stern, 1971---$7.95)

Bob Harmon's Hollywood Panorama (New York: Dutton, 1971---$3.95, paper)     

The potential of supplemental instruction in engineering education: creating additional peer-guided learning opportunities in difficult compulsory courses for first-year students

Supplemental Instruction (SI) can be an efficient way of improving student success in difficult courses. Here, a study is made on SI attached to difficult first-year engineering courses. The results show that both the percentage of students passing a difficult first-year engineering course, and scores on the course exams are considerably higher for students attending SI, compared to students not attending. The study also shows that a higher percentage of female students attend SI, compared to male students. However, both genders seem to benefit to the same degree as a result of attending SI meetings. Also all students, independent of prior academic ability, benefit from attending SI. A qualitative study suggests that SI meetings provide elements important for understanding course material, which are missing from other scheduled learning opportunities in the courses.     

Validating simulated interaction for retrieval evaluation

A searcher’s interaction with a retrieval system consists of actions such as query formulation, search result list interaction and document interaction. The simulation of searcher interaction has recently gained momentum in the analysis and evaluation of interactive information retrieval (IIR). However, a key issue that has not yet been adequately addressed is the validity of such IIR simulations and whether they reliably predict the performance obtained by a searcher across the session. The aim of this paper is to determine the validity of the common interaction model (CIM) typically used for simulating multi-query sessions. We focus on search result interactions, i.e., inspecting snippets, examining documents and deciding when to stop examining the results of a single query, or when to stop the whole session. To this end, we run a series of simulations grounded by real world behavioral data to show how accurate and responsive the model is to various experimental conditions under which the data were produced. We then validate on a second real world data set derived under similar experimental conditions. We seek to predict cumulated gain across the session. We find that the interaction model with a query-level stopping strategy based on consecutive non-relevant snippets leads to the highest prediction accuracy, and lowest deviation from ground truth, around 9 to 15% depending on the experimental conditions. To our knowledge, the present study is the first validation effort of the CIM that shows that the model’s acceptance and use is justified within IIR evaluations. We also identify and discuss ways to further improve the CIM and its behavioral parameters for more accurate simulations.