Statistical biases in Information Retrieval metrics for recommender systems

There is an increasing consensus in the Recommender Systems community that the dominant error-based evaluation metrics are insufficient, and mostly inadequate, to properly assess the practical effectiveness of recommendations. Seeking to evaluate recommendation rankings—which largely determine the effective accuracy in matching user needs—rather than predicted rating values, Information Retrieval metrics have started to be applied for the evaluation of recommender systems. In this paper we analyse the main issues and potential divergences in the application of Information Retrieval methodologies to recommender system evaluation, and provide a systematic characterisation of experimental design alternatives for this adaptation. We lay out an experimental configuration framework upon which we identify and analyse specific statistical biases arising in the adaptation of Information Retrieval metrics to recommendation tasks, namely sparsity and popularity biases. These biases considerably distort the empirical measurements, hindering the interpretation and comparison of results across experiments. We develop a formal characterisation and analysis of the biases upon which we analyse their causes and main factors, as well as their impact on evaluation metrics under different experimental configurations, illustrating the theoretical findings with empirical evidence. We propose two experimental design approaches that effectively neutralise such biases to a large extent. We report experiments validating our proposed experimental variants, and comparing them to alternative approaches and metrics that have been defined in the literature with similar or related purposes.     

An assessment model for virtual communities of practice: a study in the oil and gas industry

This paper proposes a model to assess the level of maturity in Virtual Communities of Practice (VCoPs). The model is based on a theoretical construction attained from the analysis of previous frameworks proposed in literature. Using this construction, we have proposed eight main organizational, managerial and technological levers to improve the performance of VCoPs. Additionally, the model presents the analysis of four performance metrics obtained from these VCoP management levers. The model presented can be used as a benchmarking tool for analysing how companies perform in their management of VCoPs. After developing the model, we applied it to a benchmarking study of four global oil and gas companies. Our results include a performance comparison among these companies as well as the main practices and technologies they use to achieve success in the management of VCoP.     

The strange case of reproducibility versus representativeness in contextual suggestion test collections

The most common approach to measuring the effectiveness of Information Retrieval systems is by using test collections. The Contextual Suggestion (CS) TREC track provides an evaluation framework for systems that recommend items to users given their geographical context. The specific nature of this track allows the participating teams to identify candidate documents either from the Open Web or from the ClueWeb12 collection, a static version of the web. In the judging pool, the documents from the Open Web and ClueWeb12 collection are distinguished. Hence, each system submission should be based only on one resource, either Open Web (identified by URLs) or ClueWeb12 (identified by ids). To achieve reproducibility, ranking web pages from ClueWeb12 should be the preferred method for scientific evaluation of CS systems, but it has been found that the systems that build their suggestion algorithms on top of input taken from the Open Web achieve consistently a higher effectiveness. Because most of the systems take a rather similar approach to making CSs, this raises the question whether systems built by researchers on top of ClueWeb12 are still representative of those that would work directly on industry-strength web search engines. Do we need to sacrifice reproducibility for the sake of representativeness? We study the difference in effectiveness between Open Web systems and ClueWeb12 systems through analyzing the relevance assessments of documents identified from both the Open Web and ClueWeb12. Then, we identify documents that overlap between the relevance assessments of the Open Web and ClueWeb12, observing a dependency between relevance assessments and the source of the document being taken from the Open Web or from ClueWeb12. After that, we identify documents from the relevance assessments of the Open Web which exist in the ClueWeb12 collection but do not exist in the ClueWeb12 relevance assessments. We use these documents to expand the ClueWeb12 relevance assessments. Our main findings are twofold. First, our empirical analysis of the relevance assessments of 2  years of CS track shows that Open Web documents receive better ratings than ClueWeb12 documents, especially if we look at the documents in the overlap. Second, our approach for selecting candidate documents from ClueWeb12 collection based on information obtained from the Open Web makes an improvement step towards partially bridging the gap in effectiveness between Open Web and ClueWeb12 systems, while at the same time we achieve reproducible results on well-known representative sample of the web.     

A Lost Decade? Laszlo Radvanyi and the Origins of Public Opinion Research in Mexico, 1941-1952

This article documents public opinion research activities inMexico in the 1940s and the role played by Hungarian professorLászló Radványi, who immigrated to thatcountry at the height of World War II. Our research relies onseveral of Radványi's publications archived in differentcountries, as well as on interviews with family, acquaintances,and experts on the work of his wife, the German poet Anna Seghers.During his years in Mexico, Radványi founded the ScientificInstitute of Mexican Public Opinion, in 1941, and the InternationalJournal of Opinion and Attitude Research, in 1947—a forefatherof today's IJPOR. He was also a founding member of WAPOR. Hisearly "sample surveys" raised important methodological issuesand recorded opinion results that reflect the vibrant timesof war and policy making in a modernizing country. However,Radványi's contribution to the profession has been virtuallyforgotten. Until now, accounts about how public opinion researchbegan in Mexico either ignored Radványi's works or reducedhis ten years of survey research to a single footnote. Thisarticle is an attempt to fill this enormous omission and highlightsome of Radványi's contributions to these early stagesof survey research.     

Effect of age on 16.1-km time-trial performance

In this study, we assessed the performance of trained senior (n = 6) and veteran (n = 6) cyclists (mean age 28 years, s = 3 and 57 years, s = 4 respectively). Each competitor completed two cycling tests, a ramped peak aerobic test and an indoor 16.1-km time-trial. The tests were performed using a Kingcycle ergometer with the cyclists riding their own bicycle fitted with an SRM powermeter. Power output, heart rate, and gas exchange variables were recorded continuously and blood lactate concentration [HLa] was assessed 3 min after the peak ramped test and at 2.5-min intervals during the time-trial. Peak values for power output (RMP(max)), heart rate (HR(peak)), oxygen uptake (VO2(peak)), and ventilation (V(Epeak)) attained during the ramped test were higher in the senior group (P < 0.05), whereas [HLa](peak), RER(peak), V(E): VO2(peak), and economy(peak) were similar between groups (P > 0.05). Time-trial values (mean for duration of race) for power output (W(TT)), heart rate (HR(TT)), VO2 (VO(2TT)), and V(E) (V(ETT)) were higher in the seniors (P < 0.05), but [HLa](TT), RER(TT), V(ETT): VO2(TT), and economy(TT) were similar between the groups (P > 0.05). Time-trial exercise intensity, expressed as %RMP(max), %HR(peak), % VO2(peak), and % V(Epeak), was similar (P > 0.05) for seniors and veterans (W(TT): 81%, s = 2 vs. 78%, s = 8; HR(TT): 96%, s = 4 vs. 94%, s = 4; VO2(TT): 92%, s = 4 vs. 95%, s = 10; V(ETT): 89%, s = 8 vs. 85%, s = 8, respectively). Overall, seniors attained higher absolute values for power output, heart rate, VO2, and V(E) but not blood lactate concentration, respiratory exchange ratio (RER), V(E): VO2, and economy. Veterans did not accommodate age-related declines in time trial performance by maintaining higher relative exercise intensity.     

The incidence and wage effects of overeducation

Data gathered from a recent national sample of workers on educational requirements and attainments are used to examine the extent and economic effects of overeducation. Nearly 40 percent of the U.S. work force-and about 50 percent of black males-have more education than their jobs require. But we also find that “surplus” education does have economic value. The individual return to an additional year of surplus education was positive and significant for all major demographic groups. The estimated return is, however, only about half the size of the return to an additional year of required education.     

Bridging memory-based collaborative filtering and text retrieval

When speaking of information retrieval, we often mean text retrieval. But there exist many other forms of information retrieval applications. A typical example is collaborative filtering that suggests interesting items to a user by taking into account other users’ preferences or tastes. Due to the uniqueness of the problem, it has been modeled and studied differently in the past, mainly drawing from the preference prediction and machine learning view point. A few attempts have yet been made to bring back collaborative filtering to information (text) retrieval modeling and subsequently new interesting collaborative filtering techniques have been thus derived. In this paper, we show that from the algorithmic view point, there is an even closer relationship between collaborative filtering and text retrieval. Specifically, major collaborative filtering algorithms, such as the memory-based, essentially calculate the dot product between the user vector (as the query vector in text retrieval) and the item rating vector (as the document vector in text retrieval). Thus, if we properly structure user preference data and employ the target user’s ratings as query input, major text retrieval algorithms and systems can be directly used without any modification. In this regard, we propose a unified formulation under a common notational framework for memory-based collaborative filtering, and a technique to use any text retrieval weighting function with collaborative filtering preference data. Besides confirming the rationale of the framework, our preliminary experimental results have also demonstrated the effectiveness of the approach in using text retrieval models and systems to perform item ranking tasks in collaborative filtering.