Fusing different information retrieval systems according to query-topics: a study based on correlation in information retrieval systems and TREC topics

To evaluate Information Retrieval Systems on their effectiveness, evaluation programs such as TREC offer a rigorous methodology as well as benchmark collections. Whatever the evaluation collection used, effectiveness is generally considered globally, averaging the results over a set of information needs. As a result, the variability of system performance is hidden as the similarities and differences from one system to another are averaged. Moreover, the topics on which a given system succeeds or fails are left unknown. In this paper we propose an approach based on data analysis methods (correspondence analysis and clustering) to discover correlations between systems and to find trends in topic/system correlations. We show that it is possible to cluster topics and systems according to system performance on these topics, some system clusters being better on some topics. Finally, we propose a new method to consider complementary systems as based on their performances which can be applied for example in the case of repeated queries. We consider the system profile based on the similarity of the set of TREC topics on which systems achieve similar levels of performance. We show that this method is effective when using the TREC ad hoc collection.     

Public health practitioners' information access and use patterns in the Maryland (USA) public health departments of Anne Arundel and Wicomico Counties

PURPOSE: To increase understanding of the information needs and use of public health practitioners. SETTING: From June 2005 to May 2006, the library offered a course in public health information resources to eighteen practitioners in two counties, access to the library's licensed electronic resources through a tailored web portal, and consulting services. EVALUATION METHOD: We combined usage statistics from the web portal, self-report and observational data collected during training and shadowing of participants. CONCLUSIONS: The data from this project indicate that usage of licensed information resources and services is infrequent but broad ranging. A few users register at the high end of the usage range, but one use of one high quality article can have a significant impact on policy decisions. Time and competing responsibilities often constrain the retrieval and use of resources for evidence-based decision making and an informationist or power-user model may be more appropriate than training all practitioners to integrate searching into their workflow. This study indicates (i) that evidence-based public health practice requires seamless and broadly based information access; and (ii) that the currently existing patchwork does not support the level of use or take into account the time constraints of information needs for public health practice.     

Methods for linking EHR notes to education materials

It has been shown that accessing the patients’ own electronic health records (EHR) can enhance their medical understanding and provide clinically relevant benefits. However, languages that are difficult for non-medical professionals to comprehend are prevalent in the EHR notes. The valuable and authoritative information contained in the EHR is thus less accessible to the patients, who ultimately stand to benefit the most from the information. To address this challenge, we are developing a system to retrieve EHR note-specific online consumer-oriented health education materials. We explored several query generation methods to convert long EHR notes to effective queries, including topic models and key concept identification. Our experiments show that queries using key concepts identified by a learning based model with pseudo-relevance feedback significantly outperform the baseline system of using the full text note.     

Predicting relevance based on assessor disagreement: analysis and practical applications for search evaluation

Evaluation of search engines relies on assessments of search results for selected test queries, from which we would ideally like to draw conclusions in terms of relevance of the results for general (e.g., future, unknown) users. In practice however, most evaluation scenarios only allow us to conclusively determine the relevance towards the particular assessor that provided the judgments. A factor that cannot be ignored when extending conclusions made from assessors towards users, is the possible disagreement on relevance, assuming that a single gold truth label does not exist. This paper presents and analyzes the predicted relevance model (PRM), which allows predicting a particular result’s relevance for a random user, based on an observed assessment and knowledge on the average disagreement between assessors. With the PRM, existing evaluation metrics designed to measure binary assessor relevance, can be transformed into more robust and effectively graded measures that evaluate relevance towards a random user. It also leads to a principled way of quantifying multiple graded or categorical relevance levels for use as gains in established graded relevance measures, such as normalized discounted cumulative gain, which nowadays often use heuristic and data-independent gain values. Given a set of test topics with graded relevance judgments, the PRM allows evaluating systems on different scenarios, such as their capability of retrieving top results, or how well they are able to filter out non-relevant ones. Its use in actual evaluation scenarios is illustrated on several information retrieval test collections.     

The Complexity of Measuring the Impact of Books

The purpose of this study was to discover whether there are scholarly evaluation metrics that can be applied to a wide range of books’ types and contents. We analyzed over 70,000 books and collected various metrics per each title including traditional and altmetrics measures. The analysis in this paper depicts the top books showing the highest rates of citations, reviews, readership, downloads, twitter and more. Our results show that books display different impact in each of the measurements and vary by types and content. There isn’t one measure that captures the impact of books across the board. We conclude that books should be evaluated by types and contents while using different measures per each.     

The New Role of the Managing Editor at Large Companies in Argentina

The purpose of this work is to address the changes in the role of the publisher, based on the analysis of the dynamics and performance of the major publishing houses in Argentina today. Based on a number of interviews, we will seek to identify and analyze the tension occurring between the cultural and business logics among stakeholders with a particular interest in the field. These actors share a “publisher’s practical sense” that outlines their habitus and refers to a series of dispositions for action consisting of technical and theoretical knowledge, intuitive processes which, together, make up the publisher’s practice.     

Four requirements for digital case study libraries

In education, case studies have been employed as a major pedagogical means to engage students with realistic materials and practical problems. The usefulness and success of case studies have been documented in literature. In this paper, we move the case study and its technical affordance a step further toward more authentic or realistic learning and toward communities of practice. Drawing on current intellectual achievement in education, developmental psychology and computer-supported collaborative learning, this paper proposes four requirements for interactive case study libraries, focusing on authentic learning, social interaction, resource accumulation, and communities of practice. With a usability engineering case study library as an example, we discuss design concerns derived from these requirements as well as its use in practice.