Predicting relevance based on assessor disagreement: analysis and practical applications for search evaluation

Evaluation of search engines relies on assessments of search results for selected test queries, from which we would ideally like to draw conclusions in terms of relevance of the results for general (e.g., future, unknown) users. In practice however, most evaluation scenarios only allow us to conclusively determine the relevance towards the particular assessor that provided the judgments. A factor that cannot be ignored when extending conclusions made from assessors towards users, is the possible disagreement on relevance, assuming that a single gold truth label does not exist. This paper presents and analyzes the predicted relevance model (PRM), which allows predicting a particular result’s relevance for a random user, based on an observed assessment and knowledge on the average disagreement between assessors. With the PRM, existing evaluation metrics designed to measure binary assessor relevance, can be transformed into more robust and effectively graded measures that evaluate relevance towards a random user. It also leads to a principled way of quantifying multiple graded or categorical relevance levels for use as gains in established graded relevance measures, such as normalized discounted cumulative gain, which nowadays often use heuristic and data-independent gain values. Given a set of test topics with graded relevance judgments, the PRM allows evaluating systems on different scenarios, such as their capability of retrieving top results, or how well they are able to filter out non-relevant ones. Its use in actual evaluation scenarios is illustrated on several information retrieval test collections.     

Work‐experience and schools: factors influencing the participation of industry

The growth of work‐experience as part of the school curriculum in such schemes as TVEI, has led to a growing body of literature concerned with the educational, social and political consequences of this trend. However, one aspect of analysis has been neglected by those working in this area. There has been a marked lack of investigation into factors which affect the supply of work placements to schools. Behind this lies an assumption that the participation of industry with schools is a straightforward and easy to accomplish process.

By focusing on some of the approaches exhibited by firms towards the provison of work‐experience, this paper seeks to make problematic the supply of work placements by industry in a market economy. I shall suggest that eductionalists and policy makers may have cause to be concerned not only with the quantity of placements which might be available to them in the future, but with the educational quality of these placements.     

Learning to use the Internet as a study tool: a review of available resources and exploration of students' priorities

BACKGROUND: The Internet is a valuable information tool, but users often struggle to locate good quality information from within the vast amount of information available. OBJECTIVE: The aim of the study was to identify the online information resources available to assist students develop Internet searching skills, and to explore the students' priorities in online guides. METHODS: A qualitative approach was adopted with two phases. The first was a structured search of available online study skills resources. The second comprised 10 group interviews with a total of 60 students at all stages of five undergraduate health and social care related courses at a UK university. RESULTS: The study found that there were good online guides available, but that, perversely, the better guides tended to require the best searching skills to locate them. A few students were enthusiastic about using online support, however the majority felt that if they had the skills to locate such resources they wouldn't use a study guide to improve these skills, and if they did not have the skills they would not think of using an online guide to develop them. CONCLUSIONS: Students wanted assistance when they had problems or questions, rather than sites that offered structured learning experiences. Personal support rather than virtual support was also considered to be most important to the students in this study.     

Leadership and inclusion: a special school perspective

In July 2002, the National College for School Leadership (NCSL) held a 'Leading Edge' seminar focused on the contribution that leaders in special schools can make to the development of Inclusive policies and practices. The day gave colleagues working in specialist contexts an opportunity to share examples and experiences of current initiatives and to speculate about future trends and developments in inclusive education. In this article, Robin Attfield and Chris Williams, Assistant Directors in Programmes and Research respectively at the NCSL, explore the context in which the seminar took place and some of the themes and issues that emerged from discussion on the day. At the end of their paper, they summarise the key messages that the special school leaders who attended the seminar wished to communicate to a wider audience and articulate a further role for leaders in special schools in the development of inclusion.