Measuring Search Engine Quality

The effectiveness of twenty public search engines is evaluated using TREC-inspired methods and a set of 54 queries taken from real Web search logs. The World Wide Web is taken as the test collection and a combination of crawler and text retrieval system is evaluated. The engines are compared on a range of measures derivable from binary relevance judgments of the first seven live results returned. Statistical testing reveals a significant difference between engines and high intercorrelations between measures. Surprisingly, given the dynamic nature of the Web and the time elapsed, there is also a high correlation between results of this study and a previous study by Gordon and Pathak. For nearly all engines, there is a gradual decline in precision at increasing cutoff after some initial fluctuation. Performance of the engines as a group is found to be inferior to the group of participants in the TREC-8 Large Web task, although the best engines approach the median of those systems. Shortcomings of current Web search evaluation methodology are identified and recommendations are made for future improvements. In particular, the present study and its predecessors deal with queries which are assumed to derive from a need to find a selection of documents relevant to a topic. By contrast, real Web search reflects a range of other information need types which require different judging and different measures.     

Accurate assessment of the brake torque on a rope-braked cycle ergometer

Sports Engineering - A Monark cycle ergometer is a device globally used in physiological studies to measure the work and energy levels of exercising humans. In this paper a rope-braked cycle...     

The impact of case characteristics on child abuse reporting decisions.

Drawing distinctions on the basis of at least some case characteristics in making reporting decisions makes considerable sense. Using data from a national survey of mandated reporters that included vignettes in which case and personal characteristics were systematically varied, this paper examines the impact of selected characteristics while controlling for others. The data reveal that respondents noticed and responded to case characteristics. Abuse-relevant judgments and reporting intentions varied, often substantially, as a function of case characteristics. Three case characteristics--previous abuse, severity of abuse, and recantation--were powerful predictors of vignette outcomes. A history of previous abuse led to judgments of greater seriousness, a more salutary impact of a report, and greater likelihood of reporting. More severe abuse was more likely to be labelled as abuse, and was more likely to be reported. When the alleged victim retracted her accusation upon questioning by an authority figure, respondents were significantly less likely to intend a report. Child age, perpetrator intent and family socioeconomic status also influenced abuse-relevant judgments and reporting intentions. Respondents were more likely to intend a report when younger children, lazy or angry perpetrators and children from poorer families were portrayed.     

Assessing the Utility of Item Response Models: Computerized Adaptive Testing

How has Item Response Theory helped solve problems in the development and use of computer-adaptive tests? Do we need to balance item content with computer-adaptive tests? Could we use IRT to evaluate unusual responses to computer-delivered tests?     

That vision thing.

The 1991 Janet Doe lecturer describes the vision held by the founders of the Medical Library Association of medical information being readily accessible to health professionals no matter how far they were located from major medical centers, and traces the pursuit of this vision to current outreach activities.     

Interlibrary loan and commercial document supply: Finding the right fit

This study examined the cost and performance of six commercial document suppliers accessible via OCLC, and comapared their service to that of traditional interlibrary loan (ILL). Results indicated that, while commercial firms cannot be recommended as a wholesale replacement for ILL, they can add much-needed flexibility to a library's overall plan for document delivery.