The PRET A Rapporter framework: Evaluating digital libraries from the perspective of information work

The strongest tradition of IR systems evaluation has focused on system effectiveness; more recently, there has been a growing interest in evaluation of Interactive IR systems, balancing system and user-oriented evaluation criteria. In this paper we shift the focus to considering how IR systems, and particularly digital libraries, can be evaluated to assess (and improve) their fit with users’ broader work activities. Taking this focus, we answer a different set of evaluation questions that reveal more about the design of interfaces, user–system interactions and how systems may be deployed in the information working context. The planning and conduct of such evaluation studies share some features with the established methods for conducting IR evaluation studies, but come with a shift in emphasis; for example, a greater range of ethical considerations may be pertinent. We present the PRET A Rapporter framework for structuring user-centred evaluation studies and illustrate its application to three evaluation studies of digital library systems.     

Acquired equivalence and generalized suppression in a virtual reality environment

Acquired equivalence was investigated using a virtual reality conditioned suppression task administered in a first-person-shooter game. Two visual cues, A1 and B1, were followed by a tone (O1), and another two cues, A2 and B2, were followed by another tone (O2). During differential Pavlovian conditioning, A1 was paired with an instructed unconditioned stimulus (US) consisting of a flashing white screen, whereas A2 was not. All cues and outcomes were then presented at test, in the absence of the US, and suppression ratios were calculated for multiple response topographies (shots, hits, and breaks). Clear evidence of the suppression of shots was seen for A1 and B1, with no suppression being seen for either A2 or B2. Presentations of O1 and O2 resulted in significant suppression of shots and hits, whereas only O1 led to the suppression of breaks. The US expectancy ratings were consistent with these behavioral results. The findings are discussed in the light of differing accounts of acquired equivalence.     

The effects of prior incremental cycle exercise on the physiological responses during incremental running to exhaustion: relevance for sprint triathlon performance

It is common for the physiological working capacity of a triathlete when cycling and running to be assessed on two separate days. The aim of this study was to establish whether an incremental running test to exhaustion has a negative effect after a 5 h recovery from an incremental cycling test. Eight moderately trained triathletes (age, 26.2 +/- 3.4 years; body mass, 67.3 +/- 9.1 kg; VO2max when cycling, 59 +/- 13 ml x kg x min(-1); mean +/- s) completed an incremental running test 5 h after an incremental cycling test (fatigue) as well as an incremental running test without previous activity (control). Maximum running speed, maximal oxygen uptake (VO2max) and the lactate threshold were determined for each incremental running test and correlated with the average speed during a 5 km run, which was performed immediately after a 20 km cycling time-trial, as in a sprint triathlon. There were no significant differences in maximum running speed, VO2max or the lactate threshold in either incremental running test (control or fatigue). Furthermore, good agreement was found for each physiological variable in both the control and fatigue tests. For the fatigue test, there were significant correlations between the average speed during a 5 km run and both VO2max expressed in absolute terms (r = 0.83) and the lactate threshold (r = 0.88). However, maximum running speed correlated most strongly with the average speed during a 5 km run (r = 0.96). The results of this study indicate that, under controlled conditions, an incremental running test can be performed successfully 5 h after an incremental cycling test to exhaustion. Also, the maximum running speed achieved during an incremental running test is the variable that correlates most strongly with the average running speed during a 5 km run after a 20 km cycling time-trial in well-trained triathletes.     

To reopen the American mind

This essay originally appeared in the July 8, 1988 issue ofThe Wall Street Journal. It is reprinted here with theJournal's permission.