Measuring Search Engine Quality

The effectiveness of twenty public search engines is evaluated using TREC-inspired methods and a set of 54 queries taken from real Web search logs. The World Wide Web is taken as the test collection and a combination of crawler and text retrieval system is evaluated. The engines are compared on a range of measures derivable from binary relevance judgments of the first seven live results returned. Statistical testing reveals a significant difference between engines and high intercorrelations between measures. Surprisingly, given the dynamic nature of the Web and the time elapsed, there is also a high correlation between results of this study and a previous study by Gordon and Pathak. For nearly all engines, there is a gradual decline in precision at increasing cutoff after some initial fluctuation. Performance of the engines as a group is found to be inferior to the group of participants in the TREC-8 Large Web task, although the best engines approach the median of those systems. Shortcomings of current Web search evaluation methodology are identified and recommendations are made for future improvements. In particular, the present study and its predecessors deal with queries which are assumed to derive from a need to find a selection of documents relevant to a topic. By contrast, real Web search reflects a range of other information need types which require different judging and different measures.     

Review of Cutscores: A Manual for Setting Standards of Performance on Educational and Occupational Tests

Combinations of five methods of equating test forms and two methods of selecting samples of students for equating were compared for accuracy. The two sampling methods were representative sampling from the population and matching samples on the anchor test score. The equating methods were the Tucker, Levine equally reliable, chained equipercentile, frequency estimation, and item response theory (IRT) 3PL methods. The tests were the Verbal and Mathematical sections of the Scholastic Aptitude Test. The criteria for accuracy were measures of agreement with an equivalent-groups equating based on more than 115,000 students taking each form. Much of the inaccuracy in the equatings could be attributed to overall bias. The results for all equating methods in the matched samples were similar to those for the Tucker and frequency estimation methods in the representative samples; these equatings made too small an adjustment for the difference in the difficulty of the test forms. In the representative samples, the chained equipercentile method showed a much smaller bias. The IRT (3PL) and Levine methods tended to agree with each other and were inconsistent in the direction of their bias.     

The Role of Study Habits in Foreign Language Courses

The purpose of this study was to determine which study habits distinguish successful from unsuccessful foreign language learners. Participants were 219 college students from a variety of disciplinary backgrounds, enrolled in either Spanish, French, German or Japanese classes. A canonical discriminant analysis (F [6, 117], p < 0.0001; canonical R = 0.92) revealed that, compared to their high-performing counterparts, students with the lowest levels of foreign language performance tended to report that: (a) they frequently include a lot of irrelevant or unimportant information in their notes; (b) when they have difficulty with their assignments, they do not seek help from their instructor; (c) they put their lecture notes away after taking the test and never consult them again; (d) they have to be in the mood before attempting to study; (e) they have a tendency to doodle or to daydream when they are trying to study; and (f) they do not look up in a dictionary the meanings of words that they do not understand. Implications are discussed.     

Lack of Relationship Between Functional and Perceived Quality of Life Outcomes Following Pulmonary Rehabilitation

Individuals with chronic obstructive pulmonary disease (COPD) have been shown to benefit from participation in pulmonary rehabilitation (PR) programs that include exercise training and education. Purpose: To examine the relationship between improvements in 6 minute walk distance and perceived quality of life in individuals with COPD following completion of a PR program. Methods: The records of 139 individuals completing a PR program (3 times a week for 8 weeks) were retrospectively examined. Prior to entry and upon completion of the program each individual completed a 6 minute walk test (6MWT), the SF-36 Health survey, and the UCSD Shortness of Breath Questionnaire (SOB). SF-36 results were analyzed according to 8 subscales [Physical Functioning (PF), Role Physical (RF), Bodily Pain (BP), General Health (GH), Vitality (V), Social Functioning (SF), Role Emotional (RE), and Mental Health (MH)]. Results: PR resulted in significant improvements in 6MWTdistance (Pre = 845 ± 37 ft, Post = 1127 ± 32 ft, p < 0.001), PF (p < 0.001), RF (p = 0.001), Vitality (p = 0.002), SF (p < 0.001), RE (p = 0.037), MH (p < 0.001) and SOB (Pre = 53 ± 2, Post = 47 ± 2, p < 0.001). The change in 6MWT distance was not related to changes in PF (r = 0.17), RF (r = 0.03), GH (r = 0.03), Vitality (r = −0.001), SF (r = 0.01), RE (r = 0.06), MH (r = −0.04) or SOB (r = 0.12). The magnitude of improvement in 6MWT distance (68%) was much greater than that observed in PF (15%), RF (16%), GH (6%), VT (18%), SF (20%), RE (14%), MH (14%), or SOB (8%). Conclusions: PR has a positive impact on 6 minute walk distance and perceived quality of life in individuals with COPD; however, changes in 6 minute walk distance appear to have no relationship to changes in perceived quality of life.     

Prolonged static stretching does not influence running economy despite changes in neuromuscular function

The aim of this study was to examine the acute effects of prolonged static stretching (SS) on running economy. Ten male runners (VO2(peak) 60.1 +/- 7.3 ml x kg(-1) x min(-1)) performed 10 min of treadmill running at 70% VO2(peak) before and after SS and no stretching interventions. For the stretching intervention, each leg was stretched unilaterally for 40 s with each of eight different exercises and this was repeated three times. Respiratory gas exchange was measured throughout the running exercise with an automated gas analysis system. On a separate day, participants were tested for sit and reach range of motion, isometric strength and countermovement jump height before and after SS. The oxygen uptake, minute ventilation, energy expenditure, respiratory exchange ratio and heart rate responses to running were unaffected by the stretching intervention. This was despite a significant effect of SS on neuromuscular function (sit and reach range of motion, +2.7 +/- 0.6 cm; isometric strength, -5.6% +/- 3.4%; countermovement jump height -5.5% +/- 3.4%; all P < 0.05). The results suggest that prolonged SS does not influence running economy despite changes in neuromuscular function.     

Standards-to-Standards Linkage Under Title III: Exploring Common Language Demands in ELD and Science Standards

A great deal of measurement research is currently being utilized to improve instruction at the elementary and secondary levels, but many opportunities for improving utilization are still untapped. The author proposes specific approaches to increased collaboration between researchers and practitioners to meet the burgeoning research needs of the nations' schools and to assist in data-based decision-making efforts. Society's increasing demands for accurate, usable data have contributed to the urgent current need for viable, appropriate research. This need can be met only by reseachers and practitioners working together to provide answers to critical educational questions.