Click-based evidence for decaying weight distributions in search effectiveness metrics

Search effectiveness metrics are used to evaluate the quality of the answer lists returned by search services, usually based on a set of relevance judgments. One plausible way of calculating an effectiveness score for a system run is to compute the inner-product of the run’s relevance vector and a “utility” vector, where the ith element in the utility vector represents the relative benefit obtained by the user of the system if they encounter a relevant document at depth i in the ranking. This paper uses such a framework to examine the user behavior patterns—and hence utility weightings—that can be inferred from a web query log. We describe a process for extrapolating user observations from query log clickthroughs, and employ this user model to measure the quality of effectiveness weighting distributions. Our results show that for measures with static distributions (that is, utility weighting schemes for which the weight vector is independent of the relevance vector), the geometric weighting model employed in the rank-biased precision effectiveness metric offers the closest fit to the user observation model. In addition, using past TREC data as to indicate likelihood of relevance, we also show that the distributions employed in the BPref and MRR metrics are the best fit out of the measures for which static distributions do not exist.     

Looking back,looking forward: celebrating a quarter century of serving distance education ‘down under’

The Open and Distance Learning Association of Australia (ODLAA) has reached 25 years of age. This article traces the development of the Association over that period. It records the major changes that have taken place in the range and types of activities of the Association and explains the circumstances that led to those changes. It concludes by attempting to identify the principal challenges facing the Association in entering its second quarter century and proposes ways in which these challenges might be addressed.     

EDUCATIONAL TELEVISION, 1969‐1979

The author, Principal of a College of Education, reviews the changes that have occurred in educational television in the past ten years in Britain, and the reasons for these changes. If educational television has had a raw deal, some at least of the responsibility for this lies at the door of its own practitioners, who need to try to look ahead with greater foresight to the coming decade.     

Mental computation performance in Australia,Japan and the United States

The study was conducted to explore performance on a variety of mental computation tasks using two presentation formats (visual and oral). Students at four grade levels between grades 2 and 9 in three countries (Australia, Japan, United States) were given a group administered mental computation test consisting of two parts (oral presentation format, visual presentation format).The sample of nearly 2000 students represents 6 classes at each of four grade levels in each country. Results indicate a wide variation in performance within the sample of each country at each grade level. Differences in performance between countries are also apparent and may reflect variations in instructional focus on mental computation. In particular, Japanese students perform at a higher level at the early grades than do students in either of the other countries sampled. However, by grade 8 this difference narrows in the American sample, and vanishes for the Australian sample. Differences in performance related to presentation format were dramatic for particular items and non-existent for other items. The most consistent effect was found in the Japanese sample where the visual presentation format resulted in higher performance levels on most items.It is hypothesised that superior results on visually presented items are attributable to a greater reliance on use of the standard written algorithm, while superior results on orally presented items indicate a greater tendency to use invented mental algorithms.     

Facilitating team-based course designing with conceptual mapping

A method of rendering the design of a course visible for the purpose of discussion amongst the members of course development teams is described. The paper begins by examining the way in which design is manifested in the area of course materials development and the approaches that have been used in attempting to document designs. The strengths and limitations of the conceptual mapping method are examined and the way in which the method has been used to facilitate the work of course development teams is described. Additional uses of the method and possibilities for future enhancements of the method are discussed.     

Excellence in doctoral supervision: an examination of authoritative sources across four countries in search of performance higher than competence

Supervision is generally recognised as playing a crucial role in the quality of a research student’s doctoral experience and their academic outcomes and, in common with most areas of higher education, there is an oft-stated desire to pursue excellence in this important area. Excellence in research degree supervision is, however, an elusive concept and on close scrutiny most of the discussions of high-quality supervision, even those that purport to be identifying excellence, refer to competence rather than excellence. This paper examines two potentially national authoritative perspectives from which excellence in research degree supervision might be explicated (codes of practice and learning and teaching awards) from Australia, New Zealand, South Africa and the United Kingdom but concludes that the complex nature of the activity and the complexity of the concept itself mean that rather than identifying excellence in supervision we can only respond to claims for excellence.     

Improving quantitative volumetric analysis skills in first year university chemistry courses

This paper describes a project in the School of Chemistry at The University of Melbourne. During 1992 thirty hours of videotapes were taken of students doing quantitative volumetric work in first year laboratories. These were viewed to find out what problems students encountered and what interactions they had with other students and with demonstrators. The data were logged on a Hypercard stack and novel visual images were produced to map student activity, both qualitatively and quantitatively. These data contributed to the design of a multimedia learning package which students worked through at the beginning of 1993. Video data of their laboratory performance after this intervention provides evidence of improved performance. Specializations: academic development in the sciences, evaluation of computer-based learning materials. Specializations: physical chemistry, chemical education at tertiary level. Specializations: design and development of computer-based learning materials. Specializations: electrochemistry, general physical chemistry at secondary and tertiary level. Specializations: chemical education at secondary and tertiary level, theoretical chemistry.     

Criterion-referenced and norm-referenced assessments: compatibility and complementarity

The tension between criterion-referenced and norm-referenced assessment is examined in the context of curriculum planning and assessment in outcomes-based approaches to higher education. This paper argues the importance of a criterion-referenced assessment approach once an outcomes-based approach has been adopted. It further discusses the implementation of criterion-referenced assessment, considering to what extent the criteria and standards adopted are implicitly norm referenced. It introduces a compatible interpretation of criterion-referenced and norm-referenced assessments in higher education, and illustrates how their combined use can avoid grade inflation and also provide useful information to educators, employers and learners. Instead of seeing criterion referencing and norm referencing as a dichotomy, assessment in higher education benefits from their synthesis through a feedback loop that emphasises alignment between learning and assessment; such feedback and alignment are essential features of quality assurance and enhancement.