Tasks, topics and relevance judging for the TREC Genomics Track: five years of experience evaluating biomedical text information retrieval systems

With the help of a team of expert biologist judges, the TREC Genomics track has generated four large sets of “gold standard” test collections, comprised of over a hundred unique topics, two kinds of ad hoc retrieval tasks, and their corresponding relevance judgments. Over the years of the track, increasingly complex tasks necessitated the creation of judging tools and training guidelines to accommodate teams of part-time short-term workers from a variety of specialized biological scientific backgrounds, and to address consistency and reproducibility of the assessment process. Important lessons were learned about factors that influenced the utility of the test collections including topic design, annotations provided by judges, methods used for identifying and training judges, and providing a central moderator “meta-judge”.     

政策和实践的困境(下)

教育政策和实践总是处在困境之中,这是困扰政策制定者和实践者的问题.本文着力从教育政策和实践所面临的困境、如何处理困境的角度,重新审视政策和实践两者的关系,并从政策目标、政策工具和实践中的能力三方面对其展开论述,以期在冲突的基础上促进教育政策和实践间的融合.     

Science Teachers’ Professional Development and Changes in Science Practical Assessment Practices: What are the Issues?

This paper considers the circumstances under which science teachers can respond positively and productively to educational policy reforms in the area of science practical assessment. To understand what might be involved in linking science teachers’ assessment capacities and their professional development, we present illustrative data from recent research studies conducted in Singapore and Hong Kong showing contrasting approaches taken in the implementation of reforms in science practical assessment. In Singapore, teachers worked together to select, discuss, clarify and refine their practices as they made decisions about what to teach and assess. In Hong Kong, teachers took a critical stance towards the new policy and learnt from their own experiences in order to build their confidence. With the same policy initiative, one group of teachers focused more on the technicalities of complying with requirements imposed on them while in the other group had their professional consciousness of what they thought was best for their students provoked so that their practices would be transformed. In an attempt to draw lessons for other contexts in supporting the implementation of assessment policy reforms through professional development work, we identify and discuss a range of factors in science teachers’ professional development that arise once in situ professional development work has started. Overall, our intent in this article is to recast assessment reform as a driver or pivot in teachers’ professional development and learning. To do this it is necessary, we argue, to afford teachers’ experiences and the processes involved in learning from them greater emphasis in order to ensure the continuance of innovation in the assessment of laboratory-based work.     

A New Compact for Higher Education: Funding and Autonomy for Reform and Accountability

Over the past few decades, America’s social compact for higher education as a public good has effectively lapsed as government support of higher education has diminished. Given the need for a highly educated workforce in today’s knowledge-based global economy, we propose a new compact for higher education that couples increased funding with increased institutional accountability. While the compact must retain academic freedom as an essential component of higher education, this autonomy must co-exist with rigorous accountability standards. Accountability systems should be multifaceted, as different methodologies have differing strengths and limitations. All systems should, however, inform the public and provide for institutional improvement.     

Age versus schooling effects on intelligence development

The effect of formal education, as opposed to chronological age, on intelligence development has suffered from inadequate empirical investigation. Most studies of this issue have relied on natural variation in exposure to school among children of the same age, thus confounding differences in schooling with differences in other intelligence-related variables. This difficulty can be overcome by a quasi-experimental paradigm involving comparison between children who differ in both chronological age and schooling. The present study applies this paradigm to the estimation of the independent effects of age and schooling in grades 5 and 6 on raw scores obtained on a variety of general ability tests. The sample included all students in Jerusalem's Hebrew-language, state-controlled elementary schools. The results unambiguously point to schooling as the major factor underlying the increase of intelligence test scores as a function of age and to the larger effect schooling has on verbal than nonverbal tests. These results contribute to our understanding of the causal model underlying intelligence development and call for reconsideration of the conceptual basis underlying the definition of deviation-IQ scores. Some implications of these results concerning the distinction between intelligence and scholastic achievement, the causal model underlying the development of "crystallized" and "fluid" abilities, and the notion of "culture-fair" tests are discussed.     

Developmental change in infants' perception of correlations among attributes

The perception of relational information such as the correlation or co-occurrence among features should play a central role in abilities ranging from the perception and recognition of a simple pattern or object to the formation of a category. 4 experiments were conducted to examine developmental change in 4-, 7-, and 10-month-old infants' perception of correlations among attributes. The results suggested a developmental progression in infants' processing of simple correlational information, ranging from the processing of independent featural information only at 4 months, the perception of relations among features of a single pattern at 7 months, to the abstraction of invariant relations from a category at 10 months.     

Diversification in Argentine higher education: Dimensions and impact of private sector growth

Private education is a remarkably dynamic area of change in post-secondary education, particularly in Latin America. Evidence of growth in the number of private institutions and enrollment suggests that higher education is becoming increasingly diversified into two sectors – public and private. While this appears to be true throughout Latin America, and much of the developing world, recent evidence from Argentina casts doubt on the degree to which private sector growth trends may actually translate into a significant increase in the impact of private institutions in higher education, where impact is proxied by the relative share of student enrollment and graduates. This study explores the dimensions and impact of private expansion in Argentina, and places the findings in a comparative (regional) and theoretical framework. The analysis shows that, despite unprecedented growth in the number of private universities, the public sector remains the dominant provider of university education. Private institutions, while certainly more numerous and training a growing number of students, do not account for a larger share of university enrollment. They have, however, developed specific niches (e.g., in graduate level training and in urban areas). Significantly, following a pattern of institutional proliferation found throughout Latin America, some private universities offer elite alternatives to prestigious public institutions. The findings serve to qualify apparent `gains' in private higher education and to underscore the need for further research into the relevant intrasectoral (public and private) dimensions of change in post-secondary education.     

Validating human and automated scoring of essays against “True” scores

ABSTRACT

In the current study, two pools of 250 essays, all written as a response to the same prompt, were rated by two groups of raters (14 or 15 raters per group), thereby providing an approximation to the essay’s true score. An automated essay scoring (AES) system was trained on the datasets and then scored the essays using a cross-validation scheme. By eliminating one, two, or three raters at a time, and by calculating an estimate of the true scores using the remaining raters, an independent criterion against which to judge the validity of the human raters and that of the AES system, as well as the interrater reliability was produced. The results of the study indicated that the automated scores correlate with human scores to the same degree as human raters correlate with each other. However, the findings regarding the validity of the ratings support a claim that the reliability and validity of AES diverge: although the AES scoring is, naturally, more consistent than the human ratings, it is less valid.     

Six-month-old infants' categorization of containment spatial relations

Six-month-old infants' ability to form an abstract category of containment was examined using a standard infant categorization task. Infants were habituated to 4 pairs of objects in a containment relation. Following habituation, infants were tested with a novel example of the familiar containment relation and an example of an unfamiliar relation. Results indicate that infants look reliably longer at the unfamiliar versus familiar relation, indicating that they can form a categorical representation of containment. A second experiment demonstrated that infants do not rely on object occlusion to discriminate containment from a support or a behind spatial relation. Together, the results indicate that by 6 months, infants can recognize a containment relation from different angles and across different pairs of objects.