Predicting relevance based on assessor disagreement: analysis and practical applications for search evaluation

Evaluation of search engines relies on assessments of search results for selected test queries, from which we would ideally like to draw conclusions in terms of relevance of the results for general (e.g., future, unknown) users. In practice however, most evaluation scenarios only allow us to conclusively determine the relevance towards the particular assessor that provided the judgments. A factor that cannot be ignored when extending conclusions made from assessors towards users, is the possible disagreement on relevance, assuming that a single gold truth label does not exist. This paper presents and analyzes the predicted relevance model (PRM), which allows predicting a particular result’s relevance for a random user, based on an observed assessment and knowledge on the average disagreement between assessors. With the PRM, existing evaluation metrics designed to measure binary assessor relevance, can be transformed into more robust and effectively graded measures that evaluate relevance towards a random user. It also leads to a principled way of quantifying multiple graded or categorical relevance levels for use as gains in established graded relevance measures, such as normalized discounted cumulative gain, which nowadays often use heuristic and data-independent gain values. Given a set of test topics with graded relevance judgments, the PRM allows evaluating systems on different scenarios, such as their capability of retrieving top results, or how well they are able to filter out non-relevant ones. Its use in actual evaluation scenarios is illustrated on several information retrieval test collections.     

Relaxation and guided imagery as an intervention for children with asthma: A replication

Asthma is a pervasive inflammatory disorder of the bronchial airways that causes the normal functioning of the airways to become overreactive. This disorder affects approximately 8 million children in schools per year and accounts for a significant amount of absences. Researchers have speculated that pharmacological interventions are alone insufficient or overaggressive in treating asthma, and numerous studies have demonstrated a connection between emotions and asthma. It has therefore been suggested that psychological interventions may be appropriate for this population. One such intervention, relaxation and guided imagery (RGI), that incorporates mental representations as well as the senses in a process of relaxing all the parts of the body has been shown to be effective with children with asthma. The current study served as a replication and extension of this previous research. The purpose of the investigation was to examine the effectiveness of RGI with an elementary school sample as well as determine the overall improvement in happiness, anxiety, and quality of life of the participants. Results demonstrated that RGI significantly improved the lung functioning of 3 out of 4 participants in the study. Furthermore, overall happiness improved for 1 participant in the study, state anxiety decreased for 2 of the 4 participants, and trait anxiety decreased for all 4 participants. © 2005 Wiley Periodicals, Inc. Psychol Schs 42: 707–720, 2005.     

Finding Diego: A Bilingual Student Integrates School,Language, and Identity

This article presents a mixed-methods case study of Diego, a bilingual teen who completed public school in Florida. During adolescence, Diego negotiated multiple identities: successful student, Mexican American, bilingual, and typical U.S. teenager. Diego provided interviews and bilingual (English/Spanish) writing (narrative/expository) in 2008 (at age 12, Grade 6) and in 2012 (at age 16, Grade 10). A qualitative analysis of his interviews and a quantitative linguistic analysis of his writing reveal central elements of Diego’s language development as related to academic English and identity. Educational implications for working with bilingual adolescents are discussed.     

Promoting Metacognitive Decision-Making in Teacher Education

Effective teachers are characterized by their abilities to make thoughtful, deliberate, and informed adaptations while teaching (Hoffman & Pearson, 2000). These in-the-moment teaching decisions are guided by a complex web of teacher knowledge. Raising teachers' awareness of the decisions they make on a moment-by-moment basis may aid in thoughtful, more deliberate decision-making and more robust instruction. We charge that preservice teachers can acquire this skill, as well. This article highlights practices employed in preservice teacher education courses as a means to promote awareness of, and engagement with, in-the-moment teaching decisions. Using examples from mathematics, social studies, and literacy methods courses, we describe our processes to intentionally cultivate preservice teachers' adaptive teaching skills through metacognitive decision-making.