Low-cost evaluation techniques for information retrieval systems: A review

For a system-based information retrieval evaluation, test collection model still remains as a costly task. Producing relevance judgments is an expensive, time consuming task which has to be performed by human assessors. It is not viable to assess the relevancy of every single document in a corpus against each topic for a large collection. In an experimental-based environment, partial judgment on the basis of a pooling method is created to substitute a complete assessment of documents for relevancy. Due to the increasing number of documents, topics, and retrieval systems, the need to perform low-cost evaluations while obtaining reliable results is essential. Researchers are seeking techniques to reduce the costs of experimental IR evaluation process by the means of reducing the number of relevance judgments to be performed or even eliminating them while still obtaining reliable results. In this paper, various state-of-the-art approaches in performing low-cost retrieval evaluation are discussed under each of the following categories; selecting the best sets of documents to be judged; calculating evaluation measures, both, robust to incomplete judgments; statistical inference of evaluation metrics; inference of judgments on relevance, query selection; techniques to test the reliability of the evaluation and reusability of the constructed collections; and other alternative methods to pooling. This paper is intended to link the reader to the corpus of ‘must read’ papers in the area of low-cost evaluation of IR systems.     

For Which Boys and Which Girls Are Reading Assessment Items Biased Against? Detection of Differential Item Functioning in Heterogeneous Gender Populations

In gender differential item functioning (DIF) research it is assumed that all members of a gender group have similar item response patterns and therefore generalizations from group level to subgroup and individual levels can be made accurately. However DIF items do not necessarily disadvantage every member of a gender group to the same degree, indicating existence of heterogeneity of response patterns within gender groups. In this article the impact of heterogeneity within gender groups on DIF investigations was investigated. Specifically, it was examined whether DIF results varied when comparing males versus females, gender × socioeconomic status subgroups and latent classes of gender. DIF analyses were conducted on reading achievement data from the Canadian sample of the Programme of International Student Assessment 2009. Results indicated considerable heterogeneity within males and females and DIF results were found to vary when heterogeneity was taken into account versus when it was not.     

Effect of increasing duration of diabetes mellitus type 2 on glycated hemoglobin and insulin sensitivity

Non-insulin dependent diabetes mellitus (NIDDM) is the most rapidly growing chronic metabolic disorder in the world. With advancement in the age and duration of diabetes there is a gradual tendency for the level of blood sugar to rise along with a subsequent increase in the HbA1c as well as in the fasting insulin level. Whether this is an aging process or increased frequency of diabetes is still controversial. The correlation between glucose and insulin sensitivity is consistent with the idea that the degree of chronic hyperglycemia is a cause of excessive insulin resistance in type 2 diabetes, i.e. the insulin resistance which characterizes type 2 diabetes but not nondiabetic subjects matched for age, gender, family history and duration of diabetes. The study comprised a total of 76 subjects out of which 30 were normal, non-diabetic persons and the rest 46 were diabetics with different duration of time in years, after being diagnosed diabetic. Data was analyzed after dividing the subjects into four groups—Group 1 comprised of one year old diabetics, Group 2 was made up of those, who had diabetes, for the past 2–5 years, Group 3 included patients who were diabetic since more than 5 years and Group 4 included non-diabetics as the normal control group. The results obtained indicated that the HbA1c levels showed a significant increase with the duration of diabetes as well as the insulin level showed a significant correlation after adjustment for age, sex and duration of diabetes.     

Media Usage and Acculturation: Asian Indian Professionals in Silicon Valley*

We examine connections between media use, intergroup contact, and acculturation in 237 Asian Indian professionals in Silicon Valley, drawing on theories of immigrant acculturation, intergroup contact, and media effects. Quality of host contact positively predicted acculturation. Indian (ethnic) media consumption negatively predicted acculturation, and American (host) media consumption was positively related to acculturation. Ethnic television viewing exacerbated the effects of negative intergroup contact on acculturation. Respondents used social networking websites to maintain both host and ethnic ties. Host cultural/informational capital was positively related to acculturation. Host media consumption was positively related to political engagement and loneliness.     

Examining Middle School Students’ Engineering Design Processes in a Design Workshop

Design thinking has an important role in STEM education. However, there has been limited research on how students engage in various modalities throughout the design process in hands-on design tasks. To promote middle school students’ engineering literacy, it is necessary to examine the use of design modalities during design. Using a case study approach, we examine middle school students’ design stages and modalities during design activities. We also identify the patterns of design processes in the teams with different design outcomes. Drawing on theories in design thinking and embodied interaction, we proposed a framework and devised a video analysis protocol to examine students’ design stages and modalities. Middle school students attending a design workshop engaged in two design activities in teams of 3–4 people. The design sessions were video recorded and analyzed using the video analysis protocol. The teams engaged in the stages of planning, building, and testing, while employing the verbal, the visual, and the physical modalities. The teams that varied in design outcomes exhibited different patterns in the use of multiple modalities during the design stages. This study contributes to research on design thinking by proposing a framework for analyzing middle school students’ multimodal design processes and presenting data visualization methods to identify patterns in design stages and modalities. The findings suggest the necessity to examine students’ use of design modalities in the context of design stages and imply the potential benefits of using multiple modalities during design. The implications for future research and education practices are also discussed.

     

Semantic effects as a function of reading skill in word naming of a transparent orthography

The highly transparent Turkish orthography was employed to examine imageability in relation to level of skill in two experiments. In experiment 1, previously skilled readers (n = 24)named 40 high frequency and 40 low frequency words manipulatedfactorially with imageability. A significant main effect was onlyfound for frequency. In experiment 2, a total of 44 adult Turkishreaders (16 very skilled and 28 skilled) named the same stimulias in experiment 1. The results showed an expected main effectfor skill and frequency. However, whilst the main effect forimageability was nonsignificant, a 2-way interaction of skill byimageability and a 3-way interaction of skill by imageability byfrequency were found to be significant. Very skilled readersnamed high imageable low frequency words significantly fasterthan matched low imageable low frequency words. Insofar as poorreaders are concerned the results are contradictory to previousfindings in English whilst data from highly skilled participantsare in line with those reported for skilled readers for English.Implications of these findings on the existing literature arediscussed.     

How children learn to use language

There have been many theories about how children learn to use language. Professor Narasimhan proposed a theory of child language acquisition based on behavioural pragmatics. In this article we present a simplified version of his theories about how children learn to communicate, to describe, manipulate and explore the world around them from exposure to variety of language utterances and non-verbal inputs such as gestures and pointing. We also discuss the method he used to substantiate his ideas, and briefly present a computational model of the ideas arising from his work. Raman Chandrasekar has been with Microsoft Research, Redmond, USA. since 1998. His current research interests are at the intersection of information retrieval, natural language processing and machine learning. He received his PhD from TIFR, Bombay.