Using a New Statistical Model for Testlets to Score TOEFL

Standard item response theory (IRT) models fit to examination responses ignore the fact that sets of items (testlets) often are matched with a single common stimulus (e.g., a reading comprehension passage). In this setting, all items given to an examinee are unlikely to be conditionally independent (given examinee proficiency). Models that assume conditional independence will overestimate the precision with which examinee proficiency is measured. Overstatement of precision may lead to inaccurate inferences as well as prematurely ended examinations in which the stopping rule is based on the estimated standard error of examinee proficiency (e.g., an adaptive test). The standard three parameter IRT model was modified to include an additional random effect for items nested within the same testlet (Wainer, Bradlow, & Du, 2000). This parameter, γ characterizes the amount of local dependence in a testlet.
We fit 86 TOEFL testlets (50 reading comprehension and 36 listening comprehension) with the new model, and obtained a value for the variance of γ for each testlet. We compared the standard parameters (discrimination (a), difficulty (b) and guessing (c)) with what is obtained through traditional modeling. We found that difficulties were well estimated either way, but estimates of both a and c were biased if conditional independence is incorrectly assumed. Of greater import, we found that test information was substantially over-estimated when conditional independence was incorrectly assumed.     

What Makes the Difference? Children and teachers talk about resilient outcomes for children 'at risk'

Research has been conducted into childhood resilience, a term which according to A. Masten, K. Best and N. Garmezy (Resilience and development: contributions from study of children who overcome adversity, Development and Psychopathology, 2, pp. 425-444, 1990) is defined as the process of, capacity for or outcome of successful adaptation despite challenging or threatening circumstances. The study to be presented here asked 9-12 year olds and their teachers from primary schools in disadvantaged areas in South Australia what it was they thought made the difference between 'kids with tough lives who do O.K.' and 'kids with tough lives who don't do O.K.' Using Bronfenbrenner's ecological theory to frame the analysis, this paper explores how the two groups of respondents discussed how the family, the school and the community might help make the difference. While both children and teachers tended to see the role of the family and the community in promoting resilient behaviour in fairly similar terms, the two groups saw very different roles for the school.     

Personalization of mathematics word problems in Taiwan

This study investigated the effects of group personalization of instruction on the mathematics achievement and attitudes of 72 fifth-grade Taiwanese students. Personalization was accomplished by incorporating personal information and preferences provided by students into their mathematics word problems. Students were blocked by ability level, then randomly assigned to a personalized or nonpersonalized version of an instructional program on mathematics word problems. Students made significant pretest-to-posttest gains across treatments and scored significantly higher on personalized than on nonpersonalized posttest problems. However, the personalized treatment did not produce a significant achievement difference over the nonpersonalized one. Significant two-way interactions reflected greater pretest-to-posttest improvement for lower-ability than for higher-ability students and a greater difference between scores on personalized over nonpersonalized posttest problems for lower-ability students. The posttest scores of high-ability students were limited by a ceiling effect. Student attitudes were significantly more favorable toward the personalized instruction.     

Learner control over full and lean computer-based instruction under differing ability levels

This study examined the effects of type of instructional control and program mode on the achievement, option use, time in the program, and attitudes of higher-ability and lower-ability university students. Ability level was determined using participants' Scholastic Aptitude Test (SAT) and American College Testing Assessment (ACT) scores. Participants were randomly assigned to four versions of a computer-delivered instructional program within higher-ability and lower-ability groups. The four versions of the instructional program were created by crossing the two control conditions (learner control, program control) with the two program modes (full, lean). Participants in the full program scored significantly higher on the posttest than those in the lean program, and higher-ability students scored significantly higher than lower-ability students. Learner-control participants chose to view 68% of the optional screens in the full program but only 35% in the lean one, and participants also spent significantly more time in the full version. Participants had more favorable attitudes toward learner control than program control.     

Sociopolitical change and perceived vitality

This paper addresses the issue of how ethnic groups cognitively represent some of the societal forces impinging upon them and relevant outgroups before and after a naturally-occurring event. By utilizing the concept of “vitality” and its attending measurement instrument (the SVQ), Chinese Hong Kong students were required to rate the perceived vitalities of Chinese and Western groups in the Colony as well as their respective languages before and after the Sino-British Treaty. Results showed that the status of the English language was perceived lower in a number of situations after the international agreement. This was complemented by a decrease in the perceived level of immigration and emigration patterns of Westerners, by an increase in their perceived political power locally, and by an increase in the status of the English language internationally. The findings, many of which manifest judgemental reversals from before to after the Treaty, are discussed in terms of the psychological acceptance of the agreement by the Hong Chinese and in the context of maintaining a positive ethnic identity.     

Conceptions of wayfinding: decolonizing science education in pursuit of Native American success

Cultural Studies of Science Education - The purpose of this forum paper is to address how conceptions of “success” in STEM education are culturally informed, and how supporting students...     

A Comparative Study of Electronic Performance Support Systems

ABSTRACT Electronic performance support systems (EPSS) deliver relevant support information to users while they are performing tasks. The present study examined the effect of different types of EPSS on user performance, attitudes, system use and time on task. Employees at a manufacturing company were asked to complete a procedural software task and received support from either an intrinsic, extrinsic, external performance support system or no system at all. Results revealed significant differences on performance, attitudes and use between several treatment groups. The study suggests that providing any kind of EPSS to support task performance is better than having none at all. In addition, designers can improve user performance, attitudes and use by creating systems that integrate with the primary work interface.     

A comparison of the Vane Kindergarten Test with the WPPSI and a measure of Self-Control

This study assessed the concurrent validity of the Vane Kindergarten Test (VKT), using scores on the Wechsler Preschool and Primary Scale of Intelligence (WPPSI) as the criterion measure. In addition, VKT scores were correlated with kindergarten grades and the Self-Control Rating Scale (SCRS). The results indicated that the VKT could be used as a valid screening measure for kindergarten children, but that it tended to overestimate ability and was not a test independent of a child's ability for self-contorl.