Using person response functions to investigate areas of person misfit related to item characteristics

Individual person fit analyses provide important information regarding the validity of test score inferences for an individual test taker. In this study, we use data from an undergraduate statistics test (N = 1135) to illustrate a two-step method that researchers and practitioners can use to examine individual person fit. First, person fit is examined numerically with several indices based on the Rasch model (i.e., Infit, Outfit, and Between-Subset statistics). Second, person misfit is presented graphically with person response functions, and these person response functions are interpreted using a heuristic. Individual person fit analysis holds promise for improving score interpretation in that it may detect potential threats to validity of score inferences for some test takers. Individual person fit analysis may also highlight particular subsets of items (on which a test taker performs unexpectedly) that can be used to further contextualize her or his test performance.     

Meta‐evaluation in clinical anatomy: A practical application of item response theory in multiple choice examinations

The nature of anatomy education has changed substantially in recent decades, though the traditional multiple‐choice written examination remains the cornerstone of assessing students' knowledge. This study sought to measure the quality of a clinical anatomy multiple‐choice final examination using item response theory (IRT) models. One hundred seventy‐six students took a multiple‐choice clinical anatomy examination. One‐ and two‐parameter IRT models (difficulty and discrimination parameters) were used to assess item quality. The two‐parameter IRT model demonstrated a wide range in item difficulty, with a median of ?1.0 and range from ?2.0 to 0.0 (25th to 75th percentile). Similar results were seen for discrimination (median 0.6; range 0.4–0.8). The test information curve achieved maximum discrimination for an ability level one standard deviation below the average. There were 15 items with standardized loading less than 0.3, which was due to several factors: two items had two correct responses, one was not well constructed, two were too easy, and the others revealed a lack of detailed knowledge by students. The test used in this study was more effective in discriminating students of lower ability than those of higher ability. Overall, the quality of the examination in clinical anatomy was confirmed by the IRT models. Anat Sci Educ 3:17–24, 2010. © 2009 American Association of Anatomists.     

利用网络教学平台开展过程性评价的研究与实践

当前,网络教学平台是开展过程性评价的良好工具,但教学实践中仍然存在很多障碍亟待解决。通过设置过程性评价的三个维度进行指标分解,在此基础上构建评价模型。对照模型指标,采用调查、观察和访谈方式了解网络教学平台的使用现状和学生的真实看法,总结出评价主体、评价内容、评价标准等方面的主要问题,并进一步分析产生原因,对应学习动机、学习过程、学习效果三个维度提出合理的改进建议。     

Using Dialogic Writing Assessment to Support the Development of Historical Literacy

Though discipline-specific approaches to literacy instruction can support adolescents' academic literacy and identity development, scant attention has been paid to ways of targeting such instruction to address individual student needs. Dialogic writing assessment is an approach to conducting writing conferences that foregrounds students' composing process so that teachers can assess and support that process with instructional feedback. Because such feedback is immediate, teachers can observe how students take it up. While dialogic assessment has shown promise as an approach to revealing and supporting students' writing processes in English Language Arts classrooms, it remains to be explored how this approach can support developing writers in other subject areas. This paper offers an analytic narrative account of how a high school social studies teacher used this method to support the writing process of one student, exploring what the method revealed about the challenges the student faced in writing about history, the gaps and misconceptions in their understanding of history and the intersection between the two. We discuss how certain ‘mediational moves’ the teacher employed enabled the student to compose collaboratively with the teacher, and in this collaborative composing, to capture ideas that she later used in her independent writing.     

A systematic review of item response theory in language assessment: Implications for the dimensionality of language ability

The present study conducted a systematic review of the item response theory (IRT) literature in language assessment to investigate the conceptualization and operationalization of the dimensionality of language ability. Sixty-two IRT-based studies published between 1985 and 2020 in language assessment and educational measurement journals were first classified into two categories based on a unidimensional and multidimensional research framework, and then reviewed to examine language dimensionality from technical and substantive perspectives. It was found that 12 quantitative techniques were adopted to assess language dimensionality. Exploratory factor analysis was the primary method of dimensionality analysis in papers that had applied unidimensional IRT models, whereas the comparison modeling approach was dominant in the multidimensional framework. In addition, there was converging evidence within the two streams of research supporting the role of a number of factors such as testlets, language skills, subskills, and linguistic elements as sources of multidimensionality, while mixed findings were reported for the role of item formats across research streams. The assessment of reading, listening, speaking, and writing skills was grounded within both unidimensional and multidimensional framework. By contrast, vocabulary and grammar knowledge was mainly conceptualized as unidimensional. Directions for continued inquiry and application of IRT in language assessment are provided.     

提升幼儿教师评价素养的制约因素及建议

幼儿教师评价素养是直接影响幼儿发展评价有效性的重要因素,也是决定评价是否发挥其实质性功能与作用的关键要素。它包括评价理念与专业品质、评价态度与主体意识、评价策略与实践智慧等三方面内容与判断标准。目前,国内幼儿教师评价素养的提升尚受到外部支持系统以及教师自身知识与技能水平的制约,因此需要通过提供积极的外部环境支持、加强对幼儿教师评价知识与技能的园本培训、探寻幼儿教师评价素养自主提升等途径,切实提高幼儿教师评价素养。     

论英语教师语言测试素养的培养

随着英语测试特别是大规模标准化测试在当代社会当中的重要性日益凸显,英语教师的语言测试素养的培养也逐渐成为国内外学者关注的问题。本文首先探讨了语言测试素养的定义,然后从英语教师的职业属性、大规模标准化测试给英语教师的冲击以及测试对于教、学的促进作用等三方面探讨培养语言教师测试素养的重要性,并提出了语言测试素养的培养途径。     

A new approach to test score equating using item response theory with fixed C-parameters

Because parameter estimates from different calibration runs under the IRT model are linearly related, a linear equation can convert IRT parameter estimates onto another scale metric without changing the probability of a correct response (Kolen & Brennan, 1995, 2004). This study was designed to explore a new approach to finding a linear equation by fixing C-parameters for anchor items in IRT equating. A rationale for fixing C-parameters for anchor items in IRT equating can be established from the fact that the C-parameters are not affected by any linear transformation. This new approach can avoid the difficulty in getting accurate C-parameters for anchor items embedded in the application of the IRT model. Based upon our findings in this study, we would recommend using the new approach to fix C-parameters for anchor items in IRT equating. This work was supported by a Korea Research Foundation Grant funded by the Korean Government (MOEHRD, Basic Research     

An Assessment of the Quantitative Literacy of Undergraduate Students

Quantitative literacy (QLT) represents an underlying higher-order construct that accounts for a person's willingness to engage in quantitative situations in everyday life. The purpose of this study is to retest the construct validity of a model of quantitative literacy (Wilkins, 2010). In this model, QLT represents a second-order factor that accounts for the interrelationship among three first-order factors: mathematical beliefs, mathematical disposition, and mathematical cognition. Using data from two samples of undergraduate students (n = 186 and n = 184), a series of confirmatory factor analyses were conducted. Results supported the hierarchical three-factor structure and confirmed its factorial invariance across multiple groups of students. Latent QLT scores from the model were used to compare and rank student QLT by gender and class standing.     

What English Counts as Writing Assessment? An Australian Move to Mainstream Critical Literacy

This article focuses on the demands of assessment tasks to address the question: What is entailed in becoming recognised as an accomplished student‐writer? In taking up this focus, the writers use authentic samples to show how tasks for assessing writing can be read as instantiations of particular approaches to English education. They investigate how the tasks inevitably draw on cultural knowledges as a primary resource and raise the issue of what is involved when writing assessment moves away, as it is doing in some Australian schools, from concerns with personal voice and individual growth to a socially critical, discourse‐oriented approach.     

质性评价的有效尝试:通过学生成长记录袋实现评定的发展性功能

现代教育评价强调要在日常教育教学中发挥评价的激励和发展功能，学生成长记录袋正是适应这种要求的一种质性评定方法。本文以国内外成功经验的基础，介绍了学生 成长记录袋的理论背景与主要特征，并探讨了通过行动研究推进这种方法的具体策略。     

Stages of moral judgment development: Applying item response theory to Defining Issues Test data

ABSTRACT

The Defining Issues Test (DIT) has been the dominant measure of moral development. The DIT has its roots in Kohlberg’s original stage theory of moral judgment development and asks respondents to rank a set of stage typed statements in order of importance on six stories. However, the question to what extent the DIT-data match the underlying stage model was never addressed with a statistical model. Therefore, we applied item response theory (IRT) to a large data set (55,319 cases). We found that the ordering of the stages as extracted from the raw data fitted the ordering in the underlying stage model good. Furthermore, difficulty differences of stages across the stories were found and their magnitude and location were visualized. These findings are compatible with the notion of one latent moral developmental dimension and lend support to the hundreds of studies that have used the DIT-1 and by implication support the renewed DIT-2.     

Interest Assessment and the Content Area Literacy Environment: Challenges for Research and Practice

Through our exploration of the assessment of interest, we highlight 5 issues that emerge in the effort to assess interest in content area literacy environments. First, we compare how interest is defined and generally how it has been assessed. Second, we focus more specifically on the assessment of text-based interest, and consider its assessment in less explored realms of the content area classroom that include hypertext and discussions. Third, we investigate how interest develops as students progress through their educational careers and how it is maintained over time. Fourth, we explore how assessing interest involves examining other student factors that are closely related to interest such as knowledge, importance, and strategic processing. Finally, we offer thoughts regarding directions educators can take in assessing interest within the complex literacy environment of today's content area classroom.     

CILT2000: Using Technology to Support Ongoing Formative Assessment in the Classroom

This article emphasizes the importance of formative assessment in the teaching and learning process and the role technology can play in advancing assessment practices. It describes work undertaken by attendees of the CILT workshop, highlighting key issues that were discussed and areas for further development including how to address important equity concerns.     

用正确的评价激励学生走向成功

为了激励学生走向成功,要学习和研究新课程标准和教改实验区的先进经验,确立新的正确评价理念,引导家长改变子女成材观,重视形成性的评价作用,尝试评价内容的多元化,以及开展多主体评价和建立学生档案柜等。     

美国加州州立大学信息素养评价的研究

通常学校中的信息素养评价方法都是基于考试的,难免有很多局限性。本则主要介绍了美国加州州立大学基于情境的信息素养评价的研究及其研究结果,并且论述了此种评价方法给我们带来的启示。     

Assessment of the quality and generalizability of the revised UCLA loneliness scale in Chinese and Korean community-dwelling elderly populations using item response theory (IRT)-Rasch modeling and hybrid IRT-logistic regression

ABSTRACT

Objectives: This study aims to test the dimensionality, reliability, and item quality of the revised UCLA loneliness scale as well as to investigate the differential item functioning (DIF) of the three dimensions of the revised UCLA loneliness scale in community-dwelling Chinese and Korean elderly individuals.

Method: Data from 493 elderly individuals (287 Chinese and 206 Korean) were used to examine the revised UCLA loneliness scale. The Research model based on item response theory (IRT) was used to test dimensionality, reliability, and item fit. The hybrid ordinal logistic regression-IRT test was used to evaluate DIF.

Results: Item separation reliability, person reliability, and Cronbach’s alpha met the benchmarks. The quality of the items in the three-dimension model met the benchmark. Eight items were detected as significant DIF items (at α < .01). The loneliness level of Chinese elderly individuals was significantly higher than that of Koreans in Dimensions 1 and 2, while Korean elderly participants showed significantly higher loneliness levels than Chinese participants in Dimension 3. Several collected demographic characteristics and loneliness levels were more highly correlated in Korean elderly individuals than in Chinese elderly individuals.

Conclusion: Analysis using the three dimensions is reasonable for the revised UCLA loneliness scale. Good item quality and the items of this measure suggest that the revised UCLA loneliness can be used to assess the preferred latent traits. Finally, the differences between the levels of loneliness in Chinese and Korean elderly individuals are associated with the factors of loneliness.     

Assessing the unidimensionality of the School and College Ability Test (SCAT,Spanish version) using non-parametric methods based on item response theory

The aim of this work is to analyze the dimensional structure of the Spanish version of the School and College Ability Test, employed in the process for the identification of students with high intellectual abilities. This test measures verbal and mathematical (or quantitative) abilities at three levels of difficulty: elementary (3rd, 4th, and 5th years in Primary school), intermediate (6th year in Primary school plus the 1st and 2nd years of Compulsory Secondary School or ESO), and advanced (3rd and 4th years of ESO plus the 1st and 2nd years of bachillerato – equivalent to High school). For each level there are two forms, X and Y. The research was undertaken with the results obtained from the application carried out for the validation and norming of the Spanish version of the test, and for which a representative sample of students from Navarre at these mentioned levels was taken. This study assessed the possible unidimensionality of the simplicity or the complexity of the structure of this test as an essential aspect of construct validity. To this end, the results were triangulated for the classic factorial techniques and non-parametric methods based on the item response theory.     

当代西方国家第二语言读写能力评价改革述评

第二语言读写能力的高低在一定程度上反映了第二语言教学的成效。文章论述了当代西方发达国家对学生第二语言读写能力评价的改革形式和共同特点,以期对我国的外语（第二语言）读写能力评价改革提供参考。     

Using Information and Communication Technology (ICT) to Foster Literacy and Facilitate Discourse Within the Classroom

The literacy skills of students is an issue of great policy importance which has attracted considerable interest in the literature. At the same time, much recent work has been done on the efficacy of learning communities. This paper examines the extent to which fostering learning communities can improve literacy skills. It is shown that it is possible to achieve large improvements in writing and reading by fostering a community of learners that focuses on scientific inquiries utilizing computer technology. The results lend support to the hypothesis that learning communities may be an important instrument in enhancing the reasoning, problem solving and learning strategies of students. This suggests that more work needs to be done in this area of educational research.

Les compétences des élèves en lecture sont un sujet d'extrème importance qui a provoqué un intérêt considérable dans la littérature. Au même moment beaucoup de travaux récents ont été menés sur l'efficacité des établissements d'enseigement. Cet article examine jusqu'à quel point il faut encourager des communautés d'apprentissage peut améliorer les compétences en lecture. Il montre qu'il est possible d'obtenir des améliorations importantes pour la lecture et l'écriture en développant une communauté d'apprenants qui se concentre sur les résultats scientifiques des enquêtes utilisant la technologie de l'ordinateur. Les résultats confirment l'hypothèse que les communautés d'apprentissage peuvent être un instrument important pour améliorer le raisonnement, la solution des problèmes et les stratégies d'apprentissage des élèves. L'article suggère qu'il faut travailler davantage dans ce domaine de la recherche en éducation.

Die Lese- und Rechtschreibfähigkeit von Studenten hat einen hohen politischen Rang, der auch ein entsprechendes Interesse in der Fachliteratur erzeugt hat. Gleichzeitig ist in der letzten Zeit viel Arbeit in die Untersuchung der Wirksamkeit von Lerngemeinschaften gesteckt worden. Dieses Papier überprüft, in welchem Umfang die Förderung der Lerngemeinschaften den Bildungsgrad verbessern kann. Es wird gezeigt, daß es möglich ist, große Verbesserungen im Schreiben und in der Lesefähigkeit durch Förderung einer Lerngemeinschaft, die sich auf wissenschaftliche Anfragen per Nutzung von Computertechnologie konzentriert, zu erreichen. Die Resultate stützen die Hypothese, dass Lerngemeinschaften ein wichtiges Instrument zur Verbesserung von Argumentationsfähigkeit, Problemlöseverhalten und das Entwickeln von Lernstrategien der Kursteilnehmer sein können. Daher sollte die Arbeit in diesem Bereich der pädagogischen Forschung verstärkt werden.