浅析阅读理解考试中的测试方法效应问题

本文研究的是不同的测试方法-单项选择和信息转移-是否会在阅读理解考试中产生测试方法效应的问题.除对学生的考试成绩(分数)进行分析外,本研究还进一步对试题的难度值进行了分析,而本研究中试题难度是通过项目反应理论(Item Response Theory)计算得到的.结果显示不同测试方法的确会影响题目难度及考生的考试表现,就试题难度而言信息转移比单项选择更难.     

Teaching social studies to learning disabled high school students: effects of a hypertext study guide

This study investigated the effectiveness of a computer-based study guide using hypertext software to increase textbook comprehension among four learning disabled students enrolled in a remedial high school social studies class. The program provided four levels of instructional cues that matched students to their highest level of independent interaction with a textbook passage, based on item-to-item responses to computer-generated questions. Using alternative forms of a 45-item multiple-choice test, a pre-test/post-test design was arranged, with a retention test given after a 30-day period. Fifteen questions were designated as control items by placing them in the 45-item tests but not in the computer treatment. The computer program consisted of three separate lessons administered across consecutive class sessions, with each followed by a written 15-item multiple choice test containing 10 computer questions and 5 control items. Results indicated a significant gain for pupils on computer items from pre-test to post-test and from pre-test to retention test, while no significant change occurred on control items across measures. A single-case analysis revealed a consistent relationship between gain scores on computer items, reading time on computer, and the number of instructional cues required by students. Two types of non-linear pathways that teacher might consider when constructing study guides are discussed.     

Reading subskill differences between students in Shanghai-China and the US: evidence from PISA 2009

Based on different language systems and educational practices of their respective countries, hypotheses were made regarding how 15-year-old students from Shanghai-China and the US might differ in the 5 reading subskills designated in the Programme for International Student Assessment (PISA) when they have the same overall reading ability (i.e., when their overall reading ability is controlled for). A multilevel analysis was conducted to test the hypotheses using the PISA 2009 reading dataset. When we controlled for students' overall reading ability, individual socioeconomic status (SES), and school mean SES, Shanghai-Chinese students performed significantly better in integrating and interpreting than US students. Further, when we controlled for students' overall reading ability and school mean SES, US students showed significantly higher performance in reading non-continuous texts than Shanghai-Chinese students, whereas US students showed significantly lower performance in reading continuous texts. The results of this study can inform reading instruction and learning in the 2 countries.     

Italian students’ results in the PISA mathematics test: does reading competence matter?

In Italy, from the 2003 reports to the present, the National Institute for the Educational Evaluation of Instruction and Training (INVALSI) has conducted research on Programme for International Student Assessment (PISA) results in order to understand Italian students’ low achievement in mathematics. In the present paper, data from a representative sample of 15-year-old Italian students who participated in PISA 2012 were analysed. This study’s primary aim is to verify how students’ linguistic competences are associated with their performance in mathematics. For the evaluation of the impact of item reading demand on students’ performance, we selected 24 mathematics items with a high reading demand and 31 mathematics items with a low reading demand, as classified by Italian language and methodology experts. Repeated measure variance analyses were conducted. The results showed differences in function of gender: females are advantaged in mathematics items with a high reading demand, independent of their level of reading literacy. In contrast, males are advantaged in mathematics items with a low reading demand, independent of their level of reading literacy. Possible policy implications are discussed.     

When Is Reading Also Writing: Sources of Individual Differences on the New Reading Performance Assessments

This research examined component processes that contribute to performance on one of the new, standards-based reading tests that have become a staple in many states. Participants were 60 Grade 4 students randomly sampled from 7 classrooms in a rural school district. The particular test we studied employed a mixture of traditional (multiple-choice) and performance assessment approaches (constructed-response items that required written responses). Our findings indicated that multiple-choice and constructed-response items enlisted different cognitive skills. Writing ability emerged as an important source of individual differences in explaining overall reading ability, but its influence was limited to performance on constructed-response items. After controlling for word identification and listening, writing ability accounted for no variance in multiple-choice reading scores. By contrast, writing ability accounted for unique variance in reading ability, even after controlling for word identification and listening skill, and explained more variance in constructed-response reading scores than did either word identification or listening skill. In addition, performance on the multiple-choice reading measure along with writing ability accounted for nearly all of the reliable variance in performance on the constructed-response reading measure.     

命题者：影响阅读理解测试效度的一个因素

本研究应用项目反应理论,从被试的阅读能力值和题目的难度值这两个方面,分析阅读理解测试中多项选择题命题者对考试效度的影响。实验设计中,将两组被试同时施测于一项“阅读水平测试”,根据测试结果估计出的两组被试能力值之间无显著性差异。再次将这两组被试分别施测于两位不同命题者所命制的题目,尽管这些题目均产生于相同的阅读材料,且题目的难度值之间并没有显著性差异,被试的表现却显著不同。Rasch模型认为,被试表现由被试能力和试题难度共同决定。因此,可以推测,这是由于不同命题者所命制的题目影响了被试的表现,并进而影响了使用多项选择题进行阅读理解测试的效度。     

Sources of difficulty in assessment: example of PISA science items

ABSTRACT

The understanding of what makes a question difficult is a crucial concern in assessment. To study the difficulty of test questions, we focus on the case of PISA, which assesses to what degree 15-year-old students have acquired knowledge and skills essential for full participation in society. Our research question is to identify PISA science item characteristics that could influence the item’s proficiency level. It is based on an a-priori item analysis and a statistical analysis. Results show that only the cognitive complexity and the format out of the different characteristics of PISA science items determined in our a-priori analysis have an explanatory power on an item’s proficiency levels. The proficiency level cannot be explained by the dependence/independence of the information provided in the unit and/or item introduction and the competence. We conclude that in PISA, it appears possible to anticipate a high proficiency level, that is, students’ low scores for items displaying a high cognitive complexity. In the case of a middle or low cognitive complexity level item, the cognitive complexity level is not sufficient to predict item difficulty. Other characteristics play a crucial role in item difficulty. We discuss anticipating the difficulties in assessment in a broader perspective.     

从批判性思维培养的视角看中美语文教材的差异——PISA测试结果的启示

批判性思维是语文素养的重要能力,而阅读是语文教学的核心。国际能力测试PISA显示,我国学生阅读总体水平世界第一,但与批判性思维相关的部分成绩较低。这种现象引起了学界的广泛重视。我国教材在自然描写、情感道德、历史文化传统、爱国主义、生活知识与技能等方面的比例与美国的教材显著不同;在连续性文本和非连续性文本比例上也与美国差异巨大。阅读内容及其文本形式关乎我国学生阅读能力中批判性思维的培养。建议教材中增加与学生生活、社会现实问题联系密切的以非连续性文本形式呈现的内容。     

Explaining the difference between PISA 2009 reading scores in Finland and Estonia

The aim of the study was to explain the difference between the Programme for International Student Assessment (PISA) 2009 reading results for Finland and Estonia using characteristics of teaching and learning, and characteristics of the overall development of these countries. PISA data were collected via a reading test and student questionnaires from 4,729 students in Estonia and 5,810 students in Finland. Regression analysis made it possible to identify the speed of the rise in PISA scores in relation to the selected variables. The speed was multiplied by the value of the variable to calculate the effect of the variable. The effects of the joy of reading and the diversity of reading materials were greater in Finland, but the effects of metacognition and online reading activities were greater in Estonia. The countries had different values for several indices of development, and this was concordant with the difference in the PISA scores.     

Investigating the Effect of Different Selected-response Item Formats for Reading Comprehension

While previous research has identified numerous factors that contribute to item difficulty, studies involving large-scale reading tests have provided mixed results. This study examined five selected-response item types used to measure reading comprehension in the Pearson Test of English Academic: a) multiple-choice (choose one answer), b) multiple-choice (choose multiple answers), c) re-order paragraphs, d) reading (fill-in-the-blanks), and e) reading and writing (fill-in-the-blanks). Utilizing a multiple regression approach, the criterion measure consisted of item difficulty scores for 172 items. 18 passage, passage-question, and response-format variables served as predictors. Overall, four significant predictors were identified for the entire group (i.e., sentence length, falsifiable distractors, number of correct options, and abstractness of information requested) and five variables were found to be significant for high-performing readers (including the four listed above and passage coherence); only the number of falsifiable distractors was a significant predictor for low-performing readers. Implications for assessing reading comprehension are discussed.     

PISA式汉语阅读测验的编制与维度评价

PISA测验着眼于学生的终生发展,其测验编制思想给各国教育评价带来了深刻的变革。本研究在PISA阅读测验理论与框架基础上,编制了PISA式汉语阅读测验。该测验包含三篇阅读材料,共18个测验项目。通过对测验难度、区分度、信度、效度的检测,并使用全息Bifactor模型进行维度评价。结果表明,编制的PISA式汉语阅读测验难度适中,具有较好区分度,信效度基本合格。同时,基本达到PISA对阅读测验能力结构的要求,较好地考查了学生的一般阅读理解能力,以及信息提取、文本解释、反思和评价等三个子维度的能力。     

Relationships between the use of test results and US students’ academic performance

In this study, we examined relationships between the use of test results and US students’ math, reading, and science performance in the Programme for International Student Assessment (PISA) 2009. Based on a literature review, we hypothesized that the 16 items in the PISA school questionnaire, which are related to the use of test results, can be categorized according to 4 factors. We validated this hypothesized factor structure using a confirmatory factor analysis and then obtained composite scores for each factor. As revealed by a multilevel analysis, when student and school demographic variables were controlled for, using test results to hold schools accountable to authority and the public was significantly positively related to students’ performance across all 3 subjects. No statistically significant relationship, however, was detected between students’ performance and the following uses of test scores: informing parents of their children’s performance, providing information for instructional purposes, and evaluating teachers and principals.     

Teacher prediction of student difficulties while solving a science inquiry task: example of PISA science items

This study focuses on the teachers’ predictions of the students’ performances – in particular the middle-low achievers – while solving tasks testing inquiry competencies. The tasks come from PISA science. More specifically we study science teachers’ predictions for several aspects: levels of difficulty of the tasks, the potential sources of difficulty and the potential difficulty in solving it for medium-low achievers. We also study what assessed competencies are identified by science teachers in the tasks. Our approach is a questionnaire-based study. A sample of French teachers in science and technology (125) responded to the questionnaire. The teachers show a rather good ability to predict inquiry task levels of difficulty for medium-low achievers and are able to identify relevant potential sources of difficulty or easiness in the items. However, they are not aware of some essential difficulties that medium-low students encounter while solving science inquiry tasks. Moreover, the teachers have difficulty identifying the competencies that are tested by an item.     

Ethnic DIF in Reading Tests With Mixed Item Formats

This article presents a study of ethnic Differential Item Functioning (DIF) for 4th-, 7th-, and 10th-grade reading items on a state criterion-referenced achievement test. The tests, administered 1997 to 2001, were composed of multiple-choice and constructed-response items. Item performance by focal groups (i.e., students from Asian/Pacific Island, Black/African American, Native American, and Latino/Hispanic origins) were compared with the performance of White students using simultaneous item bias and Rasch procedures. Flagged multiple-choice items generally favored White students, whereas flagged constructed-response items generally favored students from Asian/Pacific Islander, Black/African American, and Latino/Hispanic origins. Content analysis of flagged reading items showed that positively and negatively flagged items typically measured inference, interpretation, or analysis of text in multiple-choice and constructed-response formats. Items that were not flagged for DIF generally measured very easy reading skills (e.g., literal comprehension) and reading skills that require higher level thinking (e.g., developing interpretations across texts and analyzing graphic elements).     

Identifying processes underlying the multimedia effect in testing: An eye-movement analysis

Test items become easier when a representational picture visualizes the text item stem; this is referred to as the multimedia effect in testing. To uncover the processes underlying this effect and to understand how pictures affect students' item-solving behavior, we recorded the eye movements of sixty-two schoolchildren solving multiple-choice (MC) science items either with or without a representational picture. Results show that the time students spent fixating the picture was compensated for by less time spent reading the corresponding text. In text-picture items, students also spent less time fixating incorrect answer options; a behavior that was associated with better test scores in general. Detailed gaze likelihood analyses revealed that the picture received particular attention right after item onset and in the later phase of item solving. Hence, comparable to learning, pictures in tests seemingly boost students' performance because they may serve as mental scaffolds, supporting comprehension and decision making.     

Gender DIF in Reading and Mathematics Tests With Mixed Item Formats

This was a study of differential item functioning (DIF) for grades 4, 7, and 10 reading and mathematics items from state criterion-referenced tests. The tests were composed of multiple-choice and constructed-response items. Gender DIF was investigated using POLYSIBTEST and a Rasch procedure. The Rasch procedure flagged more items for DIF than did the simultaneous item bias procedure—particularly multiple-choice items. For both reading and mathematics tests, multiple-choice items generally favored males while constructed-response items generally favored females. Content analyses showed that flagged reading items typically measured text interpretations or implied meanings; males tended to benefit from items that asked them to identify reasonable interpretations and analyses of informational text. Most items that favored females asked students to make their own interpretations and analyses, of both literary and informational text, supported by text-based evidence. Content analysis of mathematics items showed that items favoring males measured geometry, probability, and algebra. Mathematics items favoring females measured statistical interpretations, multistep problem solving, and mathematical reasoning.     

Description and interactions of informative text structure knowledge and skills of French-speaking Grade 6 students

This study was conducted in the province of Québec, Canada, among French-speaking Grade 6 students (n?=?175) in the context of a school curriculum that does not clearly address text structure and main idea instruction. It aims to understand whether these students can identify informative text structures and main ideas in isolated paragraphs, comprehend main ideas and text structure in an informative text, and write a short structured informative text. It also describes relationships between these knowledge and skills coming from different reading and writing tasks. Three assessments relative to informative text structures were administered: a multiple-choice test on text structure knowledge and identification of main ideas, a reading comprehension test, and a short writing task. Results revealed that students performed better in the multiple-choice assessment compared to other assessments. Correlations between variables stemming from the three assessments were significant but their effect sizes were low to moderate. A hypothesized model was investigated via a path analysis suggesting that structure knowledge and main idea identification influence reading comprehension, which then influence writing.     

阅读任务对中学生阅读素养的影响与启示--基于PISA 2018的数据分析

为了探究阅读任务对学生阅读素养的影响,基于PISA 2018中国北京、上海、江苏和浙江四省市学生的测评数据,将经常与较少执行阅读任务的学生的阅读素养测评结果进行比较。研究发现:不同的阅读任务对学生阅读能力的影响不一,其中“提出看法”与“回答问题”两项任务更有利于提升学生阅读能力;9项阅读任务都有助于发展学生的阅读兴趣,优化学生阅读自我概念;在培养学生阅读元认知能力方面,9项阅读任务的效果并不如人意。这些启示语文教师要基于不同阅读任务的特性,设计最佳任务组合优化教学;反思阅读任务的执行过程,提高阅读任务有效性;强化阅读策略教学,提升学生阅读元认知能力。     

Does difficulty-based item order matter in multiple-choice exams? (Empirical evidence from university students)

This empirical study aimed to investigate the impact of easy first vs. hard first ordering of the same items in a paper and-pencil multiple-choice exam on the performances of low, moderate, and high achiever examinees, as well as on the item statistics. Data were collected from 554 Turkish university students using two test forms, which included the same multiple-choice items ordered reversely, i.e. easy first vs. hard first. Tests included 26 multiple-choice items about the introductory unit of “Measurement and Assessment” course. The results suggested that sequencing the multiple-choice items in either direction from easy to hard or vice versa did not affect the test performances of the examinees no matter whether they are low, moderate or high achiever examinees. Finally, no statistically significant difference was observed between item statistics of both forms, i.e. the difficulty (p), discrimination (d), point biserial (r), and adjusted point biserial (adj. r) coefficients.     

Gotcha! Catching Kids During Mindless Reading

The purpose of the current study was to examine the mindless reading behavior of children. Across two studies, 2nd-grade students read passages while their eye movements were monitored. Trained raters then identified mindless reading behaviors from the eye movement records. Several important findings emerged. We were able to reliably identify mindless reading behavior in children using eye-tracking methodology, which was characterized by shorter gaze durations and total time, more skipping, and in general a more erratic reading pattern than on-task reading behavior. On the other hand, on-task reading behavior was characterized by an increase in fixations and regressions, especially intraword regressions. Word frequency effects were attenuated during mindless reading. In addition, the children who engaged in mindless reading had weaker reading achievement profiles compared to children who read the entire passage.