A Bifactor Analysis for a Mode-of-Administration Effect

This study investigated the usefulness of the bifactor model in the investigation of score equivalence from computerized and paper-and-pencil formats of the same reading tests. Concerns about the equivalence of the paper-and-pencil and computerized formats were warranted because of the use of reading passages, computer unfamiliarity of primary school students, teacher versus computer administration of the tests, and slightly lower scores on the computerized format than on the paper-and-pencil format across all 4 grades. A confirmatory item factor analysis implemented through the bifactor model in TESTFACT indicated that the best-fitting model had a general factor as well as skill-group factors. This model was more consistent with the data than a model with 2 method factors, paper-and-pencil and computer administration. In addition, the general and skill factor loadings for most of the items were reasonable. Although several instances of negative loadings were found for items on the skill factors, these did not appear to have any practical importance. As a result, the bifactor model proved useful for studying paper-and-pencil and computerized score equivalence because of the reasonable results and delineation of loadings for the method and skill factors at the item level as well as for the general factor.     

Response Time Differences Between Computers and Tablets

Item response time data were used in investigating the differences in student test-taking behavior between two device conditions: computer and tablet. Analyses were conducted to address the questions of whether or not the device condition had a differential impact on rapid guessing and solution behaviors (with response time effort used as an indicator) as well as on the time that students spent on the test (reading, mathematics, and science) or a given item type (such as drag-and-drop and fill in blank). Further analyses were conducted to examine if the potential impact of device conditions varied by gender and ethnicity groups. Overall there were no significant differences in response time effort related to device, although some differences related to item type and test sequence were noted. Students tended to spend slightly more time when taking the tests and certain types of items on the tablet than on the computer. No interactions of device with gender or ethnicity were observed. Follow-up research on the item time thresholds is discussed.     

Test item linguistic complexity and assessments for deaf students

Linguistic complexity of test items is one test format element that has been studied in the context of struggling readers and their participation in paper-and-pencil tests. The present article presents findings from an exploratory study on the potential relationship between linguistic complexity and test performance for deaf readers. A total of 64 students completed 52 multiple-choice items, 32 in mathematics and 20 in reading. These items were coded for linguistic complexity components of vocabulary, syntax, and discourse. Mathematics items had higher linguistic complexity ratings than reading items, but there were no significant relationships between item linguistic complexity scores and student performance on the test items. The discussion addresses issues related to the subject area, student proficiency levels in the test content, factors to look for in determining a "linguistic complexity effect," and areas for further research in test item development and deaf students.     

Measuring Socioeconomic Status at Individual and Collective Levels

This study investigated the multilevel dimensionality of socioeconomic status and its relationship to reading achievement in 23 countries. Different factor structures of SES were found at different levels’ observations and in different countries. The study showed that the cultural dimension strongly related to student reading performance, while the school general capital dimension explained a large part of the between-school reading achievement differences. Most interestingly, the factor relationship between SES and reading achievement at the school level varies greatly across countries. It was argued that these variations might be due to the differences in the centralized versus decentralized educational finance, tracking mechanism and some social characteristics in different countries.     

The Role of Students’ Home Language in Science Achievement: A multilevel approach

The present study aimed to identify the role of both student- and school-level characteristics in primary school students’ achievement in the science curriculum. As societies become more culturally and linguistically diverse, many students enter the classroom with a home language that is different from the language of instruction used at school. This study takes into account both the home language and literacy in the language of instruction in relation to student achievement in science subjects. Questionnaires, reading performance tests, and science achievement tests were administered to 1,761 fourth-grade students from 67 schools across Flanders (Belgium). Multilevel hierarchical regression analyses show that the home language and literacy in the language of instruction play an important role in science achievement at the student level, next to gender and socioeconomic status. Students with a home language that is different from the language of instruction experience difficulties with science subjects. Moreover, the higher students’ performance on reading comprehension and self-assessed proficiency in the language of instruction, the higher their score on science achievement tests. At the school level, a school's teachability expectations are one of the key factors related to students’ science achievement. Limitations of this study and future directions for research are discussed.     

The Development of Metacognition in Primary School Learning Environments

Constructivist ideas have influenced recent major innovations in Dutch secondary education and new curricula for reading and math in primary education, for example, pay much more attention to metacognition than before. In our study, we compared the growth of student metacognition in varying learning environments, direct instruction, and cognitive apprenticeship in primary school. The study also included a control group of teachers. In order to measure metacognition we developed a questionnaire, with separate parts for metacognitive skills and metacognitive knowledge. In the item selection procedure we made use of item response modeling. It was found that in the direct instruction and the cognitive apprenticeship group the pupils had higher scores on metacognitive skills and metacognitive knowledge compared to the control group pupils. No clear differences were found between direct instruction and cognitive apprenticeship. Interactions of learning environment and student intelligence were non-significant for both output measures.     

An Exploratory Analysis of Differential Item Functioning and Its Possible Sources in a Higher Education Admissions Context

We analyzed a pool of items from an admissions test for differential item functioning (DIF) for groups based on age, socioeconomic status, citizenship, or English language status using Mantel-Haenszel and item response theory. DIF items were systematically examined to identify its possible sources by item type, content, and wording. DIF was primarily found in the citizenship group. As suggested by expert reviewers, possible sources of DIF in the direction of U.S. citizens was often in Quantitative Reasoning in items containing figures, charts, tables depicting real-world (as opposed to abstract) contexts. DIF items in the direction of non-U.S. citizens included “mathematical” items containing few words. DIF for the Verbal Reasoning items included geocultural references and proper names that may be differentially familiar for non-U.S. citizens. This study is responsive to foundational changes in the fairness section of the Standards for Educational and Psychological Testing, which now consider additional groups in sensitivity analyses, given the increasing demographic diversity in test-taker populations.     

Embedded Field Test Item Statistics: Can They Be Trusted for Estimating Student Proficiency?

Assessment items are commonly field tested prior to operational use to observe statistical item properties such as difficulty. Item parameter estimates from field testing may be used to assign scores via pre-equating or computer adaptive designs. This study examined differences between item difficulty estimates based on field test and operational data and the relationship of such differences to item position changes and student proficiency estimates. Item position effects were observed for 20 assessments, with items in later positions tending to be more difficult. Moreover, field test estimates of item difficulty were biased slightly upward, which may indicate examinee knowledge of which items were being field tested. Nevertheless, errors in field test item difficulty estimates had negligible impacts on student proficiency estimates for most assessments. Caution is still warranted when using field test statistics for scoring, and testing programs should conduct investigations to determine whether the effects on scoring are inconsequential.     

Computer Use and Academic Development in Secondary Schools

Several studies provide preliminary evidence that computer use is positively related to academic performance; however, no clear relationship has yet been established. Using a national database, we analyzed how students’ school behavior (i.e., evaluated by English and math teachers) and standardized test scores (e.g., math and reading) are related to computer use for school work or other than school work for the tenth grade student. While controlling socioeconomic status (SES), home computer access, parental involvement, and students’ academic expectation variables, the students who used a computer for one hour per day showed more positive school behaviors and higher reading and math test scores. This article concludes with implications for future study to better understand the impact of computer use on adolescent academic development.     

Motivation across domains: Do goals and attributions change with subject matter for Grades 4 and 5 students?

This study explored a theory for motivation which included aspects of both attribution theory and goal theory. Motivational variables included beliefs about intelligence (entity or incremental), goal orientation (mastery/learning, performance-approach, performance-avoidance) and avoidant behaviours. Grades 4 and 5 students from a large, metropolitan school district were surveyed regarding these motivational variables across the academic domains of reading and mathematics. The relationships among these motivational variables were explored, as well as differences across domains. A diverse sample allowed differences across ethnic groups and socioeconomic status to be studied. Results indicate that children could have a generalised notion of motivation that becomes differentiated when students are asked to reflect on these variables within specified domains. The existence of few differences across ethnic and socioeconomic groups suggest that instructional style could be a more powerful influence than either of these variables.     

The Impact of Item Stem Format on the Dimensional Structure of Mathematics Assessments

Item stem formats can alter the cognitive complexity as well as the type of abilities required for solving mathematics items. Consequently, it is possible that item stem formats can affect the dimensional structure of mathematics assessments. This empirical study investigated the relationship between item stem format and the dimensionality of mathematics assessments. A sample of 671 sixth-grade students was given two forms of a mathematics assessment in which mathematical expression (ME) items and word problems (WP) were used to measure the same content. The effects of mathematical language and reading abilities in responding to ME and WP items were explored using unidimensional and multidimensional item response theory models. The results showed that WP and ME items appear to differ with regard to the underlying abilities required to answer these items. Hence, the multidimensional model fit the response data better than the unidimensional model. For the accurate assessment of mathematics achievement, students’ reading and mathematical language abilities should also be considered when implementing mathematics assessments with ME and WP items.     

The Effects of Within-Class Ability Grouping on Academic Achievement in Early Elementary Years

Abstract

By incorporating two theoretical frameworks this study examines how school characteristics shape first-grade reading ability-grouping practices, and how this, in turn, affects students’ reading achievement. The author uses the data from the Early Childhood Longitudinal Study and applies the propensity-score method to examine whether first-grade ability grouping improves student achievement, whether ability grouping increases achievement inequalities, and whether its effects vary by student initial abilities and/or school contexts. Findings support an argument that ability grouping is an organizational response to problems of diversity in the student body. Schools that use ability grouping are likely to have heterogeneous ability compositions. They are also public, low-performing, low socioeconomic status, and high-minority schools. In these schools, ability grouping has no effects or negative effects, particularly for low-ability students. In contrast, ability grouping may improve achievement for all students in schools with advantageous characteristics, mostly private schools, and may reduce achievement inequalities, because low-ability students benefit the most from this practice.     

The Role of Extended Time and Item Content on a High‐Stakes Mathematics Test

The premise of a great deal of current research guiding policy development has been that accommodations are the catalyst for student performance differences. Rather than accepting this premise, two studies were conducted to investigate the influence of extended time and content knowledge on the performance of ninth‐grade students who took a statewide mathematics test with and without accommodations. Each study involved 1,250 accommodated students (extended time only) with learning disabilities and 1,250 nonaccommodated students demonstrating no disabilities. In Study One, a standard differential item functioning (DIF) analysis illustrated that the usual approach to studying the effects of accommodations contributes little to our understanding of the reason for performance differences across students. Next, a mixture item response theory DIF model was used to explore the most likely cause(s) for performance differences across the population. The results from both studies suggest that students for whom items were functioning differently were not accurately characterized by their accommodation status but rather by their content knowledge. That is, knowing students' accommodation status (i.e., accommodated or nonaccommodated) contributed little to understanding why accommodated and nonaccommodated students differed in their test performance. Rather, the data would suggest that a more likely explanation is that mathematics competency differentiated the groups of student learners regardless of their accommodation and/or reading levels.     

The Effect of Linguistic Simplification of Science Test Items on Score Comparability

The use of accommodations has been widely proposed as a means of including English language learners (ELLs) or limited English proficient (LEP) students in state and districtwide assessments. However, very little experimental research has been done on specific accommodations to determine whether these pose a threat to score comparability. This study examined the effects of linguistic simplification of 4th- and 6th-grade science test items on a state assessment. At each grade level, 4 experimental 10-item testlets were included on operational forms of a statewide science assessment. Two testlets contained regular field-test items, but in a linguistically simplified condition. The testlets were randomly assigned to LEP and non-LEP students through the spiraling of test booklets. For non-LEP students, in 4 t-test analyses of the differences in means for each corresponding testlet, 3 of the mean score comparisons were not significantly different, and the 4th showed the regular version to be slightly easier than the simplified version. Analysis of variance (ANOVA), followed by pairwise comparisons of the testlets, showed no significant differences in the scores of non-LEP students across the 2 item types. Among the 40 items administered in both regular and simplified format, item difficulty did not vary consistently in favor of either format. Qualitative analyses of items that displayed significant differences in p values were not informative, because the differences were typically very small. For LEP students, there was 1 significant difference in student means, and it favored the regular version. However, because the study was conducted in a state with a small number of LEP students, the analyses of LEP student responses lacked statistical power. The results of this study show that linguistic simplification is not helpful to monolingual English-speaking students who receive the accommodation. Therefore, the results provide evidence that linguistic simplification is not a threat to the comparability of scores of LEP and monolingual English-speaking students when offered as an accommodation to LEP students. The study findings may also have implications for the use of linguistic simplification accommodations in science assessments in other states and in content areas other than science.     

Learning about students’ knowledge and thinking in science through large-scale quantitative studies

The main issue addressed in this article is that there is much to learn about students’ knowledge and thinking in science from largescale international quantitative studies beyond overall score measures. Response patterns on individual or groups of items can give valuable diagnostic insight into students’ conceptual understanding, but there is also a danger of drawing conclusions that may be too simple and nonvalid. We discuss how responses to multiple-choice items could be interpreted, and we also show how responses on constructed-response items can be systematised and analysed. Finally, we study, empirically, interactions between item characteristics and student responses. It is demonstrated that even small changes in the item wording and/or the item format may have a substantial influence on the response pattern. Therefore, we argue that interpretations of results from these kinds of studies should be based on a thorough analysis of the actual items used. We further argue that diagnostic information should be an integrated part of the international research aims of such large-scale studies. Examples of items and student responses presented are taken from The Third International Mathematics and Science Study (TIMSS).     

Ecological, psychological, and cognitive components of reading difficulties: testing the component model of reading in fourth graders across 38 countries

The authors tested the component model of reading (CMR) among 186,725 fourth grade students from 38 countries (45 regions) on five continents by analyzing the 2006 Progress in International Reading Literacy Study data using measures of ecological (country, family, school, teacher), psychological, and cognitive components. More than 91% of the differences in student difficulty occurred at the country (61%) and classroom (30%) levels (ecological), with less than 9% at the student level (cognitive and psychological). All three components were negatively associated with reading difficulties: cognitive (student's early literacy skills), ecological (family characteristics [socioeconomic status, number of books at home, and attitudes about reading], school characteristics [school climate and resources]), and psychological (students' attitudes about reading, reading self-concept, and being a girl). These results extend the CMR by demonstrating the importance of multiple levels of factors for reading deficits across diverse cultures.     

Differential Item Functioning on the SAT-M Braille Edition

This study attempted to pinpoint the causes of differential item difficulty for blind students taking the braille edition of the Scholastic Aptitude Test's Mathematical section (SAT-M). The study method involved reviewing the literature to identify factors that might cause differential item functioning for these examinees, forming item categories based on these factors, identifying categories that functioned differentially, and assessing the functioning o f the items comprising deviant categories to determine if the differential effect was pervasive. Results showed an association between selected item categories and differential functioning, particularly for items that included figures in the stimulus, items for which spatial estimation was helpful in eliminating at least two of the options, and items that presented figures that were small or medium in size. The precise meaning of this association was unclear, however, because some items from the suspected categories functioned normally, factors other than the hypothesized ones might have caused the observed aberrant item behavior, and the differential difficulty might reflect real population differences in relevant content knowledge     

Exploring features that affect the difficulty and functioning of science exam questions for those with reading difficulties

This research explored the measurement characteristics of two science examinations and the potential to use access arrangements data to investigate how students requiring reading support are affected by features of exam questions. For two science examinations, traditional and Rasch analyses provided estimates of difficulty and information on item functioning. For one examination, the performance of students eligible for support from a reader in exams was compared to a ‘norm’ group. For selected items a sample of student responses were analysed. A number of factors potentially making questions easier, more difficult or potentially contributing to problems with item functioning were identified. A number of features that may particularly influence those requiring reading support were also identified.     

Item-Level Comparative Analysis of Online and Paper Administrations of the Texas Assessment of Knowledge and Skills

This article describes a comparative study conducted at the item level for paper and online administrations of a statewide high stakes assessment. The goal was to identify characteristics of items that may have contributed to mode effects. Item-level analyses compared two modes of the Texas Assessment of Knowledge and Skills (TAKS) for up to four subjects at two grade levels. The analyses included significance tests of p-value differences, DIF, and response distributions for each item. Additional analyses investigated item position effects and objective-level mode differences. No evidence of item position effects emerged, but significant differences were found for several items and objectives in all subjects at grade 8 and in mathematics and English language arts (ELA) at grade 11. Differences generally favored the paper group. ELA items that were longer in passage length and math items that required graphing and geometric manipulations or involved scrolling in the online administration tended to be the items showing mode differences.     

Socioeconomic Gaps in Academic Achievement Within Schools: Are They Consistent Across Subject Areas?

Few studies have examined the correlates of within-school socioeconomic gaps in academic achievement corresponding to subject areas across schools. This study addressed this limitation with data from the New Brunswick School Climate Study (N = 6,883 students from 148 schools) which contained measures on academic achievement in four subject areas (mathematics, science, reading, and writing) as well as student and school background characteristics. Results of multivariate, multilevel analyses showed that within-school socioeconomic gaps were similar between reading and writing as well as between mathematics and science. Furthermore, the interrelationships of within-school socioeconomic gaps in academic achievement corresponding to the four subject areas across schools were not much influenced by student background characteristics (gender, Native status, number of parents, and number of siblings) and characteristics of school context and climate (school size, school mean SES, disciplinary climate, academic expectation, and parental involvement).