Effects of Curricular Differences on Achievement Test Data at Item and Objective Levels

The objectives of this study were to examine the impact of different curricula on standardized achievement test scores at item and objective levels and to determine if different curricula generate different patterns of item factor loadings. School buildings from a middle-sized district were rated regarding the degree to which their curricula matched the content of the standardized test, and the actual textbook series used within each building (classroom) was determined. Covariate analyses of objective scores and plots and correlations of item p values indicated very small, nonsignificant differential effects across ratings and textbook series. Factor patterns indicated no curricular effects on large first factors. These findings parallel the results of a previous study conducted at the subtest level. We conclude that educators need not be unduly concerned about the impact of specific and generally small differences in curricular offerings within a district on standardized test scores or inferences to a broad content domain.     

IRT Ability Estimates from Customized Achievement Tests Without Representative Content Sampling

This study examines the effects of using item response theory (IRT) ability estimates based on customized tests that were formed by selecting specific content areas from a nationally standardized achievement test. Subsets of items were selected from four different subtests of the Iowa Tests of Basic Skills (Hieronymus, Hoover, & Lindquist, 1985) on the basis of (a) selected content areas (content-customized tests) and (b) a representative sampling of content areas (representative-customized tests). For three of the four tests examined, ability estimates and estimated national percentile ranks based on the content-customized tests in school samples tended to be systematically higher than those based on the full tests. The results of the study suggested that for certain populations, IRT ability estimates and corresponding normative scores on content-customized versions of standardized achievement tests cannot be expected to be equivalent to scores based on the full-length tests.     

DETECTING IMPACTS OF CURRICULAR DIFFERENCES IN ACHIEVEMENT TEST DATA

Standardized tests are designed to measure broad goals. But many professionals have been concerned with the lack of fairly specific matches between items (or objectives) on a test and the curriculum (instruction). This study assessed the differences in standardized test scores resulting from curricular differences in two school systems. The degree of curriculum-test match for reading and math in grades 3 and 6 was based on ratings of that match by qualified district personnel. Further, results of using different textbook series were analyzed. The dependent variables of test and subtest scores were analyzed using a two-factor MANCOVA where textbook series and school personnel ratings were the two factors, and pretest scores and percent eligible for Aid to Families with Dependent Children (AFDC) were the covariates. None of the multivariate F tests were significant at the .05 level. It was concluded that neither the curricular match as judged by district personnel or the textbook series used had a significant impact on standardized test scores.     

Instructional Sensitivity of a State's Standards-Based Assessment

Abstract

The accuracy of achievement test score inferences largely depends on the sensitivity of scores to instruction focused on tested objectives. Sensitivity requirements are particularly challenging for standards-based assessments because a variety of plausible instructional differences across classrooms must be detected. For this study, we developed a new method for capturing the alignment between how teachers bring standards to life in their classrooms and how the standards are defined on a test. Teachers were asked to report the degree to which they emphasized the state's academic standards, and to describe how they taught certain objectives from the standards. Two curriculum experts judged the alignment between how teachers brought the objectives to life in their classrooms and how the objectives were operationalized on the state test. Emphasis alone did not account for achievement differences among classrooms. The best predictors of classroom achievement were the match between how the standards were taught and tested, and the interaction between emphasis and match, indicating that test scores were sensitive to instruction of the standards, but in a narrow sense.     

Format changes in reading achievement tests: Implications for learning disabled students

It has been seen that children's scores on reading achievement tests vary not only with knowledge of content, but also with the differing formats of test items. Teachers working with learning disabled children or children with attention problems may wish to choose standardized tests with fewer, rather than more, format changes. The present study evaluated the number of format and direction changes across tests and grade levels of the major elementary standardized reading achievement tests. The number of format changes varies from one change every 1.2 minutes on the Metropolitan Achievement Test Level E1 to one change every 21.3 minutes on the P1 level of the Stanford Achievement Test. Teachers may wish to take this evaluation into account when considering use of standardized reading achievement tests for their students.     

Three Options Are Optimal for Multiple-Choice Items: A Meta-Analysis of 80 Years of Research

Multiple-choice items are a mainstay of achievement testing. The need to adequately cover the content domain to certify achievement proficiency by producing meaningful precise scores requires many high-quality items. More 3-option items can be administered than 4- or 5-option items per testing time while improving content coverage, without detrimental effects on psychometric quality of test scores. Researchers have endorsed 3-option items for over 80 years with empirical evidence—the results of which have been synthesized in an effort to unify this endorsement and encourage its adoption.     

Educational Assessment as a Promising Area for Psychometric Research

Tests of educational achievement typically present items in the multiple-choice format. Some achievement test items may be so "saturated with aptitude" (Willingham, 1980) as to be insensitive to skills acquired through education. Multiple-choice tests are ill-suited for assessing productive thinking and problem-solving skills, skills that often constitute important objectives of education. Viewed as incentives for learning, multiple-choice tests may impede student progress toward these objectives. There is need for accelerated research to develop alternatives to multiple-choice achievement tests, with content selected to match the specified educational objectives.     

初中生物课程中健康教育渗透情况研究--中考生物学试卷健康教育内容分析

学校健康教育课程是健康教育的主要形式,我国现阶段学校健康教育课程主要采取与其它课程渗透的形式,在生物课中渗透健康教育更已成为传统做法。分析中考生物学考试试卷中与健康教育相关的试题,结果表明:健康教育内容在生物学教育中占有相当比例,主要是身体健康和疾病防治的相关内容;这些内容主要是在生理卫生部分讲授;对于健康教育内容的考核仍以知识为主。目前,生物学中的健康教育在内容、目的、方法等方面与现代健康教育的要求尚有一定的差距。     

Standardized test outcomes for students engaged in inquiry‐based science curricula in the context of urban reform

Considerable effort has been made over the past decade to address the needs of learners in large urban districts through scaleable reform initiatives. We examine the effects of a multifaceted scaling reform that focuses on supporting standards based science teaching in urban middle schools. The effort was one component of a systemic reform effort in the Detroit Public Schools, and was centered on highly specified and developed project‐based inquiry science units supported by aligned professional development and learning technologies. Two cohorts of 7th and 8th graders that participated in the project units are compared with the remainder of the district population, using results from the high‐stakes state standardized test in science. Both the initial and scaled up cohorts show increases in science content understanding and process skills over their peers, and significantly higher pass rates on the statewide test. The relative gains occur up to a year and a half after participation in the curriculum, and show little attenuation with in the second cohort when scaling occurred and the number of teachers involved increased. The effect of participation in units at different grade levels is independent and cumulative, with higher levels of participation associated with similarly higher achievement scores. Examination of results by gender reveals that the curriculum effort succeeds in reducing the gender gap in achievement experienced by urban African‐American boys. These findings demonstrate that standards‐based, inquiry science curriculum can lead to standardized achievement test gains in historically underserved urban students, when the curriculum is highly specified, developed, and aligned with professional development and administrative support. © 2008 Wiley Periodicals, Inc. J Res Sci Teach 45: 922–939, 2008     

A critique of “unequal educational opportunity" 1

Using the 159 counties of Georgia as statistical units Osborne has presented data indicating (a) a positive relationship between percentage of non‐Whites and per pupil expenditures; (b) a negative relationship between per pupil expenditures and mean achievement and intelligence scores; and (c) a negative relationship between percentage of non‐Whites and mean intelligence and achievement scores. Osborne has interpreted these findings as indicating that the attempt to raise the achievement levels of previously deprived groups has been a failure, and has used this interpretation as a basis for a more general negative conclusion concerning the effectiveness of compensatory efforts. The present analysis indicates, however, that neither of these conclusions follows from his data. The evidence suggests rather that Osborne's negative correlations reflect in large measure long‐range effects of earlier conditions.     

Curriculum bias and reading achievement test performance

The overlap of words specifically taught in reading textbooks with the contents of standardized reading achievement tests may be a source of bias that is frequently overlooked in psychoeducational assessments. This study compares the standardized achievement test performance of 62 second graders receiving instruction in two different reading curricula (Open Court and Houghton-Mifflin) to determine whether either curriculum generates different quantitative estimates of reading achievement. Reading subtest scores derived from the Kaufman Test of Educational Achievement-Brief Form (K-TEA), the Wide Range Achievement Test-Revised (WRAT-R), and the Reading Recognition and Reading Comprehension subtests from the Peabody Individual Achievement Test (PIAT) were examined. Grade level equivalents and scaled scores from the California Achievement Test (CAT) were also examined. Three Curriculum × Test repeated measures ANOVAs were conducted using grade level scores (2×7), standard scores (2×4), and CAT scaled scores (2 × 5) as dependent measures. A significant Curriculum × Test interaction was identified, suggesting differences among tests in estimates of reading ability as a function of the reading program.     

The Generalizability of Content Validity Ratings

The problem of assessing the content validity (or relevance) of standardized achievement tests is considered within the framework of generalizability theory. Four illustrative designs are described that may be used to assess test-item fit to a curriculum. For each design, appropriate variance components are identified for making relative and absolute item (or test) selection decisions. Special consideration is given to use of these procedures for determining the number of raters and/or schools needed in a content-validation decisionmaking study. Application of these procedures is illustrated using data from an international assessment of mathematics achievement     

Parental Involvement,Instructional Expenditures,Family Socioeconomic Attributes,and Student Achievement

Abstract

The influence of parental involvement, socioeconomic status of parents, and instructional supplies expenditures on mathematics achievement scores of Grade 4 students in a low-income county in North Carolina were examined. An educational production function framework was used to analyze the influence of educational resources on mathematics achievement scores. Pearson product-moment correlation and ordinary least squares regression were used to determine the overall strength of each relation and the variables with the greatest impact on mathematics achievement. Results indicated that instructional supplies expenditures per pupil and parental volunteer hours were not statistically significant in explaining mathematics test scores. Furthermore, results showed that the percentage of students in free/reduced-price lunch programs was related negatively to students' academic performance in mathematics. This finding supports the notion that economic circumstances are correlated with academic achievement.     

The Effect of Training in Test Item Writing on Test Performance of Junior High Students

High stakes testing, a phenomena born out of intense accountability across the United States, produces instructional settings that marginalize both curriculum and instruction. Teachers and other school personnel have minimized instruction to drill and practice in an effort to raise standardized and criterion referenced test scores. This study presents an alternative to current practice that engages students in learning and increases their awareness of the internal aspects of standardized tests. The Test Item Construction Model (TICM) guides students through the process of studying test item stems and subsequently creating items using a 12 week process of incrementing from understanding to creating test items. Students grew in their understanding of the test item stems and the generation of these. An ANOVA did not yield significant differences between random groups of trained and untrained test writers. However, students in the experimental group demonstrated gains in understanding of test items.     

The Effect of Item Response Changes on Scores on an Elementary Reading Achievement Test

Abstract

The effect of changing item responses on scores of elementary school children on a standardized achievement test was studied. Previous research, primarily involving non-standardized instruments and adult samples, indicates that changed responses are more likely to be correct than not. Subjects were 165 third grade students using the Metropolitan Reading Tests. Students received no special instructions regarding changing responses. Changes were identified visually and were independently verified. While frequency of response changes was low, such changes generally improved scores. Sex differences in number and success of changes were non-significant. The relationship between frequency of response change and test score was minimal. Responses to difficult items were changed more frequently with less success than changes on easy items. High scorers made more successful changes than did low scorers. Within the limits of the methodology, results clearly indicated that response changes of elementary students on multiple-choice items tend to improve test scores.     

Effort Analysis: Individual Score Validation of Achievement Test Data

Whenever the purpose of measurement is to inform an inference about a student’s achievement level, it is important that we be able to trust that the student’s test score accurately reflects what that student knows and can do. Such trust requires the assumption that a student’s test event is not unduly influenced by construct-irrelevant factors that could distort his score. This article examines one such factor—test-taking motivation—that tends to induce a person-specific, systematic negative bias on test scores. Because current measurement models underlying achievement testing assume students respond effortfully to test items, it is important to identify test scores that have been materially distorted by non-effortful test taking. A method for conducting effort-related individual score validation is presented, and it is recommended that measurement professionals have a responsibility to identify invalid scores to individuals who make inferences about student achievement on the basis of those scores.     

Are effective or meritorious schools meretricious?

Data on student achievement gain in a large urban school district during the 1984–85 school year were analyzed to identify schools with achievement consistently higher or lower than schools with similar student and school characteristics. Before calculating gain scores, students were grouped with others of the same sex, ethnicity, initial grade level, and initial achievement level. After residuals for students who were above or below similar students were summed and averaged for reading and math at each grade, school-level regression analysis was used to further control for student and school characteristics. Results indicated that schools with reading gains one standard deviation above the mean had average raw scores less than two items higher than the district average on the sixty-item reading test, while schools with math gains this large had raw scores little more than one item higher than the average on the 44-item math test. In addition, examination of data on schools with residual (combined) reading and math gain scores 1.5 or more standard deviations above the average showed that after taking account of student background information and salient school characteristics, only one elementary school among 173 and no secondary school among 71 had unusually high achievement scores. Implications are discussed regarding efforts to identify and reward meritorious schools which allegedly have improved achievement more than other similar schools.     

Investigating the relationship between instructional practices and science achievement in an inquiry-based learning environment

ABSTRACT

This study investigates the discrete effects of inquiry-based instructional practices that described the PISA 2015 construct ‘inquiry-based instruction’ and how each practice, and the frequency of each practice, is related to science achievement across 69 countries. The data for this study were drawn from the PISA 2015 database and analysed using hierarchical linear modelling (HLM). HLMs were estimated to test the contribution of each item to students’ science achievement scores. Some inquiry practices demonstrated a significant, linear, positive relationship to science achievement (particularly items involving contextualising science learning). Two of the negatively associated items (explaining their ideas and doing experiments) were found to have a curvilinear relationship to science achievement. All nine items were dummy coded by the reported frequency of use and an optimum frequency was determined using the categorical model and by calculating the inflection point of the curvilinear associations in the previous model e.g. students that carry out experiments in the lab in some lessons have higher achievement scores than students who perform experiments in all lessons. These findings, accompanied by detailed analyses of the items and their relationships to science outcomes, give stakeholders clear guidance regarding the effective use of inquiry-based approaches in the classroom.     

Simultaneous testing of four decades of academic self-concept models

In separate studies on academic self-concept, previous research has shown: (1) the distinctiveness of a cognitive and an affective component, (2) the domain specificity of self-concepts, (3) the reciprocal effects of self-concept and achievement, (4) the internal/external frame of reference in self-concept development, (5) the reciprocal effects of the internal/external frame of reference, (6) the big-fish-little-pond effect, and (7) the interrelatedness of self-concepts in similar domains. The present study demonstrates that all of these seven findings are replicable and may be synthesized in a single study with a sample of students in Singapore. Secondary 1 students (7th graders; N = 275) were surveyed with 24 items about their academic self-concepts in physics, English, and math in two components (cognitive and affective), and their respective achievement scores were recorded over two time points. Confirmatory factor analysis found that the cognitive and affective components of academic self-concept were separable. The students’ self-concepts in different curriculum domains were distinct, supporting the domain specificity of self-concepts. The frame of reference and reciprocal effects were both supported, but only for the cognitive component of self-concept. Positive and statistically significant correlations between physics and math suggest that these curriculum domains were interrelated. Results of self-concept studies in schools can encourage and guide the design of interventions that could enhance students’ self-concept for positive sustainable effects on desirable educational outcomes. Attempts to improve learning outcomes should emphasize an enhancement of specific components of academic self-concept in domain-specific and related curriculum domains for optimal effects.