Constructing a Universal Scale of High School Course Difficulty

This study examined the usefulness of applying the Rasch rating scale model (Andrich, 1978) to high school grade data. ACT Assessment test scores (English, Mathematics, Reading, and Science Reasoning) were used as "common items" to adjust for different grading standards in individual high school courses both within and across schools. This scaling approach yielded an ACT Assessment-adjusted high school grade point average (AA-HSGPA) on a common scale across high schools and cohorts within a large public university. AA-HSGPA was a better predictor of first-year college grade point average (CGPA) than the regular high school grade point average. The best model for predicting CGPA included both the ACT composite score and AA-HSGPA.     

Test Score or Student Progress? A Value-Added Evaluation of School Effectiveness in Urban China

Outcome-oriented evaluation of school effectiveness is often based on student test scores in certain critical examinations. This study provides another method of evaluation—value-added—which is based on student achievement progress. This paper introduces the method of estimating the value-added score of schools in multi-level models. Based on longitudinal student achievement data, two measures of school effectiveness in one local education authority in China are compared. It is found that the between-school difference in both test-score and value-added is large comparable with that of Western countries. The results of the two measures of school effectiveness are highly different. The value-added measures lack consistency across different subject areas within schools while the test score measures are highly correlated between subjects. Teachers show their preference for value-added measures over test-score measures of education quality. It is suggested that value-added measures of school effectiveness should be used as a complement to rather than a substitute for test-score measures. The shortcomings of value-added approach are also discussed.     

Examination of Indices of High School Performance Based on the Graded Response Model

We examined summary indices of high school performance (coursework, grades, and test scores) based on the graded response model (GRM). The indices varied by inclusion of ACT test scores and whether high school courses were constrained to have the same difficulty and discrimination across groups of schools. The indices were examined with respect to skewness, incremental prediction of college degree attainment, and differences across racial/ethnic and socioeconomic subgroups. The most difficult high school courses to earn an “A” grade included calculus, chemistry, trigonometry, other advanced math, physics, algebra 2, and geometry. The GRM‐based indices were less skewed than simple high school grade point average (HSGPA) and had higher correlations with ACT Composite score. The index that included ACT test scores and allowed item parameters to vary by school group was most predictive of college degree attainment, but had larger subgroup differences. Implications for implementing multiple measure models for college readiness are discussed.     

Skilled Students and Effective Schools: Reading Achievement in Denmark,Sweden, and France

This study investigates how reading achievement relates to student and school characteristics in countries with different reading scores at the fourth grade level. Data comes from the Progress in International Reading Literacy Study (PIRLS) 2011 for Denmark, Sweden, and France and the multilevel analysis includes two levels: student/home and schools. The school effectiveness and the home literacy models informed the selection of the independent variables. Results show that students’ early literacy skills, home literacy practices and resources, and reading behavior are associated with reading scores in all countries. Furthermore, across different countries there are student/home universals and school particulars that explain variation in reading achievement. Educational policies should address home and school literacy skills and practices, school climate, and school composition to improve students’ reading ability.     

Development of low-stakes mathematics and literacy test scores during lower secondary school – A multilevel pattern-centered analysis of student and classroom differences

The development of students’ learning and test-taking behavior may derive from the social context and the group of peers they associate with daily for years. Consequently, it is assumed that students’ academic achievements are to some degree affected by their classmates and the composition of the classroom. The present study provides evidence on how Finnish students (N = 5071) from different classrooms (N = 435) develop distinct patterns regarding their mathematics and literacy achievement during lower secondary school. We analysed longitudinal large-scale educational assessment data using a multilevel latent profile analysis (MLPA) to investigate the impact of classroom effect on students’ achievement patterns, that is, on the development of students’ low-stakes mathematics and literacy test scores from 7th to 9th grade. The results demonstrated the added value of modelling the multilevel structure inherent in educational assessment data: we identified four student achievement patterns that displayed different distributions across the school classes. More precisely, besides individual characteristics, the development of students’ low-stakes mathematics and literacy test scores was associated with class-level factors and some of the classrooms seemed to have a stronger effect on students’ test scores. These results suggest that classroom context is associated with students’ achievement patterns, especially regarding the worst achieving students. The findings may reflect a combination of class placement practices as well as classroom and peer effect. Although the differences between Finnish schools have been one of the lowest in the OECD countries, the findings of the present study suggest that the classroom membership may create class level quality differences in both the preconditions and the development of learning.     

Differential Prediction of Study Success Across Academic Programs in the Swedish Context: The Validity of Grades and Tests as Selection Instruments for Higher Education

The purpose of the study is to investigate the predictive validity of criterion- and norm-referenced grades and the Swedish Scholastic Aptitude Test (SweSAT) and, in particular, possible differences in the prediction of achievement in higher education across academic programs. The analyses were based on credit points obtained by 164,106 Swedish students during the years 1993 to 2001. Two-level modeling with randomly varying slopes with academic program as cluster variable was used. The results provide means and variances of the slopes across the different programs. Variability in the slopes because of program subject area was also investigated. The results indicate that the validity of grades, irrespective of grading system, is stronger in comparison with SweSAT scores. The results also indicate considerable differences in predictive power across programs for the SweSAT, whereas there are much smaller differences for norm-referenced grades and relatively modest differences for criterion-referenced grades. The impact of program subject area on the variability of prediction was substantial for SweSAT scores.     

Predicting Freshman Grade‐Point Average from Test Scores: Effects of Variation Within and Between High Schools

Most studies predicting college performance from high‐school grade point average (HSGPA) and college admissions test scores use single‐level regression models that conflate relationships within and between high schools. Because grading standards vary among high schools, these relationships are likely to differ within and between schools. We used two‐level regression models to predict freshman grade point average from HSGPA and scores on both college admissions and state tests. When HSGPA and scores are considered together, HSGPA predicts more strongly within high schools than between, as expected in the light of variations in grading standards. In contrast, test scores, particularly mathematics scores, predict more strongly between schools than within. Within‐school variation in mathematics scores has no net predictive value, but between‐school variation is substantially predictive. Whereas other studies have shown that adding test scores to HSGPA yields only a minor improvement in aggregate prediction, our findings suggest that a potentially more important effect of admissions tests is statistical moderation, that is, partially offsetting differences in grading standards across high schools.     

The Stability of School Effectiveness Indices Across Grade Levels and Subject Areas

School effectiveness indices (SEIs) based on regressing achievement test performance onto earlier test performance and a socioeconomic status (SES) measure were obtained for eight subject–grade level combinations for a large sample of elementary schools. School means based on longitudinally matched student scores comprised the data set used in the analysis. The resulting SEIs were found to be somewhat unstable across subject areas (reading and mathematics) and very unstable across grade levels (1 through 4). Grade-to-grade correlations of the SEIs measuring mathematics performance, although small, tended to be statistically significant, whereas those measuring reading performance were generally nonsignificant. Thus, school effects may be more readily discernible in some subject areas than in others. Implications for research on effective schools and for the design of school recognition programs based on student test performance are discussed.     

Comparing dropout predictors for two state-level panels using Grade 6 and Grade 8 data

The purpose of this study was to examine the effectiveness of dropout predictors across time. Two state-level high school graduation panels were selected to begin with the seventh and ninth grades but end at the same time. The first panel (seventh grade) contained 29,554 students and used sixth grade predictors. The second panel (ninth grade) included 31,641 students and used eighth grade predictors. The predictors studied were age, poverty, attendance, gender, and standardized test scores. The data were analyzed using logistic regression. All variables were predictors of dropping out of high school. Age and poverty proved to be the most effective at discriminating between dropouts and graduates within each panel. Age became more effective with time. Attendance and test scores were stable indicators between panels. Gender predicted dropouts for only the ninth grade panel. Eighth graders that were female were approximately 22% less likely to drop out.     

School Effectiveness Indices Revisited: Cross-Year Stability

School effectiveness indices (SEIs) based on residuals from regressing test performance onto prior test performance and a socioeconomic status (SES) measure were obtained for 2 consecutive years for 431 elementary schools. The resulting SEIs were found to be reasonably stable year to year, the correlations ranging from. 34 to .66, depending on grade level (1–4) and subject (reading and mathematics). To aid in the identification of the factors that affect the stability of school achievement, correlations of the SEIs across subjects and grade levels were obtained also. It was determined that SEIs reflecting the performance of students at the same grade level were relatively stable, whether the same or different students were involved. However, SEIs reflecting the performance of students at different grade levels were very unstable. This suggests that grade-within-school effects dominate whatever global school effects operate in elementary schools. Implications for effective schools research, the design of school recognition/reward programs, and research and measurement specialists in general are discussed     

School effectiveness research findings in the Portuguese speaking countries: Brazil and Portugal

This paper provides findings of research on school effectiveness and discusses implications for evaluation in Brazil and Portugal. Most findings reported over the last decade have been published in Brazilian or Portuguese refereed journals. Thus, a brief literature review of such studies enables that knowledge to reach international scholars and researchers. The magnitude of school effects obtained from longitudinal and cross-sectional data modelling is presented and discussed. In particular, a value-added approach based on multilevel models is used to explore consistency across models with different controlling variables and across different curricular contents, and stability over time. The results show a great deal of regional disparities regarding educational outcomes that are related to pupils’ socioeconomic and prior achievement heterogeneity. In addition, evidence is given for stronger school effects in primary than in lower secondary education, larger amplitude of the variance partition coefficient in primary than in lower secondary education, stronger consistency of value-added estimates across models with different controlling variables and moderate consistency across different curricular contents. Weak-to-moderate stability of value-added estimates is also shown when yearly measured compared to when measured by cycles of education, and moderate-to-strong stability when measured in different curricular contents. Recommendations are outlined in terms of how the results could be used to mentoring evaluation in Portuguese speaking countries and enhance school improvement.     

Substituting SAT II: Subject Tests for SAT I: Reasoning Tests: Impact on Admitted Class Composition and Quality

Using data from a sample of 10 colleges at which most students had taken both SAT I: Reasoning tests and SAT II: Subject tests, we simulated the effects of making selection decisions using SAT II scores in place of SAT I scores. Specifically, we treated the students in each college as forming the applicant pool for a more select college, and then selected the top two thirds (and top one third) of the students using high school grade point average combined with either SAT I scores or the average of SAT II scores. Success rates, in terms of first-year grade point averages, were virtually identical for students selected by the different models. The percentage of African American, Asian American, and White students selected varied only slightly across models. Appreciably more Mexican American and Other Latino students were selected with the model that used SAT II scores in place of SAT I scores because these students submitted subject test scores for the Spanish test on which they had high scores.     

The effect of different types of nomination forms on teachers' identification of gifted children

The present study sought to determine the effect of three different types of teacher nomination forms on a group of teachers' effectiveness and efficiency in identifying gifted children; to compare the effectiveness and efficiency of the three forms with each other, with that of the Renzulli-Hartman Scale for Rating Behavioral Characteristics of Superior Students, and with that of a form that requested teachers to identify their gifted students; and to examine the relationship between the scores on the intelligence test, on the various nomination forms, and on the California Achievement Test (CAT). The subjects of the investigation were 183 children in grades K-5 and their teachers. Based on the findings of the study, the following conclusions were reached. First, the use of a behavior rating scale teacher nomination form will result in the greatest number of gifted children being correctly identified. Secondly, it is possible to increase the effectiveness, without overly affecting the efficiency, of any teacher nomination form by making the criterion standard for giftedness sensitive to the specific population it is screening. Thirdly, while the relationship between the scores was relatively low, because of its high effectiveness, its acceptable completion time, and its scores having the highest positive relationship with the intelligence test scores of any of the forms used, Form C is recommended as the teacher nomination form to be considered to assist teachers in the identification of gifted children. Fourthly, in schools where a large majority of the children score very high on a standardized achievement test, some other measure of academic success must be found if academic achievement is to be a component in the screening process. Finally, while, as a group, teachers in a school all may appear to be very good identifiers of gifted children, careful examination across grade levels, and within grade levels if the teacher sample size per grade is large enough, may assist in the identification of groups of teachers for in-service training.     

Klassenwiederholen in PISA-I-Plus: Was lernen Sitzenbleiber in Mathematik dazu?

This study analyses the effects which repeating a class has on ninth grade students’ development of mathematical competency. The following research questions were addressed: How many students repeat grades in the different types of schools? How do students who repeat a grade differ from those who do not in their performance and background characteristics? How much extra mathematics do students repeating a grade learn in one school year? What are the differences between various types of school? Can students with poor mathematics grades in particular profit from repeating a grade? The sample is a sub-sample of the PISA-I-Plus study and comprises N = 360 ninth grade students. The total sample of PISA-I-Plus is representative for all ninth/tenth grade students from the different school types in Germany. The data survey was carried out in the ninth grade and then repeated after the students had repeated a year. The results document differences in the amount of grade repeat quotas between types of school. Furthermore, compared to students not repeating, those repeating a grade had lower mathematics (d = 1.02) and german (d = 1.14) grades, a lower level of mathematical literacy (d = 0.51), and lower test results with regard to basic cognitive abilities (d = 0.32). In terms of the development of mathematical literacy, the students repeating a grade could improve by an average of 23 points (d = 0.27) on the PISA mathematics scale. However, the results identify 38 percent of students repeating a grade who do not make any significant improvement in mathematics or even get worse. A differentiation according to school types shows that students repeating a grade in integrated comprehensive secondary schools and in schools with several educational levels in particular do not, on average, show any noteworthy improvement in their mathematical literacy. The analysis of the school grades received in mathematics shows that students whose mathematics grades are unsatisfactory do not benefit more from repeating a grade than students whose mathematics performance has been rated as being “satisfactory” or better. The article concludes with a discussion of the possible consequences of changing the way in which repetitions of grades are dealt with.     

Using growth models to monitor school performance: comparing the effect of the metric and the assessment

This paper investigates whether inferences about school performance based on longitudinal models are consistent when different assessments and metrics are used as the basis for analysis. Using norm-referenced (NRT) and standards-based (SBT) assessment results from panel data of a large heterogeneous school district, we examine inferences based on vertically equated scale scores, normal curve equivalents (NCEs), and nonvertically equated scale scores. The results indicate that the effect of the metric depends upon the evaluation objective. NCEs significantly underestimate absolute individual growth, but NCEs and scale scores yield highly correlated (r >.90) school-level results based on mean initial status and growth estimates. SBT and NRT results are highly correlated for status but only moderately correlated for growth. We also find that as few as 30 students per school provide consistent results and that mobility tends to affect inferences based on status but not growth – irrespective of the assessment or metric used.     

Impact of North Carolina's Early Childhood Programs and Policies on Educational Outcomes in Elementary School

North Carolina's Smart Start and More at Four (MAF) early childhood programs were evaluated through the end of elementary school (age 11) by estimating the impact of state funding allocations to programs in each of 100 counties across 13 consecutive years on outcomes for all children in each county‐year group (n = 1,004,571; 49% female; 61% non‐Latinx White, 30% African American, 4% Latinx, 5% other). Student‐level regression models with county and year fixed effects indicated significant positive impacts of each program on reading and math test scores and reductions in special education and grade retention in each grade. Effect sizes grew or held steady across years. Positive effects held for both high‐ and low‐poverty families, suggesting spillover of effects to nonparticipating peers.     

Monitoring Student Achievement for Accountability: The Demonstration of a Model

Abstract

This study demonstrated a procedural model that can be applied by any school to assess, guide, and account for the progress of its students as well as to analyze its own effectiveness. The model uses equivalent achievement tests to monitor student achievement in subject areas at grade levels, between grade levels, and across subgroups of students. Multiple regression analyses of test scores between grades identify factors associated with achievement Using sixth and eighth grade Comprehensive Tests of Basic Skills scores in a matched longitudinal sample of 208 students, the study found small differences in average achievement between boys and girls. Differences between corresponding sixth and eighth grade test means were higher in mathematics than in language. From the sixth grade to the eighth, there was a widening gap in average achievement between high and low I.Q. groups. In multiple regressions of eighth grade test scores on sixth grade measures, I.Q., study skills, and reading were prevalent in the regression equations, but clusters of measures associated with achievement differed between high and low’ LQ. groups. The results of the study have implications for developing and evaluating the achievement of students with varying mental abilities.     

Book review of Restructuring Schools for Linguistic Diversity

This study evaluated the feasibility of using classroom assistants as tutors of 1st-grade struggling readers in a school with limited financial and personnel resources. The tutoring program, Partners-in-Reading (PIR), offered assistance to 54 first graders in 2 cohorts. Classroom assistants scheduled tutoring a minimum of 4 times per week for 30 to 40 min per session: A typical session included the reading and rereading of familiar texts, an introduction of texts at or slightly above a student's instructional level, and various word recognition activities. PIR students' word recognition and development spelling scores were compared with Reading-Recovery (RR) students (n = 62) and a control group (n = 58). Although equivalent at the year's start, PIR and RR students outperformed controls on these measures at the end of 1st grade. They also scored higher than did the controls on a norm-referenced word recognition subtest and were less likely to be retained. PIR students also outperformed the controls on a norm-referenced comprehension subtest. This discussion focuses on the benefits of using classroom assistants as tutors and the related questions of when tutoring should be offered, its duration, and its evaluation.     

Mathematics attainment at inner-city schools: Establishing the need for systematic formative evaluation practices

Recently there has been a great amount of research and professional educator interest in at-risk, poor academically attaining students, especially low socioeconomic status students at U.S. inner-city schools. A major factor that has been hypothesized in the research literature as being associated with poor academic attainment is the lack of critical and timely instructional feedback or formative evaluation. Using a sample of 130 inner-city senior high school students, the perceived quality and quantity of formative evaluation received by these students at their elementary and secondary school levels were assessed. in addition, each student was given a mathematics (pre-algebra) assessment using both a one and two-dimensional format (recognition plus confidence) to determine present levels of mathematics attainment. Finally data were collected from the cumulative grade-level folders of a subset of these students, especially norm-referenced data (NRT) in mathematics, to examine their relationship to scores on the Scholastic Aptitude Test-Quantitiative portion. The study finds that in addition to extremely poor mathematics attainment and poor formative evaluation practices there is little association between SAT (quantitative) scores and the grade-level (mathematics) NRT scores. These findings suggest that parents cannot depend on traditional norm-referenced measures to indicate actual mathematics attainment as these students are progressing through the schools. These findings also challenge urban school administrative personnel to reassess the use of NRT measures to monitor student progress and to develop more comprehensive and systematic formative evaluation procedures and practices for individual students as they progress through each grade level.     

SAT Validity for Linguistic Minorities at the University of California, Santa Barbara

The validity of the SAT as an admissions criterion for Latinos and Asian Americans who are not native English speakers was examined. The analyses, based on 1997 and 1998 UCSB freshmen, focused on the effectiveness of SAT scores and high school grade-point average (HSGPA) in predicting college freshman grade-point average (FGPA). When regression equations were estimated based on all students combined, some systematic prediction errors occurred. For language minorities, using only high school grades as a predictor led to predicted FGPAs that tended to exceed actual FGPAs, particularly for Latinos. Including SAT scores in the equation notably reduced prediction bias. Further analyses showed that, while HSGPA had the highest correlation with FGPA for most groups, SAT verbal score was the strongest predictor of FGPA for language minorities in 1998. An overriding conclusion is that combining data across language groups can obscure important test validity information.