Performance Trajectories and Performance Gaps as Achievement Effect-Size Benchmarks for Educational Interventions

Abstract

Two complementary approaches to developing empirical benchmarks for achievement effect sizes in educational interventions are explored. The first approach characterizes the natural developmental progress in achievement made by students from one year to the next as effect sizes. Data for seven nationally standardized achievement tests show large annual gains in the early elementary grades followed by gradually declining gains in later grades. A given intervention effect will therefore look quite different when compared to the annual progress for different grade levels. The second approach explores achievement gaps for policy-relevant subgroups of students or schools. Data from national- and district-level achievement tests show that, when represented as effect sizes, student gaps are relatively small for gender and much larger for economic disadvantage and race/ethnicity. For schools, the differences between weak schools and average schools are surprisingly modest when expressed as student-level effect sizes. A given intervention effect viewed in terms of its potential for closing one of these performance gaps will therefore look very different depending on which gap is considered.     

Does Early Mathematics Intervention Change the Processes Underlying Children's Learning?

Early educational intervention effects typically fade in the years following treatment, and few studies have investigated why achievement impacts diminish over time. The current study tested the effects of a preschool mathematics intervention on two aspects of children's mathematical development. We tested for separate effects of the intervention on “state” (occasion-specific) and “trait” (relatively stable) variability in mathematics achievement. Results indicated that, although the treatment had a large impact on state mathematics, the treatment had no effect on trait mathematics, or the aspect of mathematics achievement that influences stable individual differences in mathematics achievement over time. Results did suggest, however, that the intervention could affect the underlying processes in children's mathematical development by inducing more transfer of knowledge immediately following the intervention for students in the treated group.     

Scaling Up the Implementation of a Pre-Kindergarten Mathematics Curriculum: Teaching for Understanding With Trajectories and Technologies

Abstract

This study used a randomized field trial design to evaluate the efficacy of a research-based model for scaling up an intervention focused on preschool mathematics. Although the successes of research-based educational practices have been documented, equally well known is the paucity of successful efforts to bring these practices to scale. The same research corpus provides guidelines to scale up successful interventions. We designed an intervention model based on that research, including mathematics curricula with an emphasis on teaching for understanding following developmental guidelines, or learning trajectories, and using technology at multiple levels. We then implemented that model and evaluated the implementation with a limited scale up study. Within a design involving 25 classrooms serving children at risk for later school failure, we examined the impact of the model, using measures of fidelity of implementation, classroom observations of mathematics environment and teaching, and child outcomes. High levels of fidelity of implementation resulted in consistently higher scores in the intervention, compared to control, classes on the observation instrument and significantly and substantially greater gains in children's mathematics achievement in the intervention, compared to the control, children (effect size = .62).     

Improving quality and equity in schools in socially disadvantaged areas

Background: Recent effectiveness studies have investigated the relationship between two dimensions of effectiveness – namely, quality and equity. Specifically, the question of whether effective schools can also reduce the initial differences in student outcomes attributed to student background factors has been examined. In this context, the Dynamic Approach to School Improvement (DASI) makes use of theory and the research findings of effectiveness studies to try to improve school effectiveness in terms of quality and equity.

Purpose: This study aimed to examine whether the implementation of DASI in primary schools in socially disadvantaged areas in four European countries (Cyprus, England, Greece and Ireland) was able to promote student learning outcomes in mathematics and to reduce the impact of student background factors on student achievement in mathematics.

Design and methods: A sample of 72 primary schools across the four countries was randomly split into experimental and control groups. At the beginning and at the end of the school year, mathematics tests were administered to all students of Grades 4–6 (n = 5560; student ages 9–12 years). The experimental group made use of DASI. Within-country multilevel regression analyses were conducted to evaluate the impact of the intervention and search for interaction effects between the use of DASI and student background factors on final achievement.

Results: In each country, the experimental group achieved better results in mathematics than the control group. At the beginning of the intervention, the achievement gap based on socio-economic status (SES) was equally large in the experimental and the control groups. Only in the experimental group did the achievement gap based on SES become smaller. However, DASI was not found to have an effect on equity when the equity dimension was examined by focusing on the achievement gap based on either gender or ethnicity.

Conclusions: Implications of findings are drawn and the importance of measuring equity in terms of student achievement gaps based on different background factors, rather than only on SES, is emphasised. We propose the evaluation of the impact of interventions on promoting equity by the use of various criteria.     

Testing the Immediate and Long-Term Efficacy of a Tier 2 Kindergarten Mathematics Intervention

ABSTRACT

This study examined the efficacy of a kindergarten mathematics intervention program, ROOTS, focused on developing whole-number understanding in the areas of counting and cardinality and operations and algebraic thinking for students at risk in mathematics. The study utilized a randomized block design with students within classrooms randomly assigned to treatment or control conditions. Measures of mathematics achievement were collected in the fall (pretest) and spring (posttest) in kindergarten and in the winter of first grade (delayed posttest). Significant differences between conditions favoring treatment students were found on four of six measures at posttest. Treatment students reduced the achievement gap with their not-at-risk peers. No effect was found on follow-up first-grade achievement scores. Implications for Tier 2 mathematics instruction in a Response to Intervention model are discussed.     

Does socioeconomic sorting of teacher qualifications exacerbate mathematics achievement inequity? Panel data estimates from 20 years of TIMSS

Recent and older studies have reported either a persistence or a widening of the socioeconomic achievement gap—the difference in performance between students in top and bottom socioeconomic groups. Using a panel data technique with country fixed effects for 32 education systems and six waves of data from the Trends in International Mathematics and Science Study, we examine whether the sorting of teachers by specialization level in mathematics education and novice status across students of different socioeconomic backgrounds exacerbates mathematics achievement inequity despite the presence of a time-varying control for socioeconomic school segregation. We find modest evidence that sorting by mathematics education is associated with achievement inequity, but no evidence supporting the importance of sorting based on teacher experience. Socioeconomic school segregation, on the other hand, clearly and persistently exacerbates achievement inequity. The results have policy implications regarding the effective distribution of educational resources.Availability of data and materialsThe data are freely available via the TIMSS data repository.     

Recognition and development of academic talent in educationally disadvantaged students

Abstract

We studied 36 students identified as “educationally disadvantaged” who scored above average on standardized achievement tests and completed a program to reinforce their academic skills in either language arts or mathematics. We pretested and posttested an additional (control) group of 28 students who received no instructional intervention. We found a significant effect of the instructional intervention for both achievement and aptitude test scores in language arts as well as mathematics. Gains in mathematics were significantly greater than in language arts. After instructional intervention, the majority of students were eligible and academically qualified for challenging gifted‐talented programs. These findings are consistent with two other similar research projects using the same instructional program and model, suggesting stability of results. We discuss implications of this study for identifying and developing academic talent in such a population.     

Parental Involvement,Instructional Expenditures,Family Socioeconomic Attributes,and Student Achievement

Abstract

The influence of parental involvement, socioeconomic status of parents, and instructional supplies expenditures on mathematics achievement scores of Grade 4 students in a low-income county in North Carolina were examined. An educational production function framework was used to analyze the influence of educational resources on mathematics achievement scores. Pearson product-moment correlation and ordinary least squares regression were used to determine the overall strength of each relation and the variables with the greatest impact on mathematics achievement. Results indicated that instructional supplies expenditures per pupil and parental volunteer hours were not statistically significant in explaining mathematics test scores. Furthermore, results showed that the percentage of students in free/reduced-price lunch programs was related negatively to students' academic performance in mathematics. This finding supports the notion that economic circumstances are correlated with academic achievement.     

Assessing Impacts of Math in Focus,a “Singapore Math” Program

ABSTRACT

This study investigates, through a cluster randomized trial, the impact of Math in Focus, a core mathematics program modeled after instructional approaches used in Singapore, on third- through fifth-grade students’ achievement in mathematics. The program is currently being used in more than 400 school districts in the United States. The program focuses on coherence of coverage of materials across grades, use of the Concrete to Pictorial to Abstract approach to instruction, and covering fewer topics, but addressing them more thoroughly. Twenty-two grade-level teams across 12 schools were randomized to the program or business as usual. Measures included indicators of fidelity to treatment, and student mathematics learning. Impacts on mathematics achievement ranged from .11 to .15 standard deviation units. No differences in impact were observed depending on level of incoming achievement, minority status, or grade level. Impact of Math in Focus did not vary across the procedures and problem-solving subscales. Discussion of findings includes the nature of the counterfactual, and possible future direction of impact studies that may focus in greater depth on the critical features of inquiry instruction unique to Math in Focus.     

A Randomized Trial of Teacher Development in Elementary Science: First-Year Achievement Effects

Abstract

This article describes 1st-year experimental effects of a large-scale reform providing professional development to elementary school teachers to implement an extended, inquiry-oriented science curriculum. Known as “immersion teaching” because it “immerses” teachers and students in the full cycle of scientific inquiry, this approach developed through a partnership involving university-based science and mathematics content experts and educators and K-12 educators from the Los Angeles Unified School District. Multilevel analyses, which examined school-level effects of assignment to the professional development intervention, nested Grade 4 students and their science achievement outcomes within the 80 study schools. The analyses revealed a statistically significant negative 1st-year treatment effect of school-level assignment to the initiative on the key science achievement outcome. We also tested whether the treatment had differential effects for English language learners, schools with large proportions of English language learners, and students of new teachers. We found an interaction effect of the treatment by teacher experience level for the teachers who were the primary target of the intervention, with the treatment having positive effects for novice teachers (3 years of experience or less) but a larger, negative effect for veteran teachers. We explore analytically three sets of explanations for the unexpected negative main effect of treatment: potential statistical and design artifacts, possible misalignment between the assessments and content of the treatment, and practical issues related to implementation of the treatment.     

Differentiation of Children's Competence Beliefs and Achievement Values,and their Relations to Children's Performance and Choice of Different Activities

ABSTRACT

Results of two studies of children's competence beliefs and achievement values for mathematics and reading are summarized. Approximately 1700 children and adolescents participated in the studies; the participants were in first through 12th grades. The studies were based on an expectancy — value model of achievement choice proposed by Eccles et al. (1983). Results indicated that children's competence beliefs and achievement task values are distinctive belief systems, even in first grade children. During the elementary school years, the strength of relations between children's competence beliefs and adult evaluations of children's competence increased across grade. Children and adolescents’ competence beliefs predict their mathematics performance, whereas their task values predict their intentions to continue taking mathematics.     

School Size Effects on Achievement in Secondary Education Evidence from the Netherlands,Sweden and the USA 1

ABSTRACT

In this article the results of an investigation into the relation between school size and achievement are reported. The findings relate to mathematics achievement in Dutch, Swedish and American secondary education and to science achievement in the Netherlands. The analyses sought to provide an answer to the following questions:

(1) Is school size related to achievement independently of student background characteristics such as sex, achievement motivation, socio‐economic status and cognitive aptitude? (2) Is the effect of school size related to any of the aforementioned background characteristics? (3) Does the effect of school size on achievement differ between the educational systems of the Netherlands, Sweden and the USA? (4) Is the effect of school size the same for different measures of student achievement (mathematics versus science)?

It was hypothesized that school size would be most strongly related to achievement in the USA. The analyses, however, revealed little empirical evidence for the existence of school size effects on achievement in any of the three countries, possibly because school size and curriculum comprehensiveness are not strongly related in these countries.

Because the investigations involved the analysis of five separate datasets, the research outcomes revealed some useful additional information with respect to the robustness of the detected relations between the five covariates and student achievement.

     

School Effects on Performances of Minority Pupils 1

The impact of determinants from the individual, school and school environment level with regard to educational performance of pupils from various ethnic origins in the Netherlands is examined in this article. From rational choice and empowerment theory crucial variables were deduced. By means of multilevel modelling school effects in secondary education on performances in mathematics and linguistics were established. Results indicate the importance of a school policy orientated towards improving achievements of minority subgroups. A more general approach, based on the notion of shared circumstances of pupils from ethnic minority and low socio‐economic background families, did not generate positive results.

     

What Is the Long‐Run Impact of Learning Mathematics During Preschool?

The current study estimated the causal links between preschool mathematics learning and late elementary school mathematics achievement using variation in treatment assignment to an early mathematics intervention as an instrument for preschool mathematics change. Estimates indicate (n = 410) that a standard deviation of intervention‐produced change at age 4 is associated with a 0.24‐SD gain in achievement in late elementary school. This impact is approximately half the size of the association produced by correlational models relating later achievement to preschool math change, and is approximately 35% smaller than the effect reported by highly controlled ordinary least squares (OLS) regression models (Claessens et al., 2009; Watts et al., 2014 ) using national data sets. Implications for developmental theory and practice are discussed.     

Grouping in Education in Light of Theory and Results in Motivation

Abstract

Homogeneous and heterogeneous grouping as well as research on either have not taken into account the implications of the pupils’ achievement motives. This theoretical contribution attempts to analyze why some pupils take advantage or disadvantage of being grouped in a homogeneous class — and why others take advantage or disadvantage of being grouped in a heterogeneous class. Suggestions for new grouping procedure in education are offered. The principles guiding the new procedure are that the pupils should be grouped in a way which releases and utilizes the potential positive influence of the motive to approach success — and avoids the possible negative effects of the motive to avoid failure. Is it possible to create such an educational setting?     

Effects of a Learning Information System on Mathematics Achievement and Classroom Structure

Abstract

The authors examined the effects of implementing an instructional system that automates application of evidence-based components of effective instruction on student mathematics achievement and on classroom behaviors known to be related to overall student achievement outcomes. A treatment group of 157 4th- and 5th-grade students used the intervention in conjunction with the Everyday Math curriculum. Student performance was compared with a within-school control group of 61 4th- und 5th-grade students, as well as all 4th- and 5th-grade students in the district (N = 6,385). The students in the control groups received only the Everyday Math curriculum. Results indicate that the implementation of the instructional management system as an enhancement to Everyday Math resulted in an increase in the amount of time spent on classroom activities that researchers have identified as contributing to positive academic outcomes. Furthermore, students who used the Accelerated Math program demonstrated greater mathematics achievement gains than did the control groups.     

Reading Recovery: Exploring the Effects on First-Graders' Reading Motivation and Achievement

ABSTRACT

This study examined the effects of Reading Recovery on children's motivational levels, and how motivation may contribute to the effect of the intervention on literacy achievement. Prior studies concluded that Reading Recovery was positively associated with increased student motivation levels, but most of those studies were limited methodologically. The achievement and motivation levels before and after the intervention of Reading Recovery students and similarly low-performing first-grade students were compared using structural equation modeling. It was found that Reading Recovery had a .31 treatment effect on achievement after controlling for baseline achievement and motivational differences among the treatment and comparison students. Reading Recovery also was associated with greater average levels of posttest motivation, and motivation was found to mediate the treatment-achievement relationship. This study highlights how important it is for early reading interventions to consider the role motivation plays in literacy acquisition.     

Physics knowledge of first semester physics students in Germany: a comparison of 1978 and 2013 cohorts

ABSTRACT

Over the last decades, the percentage of the age group choosing to pursue university studies has increased significantly across the world. At the same time, there are university teachers who believe that the standards have fallen. There is little research on whether students nowadays demonstrate knowledge or abilities similar to that of the preceding cohorts. However, in times of educational expansion, empirical evidence on student test performance is extremely helpful in evaluating how well educational systems cope with the increasing numbers of students. In this study, we compared a sample of 2322 physics freshmen from 2013 with another sample of 2718 physics freshmen from 1978 at universities in Germany with regard to their physics knowledge based on their results in the same entrance test. Previous results on mathematics knowledge and abilities in the same sample of students indicated that there was no severe decline in their average achievement. This paper compares the physics knowledge of the same two samples of students. Contrary to their mathematics results, their physics results showed a substantial decrease in physics knowledge as measured by the test.     

Assessing Effects of Schooling With Cross-Sectional Data: Between-Grades Differences Addressed as a Selection-Bias Problem

Abstract

This article details how cross-sectional data may be used to assess the effect of schooling. The methodology proposed requires that the data derive from two (or more) adjacent grades. Differences in learning outcomes between grades are assumed to be accounted for by differences in age, the education received, and unmeasured variables affecting both assignment to grades and learning outcomes. The approach presented includes an estimation of the impact of these unmeasured variables. It was applied to the TIMSS-95 data, which relate to student achievement and attitudes to both mathematics and science in 15 different countries. This research illustrated that the proposed approach yielded results that were very close to estimates based on an instrumental variables method.     

Relationship between affect and achievement in science and mathematics in Malaysia and Singapore

Background : The Trends in International Mathematics and Science Study (TIMSS) assesses the quality of the teaching and learning of science and mathematics among Grades 4 and 8 students across participating countries.

Purpose : This study explored the relationship between positive affect towards science and mathematics and achievement in science and mathematics among Malaysian and Singaporean Grade 8 students.

Sample : In total, 4466 Malaysia students and 4599 Singaporean students from Grade 8 who participated in TIMSS 2007 were involved in this study.

Design and method : Students’ achievement scores on eight items in the survey instrument that were reported in TIMSS 2007 were used as the dependent variable in the analysis. Students’ scores on four items in the TIMSS 2007 survey instrument pertaining to students’ affect towards science and mathematics together with students’ gender, language spoken at home and parental education were used as the independent variables.

Results : Positive affect towards science and mathematics indicated statistically significant predictive effects on achievement in the two subjects for both Malaysian and Singaporean Grade 8 students. There were statistically significant predictive effects on mathematics achievement for the students’ gender, language spoken at home and parental education for both Malaysian and Singaporean students, with R ² = 0.18 and 0.21, respectively. However, only parental education showed statistically significant predictive effects on science achievement for both countries. For Singapore, language spoken at home also demonstrated statistically significant predictive effects on science achievement, whereas gender did not. For Malaysia, neither gender nor language spoken at home had statistically significant predictive effects on science achievement.

Conclusions : It is important for educators to consider implementing self-concept enhancement intervention programmes by incorporating ‘affect’ components of academic self-concept in order to develop students’ talents and promote academic excellence in science and mathematics.