The causal effect of an extra year of schooling on skills and knowledge in Latin America. Evidence from PISA

Abstract

In this paper, we estimate the causal effect of an extra year of schooling on mathematics performance for seven Latin American countries based on PISA 2012. To that end we exploit exogenous variation in students’ birthdates around the school entry cut-off date using both sharp and fuzzy Regression Discontinuity designs. We find strong effects of an extra year of schooling in most countries, which amount to a 30% increase in PISA test scores in Brazil, 18% in Uruguay, 7% in Argentina and 6% in Costa Rica. These effects differ from the typical estimates obtained from simple regressions or multilevel models and are large enough to allow 15-year-old students to reach higher proficiency levels, suggesting significant potential gains of reducing dropout rates in the region. Finally, we stress the importance of taking into account the effects of school entry cut-off dates on PISA samples to avoid making unfair international comparisons.     

Regression Discontinuity Designs With Multiple Rating-Score Variables

Abstract

In the absence of a randomized control trial, regression discontinuity (RD) designs can produce plausible estimates of the treatment effect on an outcome for individuals near a cutoff score. In the standard RD design, individuals with rating scores higher than some exogenously determined cutoff score are assigned to one treatment condition; those with rating scores below the cutoff score are assigned to an alternate treatment condition. Many education policies, however, assign treatment status on the basis of more than one rating-score dimension. We refer to this class of RD designs as “multiple rating score regression discontinuity” (MRSRD) designs. In this paper, we discuss five different approaches to estimating treatment effects using MRSRD designs (response surface RD; frontier RD; fuzzy frontier RD; distance-based RD; and binding-score RD). We discuss differences among them in terms of their estimands, applications, statistical power, and potential extensions for studying heterogeneity of treatment effects.     

Intervening with Struggling Readers in Seventh Grade: Impact Evidence from Six School Districts

Abstract

This study uses a regression discontinuity design to evaluate the impact of seventh-grade interventions delivered by 25 schools on the reading and school engagement outcomes of struggling readers. Students in participating schools were assigned to intervention (n?=?1,495) or comparison (n?=?4,397) conditions, based on their state reading performance and reading fluency scores in sixth grade. On the state reading test, intervention impact favoring the treatment group approached, but did not reach, statistical significance (p?=?.056, g?=?.15). On a measure of reading fluency, there was a significant interaction between the cut points schools chose for condition assignment and intervention impact (p?=?.015). In schools that chose a lower cut point, intervention students outperformed comparison students, while in schools that chose a higher cut point, comparison students outperformed intervention students. On a student self-report measure of school engagement, no significant differences between the intervention and comparison groups were observed (p?=?.83, g?=?.02). Implications for rigorous evaluation of interventions implemented by districts and schools under naturalistic conditions are discussed.     

Does NCLB Improve the Achievement of Students With Disabilities? A Regression Discontinuity Design

Abstract

The No Child Left Behind Act (NCLB) holds public schools accountable for the academic achievement of student subgroups that are larger than the state minimum-subgroup-size threshold. In 2004, California added students with disabilities to the NCLB subgroup categories. Using a regression discontinuity design, this study compared the academic achievement of students-with-disabilities subgroups that were just above the minimum-subgroup-size threshold to those just below the threshold. The results showed no effects of holding schools and teachers accountable for the achievement of students with disabilities after controlling for student performance and school characteristics of the previous academic year. Policy implications are discussed.     

Modern Regression Discontinuity Analysis

Abstract

This article provides a detailed discussion of the theory and practice of modern regression discontinuity (RD) analysis for estimating the effects of interventions or treatments. Part 1 briefly chronicles the history of RD analysis and summarizes its past applications. Part 2 explains how in theory an RD analysis can identify an average effect of treatment for a population and how different types of RD analyses—“sharp” versus “fuzzy”—can identify average treatment effects for different conceptual subpopulations. Part 3 of the article introduces graphical methods, parametric statistical methods, and nonparametric statistical methods for estimating treatment effects in practice from regression discontinuity data plus validation tests and robustness tests for assessing these estimates. Section 4 considers generalizing RD findings and presents several different views on and approaches to the issue. Part 5 notes some important issues to pursue in future research about or applications of RD analysis.     

Evaluating Paraeducator-led Reading Interventions in Elementary School: A Multi-Cutoff Regression-Discontinuity Analysis

Abstract

A two-cutoff regression discontinuity design (RDD) was used to assign 321 students in grades 1 through 6 at a Title I elementary school to two types of Tier 2 reading interventions administered by paraeducators: (a) direct instruction (DI) and (b) computer-assisted instruction (CAI). Students scoring at or below a lower cutoff pretest score were assigned to the DI reading intervention. Students scoring between the lower cutoff score and an upper cutoff score on the pretest were assigned to a CAI reading intervention. Student reading ability was reassessed in January and May. Results indicated that the DI intervention was significantly more effective than the CAI interventions at the lower cutoff (p < .01). No significant treatment effect was detected at the upper cutoff, but the estimation power of the design at this cutoff was limited to medium-to-large effect size. Findings suggest that the DI intervention was superior to the CAI interventions for at-risk readers. Implications for practice, including fidelity of paraeducator implementation, are discussed.     

Optimal Design for Two-Level Random Assignment and Regression Discontinuity Studies

An important concern when planning research studies is to obtain maximum precision of an estimate of a treatment effect given a budget constraint. When research designs have a multilevel or hierarchical structure changes in sample size at different levels of the design will impact precision differently. Furthermore, there will typically be differential costs of enrolling additional units at different levels of the hierarchy. The optimal design problem in multilevel research studies involves determining the optimal sample size at each level of the design given specified design parameters and a specified marginal cost of recruitment at each level. The current work extends existing results by considering optimal design for (a) unbalanced random assignment designs and (b) regression discontinuity designs.     

Class Size Effects on Fourth-Grade Mathematics Achievement: Evidence From TIMSS 2011

ABSTRACT

Class size reduction policies have been widely implemented around the world in recent years. However, findings about the effects of class size on student achievement have been mixed. This study examines class size effects on fourth-grade mathematics achievement in 14 European countries using data from TIMSS (Trends in International Mathematics and Science Study) 2011. We employ quasi-experimental methodology (i.e., instrumental variables and regression discontinuity) to facilitate causal inferences of class size effects. Although we find some evidence of class size effects in Romania and the Slovak Republic, overall there are no systematic patterns of class size effects across countries. The results indicate that in most European countries class size reduction may not improve mathematics achievement in fourth grade.     

Inter-method reliability of school effectiveness measures: a comparison of value-added and regression discontinuity estimates

Value-added (VA) measures are currently the predominant approach used to compare the effectiveness of schools. Recent educational effectiveness research, however, has developed alternative approaches including the regression discontinuity (RD) design, which also allows estimation of absolute school effects. Initial research suggests RD is a viable approach to measuring school effectiveness. The present study builds on this pioneering work by using RD and VA designs to estimate school effects at system and school level, comparing estimates from several measurement designs. The study uses a large English dataset (N = 148,135) spanning 342 schools, 10 local authorities, 6 consecutive school year groups (UK Years 3–9) across 3 years. RD is found to be a suitable approach for system-level absolute school effect estimates. Cross-sectional and longitudinal measures are found to lead to markedly different estimates when comparing individual schools. The results also reinforce the need to treat measures based on a single cohort with extreme caution.     

The Causal Effects of Grade Retention on Behavioral Outcomes

This study examines the impact of grade retention on behavioral outcomes under a comprehensive assessment-based student promotion policy in New York City. To isolate the causal effect of grade retention, we implement a fuzzy regression discontinuity (RD) design that exploits the fact that grade retention is largely determined by whether a student scores below a cutoff on a standardized test score. We use data on students subject to the policy over a nine-year span to examine impacts on attendance and disciplinary event outcomes. We do not find evidence of systematic effects of retention on behavioral outcomes in either direction. We do find sporadic nonsustained significant effects of retention on behavioral outcomes. When present, these isolated nonpersistent effects tend to be beneficial when found for retained elementary school students and mixed for retained middle school students.     

Using Regression Discontinuity to Test the Impact of a Tier 2 Reading Intervention in First Grade

Abstract

?Multitiered systems of reading instruction and intervention, including response to intervention, are widely used in early reading by schools to provide more intense services to students who need them. Research using randomized controlled trials has compared innovative Tier 2 interventions to business-as-usual Tier 2 approaches and established a number of important components that compose effective Tier 2 interventions in early reading. The purpose of this study was to test the impact of a Tier 2 intervention with Tier 2 compared to Tier 1 instruction alone using regression discontinuity. A cut score was used to assign first-grade students at risk for reading difficulties to Tier 2 intervention plus Tier 1 instruction. Students who missed the cut score in the control group received Tier 1 instruction only. Students in the treatment group, just below the cut score, made greater gains on the SAT10 total score and the individual subtests than students just above the cut score. Outcomes were not significant on oral reading fluency.     

Editorial

Purpose: The purpose of this study was to assess the quality of scientific arguments developed by pre-service physics teachers.

Sample: The participants were 171 pre-service physics teachers recruited from two universities: 86 from University A and 85 from University B.

Design and method: Participants were prompted to develop a written argument to either support or challenge the Turkish government’s decision to invest in nuclear power plants. Data consist of written arguments developed by the participants and information on participants’ knowledge of the topic, their confidence in their knowledge and the source of their knowledge related to the topic. Data were analyzed using the CER framework.

Results: The results show that participants did not perform at the expected level. The majority of students failed to develop strong scientific arguments. While almost all of the participants provided evidence to justify their claims, they failed to effectively coordinate evidence, claim and theory to develop an argument. Students struggled the most in the warrant/reasoning category of the CER framework. We also identified several misconceptions that students held related to nuclear power plants.

Conclusions: In our discussion we problematize college science teaching and advocate integration of instructional strategies such as argumentation that can effectively engage students in construction, evaluation and justification of knowledge.     

A jigsaw cooperative learning application in elementary science and technology lessons: physical and chemical changes

Cooperative learning is an active learning approach in which students work together in small groups to complete an assigned task. Students commonly find the subject of ‘physical and chemical changes’ difficult and abstract, and thus they generally have many misconceptions about it.

Purpose

This study aimed to investigate the effects of jigsaw cooperative learning activities developed by the researchers on sixth grade students’ understanding of physical and chemical changes.

Sample

Participants in the study were 61 sixth grade students in a public elementary school in Izmir, Turkey.

Design and methods

A pre-test and post-test experimental design with a control group was used, and students were randomly assigned to the experimental and control groups. Instruction of the subject was conducted via jigsaw cooperative learning in the experimental group and via teacher-centered instruction in the control group. During the jigsaw process, experimental group students studied the subjects of changes of state, changes in shape and molecular solubility from physical changes, and acid–base reactions, combustion reactions and changes depending on heating from chemical changes in their jigsaw groups.

Results

The concept test results showed that jigsaw cooperative learning instruction yielded significantly better acquisition of scientific concepts related to physical and chemical changes, compared to traditional learning. Students in the experimental group had a lower proportion of misconceptions than those in the control group, and some misconceptions in the control group were identified for the first time in this study.

Conclusions

Jigsaw cooperative learning is an effective teaching technique for challenging sixth grade students’ misconceptions in the context of physical and chemical changes, and enhancing their motivation, learning achievements, self-confidence and willingness in the science and technology lesson. This technique could be applied to other chemistry subjects and other grade levels.     

Student achievement and attitude in a satellite‐delivered high school science course

Abstract

This study investigated the effect of satellite‐delivered instruction on student achievement and attitude in a high school anatomy and physiology course. The experimental group included students from seven high schools enrolled in the satellite‐delivered course. The control group consisted of students from seven high schools in which classroom teachers provided instruction. An experimental versus control matched‐pair design was used in the study. Two hypotheses were tested using the t test for dependent samples. The findings show that there was no significant difference between the experimental group and control group in attitude toward anatomy and physiology. However, on the achievement test, the mean post‐test score of the experimental group was significantly higher than that of the control group.     

Middle school students’ learning of mechanics concepts through engagement in different sequences of physical and virtual experiments

ABSTRACT

Physical and virtual experimentation are thought to have different affordances for supporting students’ learning. Research investigating the use of physical and virtual experiments to support students’ learning has identified a variety of, sometimes conflicting, outcomes. Unanswered questions remain about how physical and virtual experiments may impact students’ learning and for which contexts and content areas they may be most effective. Using a quasi-experimental design, we examined eighth grade students’ (N?=?100) learning of physics concepts related to pulleys depending on the sequence of physical and virtual labs they engaged in. Five classes of students were assigned to either the: physical first condition (PF) (n?=?55), where students performed a physical pulley experiment and then performed the same experiment virtually, or virtual first condition (VF) (n?=?45), with the opposite sequence. Repeated measures ANOVA’s were conducted to examine how physical and virtual labs impacted students’ learning of specific physics concepts. While we did not find clear-cut support that one sequence was better, we did find evidence that participating in virtual experiments may be more beneficial for learning certain physics concepts, such as work and mechanical advantage. Our findings support the idea that if time or physical materials are limited, using virtual experiments may help students understand work and mechanical advantage.     

Prediction/discussion-based learning cycle versus conceptual change text: comparative effects on students’ understanding of genetics

Background and purpose:

The purpose of this study was to investigate the comparative effects of a prediction/discussion-based learning cycle (HPD-LC), conceptual change text (CCT) and traditional instruction on 10th grade students’ understanding of genetics concepts.

Sample:

Participants were 112 10th basic grade male students in three classes of the same school located in an urban area. The three classes taught by the same biology teacher were randomly assigned as a prediction/discussion-based learning cycle class (n?=?39), conceptual change text class (n?=?37) and traditional class (n?=?36).

Design and method:

A quasi-experimental research design of pre-test–post-test non-equivalent control group was adopted. Participants completed the Genetics Concept Test as pre-test–post-test, to examine the effects of instructional strategies on their genetics understanding. Pre-test scores and Test of Logical Thinking scores were used as covariates.

Results:

The analysis of covariance showed a statistically significant difference between the experimental and control groups in the favor of experimental groups after treatment. However, no statistically significant difference between the experimental groups (HPD-LC versus CCT instruction) was found.

Conclusions:

Overall, the findings of this study support the use of the prediction/discussion-based learning cycle and conceptual change text in both research and teaching. The findings may be useful for improving classroom practices in teaching science concepts and for the development of suitable materials promoting students’ understanding of science.     

According to Plan? Examining the Intended and Unintended Treatment Effects of EL Classification in Early Elementary and the Transition to Middle School

Abstract

Research has demonstrated that classification as an English learner (EL) is consequential for students, with important effects on educational achievement and attainment. Yet we know little about the mechanisms that underlie the effects of EL classification. This study explores whether, and to what extent, EL classification results in the treatments it is designed to trigger, and also whether, and to what extent, EL classification triggers unintended treatments. The study uses longitudinal school district data and regression discontinuity analyses to evaluate the treatment effects of EL classification. It examines two moments: initial classification as students enter kindergarten, and retained EL classification as students transition into middle school. Findings suggest that EL classification, for students near the margin of classification, results in many treatments it is intended to trigger including language assessments in early elementary, and English language development instruction and modified content instruction in middle school. In addition, findings show that EL classification triggers two unintended treatments among students; Classification limits access to English speaking peers in early elementary, and negatively impacts enrollment in a full academic course load in middle school.     

Meta-Analysis of the Impact of Reading Interventions for Students in the Primary Grades

Abstract

This meta-analysis systematically reviewed the most up-to-date literature to determine the effectiveness of reading interventions on measures of word and pseudoword reading, reading comprehension, and passage fluency, and to determine the role intervention and study variables play in moderating the impacts for students at risk for reading difficulties in Grades 1–3. We used random-effects meta-regression models with robust variance estimates to summarize overall effects and to explore potential moderator effects. Results from a total of 33 rigorous experimental and quasi-experimental studies conducted between 2002 and 2017 that met WWC evidence standards revealed a significant positive effect for reading interventions on reading outcomes, with a mean effect size of 0.39 (SE = .04, p < .001, 95% CI [0.32, 0.46]). Moderator analyses demonstrated that mean effects varied across outcome domains and areas of instruction.     

The relationship between design reflectivity and conceptions of informed design among high school students

ABSTRACT

Although reflection is a key behaviour of expert designers, it is often a challenging task for new designers. In addition, research on the reflectivity of student designers is limited. The purpose of this study is twofold: (1) to identify the levels of reflectivity while designing, and (2) to study the relationship between reflectivity and conceptions of informed design. We collected data from high school students engaged in an engineering design project. We developed a coding protocol to score levels of reflectivity in student reflections at three levels (low, medium, and high), and used the conceptions of design test to assess changes in student understanding of design activities. Using Wilcoxon signed-rank tests, we determined if students tended to select more ‘key’ design activities and fewer ‘distractors’ within each reflection group. We also performed McNemar’s tests to determine which specific design activities were important within each reflection group after the design project. The results show moderately reflective students had higher gains in understanding of informed design activities compared to those with high or low reflectivity. Results also indicate that different design activities became important for students within each of the three reflective groups. Implications from this research indicate that groups of students experience changing conceptions of design in different ways. An understanding of what students deem important while designing would better allow teachers to encourage behaviours that are like those of informed designers.