Does Work Harm Academic Performance of Students? Evidence Using Propensity Score Matching

In this article we analyze the effects of student work on academic performance for college students. In order to reduce the endogeneity bias due to selection into treatment, we use propensity score matching technique. This approach allows us to estimate the effects of student work separately for different years of study, which is not possible when inside instruments are used to deal with endogeneity of student work. We find predominantly negative effects of student work for all measures of academic performance (GPA, exam attempts, exams passed, and likelihood of passing a year), although many of these are economically and statistically insignificant. We supplement existing studies that do not estimate separate treatment effects for different years of study by showing that work while in college harms study outcomes mostly in the first year of study—by passing smaller number of exams and thereby increasing the likelihood of failing a year. Our results are consistent with evidence on difficulty with adjusting to college studies of first-year students, who face many uncertainties that affect finding the optimal allocation of time between studies, work and leisure.     

班干部身份对学习机会获得的影响——基于4026位初中生的倾向值匹配法研究

已有研究表明,相比普通同学,班干部能获得更多的学习机会。但因为研究方法的局限,我们无法知道,这两者的相关是因为班干部"身份"导致更多学习机会,还是因为它们两者共同受作用于其他因素。研究利用4 026位初中学生的有效数据,运用倾向值匹配法来解决班干部"身份"估计效应的推断问题。结果表明,班干部"身份"确实可以独立地增加学生的学习机会,而且效应不小。研究发现对于我们改进班干部制度具有直接的政策启示。     

Narrowing in on Educational Resources That Do Affect Student Achievement

In an era dominated by issues of school finance adequacy, it seems particularly important to provide evidence that, despite a number of claims to the contrary, educational resources are indeed positively related to improved student achievement. One of the hypotheses of this article is that expenditures per pupil must be disaggregated into more meaningful categories to discern the relationship between resources and student achievement. To explore this question, this article uses data from the Washoe County School District in Reno, Nevada, which reports its school expenditures using a program called InSite. This program disaggregates expenditures into 4 categories: instruction, instructional support, leadership, and operations and maintenance. This school-level variable is the primary explanatory variable in this covariate adjustment model using a 3-level hierarchical linear modeling analysis of students (approximately 7,000) nested in classrooms (approximately 420) nested in schools (approximately 55). This model also includes a number of contextual and school compositional factors that research tells us affect student achievement, including student demographic characteristics and pretest score; teacher experience, education, and a measure of his or her instructional practice; and school size, school-level poverty, and expenditures broken out into 4 categories: instruction, instructional support, leadership, and operations and maintenance. The results show that expenditures for instruction and instructional support were positively related and statistically significant for the reading achievement of 3rd, 4th, 5th, and 6th graders in the 2002-03 school year.     

‘Perceived’ competition and performance in Italian secondary schools: New evidence from OECD—PISA 2006

In this paper, we investigate the effects of competition on the performance of Italian secondary schools as measured by maths achievement scores (PISA 2006 dataset). Competition is measured by an indicator of ‘perceived’ competition (generated from an answer provided by the schools’ principals). The methodology employed is a propensity score matching that is corrected to take into account heteroskedasticity and finite sample bias. The results show a positive effect of competition on school performance. Nevertheless, this effect is quite low (between 3.62% and 4.05% computed at the average score level) and is consistent with previous findings about educational systems in Italy and worldwide. This is relevant for policy‐making because competition appears to impact school performance even in a country like Italy where specific pro‐competitive policies are quite absent.     

Heterogeneous Subgroup Identification in Observational Studies

Abstract

Despite the overwhelming focus on the overall average treatment effect in the methodological and statistical literature, in many cases the efficacy of an educational program or intervention might vary based on unit background characteristics. The identification of subgroups for which an educational intervention is particularly effective or, on the other hand, has no effect or is possibly harmful, may have important practical implications, especially in terms of allocation of resources. We propose a five-step approach using propensity score matching and regression trees to identify subgroups with heterogeneous treatment effects in observational studies. Results of two Monte Carlo simulation studies demonstrate that the proposed approach can accurately identify heterogeneous subgroups while maintaining Type I error rate. In a case study with Early Childhood Longitudinal Study-Kindergarten cohort data, we find that the effect of exposure to special education services on fifth-grade mathematics achievement varies based on kindergarten mathematics achievement and student gender.     

Student Employment and Persistence: Evidence of Effect Heterogeneity of Student Employment on College Dropout

This study explores how student employment affects college persistence and how these effects differ by individual likelihood of participating in student employment. I analyze data from the National Longitudinal Survey of Youth 1997 using propensity score matching and stratification-multilevel analysis. This study finds that engaging in intense work has deleterious effects on college persistence. However, these negative effects vary significantly according to likelihood of participation in intense work. The results indicate that employment has less negative impacts on completion for those most likely to participate in intense work, who are typically those from the most disadvantaged social backgrounds. This finding suggests that efforts to reduce the deleterious effects of intense work on persistence should be practiced with careful consideration for sub-populations that may have different reasons for and effects of student employment.     

Detecting selection bias,using propensity score matching,and estimating treatment effects: an application to the private returns to a master’s degree

Most research in the area of higher education is plagued by the problem of endogeneity or self-selection bias. Unlike ordinary least squares (OLS) regression, propensity score matching addresses the issue of self-selection bias and allows for a decomposition of treatment effects on outcomes. Using panel data from a national survey of bachelor’s degree recipients, this approach is illustrated via an analysis of the effect of receiving a master’s degree, in various program areas, on wage earning outcomes. The results of this study reveal that substantial self-selection bias is undetected when using OLS regression techniques. This article also shows that, unlike OLS regression, propensity score matching allows for estimates of the average treatment effect, average treatment on the treated effect, and the average treatment on the untreated effect on student outcomes such as wage earnings.     

The heterogeneous effects of ability grouping on national college entrance exam performance – evidence from a large city in China

This study attempts to evaluate the achievement effect of ability grouping on student performance on the National College Entrance Exam in China. The context of this study is the ongoing school reform movement occurring in many Chinese municipalities. The current reform movement is striving to achieve educational equity and quality by integrating initial low achievers into high-performing schools. The propensity score matching method is employed as the identification strategy. After controlling for self-selection bias in high school assignment, this study finds that while there is no effect of high ability grouping at the school level on academic achievement, initial low achievers’ academic performances can be significantly improved when integrated with high performing students at the school level. In addition, between-class grouping significantly improves student performance in a heterogeneous school-level grouping. Upon analyzing the results, rich information on school level input is reported.     

“Screening” or “Meritocracy”: Does the Choice of a Private School Help Students Achieve Higher Academic Performance? —Based on the PISA 2018 Analysis of Four Provinces and Municipalities in China

The introduced policy of “synchronized enrollment of public and private schools” has once again triggered the debate on the effectiveness of public and private schools. Based on the data of the Program for International Student Assessment (PISA) 2018 from four Chinese provinces and municipalities, this paper explores whether private schools gain a relative advantage in student academic performance through student “screening” or academic “meritocracy,” through a hierarchical linear model (HLM) and an empirical test of the propensity score matching (PSM). It has been found that the academic performance of students in private schools is significantly better than that in public schools. But with background, metacognitive ability, and learning hours of students in private schools controlled for, such academic performance is not significantly superior, suggesting that private schools rely heavily on student “screening” to achieve a relative advantage in student academic performance. This finding has also verified the scientific nature of the above policy.     

The Implications of Contamination for Educational Experiments With Two Levels of Nesting

ABSTRACT

Experimental evaluations that involve the educational system usually involve a hierarchical structure (students are nested within classrooms that are nested within schools, etc.). Concerns about contamination, where research subjects receive certain features of an intervention intended for subjects in a different experimental group, have often led researchers to randomize units at a higher level of the educational hierarchy. Existing work on two-level designs suggests that situations where contamination should lead to randomization at a higher level are likely to be rare. This article extends these results to the case of three-level designs. In order to understand the implications of mathematical results, existing information about the size of intracluster correlation coefficients (ICCs) in educational studies with three levels and about the extent of treatment effect heterogeneity across schools is discussed. Better empirical estimates of ICCs, treatment effect heterogeneity, and plausible contamination values are necessary to make full use of the results in this article. However, it seems likely that situations where contamination should lead to randomization at a higher level in three-level designs are rare.     

Investigating the effect of school‐wide positive behavioral interventions and supports on student learning and behavioral problems in elementary and middle schools

Many states have scaled up School‐Wide Positive Behavioral Interventions and Supports (SW‐PBIS) with the goal of improving student behavior and academic outcomes. Although the effects of SW‐PBIS on behavioral and discipline outcomes have been promising, the findings for academic achievement have been inconclusive and are often limited to cross‐sectional data. This paper examined the longitudinal effect of SW‐PBIS on student behavioral problems and academic achievement growth in elementary and middle schools using propensity score matching. We found no statistically significant longitudinal effects of SW‐PBIS on either student behavior problems or academic achievement among elementary and middle schools in this quasi‐experimental design, even though the positive direction of longitudinal changes would be identified. The result raised questions about the efficacy of SW‐PBIS using a statewide achievement test when scaled up to a state level. Additional research is needed to better understand the impact of SW‐PBIS using rigorous scale‐up designs.     

Exploring the relationship between child physical abuse and adult dating violence using a causal inference approach in an emerging adult population in South Korea

Child maltreatment is one of the most commonly examined risk factors for violence in dating relationships. Often referred to as the intergenerational transmission of violence or cycle of violence, a fair amount of research suggests that experiencing abuse during childhood significantly increases the likelihood of involvement in violent relationships later, but these conclusions are primarily based on correlational research designs. Furthermore, the majority of research linking childhood maltreatment and dating violence has focused on samples of young people from the United States. Considering these limitations, the current study uses a rigorous, propensity score matching approach to estimate the causal effect of experiencing child physical abuse on adult dating violence among a large sample of South Korean emerging adults. Results indicate that the link between child physical abuse and adult dating violence is spurious rather than causal. Study limitations and implications are discussed.     

The Effect of Propensity Score Matching Method on the Quantity and Quality of Matches

Often program administrators are interested in knowing how students benefit from participation in programs compared to students who do not participate. Such comparisons may be sullied by the fact that participants self-select into programs, resulting in differences between groups prior to programming. By controlling for researcher-identified–self-selection variables, propensity score matching enables researchers to create comparable matched groups. However, when employing propensity score matching, researchers are faced with a plethora of matching options. In the current study, we compared the quantity and quality of matches obtained when applying common matching techniques to real data. The methods produced matches of varying quantity and quality. Moreover, group comparisons on the outcome led to different conclusions depending on the matching method employed.     

Youth Participatory Action Research in the High School Curriculum: Education Outcomes for Student Participants in a District-Wide Initiative

This study employed a quasi-experimental design to examine the effects of a school-based youth participatory action research program on the education outcomes of participating high school students. The program was a year-long elective course in six high schools in the same California district whose student population is predominantly low-income youth of color. A propensity score matching approach compared the reading achievement, attendance rates, and discipline referral rates of 153 students who participated in the program to nonparticipating peers with similar demographics and baseline scores on the outcomes. Results showed that the program significantly improved students’ attendance rates with less compelling evidence for its effect on reading achievement. These findings suggest that youth participatory action research may be an effective pedagogical practice for high school students, particularly low-income students of color.     

Associations between students' perceptions of classroom environment and academic efficacy in Australian and British secondary schools

Often, when an evaluation is needed, there is always a tendency to jump into data collection without first considering what might be already available. This paper demonstrates how an existing student database from a large school district could be used to provide some rigorous evidence regarding the impact of a supplementary teacher professional development programme on student achievement. Randomization was not possible in this study but to ensure rigour, matching of the treatment and the comparison groups was done on variables that were considered highly predictive of future achievement. The two groups were matched at the class level on course type, grade level and prior achievement. Next, hierarchical linear modelling was used to further control for selection bias between the two groups. According to the statistical theory on selection bias, this two‐part process of matching and modelling removes all the observed bias. This illustration shows that performing a rigorous quasi‐experimental study is possible in education even when there are limiting circumstances to conducting a randomized experiment.     

Propensity Score Matching for Education Data: Worked Examples

Abstract

Randomized controlled trials are not always feasible in educational research, so researchers must use alternative methods to study treatment effects. Propensity score matching is one such method for observational studies that has shown considerable growth in popularity since it was first introduced in the early 1980s. This paper outlines the concept of propensity scores by explaining their theoretical principles and providing two examples of their usefulness within the realm of educational research. Through worked examples, we highlight the effectiveness of propensity scores as a method for reducing bias and increasing the balance between treatment and comparison groups. To aid in the understanding and future use of propensity scores, we provide R syntax for all our analyses.     

The Impact of Education Tax Benefits on College Completion

This paper uses a nationally representative sample from the 2004–09 Beginning Postsecondary Students (BPS) survey to examine the effect of education tax benefits on college completion. The paper employs a propensity score matching approach to correct for differences between eligible and ineligible students. Results suggest that tax benefits increase the likelihood of completing a college degree by 8 percentage points. The effect of tax benefits is largest for students who attended private and four-year institutions.     

THE DIFFERENCES IN SCORES AND SELF-EFFICACY BY STUDENT GENDER IN MATHEMATICS AND SCIENCE

Typically, mathematics and science are seen as linked together, where both subjects involve numbers, critical thinking, and problem solving. Our study aims to develop a better understanding of the connections between student’s achievement scores in mathematics and science, student gender, and self-efficacy. We used the Trends in International Mathematics and Science Study 2007 eighth grade data to answer our research questions and were able to demonstrate that when controlling for self-efficacy, there is a statistically significant difference in the achievement scores between males and females by subject, where females score higher Algebra, but males score higher in the other mathematics subjects. Likewise, we were also able to demonstrate that there is a statistically significant difference in the achievement scores in Earth Science, Physics, and Biology, between males and females where males score higher in science subjects. In both mathematics and science examinations, we controlled for self-efficacy where in mathematics females hold lower self-efficacy then males and in science there is no difference between females and males in terms of self-efficacy. We conjecture that mathematics and science classrooms that consider self-efficacy may impact student’s achievement scores by subject, which can ultimately impact career choices in mathematics- and science-based fields.     

The Impact of Recent Academic Due Process Decisions on Counselor Education Programs

Although fairly recent to higher education, the broad concept and specifications of due process have had a major impact on the day-to-day operations of educational institutions. The due process concept and its accompanying rights and responsibilities is particularly applicable to counselor education programs in situations where a student is terminated from a degree program. This article discusses a recent United States Supreme Court decision involving a medical student in which the Court established guidelines for academic dismissals and terminations that are now applicable to counselor education programs.     

Goals,data use,and instruction: the effect of a teacher professional development program on reading achievement

In this paper, we investigated whether student reading comprehension could be improved with help of a teacher Professional Development (PD) program targeting goals, data use, and instruction. The effect of this PD program on 2nd- and 3rd-grade student achievement was examined using a pretest-posttest control group design. Applying propensity score matching, 35 groups in the experimental condition were matched to 35 control groups. Students in the experimental condition (n = 420) scored significantly higher on a standardized assessment than the control condition (n = 399), the effect size being d = .37. No differential effects of the PD program were found in relation to initial reading performance or grade. Different model specifications yielded similar albeit smaller effect sizes (d = .29 and d = .30). At the end of the program, students in the experimental condition were more than half a year ahead of students in the control condition.