The Effect of Propensity Score Matching Method on the Quantity and Quality of Matches

Often program administrators are interested in knowing how students benefit from participation in programs compared to students who do not participate. Such comparisons may be sullied by the fact that participants self-select into programs, resulting in differences between groups prior to programming. By controlling for researcher-identified–self-selection variables, propensity score matching enables researchers to create comparable matched groups. However, when employing propensity score matching, researchers are faced with a plethora of matching options. In the current study, we compared the quantity and quality of matches obtained when applying common matching techniques to real data. The methods produced matches of varying quantity and quality. Moreover, group comparisons on the outcome led to different conclusions depending on the matching method employed.     

Bias Reduction in Quasi-Experiments With Little Selection Theory but Many Covariates

Abstract: In observational studies, selection bias will be completely removed only if the selection mechanism is ignorable, namely, all confounders of treatment selection and potential outcomes are reliably measured. Ideally, well-grounded substantive theories about the selection process and outcome-generating model are used to generate the sample of covariates. However, covariate selection is more heuristic in actual practice. Using two empirical data sets in a simulation study, we investigate four research questions about bias reduction when the selection mechanism is not known but many covariates are measured: (1) How important is the conceptual heterogeneity of the covariate domains in the data set? (2) How important is the number of covariates assessing each domain? (3) What are the joint effects of this conceptual heterogeneity and of the number of covariates per domain? (4) What happens to bias reduction when the set of covariates is deliberately impoverished by removing the covariates most responsible for selection bias, thus ensuring a slightly smaller but still heterogeneous set of covariates? The results indicate: (1) increasingly more bias is reduced as the number of covariate domains and the number of covariates per domain increase, though the rate of bias reduction is diminishing in each case; (2) sampling covariates from multiple heterogeneous covariate domains is more important than choosing many measures from fewer domains; (3) the most heterogeneous set of covariate domains removes almost all of the selection bias when at least five covariates are assessed in each domain; and (4) omitting the most crucial covariates generally replicates the pattern of results due to the number of domains and the number of covariates per domain, but the amount of bias reduction is less than when all variables are included and will surely not satisfy all consumers of causal research.     

Detecting selection bias,using propensity score matching,and estimating treatment effects: an application to the private returns to a master’s degree

Most research in the area of higher education is plagued by the problem of endogeneity or self-selection bias. Unlike ordinary least squares (OLS) regression, propensity score matching addresses the issue of self-selection bias and allows for a decomposition of treatment effects on outcomes. Using panel data from a national survey of bachelor’s degree recipients, this approach is illustrated via an analysis of the effect of receiving a master’s degree, in various program areas, on wage earning outcomes. The results of this study reveal that substantial self-selection bias is undetected when using OLS regression techniques. This article also shows that, unlike OLS regression, propensity score matching allows for estimates of the average treatment effect, average treatment on the treated effect, and the average treatment on the untreated effect on student outcomes such as wage earnings.     

Empirical Performance of Covariates in Education Observational Studies

This article summarizes results from 12 empirical evaluations of observational methods in education contexts. We look at the performance of three common covariate-types in observational studies where the outcome is a standardized reading or math test. They are: pretest measures, local geographic matching, and rich covariate sets with a strong theory of treatment selection. Overall, the review demonstrates that although the pretest often reduces bias in observational studies, it does not always eliminate it. Its performance depends on the pretest's correlation with treatment selection and the outcome, and whether preintervention trends are present. We also find that although local comparisons are prioritized for matching, its performance depends on whether comparable no-treatment cases are available. Otherwise, local comparisons may produce badly biased results. In cases where researchers have a strong theory of selection and rich covariate sets, observational methods perform well, but additional replication studies are needed. Finally, observational methods that rely on demographic covariates without a theory of selection rarely produce unbiased treatment effects. The article concludes by offering education researchers empirically based guidance on covariate selection in observational studies.     

How Does Training Abroad Affect Faculty Income? An Analysis Based on the “2014 Faculty Survey in China”

Abstract

Training abroad is an important avenue for promoting the specialized development of faculty and improving international accomplishments. On the basis of the data from the 2014 Faculty Survey in China, this paper applies the technique of propensity score matching to control for the self-selection bias in the sample, so as to quantitatively evaluate the economic benefits of training abroad for faculty. The study finds that training abroad presents significant economic returns, while the potential benefits of training abroad for faculty members who have not yet pursued training abroad are higher than the benefits for faculty members who have pursued training abroad; a nonlinear relationship exists between the period of time for which a faculty member pursues training abroad and the economic benefits thereof, with the order of the economic benefits of training abroad from least to most for different time periods being: 1?year, less than one half year, more than one half year to less than 1?year, and more than 1?year. The conclusions of the study provide an empirical basis for the selection of pathways for the professional development of faculty in the future, as well as the design and refinement of training abroad programs, et cetera.     

A Comparison of Propensity Score Weighting Methods for Evaluating the Effects of Programs With Multiple Versions

This Monte Carlo simulation study compares methods to estimate the effects of programs with multiple versions when assignment of individuals to program version is not random. These methods use generalized propensity scores, which are predicted probabilities of receiving a particular level of the treatment conditional on covariates, to remove selection bias. The results indicate that inverse probability of treatment weighting (IPTW) removes the most bias, followed by optimal full matching (OFM), and marginal mean weighting through stratification (MMWTS). The study also compared standard error estimation with Taylor series linearization, bootstrapping and the jackknife across propensity score methods. With IPTW, these standard error estimation methods performed adequately, but standard errors estimates were biased in most conditions with OFM and MMWTS.     

Effects of kindergarten retention for at-risk children’s psychosocial development

Kindergarten retention is a popular practice for children who are considered unready for primary school. However, past research has not yet succeeded to find consistent, strong empirical evidence supporting the practice. In the current study, kindergarten repeaters’ development in nine psychosocial domains is compared with that of equally at risk but (1) continuously promoted age-mates and (2) promoted age-mates who repeated first grade instead. Analysing data from a large-scale longitudinal study using propensity score matching and multilevel modelling, the findings reveal no harm of kindergarten retention for at-risk children’s long term psychosocial development. Rather, we find that, relative to equally at-risk but continuously promoted children, kindergarten repeaters benefit from retention with respect to higher levels of well-being, and peer relations, and lower levels of hyperactivity, aggression and asocial behaviour. Compared to similar children who were promoted but who were retained in first grade instead, kindergarten repeaters are found to benefit more from retention with respect to higher levels of well-being, self-confidence, attitude to work and independent behaviour, and lower levels of hyperactivity.     

The Principle of the Nomograph in Education

ABSTRACT

Educational researchers frequently study the impact of treatments or interventions on educational outcomes. However, when observational or quasiexperimental data are used for such investigations, selection bias can adversely impact researchers’ abilities to make causal inferences about treatment effects. One way to deal with selection bias is to use propensity score methods. The authors introduce educational researchers to the general principles underlying propensity score methods, describe 2 practical applications of these methods, and discuss their limitations.     

Assessing Methods for Generalizing Experimental Impact Estimates to Target Populations

ABSTRACT

Randomized experiments are considered the gold standard for causal inference because they can provide unbiased estimates of treatment effects for the experimental participants. However, researchers and policymakers are often interested in using a specific experiment to inform decisions about other target populations. In education research, increasing attention is being paid to the potential lack of generalizability of randomized experiments because the experimental participants may be unrepresentative of the target population of interest. This article examines whether generalization may be assisted by statistical methods that adjust for observed differences between the experimental participants and members of a target population. The methods examined include approaches that reweight the experimental data so that participants more closely resemble the target population and methods that utilize models of the outcome. Two simulation studies and one empirical analysis investigate and compare the methods’ performance. One simulation uses purely simulated data while the other utilizes data from an evaluation of a school-based dropout prevention program. Our simulations suggest that machine learning methods outperform regression-based methods when the required structural (ignorability) assumptions are satisfied. When these assumptions are violated, all of the methods examined perform poorly. Our empirical analysis uses data from a multisite experiment to assess how well results from a given site predict impacts in other sites. Using a variety of extrapolation methods, predicted effects for each site are compared to actual benchmarks. Flexible modeling approaches perform best, although linear regression is not far behind. Taken together, these results suggest that flexible modeling techniques can aid generalization while underscoring the fact that even state-of-the-art statistical techniques still rely on strong assumptions.     

青少年体重影响学业成绩吗--来自中国教育追踪调查的经验证据

超重、肥胖、低体重已成为青少年群体面临的主要健康问题。国内学界有关体质健康与学业成绩的探讨方兴未艾,而聚焦体重状态、使用全国性样本的经验分析较为缺乏。研究基于2013—2014学年、2014—2015学年中国教育追踪调查(CEPS)的数据分析,发现:我国初中生体重水平存在性别与城乡差异,男生较女生更易处于异常体重状态,城镇学生更易超重及肥胖,农村学生更易低体重;OLS和倾向值匹配估计(PSM)结果表明,超重及肥胖学生的学业成绩受到体重状态的显著负向影响,低体重对学业成绩的影响不显著;分性别对比结果显示,体重状态对学业成绩的影响存在性别差异,女生群体学业成绩受到超重及肥胖的消极冲击程度更大。     

Propensity Score Matching for Education Data: Worked Examples

Abstract

Randomized controlled trials are not always feasible in educational research, so researchers must use alternative methods to study treatment effects. Propensity score matching is one such method for observational studies that has shown considerable growth in popularity since it was first introduced in the early 1980s. This paper outlines the concept of propensity scores by explaining their theoretical principles and providing two examples of their usefulness within the realm of educational research. Through worked examples, we highlight the effectiveness of propensity scores as a method for reducing bias and increasing the balance between treatment and comparison groups. To aid in the understanding and future use of propensity scores, we provide R syntax for all our analyses.     

Time spent on the Internet by multicultural adolescents in Korea

This study analysed The 8th Online Survey of Adolescent Health and Behaviour in 2012, which is a government-approved statistical survey of Internet use and patterns in Korea. We conducted a propensity score matching (PSM) to control for economic status differences between monocultural and multicultural families and an ANOVA to estimate the effects of multicultural family status, adolescents’ gender, school grades, locality, and parents’ academic background on adolescents’ Internet use (for studying and entertainment). The study revealed the following: with regard to Internet use patterns, adolescents from both monocultural and multicultural families selected gaming as their first choice, followed by studying and data searching. Furthermore, main and interaction effects of independent variables on adolescents’ time spent on the Internet for studying were not significant. However, the interaction effects of multicultural family status and mothers’ academic background on adolescents’ time spent on the Internet for entertainment were statistically significant. Specifically, adolescents’ time spent on the Internet for entertainment was higher in multicultural families with mothers whose education level was lower than middle school. Finally, we suggested that the government should provide full support to less educated mothers in multicultural families for effectively and constructively controlling their children’s Internet use and teaching it to them.     

体育锻炼能够促进青少年的认知能力发展吗?——基于中国教育追踪调查数据的实证研究

本文利用中国教育追踪调查数据(2014—2015),采用普通最小二乘法、分位数回归以及倾向得分匹配法,实证检验了体育锻炼对青少年认知能力的影响,结果发现:(1)利用普通最小二乘法测得经常参与体育锻炼的青少年在认知能力测试上比不经常参与的青少年高出0.149个标准分,当控制前期能力后,青少年参加体育锻炼的时间每上升1个单位值,能将认知能力提高0.049个标准分;(2)利用分位数回归发现参加体育锻炼对认知能力处于中位数上的青少年具有最强的促进作用,体育锻炼的异质性特征突出表现为“倒V”型曲线,而利用无条件分位数回归发现经常参与体育锻炼能够缩小低–高分位点上认知能力的组内差异;(3)利用Probit模型发现学校作业并非青少年参与体育锻炼的决定因素,看电视、玩网络游戏等课余活动的时间则对青少年参与体育锻炼构成了挤出关系,文化资本、人力资本以及教育期望与青少年的体育运动参与率呈正相关;(4)利用倾向得分匹配法发现体育锻炼影响认知能力的平均处理效应落入[0.095,0.117]的取值区间,而忽视选择性偏差将会高估体育锻炼的净效应。     

An Evaluation of Latent Growth Models for Propensity Score Matched Groups

Because random assignment is not possible in observational studies, estimates of treatment effects might be biased due to selection on observable and unobservable variables. To strengthen causal inference in longitudinal observational studies of multiple treatments, we present 4 latent growth models for propensity score matched groups, and evaluate their performance with a Monte Carlo simulation study. We found that the 4 models performed similarly with respect to model fit, bias of parameter estimates, Type I error, and power to test the treatment effect. To demonstrate a multigroup latent growth model with dummy treatment indicators, we estimated the effect of students changing schools during elementary school years on their reading and mathematics achievement, using data from the Early Childhood Longitudinal Study Kindergarten Cohort.     

The Effect of Catholic Schooling on Math and Reading Development in Kindergarten Through Fifth Grade

Abstract

Prior research estimating the effect of Catholic schooling has focused on high school, where evidence suggests a positive effect of Catholic versus public schooling. In this article, we estimate the effect of attending a Catholic elementary school rather than a public school on the math and reading skills of children in kindergarten through fifth grade. We use nationally representative data and a set of matching estimators to estimate the average effect of Catholic schooling and the extent to which the effect varies across educational markets. When we use public school students nationwide or within the same county to provide a counterfactual estimate of how Catholic school students would have performed in public schools, we find strong evidence indicating that Catholic elementary schools are less successful at teaching math skills than public schools (Catholic school students are 3–4 months behind public school students by third and fifth grade), but no more or less successful at teaching reading skills. Moreover, unlike prior research, we find no consistent evidence that the effects of Catholic schooling vary substantially by race or urbanicity, though our power to detect such differences is weak.     

The relative effectiveness of private and public schools: evidence from Kenya

ABSTRACT

This paper estimates the relative effectiveness of private and public primary schools in Kenya using data from 4,433 Grade 6 schoolchildren. Using ordinary least squares as a baseline model, we use the proportion of private schools in a district as an instrument in a Heckman two-stage correction framework, as well as propensity score matching models to correct for selection bias. There is a positive private school effect across all models. In the corrected models, we find that private school pupils outperform their public school counterparts by between .24 and .52 standard deviations.     

Generalized Linear Factor Score Regression: A Comparison of Four Methods

Factor score regression has recently received growing interest as an alternative for structural equation modeling. However, many applications are left without guidance because of the focus on normally distributed outcomes in the literature. We perform a simulation study to examine how a selection of factor scoring methods compare when estimating regression coefficients in generalized linear factor score regression. The current study evaluates the regression method and the correlation-preserving method as well as two sum score methods in ordinary, logistic, and Poisson factor score regression. Our results show that scoring method performance can differ notably across the considered regression models. In addition, the results indicate that the choice of scoring method can substantially influence research conclusions. The regression method generally performs the best in terms of coefficient and standard error bias, accuracy, and empirical Type I error rates. Moreover, the regression method and the correlation-preserving method mostly outperform the sum score methods.     

大学生就业市场景气指数的建构与分析

利用北京大学教育经济研究所历年全国高等学校毕业生就业状况调查数据和多值倾向评分匹配的方法,构建了一个动态化、定量化、具备可比性的大学生就业市场景气测量体系。测量结果表明,大学生就业景气指数呈现上下波动的变化趋势,2003年和2009年是指数的波谷年份,2005年和2015年则是指数的波峰年份;大学生就业景气指数的跨年变动幅度平稳,且指数波动受供需两侧因素的共同影响。在分析往年大学生就业景气指数的变动趋势和影响机制后发现,近年来大学生就业景气又进入了新的下降通道,2020年全球经济萎靡、国内经济下行压力和新型冠状病毒肺炎疫情叠加影响使得高等学校毕业生就业形势更加严峻,但在政府政策干预、新经济兴起以及求职方式转变的影响下,大学生就业景气程度短期受负面影响严重,但就业率并未出现断崖式下降。     

Assessing the effectiveness of a hybrid-flipped model of learning on fluid mechanics instruction: overall course performance,homework, and far- and near-transfer of learning

To examine the impact of a hybrid-flipped model utilising active learning techniques, the researchers inverted one section of an undergraduate fluid mechanics course, reduced seat time, and engaged in active learning sessions in the classroom. We compared this model to the traditional section on four performance measures. We employed a propensity score method entailing a two-stage regression analysis that considered eight covariates to address the potential bias of treatment selection. First, we estimated the probability score based on the eight covariates, and second, we used the inverse of the probability score as a regression weight on the performance of learners who did not select into the hybrid course. Results suggest that enrolment in the hybrid-flipped section had a marginally significant negative impact on the total course score and a significant negative impact on homework performance, possibly because of poor video usage by the hybrid-flipped learners. Suggested considerations are also discussed.     

Adding up or Down: A Discussion

ABSTRACT

Increased access to algebra was a focal point of the National Mathematics Advisory Panel's 2008 report on improving mathematics learning in the United States. Past research found positive effects for early access to algebra, but the focus on average effects may mask important variation across student subgroups. The author addresses whether these positive effects hold up when the analysis is expanded to examine effect heterogeneity. Using a nationally representative sample of eighth-grade students in 1988, the author examined sensitivity of findings to methods for selection bias adjustment, heterogeneity across the propensity to take algebra in Grade 8, and across schools. The findings support past research regarding positive benefits to Grade 8 algebra and are consistent with policies that increase access to algebra in middle school.