The Power of the Test for Treatment Effects in Three-Level Cluster Randomized Designs

Abstract

Experiments that involve nested structures often assign entire groups (such as schools) to treatment conditions. Key aspects of the design of such experiments include knowledge of the intraclass correlation structure and the sample sizes necessary to achieve adequate power to detect the treatment effect. This study provides methods for computing power in three-level cluster randomized balanced designs (with two levels of nesting), where, for example, students are nested within classrooms and classrooms are nested within schools and schools are assigned to treatments. The power computations take into account nesting effects at the second (classroom) and at the third (school) level, sample size effects (e.g., number of schools, classrooms, and individuals), and covariate effects (e.g., pretreatment measures). The methods are applicable to quasi-experimental studies that examine group differences in an outcome.     

A More Powerful Test in Three-Level Cluster Randomized Designs

Abstract

Field experiments that involve nested structures frequently assign treatment conditions to entire groups (such as schools). A key aspect of the design of such experiments includes knowledge of the clustering effects that are often expressed via intraclass correlation. This study provides methods for constructing a more powerful test for the treatment effect in three-level cluster randomized designs with two levels of nesting (at the second and third levels). When the intraclass correlation structure at the second and third level is assumed to be known, the proposed test provides higher estimates of power than those obtained from the typical test based on level-3 unit means, because it preserves the degrees of freedom associated with the number of level-2 and level-1 units. The advantage in power estimates is more pronounced when the number of level-3 units (e.g., schools) is small and the samples are homogeneous (e.g., low-achieving schools).     

Power Computations for Polynomial Change Models in Block-Randomized Designs

Education experiments frequently assign students to treatment or control conditions within schools. Longitudinal components added in these studies (e.g., students followed over time) allow researchers to assess treatment effects in average rates of change (e.g., linear or quadratic). We provide methods for a priori power analysis in three-level polynomial change models for block-randomized designs. We discuss unconditional models and models with covariates at the second and third level. We illustrate how power is influenced by the number of measurement occasions, the sample sizes at the second and third levels, and the covariates at the second and third levels.     

The Growth and Nature of Research Interest in Arithmetic and Reading

ABSTRACT

The author examined whether mathematics instruction provided by kindergarten teachers is related to children's mathematics learning during the kindergarten year based on the children's socioeconomic status and race. Hierarchical linear modeling was employed using a large sample of kindergarten students to estimate relationships between the teacher's instructional approach (e.g., concrete-spatial, interpersonal, linguistic) and children's mathematics skills. Results showed that the teacher's instructional approach was selectively related to children's mathematics learning. Students in classrooms where teachers frequently employed a concrete-spatial instructional approach learned more during the kindergarten year. Students from lower socioeconomic backgrounds benefitted more from teachers who often employed an interpersonal approach, and Black/African American students benefitted less from classrooms where teachers relied more on a linguistic approach. The findings suggest that kindergarten teachers’ instruction needs to employ varied methods that take into account students’ mathematics skills and background characteristics.     

The developmental influence of sex composition in preschool classrooms: Boys fare worse in preschool classrooms with more boys

A multilevel modeling approach was employed to investigate the relation between sex composition and developmental change in 70 urban preschool classrooms. The research represents a unique contribution as (1) few studies have examined the influence of sex composition during the preschool years, (2) it represents the first research to use a continuous (i.e., sex ratio) as opposed to binary (i.e., mixed- versus single-sex) indicator for classroom sex composition, and (3) the sample represents an important and often neglected group (i.e., low-income children from urban schools). A series of HLM models were run, addressing the nested nature of the data (children within classrooms), and relating classroom sex composition to developmental change using the cognitive, motor, and social subscales from the Child Observation Record (COR). Overall, there were no main effects at the classroom-level for sex composition. However, a cross-level interaction indicated that, while girls’ development was not influenced by classroom sex composition, boys in classrooms with proportionally more boys fared significantly worse in terms of development as assessed by combined score on the COR. More specifically, this interaction was significant when predicting the COR cognitive subscale, but nonsignificant when predicting the COR social and motor subscales. This was true when controlling for the number of students at the classroom-level, as well as child's age and baseline ability (i.e., Time 1 COR) at the child-level. Implications for early childhood education policy are discussed.     

Cognition and classroom quality as predictors of math achievement in the kindergarten year

Using a sample of 171 children, we examined classroom quality as a potential moderator of the link between three distinct but related aspects of cognition (fluid intelligence, crystallized intelligence, and executive functioning) and math achievement across the kindergarten year. Multilevel modeling analyses were conducted to account for nesting of students within classrooms. Results revealed significant aptitude by treatment interactions for fluid and crystallized intelligence, suggesting that classroom practices may affect children differently depending on their abilities. Children with higher levels of fluid intelligence and of crystallized intelligence fared better in higher quality classrooms. Results also provide some support for Cattell's investment hypothesis. Implications of the results are discussed.     

Estimating the Standard Error of the Impact Estimator in Individually Randomized Trials With Clustering

ABSTRACT

In the “individually randomized group treatment” (IRGT) experimental design, individuals are first randomly assigned to a treatment arm or a control arm, but then within each arm, are grouped together (e.g., within classrooms/schools, through shared case managers, in group therapy sessions, through shared doctors, etc.) to receive services. In the common setting where the second-stage grouping is not experimentally controlled, we demonstrate that the standard error estimators for the estimated treatment effects from both random and fixed effects models are biased because parameters required to compute the standard errors are not identifiable from the data. We discuss implications for designing and analyzing IRGTs and provide suggestions for mitigating the bias.     

Hard Cash and Soft Skills: Experimental Evidence on Combining Scholarships and Mentoring in Argentina

Abstract

We evaluate a long-standing program run by one of the largest education foundations in Argentina that offers scholarships and nonacademic mentoring to secondary school students. We randomly assigned 408 grade 6 students within 10 public schools in the Province of Buenos Aires to either receive the program throughout secondary school or not to receive it. After three years, the program improved students’ academic behaviors (e.g., studying before an exam or catching up on missed work), but we find little evidence that these changes translated into broader improvements in students’ academic mindsets (e.g., self-beliefs about performance and efficacy), perseverance (e.g., grit), or learning strategies (e.g., metacognition). The program also improved students’ performance in school during the first year (e.g., grades, attendance, and passing rates), but we do not find similar gains in subsequent years. This may be due to a large share of treatment students being expelled from the program for not meeting its requirements. The program did not improve student learning or personality traits (e.g., conscientiousness). Finally, we find some heterogeneous effects for female students and students from low-income families.     

Teaching for All? Teach For America's Effects Across the Distribution of Student Achievement

ABSTRACT

This article examines the effect of Teach For America (TFA) on the distribution of student achievement in elementary school. It extends previous research by estimating quantile treatment effects (QTE) to examine how student achievement in TFA and non-TFA classrooms differs across the broader distribution of student achievement. It also updates prior distributional work on TFA by correcting for previously unidentified missing data and estimating unconditional rather than conditional QTE. Consistent with previous findings, results reveal a positive impact of TFA teachers across the distribution of math achievement. In reading, however, relative to veteran non-TFA teachers, students at the bottom of the reading distribution score worse in TFA classrooms, and students in the upper half of the distribution perform better.     

Effects of Multilevel Versus Unilevel Metacognitive Training on Mathematical Reasoning

Abstract

The effects of 3 instructional methods on mathematical reasoning were investigated. The methods are (a) cooperative learning embedded within multilevel metacognitive training (MMT), (b) cooperative learning embedded within unilevel metacognitive training (UMT), and (c) learning in the whole class with no metacognitive training. MMT was implemented in mathematics and English classrooms; UMT was used only in mathematics classrooms; and the whole class with no metacognitive training served as a control group. Results indicated that students who were exposed to MMT significantly outperformed their counterparts who were exposed to UMT who, in turn, significantly outperformed the control group. Effects of MMT were observed on students while they solved mathematical problems. Theoretical and practical implications of the study are discussed.     

The Implications of Teacher Selection and the Teacher Effect in Individually Randomized Group Treatment Trials

Abstract

In some experimental evaluations of classroom-level interventions it is not practically feasible to randomly assign teachers to experimental conditions. Given such restrictions, researchers may randomly assign students to experimental conditions and consider the teacher to be a part of the intervention. However, in an individually randomized evaluation of a classroom-level intervention, unless teachers are randomized to experimental conditions, it will not be clear whether differences in outcomes between program and control group students are a result of the core components of the intervention or the teachers (i.e., teacher effects). This article clarifies the correct interpretation of “program impacts” when this study design is used. In addition, using the magnitude of estimated teacher effects from past research, this article demonstrates that, if teachers are not randomly assigned to experimental conditions, it is difficult to establish whether the program works or whether the types of teachers selected to teach in program classrooms are simply more or less effective than their control group counterparts. The significant implications of the correct causal interpretation are discussed, and the limitations of this research design are explored.     

Scholar commentary

ABSTRACT

This commentary reflects upon the articles and purposes of this special issue, in the context of, growing numbers of students of color (SOCs) in K-12 schools who continue to be modally placed in classrooms with White teachers (e.g., Berry and Pour-Khorshid papers of this issue). Few institutional programs of Teacher Education seem able to effectually address this longstanding demographic challenge (e.g., Whitaker, et al. paper). Fewer still, impactfully retain SOCs for careers in K-12 settings (e.g., Morales paper). Nevertheless, second wave teacher identity studies are encouraging nuanced professional learning for White and other, K-12, candidates and teachers that encourages: critical consciousness about race and whiteness, metalogic talk about colorblindness, and closing the gap between rhetoric and genuinely antiracist actions in praxis with SOCs (e.g., Alvarez & Milner; Caldas; McManimon & Casey papers. Demonstrably, second-wave analyses are revealing much about evidentiary approaches for teacher education/learning that emphasize differential capacities for teaching SOCs. Nonetheless, further research is needed on strategies for addressing formidable challenges, such as those of defensiveness and White Fragility (e.g., Blaisedell paper). The papers of this special issue indicate that combining such research with ground-breaking approaches to race-visible education offers a promising path to the knowledge and praxis needed.     

The jigsaw technique and self-efficacy of vocational training students: a practice report

Can teenagers’ self-efficacy be improved in a short time? Previous research has shown the positive effect of cooperative learning methods, including “jigsaw classrooms” (Aronson and Patnoe, 1997), on various outcomes (e.g., the liking of school, self-esteem, and reduction of prejudices). The present practice report investigated the effects of jigsaw technique in boosting the self-efficacy of students enrolled in a vocational curriculum. Over a period of four sessions, 33 male participants studied school materials either in jigsaw groups or in a traditional class (individual work). Their academic self-efficacy in math and French was measured before and after treatment. Results indicated that students’ self-efficacy increased after the four sessions, but only in the jigsaw group. This report provides additional evidence supporting the benefit of jigsaw classrooms based on a different outcome than the one used in previous research—namely, self-efficacy—and among a particular population—namely, vocational trainees. Implications for classroom practice are discussed. In particular, the present practice report demonstrates that implementing the jigsaw approach in classrooms might be an effective tool for enhancing the quality of vocational students’ school experience.     

Peer relationships of students with intellectual disabilities in special needs classrooms – a systematic review

Students with intellectual disabilities (ID) often experience difficulties in their peer relationships at school. Although a broad knowledge base on peer relationships in inclusive classrooms exists, much less is known about peer relationships in special needs classrooms. We conducted a systematic review of the literature on peer relationships in special needs classrooms for students with ID. Studies that provided information on social status, social interactions, friendships, or the classroom networks of students with ID in special needs classrooms were included. Findings from 36 studies suggest that students with ID in special needs classrooms develop and maintain differentiated peer relationships. The characteristics of these peer relationships vary depending on student (e.g., cognitive skills) and classroom factors (e.g., type of special needs classroom). Results are discussed in terms of their relevance for supporting peer relationships between students with ID. Future directions for studying peer relationships in special needs classrooms are proposed.     

Conducting statistical tests with data from clustered school samples

This article discusses issues associated with statistical testing conducted with data from clustered school samples. Empirical researchers often conduct tests of statistical inference on sample data to ascertain the extent to which differences exist within groups in the population. Typically, much school‐related data are collected from students. These data are hierarchical because students are nested within classes within schools. This article studies the influence of this nesting on tests of statistical significance conducted with the student as the unit of analysis. Theory that adjusts F‐test scores for nested data in multi‐group comparisons is presented and applied to a teacher interaction dataset. The article demonstrates the potential impact of data hierarchy on the results of statistical testing if clustering is ignored. Data analysis techniques that recognize the clustering of students in classes are essential, and it is recommended that either multilevel analysis or adjustments to statistical parameters be undertaken in studies involving nested data.     

The Implications of Contamination for Educational Experiments With Two Levels of Nesting

ABSTRACT

Experimental evaluations that involve the educational system usually involve a hierarchical structure (students are nested within classrooms that are nested within schools, etc.). Concerns about contamination, where research subjects receive certain features of an intervention intended for subjects in a different experimental group, have often led researchers to randomize units at a higher level of the educational hierarchy. Existing work on two-level designs suggests that situations where contamination should lead to randomization at a higher level are likely to be rare. This article extends these results to the case of three-level designs. In order to understand the implications of mathematical results, existing information about the size of intracluster correlation coefficients (ICCs) in educational studies with three levels and about the extent of treatment effect heterogeneity across schools is discussed. Better empirical estimates of ICCs, treatment effect heterogeneity, and plausible contamination values are necessary to make full use of the results in this article. However, it seems likely that situations where contamination should lead to randomization at a higher level in three-level designs are rare.     

Fostering complex problem solving for diverse learners: engaging an ethos of intentionality toward equitable access

Complex problem solving is an effective means to engage students in disciplinary content while also furnishing critical non-cognitive and life skills. Despite increased adoption of complex problem-solving methods in K-12 classrooms today (e.g., case-, project-, or problem-based learning), we know little about how to make these approaches accessible to linguistically and culturally diverse (LCD) students. In this paper, we promote a conceptual framework, based on an ethos of intentionality, that supports culturally responsive teaching (CRT). We provide specific questions to guide teachers’ implementation of an ethos of intentionality, through critical reflection and meaningful action, and discuss a framework for culturally relevant practice that operationalizes key central tenets (e.g., high expectations, cultural competence, and critical consciousness). Finally, we include strategies that can help teachers and designers translate the principles of the CRT framework into action with a specific focus on complex problem solving in classrooms.

     

School effects on Chilean children’s achievement growth in language and mathematics: An accelerated growth curve model

This study investigates school effects on primary school students’ language and mathematics achievement trajectories in Chile, a context of particular interest given its large between-school variability in educational outcomes. The sample features an accelerated longitudinal design (3 time points, 4 cohorts) together spanning Grades 3 to 8 (n = 19,704 students in 156 schools). The magnitudes of school effects on students’ growth trajectories were found to be sizeable (generally larger than school effects in Western industrialised countries) and moderately consistent across school subjects. School composition effects on student achievement status were found for both school subjects. However, there was no evidence of composition effects on student achievement growth. The study provides new evidence on the size and nature of school effects in a developing country context based on state-of-the-art methods (i.e., accelerated longitudinal and growth curve models).     

Collaborative Strategic Reading: Findings From Experienced Implementers

Abstract

This study examined the effects and fidelity of collaborative strategic reading (CSR) implemented by experienced CSR teachers (participated in previous study; Vaughn et al., 2011) on the reading comprehension outcomes of students in English/Language Arts (ELA) or Reading classes. Eligible teachers (12 of 17; others reassigned to teach grades/subjects not eligible for inclusion) in middle schools in Texas and Colorado who participated in the previous year in a study examining the effects of CSR on the reading comprehension outcomes of their students participated in a 2nd-year, new cohort of students. Teachers taught multiple sections of ELA or reading; sections were randomly assigned to a treatment or comparison condition, and any extra classes were assigned to the treatment condition. There were 26 CSR and 22 comparison classes. Teachers were asked to implement CSR in their treatment classes only for approximately two 50-min sessions per week for 18 weeks. Examining the role of fidelity revealed that CSR was more prevalent in treatment classes than the comparison classes and that ELA teachers had significantly less treatment spillover than the Reading teachers. Findings indicate that CSR was associated with a greater effect when implemented in ELA classrooms compared to Reading classrooms.     

Extending classrooms through teacher collaboration in Virtual Learning Environments

ABSTRACT

This paper explores arrangements and conditions enabling and constraining teacher collaboration to extend classrooms through Virtual Learning Environments (VLEs). It draws implications from a participatory action research project in three small, rural Finnish schools funded by the European Social Fund (2015–2017). Data were mainly collected through discussions and interviews with five teachers. The theory of practice architectures serves as a theoretical and analytical framework. The collaboration challenged and enabled cultural-discursive, material-economic, and social-political arrangements on three levels. On a classroom level, the VLE enabled new teaching practices (e.g., student collaboration, formative assessment). However, a lack of digital competence among teachers and students hindered the cross-classroom collaboration. On a school level, the school leaders enabled or constrained the arrangements depending on the provided support or lack thereof (e.g., schedules, faculty involvement). On a regional level, the infrastructure (e.g., transportation) for collaboration would require arrangements developed in dialogue with the schools (e.g., digital systems). The paper concludes that changing teaching practices in relation to VLEs requires slowly evolving previous and new architectures simultaneously on all three levels.