A NOTE ON THE POWER OF STATISTICAL TESTS IN THE JOURNAL OF EDUCATIONAL MEASUREMENT

The power of statistical tests recently appearing in the JEM was determined using the power calculation guidelines proposed by Cohen (1969). All the articles containing tests of significance were surveyed. The results indicated that power was generally below .50 for small effect sizes and above .50 for medium and large effect sizes. A suggestion for reporting statistical results to include power of the tests was made.     

Solution-Focused Brief Therapy: Impacts on Academic and Emotional Difficulties

ABSTRACT

This randomized control trial study evaluated the effectiveness of the solution-focused approach in addressing academic, motivational, and socioemotional needs of 14 children with reading difficulties. The intervention group received five 40-min solution-focused sessions. The control group received academic homework support. Results showed advantages for the intervention condition in 26 out of 38 measures. The mean eta-squared effect size for intervention was .20 (very) large. For the control group, there were only 10 effects favoring it and the mean was .09, a medium sized effect, both significantly greater than 0 (p < .01). Comparisons of the solution-focused brief therapy (SFBT) effect sizes to the mean of the control showed it was significantly larger (p < .001), confirming that SFBT was an efficacious intervention in this sample.     

Statistical Power to Detect the Correct Number of Classes in Latent Profile Analysis

Little research has examined factors influencing statistical power to detect the correct number of latent classes using latent profile analysis (LPA). This simulation study examined power related to interclass distance between latent classes given true number of classes, sample size, and number of indicators. Seven model selection methods were evaluated. None had adequate power to select the correct number of classes with a small (Cohen's d = .2) or medium (d = .5) degree of separation. With a very large degree of separation (d = 1.5), the Lo–Mendell–Rubin test (LMR), adjusted LMR, bootstrap likelihood ratio test, Bayesian Information Criterion (BIC), and sample-size-adjusted BIC were good at selecting the correct number of classes. However, with a large degree of separation (d = .8), power depended on number of indicators and sample size. Akaike's Information Criterion and entropy poorly selected the correct number of classes, regardless of degree of separation, number of indicators, or sample size.     

A meta‐analysis of the relationship between student attitudes towards reading and achievement in reading

A meta‐analysis of the relationship between attitudes in reading and achievement in reading was conducted to provide a statistical summary to the observed variability in the magnitude of previously reported effect sizes. A total of 32 studies, with a total sample size of 224,615 were used, and included a total of 118 effect sizes. A multi‐level approach was used in meta‐analysis to determine if variance in the magnitude of effect sizes could be partitioned to study (level 1) and moderator (level 2) levels by using a mixed model approach. Results from the meta‐analysis indicated that the mean strength of the relationship between reading attitudes and achievement is moderate (Z_r=.32), while stronger for students in elementary school (Z_r=.44) when compared with middle school students (Z_r=.24). Findings related to selected moderator variables are discussed, with suggestions for future research.     

A note on attrition among college presidents, 1971–1974

Returning to the same stratified random sample of American colleges and universities studied by Cohen and March (1974) during the 1969–1970 academic year, the authors explore the extent to which theoretical estimates of attrition rates presented by Cohen and March predict recent presidential departures within their sample. They find that in the past four years (1971–1974) there has been little change in the attrition pattern among college presidents in this national sample. If there has been any change it is very small and in the direction of slightly longer tenure among presidents of large universities.     

Revisiting the effects of project-based learning on students’ academic achievement: A meta-analysis investigating moderators

Project-based learning is generally considered an alternative to traditional, teacher-led instruction. However, there is a noticeable lack of meta-analyses with regard to determining its overall effects on students' academic achievement, and what study features may moderate the impacts of project-based learning. This study thus performed a meta-analysis to synthesize existing research that compared the effects of project-based learning and those of traditional instruction on student academic achievement. Forty-six effect sizes (comparisons) extracted from 30 eligible journal articles published from 1998 to 2017 were analyzed, representing 12,585 students from 189 schools in nine countries. The results showed that the overall mean weighted effect size (d₊) was 0.71, indicating that project-based learning has a medium to large positive effect on students' academic achievement compared with traditional instruction. In addition, the mean effect size was affected by subject area, school location, hours of instruction, and information technology support, but not by educational stage and small group size.     

A study of the power associated with testing factor mean differences under violations of factorial invariance

We examine the power associated with the test of factor mean differences when the assumption of factorial invariance is violated. Utilizing the Wald test for obtaining power, issues of model size, sample size, and total versus partial noninvariance are considered along with variation of actual factor mean differences. Results of a population study show that power is profoundly affected by true factor mean differences but is relatively unaffected by the degree of factor loading noninvariance. Inequality of sample size has a profound effect on power probabilities with power decreasing as sample sizes become increasingly disparate. Sample size variations operate such that power is uniformly lower when the group with the smaller generalized variance is associated with the smaller sample size. An increase in the number of variables yields uniformly larger power probabilities. No substantial differences are found between total and partial noninvariance. Results are related to work in the area of robustness of Hotelling's T ² statistic and discussed in terms of asymptotic covariability of factor means and factor loadings. Implications for practice are considered.     

Effects of cognitive learning strategies for Korean learners: A meta-analysis

The purpose of this study was to synthesize the cognitive learning strategy intervention studies conducted in Korea between 1990 and 2006, using meta-analysis. By means of pre-established systematic criteria, 50 articles were selected and 97 effect sizes were calculated. Effect size was calculated using ‘the Cohen’s d’ (Cooper &; Hedges, 1994). The research questions of the present study were as follows: (a) Are cognitive learning strategies generally effective? (b) What type of cognitive learning strategy is most effective? (c) Are effect sizes of different types of cognitive learning strategies different according to the applied domains, grade levels, and achievement levels? The results of the study indicate that, first of all, the overall cognitive learning strategies (97 ESs) yielded a large effect size (ESsm=.96), which was not homogenous (Q=55.19,p <.05). Thus, in each subcategory of learners’ characteristics and applied domains, we calculated effect sizes and conducted the test of homogeneity separately. Except for grade level, the effect sizes were generally homogenous in each subcategory. The findings revealed that cognitive strategies had large effect sizes (.82–1.69). For average achieving students as well as underachieving students (Learning Disabilities), cognitive learning strategies were very effective (.82–1.42). The effect of cognitive learning strategies was very large in terms of students in all grades (1.02–1.34), except for middle school students (.70). Lastly, the implications for the application of different cognitive learning strategies were discussed.     

Brief Research Report: A User Friendly Effect Size—When Normality Matters

There is a need for effect sizes that are readily interpretable by a broad audience. One index that might fill this need is π, which represents the proportion of scores in one group that exceed the mean of another group. The robustness of estimates of π to violations of normality had not been explored. Using simulated data, three estimates of π (π? _direct, r, and r_robust) were studied under varying conditions of sample size, distribution shape, and group mean difference. This study demonstrated that r and r_robust were biased estimates of π when data were nonnormal. We recommend that neither be used in estimating π unless data are normally distributed.     

Parent-Child Relationships from the Child’S Point of View

The authors examined the distributional properties of 3 improvement-over-chance, I, effect sizes each derived from linear and quadratic predictive discriminant analysis and from logistic regression analysis for the 2-group univariate classification. These 3 classification methods (3 levels) were studied under varying levels of data conditions, including population separation (3 levels), variance pattern (3 levels), total sample size (3 levels), and prior probabilities (5 levels). The results indicated that the decision of which effect size to choose is primarily determined by the variance pattern and prior probabilities. Some of the I indices performed well for some small sample cases and quadratic predictive discriminant analysis I tended to work well with extreme variance heterogeneity and differing prior probabilities.     

Science Achievement and Students’ Self‐confidence and Interest in Science: A Taiwanese representative sample study

The interrelationship between senior high school students’ science achievement (SA) and their self‐confidence and interest in science (SCIS) was explored with a representative sample of approximately 1,044 11th‐grade students from 30 classes attending four high schools throughout Taiwan. Statistical analyses indicated that a statistically significant correlation existed between students’ SA and their SCIS with a moderate effect size; the correlation is even higher with almost large effect sizes for a subsample of higher‐SCIS and lower‐SCIS students. Results of t‐test analysis also revealed that there were significant mean differences in students’ SA and their knowledge (including physics, chemistry, biology, and earth sciences subscales) and reasoning skill subtests scores between higher‐SCIS and lower‐SCIS students, with generally large effect sizes. Stepwise regression analyses on higher‐SCIS and lower‐SCIS students also suggested that both students’ SCIS subscales significantly explain the variance of their SA, knowledge, and reasoning ability with large effect sizes.     

Statistical Power in Evaluations That Investigate Effects on Multiple Outcomes: A Guide for Researchers

Researchers are often interested in testing the effectiveness of an intervention on multiple outcomes, for multiple subgroups, at multiple points in time, or across multiple treatment groups. The resulting multiplicity of statistical hypothesis tests can lead to spurious findings of effects. Multiple testing procedures (MTPs) are statistical procedures that counteract this problem by adjusting p values for effect estimates upward. Although MTPs are increasingly used in impact evaluations in education and other areas, an important consequence of their use is a change in statistical power that can be substantial. Unfortunately, researchers frequently ignore the power implications of MTPs when designing studies. Consequently, in some cases, sample sizes may be too small, and studies may be underpowered to detect effects as small as a desired size. In other cases, sample sizes may be larger than needed, or studies may be powered to detect smaller effects than anticipated. This paper presents methods for estimating statistical power for multiple definitions of statistical power and presents empirical findings on how power is affected by the use of MTPs.     

Accuracy of Estimates and Statistical Power for Testing Meditation in Latent Growth Curve Modeling

The latent growth curve modeling (LGCM) approach has been increasingly utilized to investigate longitudinal mediation. However, little is known about the accuracy of the estimates and statistical power when mediation is evaluated in the LGCM framework. A simulation study was conducted to address these issues under various conditions including sample size, effect size of mediated effect, number of measurement occasions, and R ₂ of measured variables. In general, the results showed that relatively large samples were needed to accurately estimate the mediated effects and to have adequate statistical power, when testing mediation in the LGCM framework. Guidelines for designing studies to examine longitudinal mediation and ways to improve the accuracy of the estimates and statistical power were discussed.     

The Place of the Fable in the Character Training of Children

Type I error rate and power for the t test, Wilcoxon-Mann-Whitney (U) test, van der Waerden Normal Scores (NS) test, and Welch-Aspin-Satterthwaite (W) test were compared for two independent random samples drawn from nonnormal distributions. Data with varying degrees of skewness (S) and kurtosis (K) were generated using Fleishman's (1978) power function. Five sample size combinations were used with both equal and unequal variances. For nonnormal data with equal variances, the power of the U test exceeded the power of the t test regardless of sample size. When the sample sizes were equal but the variances were unequal, the t test proved to be the most powerful test. When variances and sample sizes were unequal, the W test became the test of choice because it was the only test that maintained its nominal Type I error rate.     

A Comparison of the English Vocabulary Used by Children of Non-American Ancestry in Hawaii before they Reach the Age of Seven Years with that of Kindergarten Children in Continental United States

The authors investigated 2 issues concerning the power of latent growth modeling (LGM) in detecting linear growth: the effect of the number of repeated measurements on LGM's power in detecting linear growth and the comparison between LGM and some other approaches in terms of power for detecting linear growth. A Monte Carlo simulation design was used, with 3 crossed factors (growth magnitude, number of repeated measurements, and sample size) and 1,000 replications within each cell condition. The major findings were as follows: For 3 repeated measurements, a substantial proportion of samples failed to converge in structural equation modeling; the number of repeated measurements did not show any effect on the statistical power of LGM in detecting linear growth; and the LGM approach outperformed both the dependent t test and repeated-measures analysis of variance (ANOVA) in terms of statistical power for detecting growth under the conditions of small growth magnitude and small to moderate sample size conditions. The multivariate repeated-measures ANOVA approach consistently underperformed the other tests.     

Examining the generalizability of administrative personnel allocation decisions

This study examined the effect of district and school size on principal teacher allocation decisions. The study tested the invariance of a personnel allocation decision making model for elementary school principals from three categories of school and district size. The sample consisted of elementary school principals from small, medium, and large schools and districts. The results confirmed the fit of the model across schools of all sizes and across small and medium size districts. For large school districts the proposed decision-making model did not fit the data. This result implies that district size has an effect on the personnel allocation decisions made by elementary school principals.     

Effectiveness of Reading-Strategy Interventions in Whole Classrooms: a Meta-Analysis

Research has demonstrated that in controlled experiments in which small groups are being tutored by researchers, reading-strategy instruction is highly effective in fostering reading comprehension (Palincsar & Brown, Cognition and Instruction, 1(2), 117–175, 1984). It is unclear, however, whether reading-strategy interventions are equally effective in whole-classroom situations in which the teacher is the sole instructor for the whole class. This meta-analysis focuses on the effects of reading-strategy interventions in whole-classroom settings. Results of studies on the effectiveness of reading-strategy interventions in whole-classroom settings were summarized (N_studies?=?52, K?=?125) to determine the overall effects on reading comprehension and strategic ability. In addition, moderator effects of intervention, study, and student characteristics were explored. The analysis demonstrated a very small effect on reading comprehension (Cohen’s d?=?.186) for standardized tests and a small effect (Cohen’s d?=?.431) on researcher-developed reading comprehension tests. A medium overall effect was found for strategic ability (Cohen’s d?=?.786). Intervention effects tended to be lower for studies that did not control for the hierarchical structure of the data (i.e. multilevel analyses).For interventions in which “setting reading goals” was part of the reading-strategy package, effects tended to be larger. In addition, effects were larger for interventions in which the trainer was the researcher as opposed to teachers and effect sizes tended to be larger for studies conducted in grades 6–8. Implications of these findings for future research and educational practice are discussed.     

Exploring the Value Added of a Guided,Silent Reading Intervention: Effects on Struggling Third-Grade Readers’ Achievement

ABSTRACT

The authors’ purpose was to explore the effects of a supplementary, guided, silent reading intervention with 80 struggling third-grade readers who were retained at grade level as a result of poor performance on the reading portion of a criterion referenced state assessment. The students were distributed in 11 elementary schools in a large, urban school district in the state of Florida. A matched, quasi-experimental design was constructed using propensity scores for this study. Students in the guided, silent reading intervention, Reading Plus, evidenced higher, statistically significant mean scores on the Florida Comprehensive Assessment Test criterion assessment measure of reading at posttest. The effect size, favoring the guided, silent reading intervention group was large, 1 full standard deviation, when comparing the 2 comparison groups’ mean posttest scores. As such, the results indicate a large advantage for providing struggling third-grade readers guided silent reading fluency practice in a computer-based practice environment. No significant difference was found between the treatment and control group on the Stanford Achievement Test–10 (SAT-10) posttest scores, although posttest scores for the treatment group trended higher than the control. After conducting a power analysis, it was determined that the sample size (n = 80) was too small to provide sufficient statistical power to detect a difference in third-grade students’ SAT-10 scores.     

Efficiency of digital and non-digital educational games: A comparative meta-analysis and a meta-thematic analysis

Abstract

The present article attempts to reinterpret the findings of most recent studies investigating effect of using games for teaching purposes. A methodological approach combining a meta-analysis of quantitative data with qualitative ones was adopted in order to present the broadest picture of the current research on educational use of games. To this end, we conducted a meta-analysis of 180 effect size comparisons out of 154 empirical studies on the effect of both digital and non-digital games on academic achievement conducted during the period from 2004 to 2019 in order to determine the overall effect size of using games for teaching various subjects. The overall sample size of the studies included a total number of 12800 participants. Some moderator analyses were also carried out to determine the exact efficiency of educational games in terms of student levels, durations of implementation of game activities, school subjects in which games were used, class sizes, kinds of games and achievement tests used. The findings suggest that educational games have a positive effect on academic achievement and this effect is at a medium level (g?=?0.695). The highest effect sizes were observed in foreign language courses (g?=?0.87), small (less than 50) class sizes (g?=?0.87), and in non-digital games (g?=?0.90). Moreover, we conducted a meta-thematic analysis based on document analysis of qualitative studies in order to further consolidate the findings of the meta-analysis. The meta-thematic dimension of our study reveals cognitive contributions as well as drawbacks of game-based teaching, and provides suggestions for conducting educational games in a better way.     

Revisiting the Model Size Effect in Structural Equation Modeling

Fitting a large structural equation modeling (SEM) model with moderate to small sample sizes results in an inflated Type I error rate for the likelihood ratio test statistic under the chi-square reference distribution, known as the model size effect. In this article, we show that the number of observed variables (p) and the number of free parameters (q) have unique effects on the Type I error rate of the likelihood ratio test statistic. In addition, the effects of p and q cannot be fully explained using degrees of freedom (df). We also evaluated the performance of 4 correctional methods for the model size effect, including Bartlett’s (1950), Swain’s (1975), and Yuan’s (2005) corrected statistics, and Yuan, Tian, and Yanagihara’s (2015) empirically corrected statistic. We found that Yuan et al.’s (2015) empirically corrected statistic generally yields the best performance in controlling the Type I error rate when fitting large SEM models.