Small Sample Properties of Bayesian Multivariate Autoregressive Time Series Models

The aim of this study was to compare the small sample (N = 1, 3, 5, 10, 15) performance of a Bayesian multivariate vector autoregressive (BVAR-SEM) time series model relative to frequentist power and parameter estimation bias. A multivariate autoregressive model was developed based on correlated autoregressive time series vectors of varying lengths (T = 25, 50, 75, 100, 125) using Statistical Analysis System (SAS) version 9.2. Autoregressive components for the 5 series vectors included coefficients of .80, .70, .65, .50 and .40. Error variance components included values of .20, .20, .10, .15, and .15, with cross-lagged coefficients of .10, .10, .15, .10, and .10. A Monte Carlo study revealed that in comparison to frequentist methods, the Bayesian approach provided increased sensitivity for hypothesis testing and detecting Type I error.     

Factors Affecting the Item Parameter Estimation and Classification Accuracy of the DINA Model

To better understand the statistical properties of the deterministic inputs, noisy “and” gate cognitive diagnosis (DINA) model, the impact of several factors on the quality of the item parameter estimates and classification accuracy was investigated. Results of the simulation study indicate that the fully Bayes approach is most accurate when the prior distribution matches the latent class structure. However, when the latent classes are of indefinite structure, the empirical Bayes method in conjunction with an unstructured prior distribution provides much better estimates and classification accuracy. Moreover, using empirical Bayes with an unstructured prior does not lead to extremely poor results as other prior-estimation method combinations do. The simulation results also show that increasing the sample size reduces the variability, and to some extent the bias, of item parameter estimates, whereas lower level of guessing and slip parameter is associated with higher quality item parameter estimation and classification accuracy.     

Evaluating Intercept-Slope Interactions in Latent Growth Modeling

The effects of misspecifying intercept-covariate interactions in a 4 time-point latent growth model were the focus of this investigation. The investigation was motivated by school growth studies in which students' entry-level skills may affect their rate of growth. We studied the latent interaction of intercept and a covariate in predicting growth with respect to 3 factors: sample size (100, 200, and 500), 4 levels of magnitude of interaction effect, and 3 correlation values between intercept and covariate (.3, .5, and .7). Correctly specified models were examined to determine power and Type I error rates, and misspecified models were examined to evaluate the effects on power, parameter estimation, bias, and fit indexes. Results showed that, under correctly specified models, power increased as expected with increasing sample size, larger magnitude of interaction, and larger intercept-covariate correlation. Under misspecification, omitting a non-null interaction results in significant change in the estimation of the direct effects of both covariate and intercept in both magnitude and direction, with results dependent on sign of parameter values for main effects and interaction. Including a spurious interaction does not affect estimation of direct effects of intercept and covariate but does increase standard errors. The primary problem in ignoring a non-null interaction lies in misinterpretation of the model, as interactions yield important insights into the nature of the processes being studied.     

Relative Performance of Categorical Diagonally Weighted Least Squares and Robust Maximum Likelihood Estimation

Robust maximum likelihood (ML) and categorical diagonally weighted least squares (cat-DWLS) estimation have both been proposed for use with categorized and nonnormally distributed data. This study compares results from the 2 methods in terms of parameter estimate and standard error bias, power, and Type I error control, with unadjusted ML and WLS estimation methods included for purposes of comparison. Conditions manipulated include model misspecification, level of asymmetry, level and categorization, sample size, and type and size of the model. Results indicate that cat-DWLS estimation method results in the least parameter estimate and standard error bias under the majority of conditions studied. Cat-DWLS parameter estimates and standard errors were generally the least affected by model misspecification of the estimation methods studied. Robust ML also performed well, yielding relatively unbiased parameter estimates and standard errors. However, both cat-DWLS and robust ML resulted in low power under conditions of high data asymmetry, small sample sizes, and mild model misspecification. For more optimal conditions, power for these estimators was adequate.     

Statistical Power to Detect the Correct Number of Classes in Latent Profile Analysis

Little research has examined factors influencing statistical power to detect the correct number of latent classes using latent profile analysis (LPA). This simulation study examined power related to interclass distance between latent classes given true number of classes, sample size, and number of indicators. Seven model selection methods were evaluated. None had adequate power to select the correct number of classes with a small (Cohen's d = .2) or medium (d = .5) degree of separation. With a very large degree of separation (d = 1.5), the Lo–Mendell–Rubin test (LMR), adjusted LMR, bootstrap likelihood ratio test, Bayesian Information Criterion (BIC), and sample-size-adjusted BIC were good at selecting the correct number of classes. However, with a large degree of separation (d = .8), power depended on number of indicators and sample size. Akaike's Information Criterion and entropy poorly selected the correct number of classes, regardless of degree of separation, number of indicators, or sample size.     

Pairwise deletion for missing data in structural equation models: Nonpositive definite matrices,parameter estimates,goodness of fit,and adjusted sample sizes

The use of sample covariance matrices constructed with pairwise deletion for data missing completely at random (S_PW) is addressed in a simulation study based on 3 sample sizes (n = 200, 500, 1,000) and 5 levels of missing data (%miss = 0, 1, 10, 25, and 50). Parameter estimates were unbiased, parameter variability was largely explicable in terms of the number of nonmissing cases, and no sample covariance matrices were nonpositive definite except when %miss was 50 and the sample size was 200. However, nominal χ² test statistics (and, thus, fit indices based on χ²s) were substantially biased by %miss and its interaction with N. Corrected χ²s based on the minimum, mean, and maximum number of nonmissing cases per measured variables and cases per covariance term (NPC) reduced but did not eliminate the bias. Empirically derived power functions did substantially better but may not generalize to other situations. Whereas the minimum NPC (the default in the SPSS version of LISREL) is probably better than most simple alternatives in many applications, the problem of how to assess fit for models fit to S_PWS has no simple solution; caution is recommended, and there is need for further research with more suitable methods for this problem.     

A Comparison of Diagonal Weighted Least Squares Robust Estimation Techniques for Ordinal Data

This study compared diagonal weighted least squares robust estimation techniques available in 2 popular statistical programs: diagonal weighted least squares (DWLS; LISREL version 8.80) and weighted least squares–mean (WLSM) and weighted least squares—mean and variance adjusted (WLSMV; Mplus version 6.11). A 20-item confirmatory factor analysis was estimated using item-level ordered categorical data. Three different nonnormality conditions were applied to 2- to 7-category data with sample sizes of 200, 400, and 800. Convergence problems were seen with nonnormal data when DWLS was used with few categories. Both DWLS and WLSMV produced accurate parameter estimates; however, bias in standard errors of parameter estimates was extreme for select conditions when nonnormal data were present. The robust estimators generally reported acceptable model–data fit, unless few categories were used with nonnormal data at smaller sample sizes; WLSMV yielded better fit than WLSM for most indices.     

Effects of Missing Data Methods in Structural Equation Modeling With Nonnormal Longitudinal Data

The purpose of this study is to investigate the effects of missing data techniques in longitudinal studies under diverse conditions. A Monte Carlo simulation examined the performance of 3 missing data methods in latent growth modeling: listwise deletion (LD), maximum likelihood estimation using the expectation and maximization algorithm with a nonnormality correction (robust ML), and the pairwise asymptotically distribution-free method (pairwise ADF). The effects of 3 independent variables (sample size, missing data mechanism, and distribution shape) were investigated on convergence rate, parameter and standard error estimation, and model fit. The results favored robust ML over LD and pairwise ADF in almost all respects. The exceptions included convergence rates under the most severe nonnormality in the missing not at random (MNAR) condition and recovery of standard error estimates across sample sizes. The results also indicate that nonnormality, small sample size, MNAR, and multicollinearity might adversely affect convergence rate and the validity of statistical inferences concerning parameter estimates and model fit statistics.     

The Impact of Omitting Random Interaction Effects in Cross-Classified Random Effect Modeling

The present study examines bias in parameter estimates and standard error in cross-classified random effect modeling (CCREM) caused by omitting the random interaction effects of the cross-classified factors, focusing on the effect of a sample size within cells and ratio of a small cell. A Monte Carlo simulation study was conducted to compare the correctly specified and the misspecified CCREM. While there was negligible bias in fixed effects, substantial biases were found in the random effects of the misspecified model depending on the number of samples within a cell and the proportion of small cells. However, in the case of the correctly specified model, no bias occurred. The present study suggests considering the random interaction effects when conducting CCREM to avoid overestimation of variance components and to calculate an accurate value of estimation. The implications of this study are to illuminate the conditions of cross-classification ratio and to provide a meaningful reference for applied researchers using CCREM.     

Accuracy of Estimates and Statistical Power for Testing Meditation in Latent Growth Curve Modeling

The latent growth curve modeling (LGCM) approach has been increasingly utilized to investigate longitudinal mediation. However, little is known about the accuracy of the estimates and statistical power when mediation is evaluated in the LGCM framework. A simulation study was conducted to address these issues under various conditions including sample size, effect size of mediated effect, number of measurement occasions, and R ₂ of measured variables. In general, the results showed that relatively large samples were needed to accurately estimate the mediated effects and to have adequate statistical power, when testing mediation in the LGCM framework. Guidelines for designing studies to examine longitudinal mediation and ways to improve the accuracy of the estimates and statistical power were discussed.     

Graphical Extension of Sample Size Planning With AIPE on RMSEA Using R

Kelley and Lai (2011) recently proposed the use of accuracy in parameter estimation (AIPE) for sample size planning in structural equation modeling. The sample size that reaches the desired width for the confidence interval of root mean square error of approximation (RMSEA) is suggested. This study proposes a graphical extension with the AIPE approach, abbreviated as GAIPE, on RMSEA to facilitate sample size planning in structural equation modeling. GAIPE simultaneously displays the expected width of a confidence interval of RMSEA, the necessary sample size to reach the desired width, and the RMSEA values covered in the confidence interval. Power analysis for hypothesis tests related to RMSEA can also be integrated into the GAIPE framework to allow for a concurrent consideration of accuracy in estimation and statistical power to plan sample sizes. A package written in R has been developed to implement GAIPE. Examples and instructions for using the GAIPE package are presented to help readers make use of this flexible framework. With the capacity of incorporating information on accuracy in RMSEA estimation, values of RMSEA, and power for hypothesis testing on RMSEA in a single graphical representation, the GAIPE extension offers an informative and practical approach for sample size planning in structural equation modeling.     

Formulation of the DETECT Population Parameter and Evaluation of DETECT Estimator Bias

The development of the DETECT procedure marked an important advancement in nonparametric dimensionality analysis. DETECT is the first nonparametric technique to estimate the number of dimensions in a data set, estimate an effect size for multidimensionality, and identify which dimension is predominantly measured by each item. The efficacy of DETECT critically depends on accurate, minimally biased estimation of the expected conditional covariances of all the item pairs. However, the amount of bias in the DETECT estimator has been studied only in a few simulated unidimensional data sets. This is because the value of the DETECT population parameter is known to be zero for this case and has been unknown for cases when multidimensionality is present. In this article, integral formulas for the DETECT population parameter are derived for the most commonly used parametric multidimensional item response theory model, the Reckase and McKinley model. These formulas are then used to evaluate the bias in DETECT by positing a multidimensional model, simulating data from the model using a very large sample size (to eliminate random error), calculating the large-sample DETECT statistic, and finally calculating the DETECT population parameter to compare with the large-sample statistic. A wide variety of two- and three-dimensional models, including both simple structure and approximate simple structure, were investigated. The results indicated that DETECT does exhibit statistical bias in the large-sample estimation of the item-pair conditional covariances; but, for the simulated tests that had 20 or more items, the bias was small enough to result in the large-sample DETECT almost always correctly partitioning the items and the DETECT effect size estimator exhibiting negligible bias.     

A comparison of community college responders and nonresponders to the VEDS student follow-up survey

This study compared a random sample of nonresponders (N=201) to the Vocational Education Data System's (VEDS) student follow-up survey to responders (N=1441) for 15 Massachusetts community colleges. The study was conducted because positive follow-up results for community colleges were challenged by officials due to low response rates (35.7 percent) and possible response bias.No significant differences were found between responders and nonresponders on 6 VEDS demographic and 5 VEDS dependent variables. Significant differences were found between responders and nonresponders on special needs status, degree of job relatedness, hourly wage of all graduates employed, and those graduates employed full-time in their area of training. All of the significant differences were in favor of the nonresponders. Therefore, if biased, the responder data are biased in the direction of underestimation of employment levels, average hourly wage, and other data rather than overestimation. A model is outlined to explain these results that distinguishes between the nominal and the effective population surveyed, and the results of this study are discussed in terms of this model, VEDS, and other surveys.     

Analysis of self-regulated learning processing using statistical models for count data

Researchers often use measures of the frequency of self-regulated learning (SRL; Zimmerman, American Educational Research Journal, 45(1), 166–183, 2000) processing as a predictor of learning gains. These frequency data, which are really counts of SRL processing events, are often non-normally distributed, and the accurate analysis of these data requires the use of specialized statistical models. In this study, we demonstrate how to implement and interpret count statistical models in path and latent profile analyses to investigate the role of SRL as a mediator of the relation between pretest and posttest conceptual understanding. Our findings from a sample of 170 college students using a computer to learn about the circulatory system show that SRL does mediate the aforementioned relation, and that count models are a more accurate representation of SRL processing data than standard analysis models based on ordinary least squares estimation. The results of our path analyses revealed which specific SRL processes were related to learning, above and beyond the effect of prior knowledge. Our latent profile analysis revealed two groups of participants, indicative of Brophy’s (2004) model of schematic and aschematic learners. We conclude with implications and future directions for basic and applied SRL research.     

Estimating the Coefficient of Cross-Validity in Multiple Regression: A Comparison of Analytical and Empirical Methods

In predictive applications of multiple regression, interest centers on the estimation of the population coefficient of cross-validation rather than the population multiple correlation. The accuracy of 3 analytical formulas for shrinkage estimation (Ezekiel, Browne, & Darlington) and 4 empirical techniques (simple cross-validation, multicross-validation, jackknife, and bootstrap) were investigated in a Monte Carlo study. Random samples of size 20 to 200 were drawn from a pseudopopulation of actual field data. Regression models were investigated with population coefficients of determination ranging from .04 to .50 and with numbers of regressors ranging from 2 to 10. For all techniques except the Browne formula and multicross-validation, substantial statistical bias was evident when the shrunken R ² values were used to estimate the coefficient of cross-validation. In addition, none of the techniques examined provided unbiased estimates with sample sizes smaller than 100, regardless of the number of regressors.     

Estimation Methods for One‐Parameter Testlet Models

This study demonstrated the equivalence between the Rasch testlet model and the three‐level one‐parameter testlet model and explored the Markov Chain Monte Carlo (MCMC) method for model parameter estimation in WINBUGS. The estimation accuracy from the MCMC method was compared with those from the marginalized maximum likelihood estimation (MMLE) with the expectation‐maximization algorithm in ConQuest and the sixth‐order Laplace approximation estimation in HLM6. The results indicated that the estimation methods had significant effects on the bias of the testlet variance and ability variance estimation, the random error in the ability parameter estimation, and the bias in the item difficulty parameter estimation. The Laplace method best recovered the testlet variance while the MMLE best recovered the ability variance. The Laplace method resulted in the smallest random error in the ability parameter estimation while the MCMC method produced the smallest bias in item parameter estimates. Analyses of three real tests generally supported the findings from the simulation and indicated that the estimates for item difficulty and ability parameters were highly correlated across estimation methods.     

Exploring the Value Added of a Guided,Silent Reading Intervention: Effects on Struggling Third-Grade Readers’ Achievement

ABSTRACT

The authors’ purpose was to explore the effects of a supplementary, guided, silent reading intervention with 80 struggling third-grade readers who were retained at grade level as a result of poor performance on the reading portion of a criterion referenced state assessment. The students were distributed in 11 elementary schools in a large, urban school district in the state of Florida. A matched, quasi-experimental design was constructed using propensity scores for this study. Students in the guided, silent reading intervention, Reading Plus, evidenced higher, statistically significant mean scores on the Florida Comprehensive Assessment Test criterion assessment measure of reading at posttest. The effect size, favoring the guided, silent reading intervention group was large, 1 full standard deviation, when comparing the 2 comparison groups’ mean posttest scores. As such, the results indicate a large advantage for providing struggling third-grade readers guided silent reading fluency practice in a computer-based practice environment. No significant difference was found between the treatment and control group on the Stanford Achievement Test–10 (SAT-10) posttest scores, although posttest scores for the treatment group trended higher than the control. After conducting a power analysis, it was determined that the sample size (n = 80) was too small to provide sufficient statistical power to detect a difference in third-grade students’ SAT-10 scores.     

Optimal Sample Allocation in Group-Randomized Mediation Studies with a Group-Level Mediator

We derive sample-allocation formulas that maximize the power of several mediation tests in two-level–group-randomized studies under a linear cost structure and fixed budget. The results suggest that the optimal individual sample size is typically smaller than that associated with the detection of a main effect and is frequently less than 10 under parameter values commonly seen in the literature. However, the optimal sample allocation can be heavily influenced by the group-to-individual cost ratio, the ratio of the treatment-mediator to mediator-outcome path coefficients, and the outcome variance structure. We illustrate these findings with a hypothetical group-randomized trial examining a school-discipline reform policy. To encourage utilization of the sample allocation formulas we implement them in the R package PowerUpR and powerupr Shiny application.     

Power of Latent Growth Modeling for Detecting Group Differences in Linear Growth Trajectory Parameters

This simulation study focused on the power for detecting group differences in linear growth trajectory parameters within the framework of structural equation modeling (SEM) and compared the latent growth modeling (LGM) approach to the more traditional repeated-measures analysis of variance (ANOVA) approach. Several patterns of group differences in linear growth trajectories were considered. SEM growth modeling consistently showed higher statistical power for detecting group differences in the linear growth slope than repeated-measures ANOVA. For small group differences in the growth trajectories, large sample size (e.g., N > 500) would be required for adequate statistical power. For medium or large group differences, moderate or small sample size would be sufficient for adequate power. Some future research directions are discussed.