False confidence in intervals: Inaccuracies in reporting confidence intervals

Confidence intervals often are recommended as a means of communicating the extent to which individual test scores may be influenced by measurement error. However, test manuals and assessment texts vary widely in their recommendations about how confidence intervals should be constructed, and several contain misinterpretations of classical test theory. The most widely used procedure for constructing confidence intervals misrepresents the likely distribution of true scores, and confidence intervals constructed with it will be inaccurate, especially when extreme scores are involved. The various procedures for constructing confidence intervals that have been suggested in measurement texts are examined in relation to their approximation to the most accurate procedure that uses the estimated true score as the center of the confidence interval and the standard error of estimate to determine the width. In addition, the problems of applying these procedures to norm-referenced scores are discussed—an issue that has been largely ignored in the assessment literature and that leads to further misinterpretations of confidence intervals.     

Significance,Effect Sizes,Stepwise Methods,and Other Issues: Strong Arguments Move the Field

The author asserts that editors should publicly declare their expectations and expose the rationales for editorial policies to public scrutiny. He argues that editorial policies ought to require effect size reporting, as those at 17 journals now do. He also argues (a) that score reliabilities should be reported; (b) that stepwise methods should not be used; (c) that structure coefficients should be interpreted; and (d) that if used wisely, confidence intervals differ from hypothesis tests in important ways. The use of noncentral t and F distributions to create confidence intervals about effect sizes also is appealing.     

Corrected confidence intervals for quotients on the WISC-R,by level of quotient

New confidence intervals for the Wechsler Intelligence Scale for Children-Revised (WISC-R) are provided to improve accuracy over existing tables (which center the confidence interval on the child's actual quotient and construct intervals upon the standard error of measurement). These new tables follow Schulte and Borich (1988) in centering confidence intervals on the estimated true IQ and constructing intervals with either the standard error of estimate or the standard error of prediction. The tables are more conservative and reflect more accurate statistical formulas for the construction of WISC-R confidence intervals for WISC-R quotients.     

Block-based versus text-based programming environments on novice student learning outcomes: a meta-analysis study

ABSTRACT

Background and Context: The use of block-based programming environments is purported to be a good way to gently introduce novice computer programmers to computer programming. A small, but growing body of research examines the differences between block-based and text-based programming environments.

Objective: Thus, the purpose of this study was to examine the overall effect of block-based versus text-based programming environments on both cognitive and affective student learning outcomes.

Method: Five academic databases were searched to identify literature meeting our inclusion criteria and resulted in 13 publications with 52 effect size comparisons on both cognitive and affective outcomes.

Findings: We found small effect size (g = 0.245; p = .137; with a 95% confidence interval of ?0.078 to 0.567) in favor of block-based programming environments on cognitive outcomes, and a trivial effect size (g = 0.195, p = .429; with a 95% confidence interval of ?0.289 to 0.678) on affective outcomes. Both effect size calculations were statistically insignificant using random effects models. The effect sizes were examined for moderating effects by education level, learning environment, and study duration. Some evidence of publication bias was detected in these data.

Implications: More research is needed to examine the utility and efficacy of block-based programming environments for novice programmers. Future studies should account for hybrid programming environments using novel research methods.     

Moderated Mediation Analysis Using Bayesian Methods

Conventionally, moderated mediation analysis is conducted through adding relevant interaction terms into a mediation model of interest. In this study, we illustrate how to conduct moderated mediation analysis by directly modeling the relation between the indirect effect components including a and b and the moderators, to permit easier specification and interpretation of moderated mediation. With this idea, we introduce a general moderated mediation model that can be used to model many different moderated mediation scenarios including the scenarios described in Preacher, Rucker, and Hayes (2007). Then we discuss how to estimate and test the conditional indirect effects and to test whether a mediation effect is moderated using Bayesian approaches. How to implement the estimation in both BUGS and Mplus is also discussed. Performance of Bayesian methods is evaluated and compared to that of frequentist methods including maximum likelihood (ML) with 1st-order and 2nd-order delta method standard errors and mL with bootstrap (percentile or bias-corrected confidence intervals) via a simulation study. The results show that Bayesian methods with diffuse (vague) priors implemented in both BUGS and Mplus yielded unbiased estimates, higher power than the ML methods with delta method standard errors, and the ML method with bootstrap percentile confidence intervals, and comparable power to the ML method with bootstrap bias-corrected confidence intervals. We also illustrate the application of these methods with the real data example used in Preacher et al. (2007). Advantages and limitations of applying Bayesian methods to moderated mediation analysis are also discussed.     

Using an R shiny to enhance the learning experience of confidence intervals

Many students find understanding confidence intervals difficult, especially because of the amalgamation of concepts such as confidence levels, standard error, point estimates and sample sizes. An R Shiny application was created to assist the learning process of confidence intervals using graphics and data from the US National Basketball Association.     

Lack of statistical significance

Criticism has been leveled against the use of statistical significance testing (SST) in many disciplines. However, the field of school psychology has been largely devoid of critiques of SST. Inspection of the primary journals in school psychology indicated numerous examples of SST with nonrandom samples and/or samples of convenience. In this article we present an argument against SST and its consequent p values in favor of the use of confidence intervals and effect sizes. Further, we present instances of common errors that impede cumulative knowledge in the literature related to school psychology. © 2007 Wiley Periodicals, Inc. Psychol Schs 44: 417–422, 2007.     

Prompting and visualising monitoring outcomes: Guiding self-regulatory processes with confidence judgments

Sensible self-regulated study decisions are largely based on monitoring learning and using this information to control learning processes, but research has found that such processes may not be initiated automatically. To support learners, we adopted prompting and visualisation methods by asking learners to assign confidence ratings to learning tasks and visualising them during re-study, and tested the effects on metacognitive and cognitive measures in an experimental study (N = 95). Results show that prompting monitoring increased study efforts while visualising monitoring outcomes during learning focussed these efforts on uncertain answers. Due to low monitoring accuracy, metacognitively sensible regulation did not lead to cognitive learning gains. While the results support the idea of using visualisation techniques to implicitly guide self-regulated learning, more needs to be done to increase monitoring accuracy. Further, our study suggests that researchers should be aware of the effect that assessing confidence judgments has on subsequent learning behaviour.     

New Confidence Intervals and Bias Comparisons Show That Maximum Likelihood Can Beat Multiple Imputation in Small Samples

When analyzing incomplete data, is it better to use multiple imputation (MI) or full information maximum likelihood (ML)? In large samples ML is clearly better, but in small samples ML’s usefulness has been limited because ML commonly uses normal test statistics and confidence intervals that require large samples. We propose small-sample t-based ML confidence intervals that have good coverage and are shorter than t-based confidence intervals under MI. We also show that ML point estimates are less biased and more efficient than MI point estimates in small samples of bivariate normal data. With our new confidence intervals, ML should be preferred over MI, even in small samples, whenever both options are available.     

A meta-analysis of association between acne vulgaris and Demodex infestation

Until now, etiology of acne vulgaris is still uncertain. Although clinicians usually deny the association between Demodex infestation and acne vulgaris, it has been proved in some clinical practices. To confirm the association between Demodex infestation and acne vulgaris, a meta-analysis was conducted. Predefined selection criteria were applied to search all published papers that analyzed the association between Demodex infestation and acne vulgaris (January 1950 to August 2011) in ISI Web of Knowledge, MEDLINE, and China National Knowledge Infrastructure (CNKI) databases. A meta-analysis was performed to calculate odds ratios (ORs) and 95% confidence intervals (CIs) based on fixed effects models or random effects models. We enrolled the 60 Chinese and 3 English papers in this meta-analysis, which covered Turkey and 25 different provinces/municipalities in China and 42 130 participants including students and residents, aged from 1 to 78 years. The pooled OR in random effects models is 2.80 (95% CI, 2.34–3.36). Stability is robust according to sensitivity analysis. The fail-safe number is 18 477, suggesting that at least 18 477 articles with negative conclusions would be needed to reverse the conclusion that acne vulgaris was related to Demodex infestation. So the effect of publication bias was insignificant and could be ignored. It was concluded that acne vulgaris is associated with Demodex infestation. This indicates that when regular treatments for acne vulgaris are ineffective, examination of Demodex mites and necessary acaricidal therapies should be considered.     

Implementing Confidence Assessment in Low-Stakes,Formative Mathematics Assessments

Confidence assessment (CA) involves students stating alongside each of their answers a confidence rating (e.g. 0 low to 10 high) to express how certain they are that their answer is correct. Each student’s score is calculated as the sum of the confidence ratings on the items that they answered correctly, minus the sum of the confidence ratings on the items that they answered incorrectly; this scoring system is designed to incentivize students to give truthful confidence ratings. Previous research found that secondary-school mathematics students readily understood the negative-marking feature of a CA instrument used during one lesson, and that they were generally positive about the CA approach. This paper reports on a quasi-experimental trial of CA in four secondary-school mathematics lessons (N = 475 students) across time periods ranging from 3 weeks up to one academic year, compared to business-as-usual controls. A meta-analysis of the effect sizes across the four schools gave an aggregated Cohen’s d of –0.02 [95% CI –0.22, 0.19] and an overall Bayes Factor B₀₁ of 8.48. This indicated substantial evidence for the null hypothesis that there was no difference between the attainment gains of the intervention group and the control group, relative to the alternative hypothesis that the gains were different. I conclude that incorporating confidence assessment into low-stakes classroom mathematics formative assessments does not appear to be detrimental to students’ attainment, and I suggest reasons why a clear positive outcome was not obtained.

     

The Relation of Moral Emotion Attributions to Prosocial and Antisocial Behavior: A Meta‐Analysis

This meta‐analytic review of 42 studies covering 8,009 participants (ages 4–20) examines the relation of moral emotion attributions to prosocial and antisocial behavior. A significant association is found between moral emotion attributions and prosocial and antisocial behaviors (d = .26, 95% CI [.15, .38]; d = .39, 95% CI [.29, .49]). Effect sizes differ considerably across studies and this heterogeneity is attributed to moderator variables. Specifically, effect sizes for predicted antisocial behavior are larger for self‐attributed moral emotions than for emotions attributed to hypothetical story characters. Effect sizes for prosocial and antisocial behaviors are associated with several other study characteristics. Results are discussed with respect to the potential significance of moral emotion attributions for the social behavior of children and adolescents.     

Primary science teacher confidence revisited: ten years on

Background This paper compares the findings from a recent, large-scale UK-wide survey of primary teachers' confidence in teaching science with the results of a seminal report carried out 10 years ago by Wynne Harlen in Scotland. Recent reports from across the UK have indicated there are still serious concerns relating to primary teachers' confidence and ability to teach science effectively.

Purpose The main research aims were to provide a clear, evidence-based analysis of the current issues facing primary science in the UK; explore primary teachers' confidence in science teaching and to evaluate the impact of science initiatives taking place in UK primary schools.

Sample The sample for the study comprised: telephone interviews with 300 primary teachers from all UK regions; seven focus groups of primary teachers held in England, Northern Ireland, Scotland and Wales to further explore the issues raised in the telephone interviews; and workshops from a two-day conference of 75 stakeholders in primary science from all UK regions (approximately half the delegates were teachers; also represented were teacher educators (initial and continuing professional development), curriculum developers and policy-makers). In addition, 100 teacher education institutions were surveyed in relation to their participation in primary science initiatives.

Design and methods The methodology for both studies comprised qualitative and quantitative elements (see sample details, above). All data were collected between June and September 2004.

Results The findings indicated that there are improvements in some areas of primary teachers' confidence in teaching science. However, the study showed that half of the teachers surveyed cited lack of teacher confidence and ability to teach science as the current issue of major concern in primary science. This paper also reports on some of the professional development initiatives carried out by higher education institutions in primary science.

Conclusions The paper concludes that there has been some progress in developing teacher confidence in primary science over the past 10 years. However, the situation is still critical for all stakeholders. Half of the teachers surveyed in the UK for the present study identified lack of teacher confidence and ability to teach science as the major issue of concern in primary science. Higher education institutions need to enhance the preparation of new primary teachers to ensure that they are all confident and effective teachers of science. They could also increase their partnership work with schools and other continuing professional development (CPD) providers in relation to primary science. The evidence demonstrates clearly that there is a need for substantially increasing science professional development for primary teachers. It also shows that such professional development could be more effectively targeted at specific aspects of science teaching that are more challenging for teachers. Further, the study shows that professional development in science works, in that teachers who have experienced science continuing professional development (CPD) are much more confident to teach science than those who have not.     

Inference by Eye: Pictures of Confidence Intervals and Thinking About Levels of Confidence

A picture of a 95% confidence interval (CI) implicitly contains pictures of CIs of all other levels of confidence, and information about the p‐value for testing a null hypothesis. This article discusses pictures, taken from interactive software, that suggest several ways to think about the level of confidence of a CI, p‐values, and what conclusions can be drawn from inspecting a CI.     

An interprofessional education initiative between psychology and dentistry: an exploration of students’ perceptions of confidence

ABSTRACT

Part of preparing future health professionals for multidisciplinary work environments involves interprofessional education (IPE). We explore students’ perceptions of confidence during IPE and how this impacts their ability to contribute to interprofessional learning. The written reflections of 115 undergraduate Psychology (n?=?58) and Dentistry (n?=?57) students aged 19–47 years (median?=?22 years) who participated in two sessions of IPE utilising motivational interviewing to encourage health behaviour change were analysed. Six themes were identified: Concerns about Confidence Resulted in Additional Prior Preparation, Lack of Confidence as an Impediment to Contribution, An Increase in Confidence as a Result of Doing, Recognising the Value of Confidence, Confidence as an Area for Further Development, and Confidence from the Outset. The importance of confidence in impacting students’ ability to actively contribute in IPE should not be underestimated. Educators should seek to maximise student confidence and to create a safe learning environment where further confidence can be built. Further research is needed to determine factors that may boost confidence including the optimal time to incorporate IPE in degree programmes, the best format in which to deliver IPE, and how much IPE to deliver.     

Serial pattern learning of temporal intervals

The problem was to determine how rats adjust the times of their lever responses to repeating sequences of interfood intervals. In Experiment 1, rats were trained on an interval schedule of reinforcement with a 12-element Fleshler-Hoffman series with a mean of 60 sec; the order was as follows: ascending, random with repetition, random with replacement, random without replacement. In Experiment 2, rats were trained with a 10-element ascending or descending series (from 2⁰ to 2⁹ sec), and in a ramp procedure in which these intervals increased and then decreased repeatedly. In the ascending, descending, and ramp conditions (but not in the random conditions), postreinforcement pause (PRP) was a function of the interval. PRP was most highly correlated with an interval later in the series. Theories of conditioning and timing based on the averaging of past experience must be modified to account for such anticipatory behavior.     

Effects of nonnormal data on parameter estimates and fit indices for a model with latent and manifest variables: An empirical study

Research in covariance structure analysis suggests that nonnormal data will invalidate chi‐square tests and produce erroneous standard errors. However, much remains unknown about the extent to and the conditions under which highly skewed and kurtotic data can affect the parameter estimates, standard errors, and fit indices. Using actual kurtotic and skewed data and varying sample sizes and estimation methods, we found that (a) normal theory maximum likelihood (ML) and generalized least squares estimators were fairly consistent and almost identical, (b) standard errors tended to underestimate the true variation of the estimators, but the problem was not very serious for large samples (n = 1,000) and conservative (99%) confidence intervals, and (c) the adjusted chi‐square tests seemed to yield acceptable results with appropriate sample sizes.     

Can Early Childhood Curriculum Enhance Social-Emotional Competence in Low-Income Children? A Meta-Analysis of the Educational Effects

Research Findings: This meta-analysis examined 29 (quasi-)experimental studies that involved low-income children ages 3 to 5 who might be subject to risks of academic failure and other negative outcomes. Compared to the controls, children who learned with social-emotional learning (SEL) curricula demonstrated significantly improved social-emotional competence, with an effect size or standardized mean difference of 0.241 (95% confidence interval [0.194, 0.287]). However, the use of other curricula that lacked an intensive focus on SEL yielded nonsignificant effects on the social-emotional competence of low-income children. Type of curriculum, fidelity of curriculum implementation, and duration of intervention were found to moderate the educational effects. Practice or Policy: The findings of this meta-analysis contribute to the growing body of empirical evidence on the positive effects of early SEL curricula and explain how curricula can produce social-emotional benefits for low-income children in their early years.     

Monte Carlo Confidence Intervals for Complex Functions of Indirect Effects

One challenge in mediation analysis is to generate a confidence interval (CI) with high coverage and power that maintains a nominal significance level for any well-defined function of indirect and direct effects in the general context of structural equation modeling (SEM). This study discusses a proposed Monte Carlo extension that finds the CIs for any well-defined function of the coefficients of SEM such as the product of k coefficients and the ratio of the contrasts of indirect effects, using the Monte Carlo method. Finally, we conduct a small-scale simulation study to compare CIs produced by the Monte Carlo, nonparametric bootstrap, and asymptotic-delta methods. Based on our simulation study, we recommend researchers use the Monte Carlo method to test a complex function of indirect effects.     

Dimensions of academic interest among undergraduate students: passion,confidence, aspiration and self-expression

We investigated psychological dimensions of academic interest among undergraduate students (N = 325) using a global academic interest scale. The scale was administered together with measures of academic performance, educational aspiration, career planning, goal setting, life satisfaction, attitudes towards leisure, personality and value. Exploratory factor analyses revealed four dimensions of academic interest: passion for learning, confidence in the future, career aspiration and self-expression. All four dimensions were significantly related to career planning, goal setting and life satisfaction. On the other hand, specific pairings of significant relationships were found between different dimensions of academic interest and personality and value, suggesting that students with different personalities and different sets of values are likely to appreciate and respond to different aspects of academic interest. We conclude with implications for research on academic interest development across a more diverse range of age groups.