A systematic review and meta‐analysis of the effectiveness of information and communication technology (ICT) on the teaching of spelling

Recent Government policy in England and Wales on Information and Communication Technology (ICT) in schools is heavily influenced by a series of non‐randomised controlled studies. The evidence from these evaluations is equivocal with respect to the effect of ICT on literacy. In order to ascertain whether there is any effect of ICT on one small area of literacy, spelling, a systematic review of all randomised controlled trials (RCTs) was undertaken. Relevant electronic databases (including BEI, ERIC, Web of Science, PsycINFO, The Cochrane Library) were searched. Seven relevant RCTs were identified and included in the review. When six of the seven studies were pooled in a meta‐analysis there was an effect, not statistically significant, in favour of computer interventions (Effect size = 0.37, 95% confidence interval =–0.02 to 0.77, p = 0.06). Sensitivity and sub‐group analyses of the results did not materially alter findings. This review suggests that the teaching of spelling by using computer software may be as effective as conventional teaching of spelling, although the possibility of computer‐taught spelling being inferior or superior cannot be confidently excluded due to the relatively small sample sizes of the identified studies. Ideally, large pragmatic randomised controlled trials need to be undertaken.     

Avoiding Bias in Randomised Controlled Trials in Educational Research

Randomised controlled trials (RCTs) are often seen as the 'gold standard' of evaluative research. However, whilst randomisation will ensure comparable groups, trials are still vulnerable to a range of biases that can undermine their internal validity. In this paper we describe a number of common threats to the internal validity of RCTs and methods of countering them. We highlight a number of examples from randomised trials in education and health care where problems of execution and analysis of the RCT has undermined their internal validity. However, awareness of these potential biases can lead to careful planning to avoid or reduce their occurrence. If good quality randomised trials are to inform policy and practice in education then rigorous trials need to be designed that are the least susceptible to threats to their validity.     

The NCTQ Selectivity Standard and Principal Evaluation of Teacher Preparation

Ratings published by the National Council on Teacher Quality (NCTQ) require selectivity in admission to educator preparation programs. NCTQ provides a list of citations to support the selectivity standard termed “strong support.” A review of each citation in the list found little or no support for the standard. Original data was collected examining employment supervisor ratings of over 11,000 teachers’ preparation and analyzed for correlations and differences based on the selectivity standard. No significant differences were found between supervisor ratings of teachers who met or did not meet the NCTQ test standard and no significant correlations were found between the selectivity tests, grade point average, and principal ratings of teachers. These findings bring into question the listed selectivity standard and the rationale for its inclusion in NCTQ's public ranking process.     

PUBLICATION BIAS: THE ACHILLES' HEEL OF SYSTEMATIC REVIEWS?

ABSTRACT:  The term 'publication bias' usually refers to the tendency for a greater proportion of statistically significant positive results of experiments to be published and, conversely, a greater proportion of statistically significant negative or null results not to be published. It is widely accepted in the fields of healthcare and psychological research to be a major threat to the validity of systematic reviews and meta-analyses. Some methodological work has previously been undertaken, by the author and others, in the field of educational research to investigate the extent of the problem. This paper describes the problem of publication bias with reference to its history in a number of fields, with special reference to the area of educational research. Informal methods for detecting publication bias in systematic reviews and meta-analyses of controlled trials are outlined and retrospective and prospective methods for dealing with the problem are suggested.     

Randomised controlled trial of incentives to improve attendance at adult literacy classes

Background: Incentives have been proposed as a method to improve attendance in adult literacy classes. In the UK, several areas have piloted the use of incentives to promote attendance at adult literacy classes. To date no rigorous evaluation of this policy has been undertaken. This paper describes (as far as we are aware) the only UK‐based randomised controlled trial to evaluate the use of financial incentives in order to promote attendance in classes for adult learners.

Methods: We used a cluster‐randomised design. Twenty‐nine adult literacy classes were randomised in two groups using minimisation. Intervention group learners received £5 (US$10) for each class attended. The main outcome was class attendance; the secondary outcome was literacy scores.

Results: After allocation, one class was found to be ineligible for the study. In the 28 remaining classes there was a statistically significant reduction of about 1.5 sessions (95% confidence interval (CI) 0.28, 2.79; p = 0.019) attended by the intervention group compared with control, after adjusting for cluster size and baseline scores. The difference in reading scores between the intervention and control group, conditioned on baseline scores, was ?2.38 (with controls scoring higher than the intervention group), but this difference was not statistically significant (95% CI ?7.40 to 2.57, p = 0.33).

Conclusion: Payments to attend adult literacy classes had an adverse effect on attendance. This trial needs urgent replication, ideally with a larger incentive, before this approach is widely used by policy makers.     

Is an intervention using computer software effective in literacy learning? A randomised controlled trial

Background: computer software is widely used to support literacy learning. There are few randomised trials to support its effectiveness. Therefore, there is an urgent need to rigorously evaluate computer software that supports literacy learning.

Methods: we undertook a pragmatic randomised controlled trial among pupils aged 11–12 within a single state comprehensive school in the North of England. The pupils were randomised to receive 10 hours of literacy learning delivered via laptop computers or to act as controls. Both groups received normal literacy learning. A pre‐test and two post‐tests were given in spelling and literacy. The main pre‐defined outcome was improvements in spelling scores.

Results: 155 pupils were randomly allocated, 77 to the ICT group and 78 to control. Four pupils left the school before post‐testing and 25 pupils did not have both pre‐ and post‐test data. Therefore, 63 and 67 pupils were included in the main analysis for the ICT and control groups respectively. After adjusting for pre‐test scores there was a slight increase in spelling scores, associated with the ICT intervention, but this was not statistically significant (0.954, 95% confidence interval (CI) – 1.83 to 3.74, p = 0.50). For reading scores there was a statistically significant decrease associated with the ICT intervention (?2.33, 95% CI ?0.96 to ?3.71, p = 0.001).

Conclusions: we found no evidence of a statistically significant benefit on spelling outcomes using a computer program for literacy learning. For reading there seemed to be a reduction in reading scores associated with the use of the program. All new literacy software needs to be tested in a rigorous trial before it is used routinely in schools.     

Examination for Bias in Principal Ratings of Teachers' Preparation

The Department of Education is moving to change accountability for teacher preparation institutions to include surveys of the graduates and their supervising principal following paid employment. This study describes one of a number of quantitative studies that examine the validity and usefulness of such follow-up surveys. Using multiple years of data, the authors examined the effect of teacher socioeconomic status and ethnicity on principals' evaluation of the teachers' preparation. The results indicated that there was no difference in ratings based on graduates' parent education, family income, or ethnicity. Post hoc evaluation showed that Latino teachers were rated better prepared to work with diversity in the classroom and to teach English learners. Bias does not appear to be part of principal evaluation. However, because principals are prone to rating teachers on a binary, satisfactory/unsatisfactory basis, follow-up surveys may not be the most useful tool for assessing some nuances of teacher preparation.     

Do Volunteers in Schools Help Children Learn to Read? A Systematic Review of Randomised Controlled Trials

The aim of unpaid volunteer classroom assistants is to give extra support to children learning to read. The impact of using volunteers to improve children's acquisition of reading skills is unknown. To assess whether volunteers are effective in improving children's reading, we undertook a systematic review of all relevant randomised controlled trials (RCTs). An exhaustive search of all the main electronic databases was carried out (i.e. BEI, PsycInfo, ASSIA, PAIS, SSCI, ERIC, SPECTR, SIGLE). We identified eight experimental studies, of which seven were RCTs. One of the RCTs was excluded because it did not meet the inclusion criteria. One RCT randomised intact classes and the other six studies randomised individual children and could therefore be included in a meta-analysis. All of the trials were fairly small, with the largest including 99 pupils. Four of the trials showed a positive outcome, while three showed a negative effect and the remaining study was equivocal. We pooled the four most homogeneous trials. The pooled data indicated an effect size of 0.19, which was not statistically significant ( p = 0.54, 95% confidence interval = -0.31 to 0.68). Overall, volunteering appeared to have a small effect on reading outcomes. However, the confidence intervals were wide, which could conceal a potentially large benefit or a harmful effect. Thus, more good quality RCTs are required in order to provide more conclusive evidence.     

The Validity of Certain Prognostic Tests in Predicting Algebraic Ability

Monte Carlo studies offer the opportunity to manipulate data sets with specified characteristics and to examine the generalizability of statistical procedures in ways that are not practical using actual (empirical) data. To determine whether computer-generated (simulated) data results accurately represent empirical data, this study replicated an investigation of the effects of item sampling plans in the application of multiple matrix sampling using both simulated and empirical data sets. Although results were similar, the empirical data results were more precise. This study suggests that, for some investigations, it may be important to confirm simulated study with empirical study.     

A systematic review of evidence on the effectiveness of interventions and strategies for widening participation in higher education

Widening participation (WP) in higher education (HE) is an increasingly important policy issue, with interventions to increase participation from minority ethnic, low-income and other under-represented groups undertaken in HE sectors in many countries. In the UK there is a large amount of WP activity but a lack of robust evidence of its effectiveness. This article presents a systematic review in the topic area of WP in HE. We included studies of systematic review, randomised controlled trial (RCT) and quasi-experimental (QE) designs, and assessed evidence of the effectiveness of university access strategies and approaches in relation to the participation of disadvantaged students at university. We searched for, quality appraised and synthesised the international evidence, that is, evidence published in any country, in the English language. The findings from 4 systematic reviews and 12 experimental studies (4 RCTs, 4 RDDs and 4 QEDs) are presented as narrative syntheses in a series of thematic sub-topics. We found some evidence of effectiveness for a number of university access interventions. ‘Black box’ WP programmes (those with multiple elements in a single programme) and financial incentives were found to be effective. However, much of the evidence had design limitations and the majority were conducted in the US. The article concludes with research recommendations in relation to UK interventions, including suggested designs for future quasi-experimental evaluation.