Reassessing RCTs as the ‘gold standard’: synergy not separatism in evaluation designs

ABSTRACT

Randomised controlled trials (RCTs) are increasingly used to evaluate educational interventions in the UK. However, RCTs remain controversial for some elements of the research community. This paper argues that the widespread use of the term ‘gold standard’ to describe RCTs is problematic, as it implies that other research methods are inferior. The usefulness of RCTs can be greatly enhanced when used in conjunction with implementation-specific measures (e.g. observation tools, attitude/engagement surveys and interviews). The proposal is advanced through case studies of two evaluations. One relates to the development of science subject leader skills and expertise at primary school level and the other to co-operative learning of primary maths. Both evaluations randomised schools to the intervention or the business-as-usual control, and compared impact using subject knowledge tests. Integral to each study was a process evaluation which looked at evidence from classroom practice along with feedback from the teachers and pupils themselves. We contend that this enabled much more holistic and richly interpretative pieces of research. The paper concludes that privilege for particular paradigms should be set aside when designing effective evaluations of educational interventions, and that it is insufficient to ask ‘what works?’ without also asking ‘why?’, ‘where?’ and ‘how?’.     

Career plateauing: Implications for career development specialists

Conclusion While career plateauing is a major challenge facing contemporary organizations, possible solutions are available. The career development specialist possessing a variety of skills geared toward both proaction and reaction can be instrumental in designing interventions which minimize the negative impact of plateauing. While career plateauing can be a devastating problem for many high achieving, self-sacrificing baby boomers, for others it may present an opportunity to escape the internal pressures for success. Much of what happens to a plateaued employee is determined by the resources possessed by the employee as well as the organization's response to the employee. The activities of the career development specialist constitute much of what is considered the organization's response. A career plateau may mark the beginning of a period characterized by frustration and depression or it may present the opportunity for a new challenge or reappraisal of life direction and goals.     

Effects of Missing Data Methods in SEM Under Conditions of Incomplete and Nonnormal Data

Using Monte Carlo simulations, this research examined the performance of four missing data methods in SEM under different multivariate distributional conditions. The effects of four independent variables (sample size, missing proportion, distribution shape, and factor loading magnitude) were investigated on six outcome variables: convergence rate, parameter estimate bias, MSE of parameter estimates, standard error coverage, model rejection rate, and model goodness of fit—RMSEA. A three-factor CFA model was used. Findings indicated that FIML outperformed the other methods in MCAR, and MI should be used to increase the plausibility of MAR. SRPI was not comparable to the other three methods in either MCAR or MAR.     

Politics,Poverty and Education in Rural South Africa

This article examines the relationship between poverty and education in South Africa, and how its conceptualisation has changed historically. By analysing two major inquiries into poverty conducted by the Carnegie Corporation (in 1929‐32 and 1982‐84), it highlights the political nature of poverty and also its racialisation in South Africa. Using material from a peri‐urban research study, it extends the analysis to include the underprovision of schooling, gender relationships of poverty and also child labour. The article illustrates how the relationship between poverty and education has been differently constructed in different discourses, and concludes by considering the challenges of developing policies to address the education/poverty nexus in the rural areas of post‐apartheid South Africa.     

The Role of Drawing in Graphic Design and the Implications for Curriculum Planning

Over a number of years a programme of research has been carried out into the role of drawing in design processes, primarily the graphic design process, and on the possible implications for curriculum planing for design students. From the findings of an original in-depth analysis of the use of drawing in the commercial environment of the design consultant, various initiatives in educational planning and assessment in England and Scotland have been fostered and a contribution has been made to the ongoing debate in drawing studies     

Engaging New Faculty in the Scholarship of Teaching

This article describes how scholarly teaching projects were conceived, implemented, and evaluated by junior faculty from a variety of disciplines at a medium-sized midwestern university. The authors explore: considerations in designing scholarly teaching projects, methods used to evaluate teaching effectiveness, and outcomes of the teaching projects. Finally, two of the teaching projects illustrate how junior faculty with diverse courses, class sizes, and teaching concerns approached the task of improving their teaching.     

Mixed methods,mixed outcomes? Combining an RCT and case studies to research the impact of a training programme for primary school science teachers

A randomised controlled trial (RCT) and a series of case studies were used to determine the impact of two variants of an intervention (a professional development programme) aimed at improving primary school science teachers’ subject and pedagogic content knowledge, and enhancing their subject leadership ability. Ninety-six schools were randomly assigned to full or partial treatment groups or a ‘business-as-usual’ control group. Quantitative data were collected from teachers and pupils through an assessment of scientific knowledge based on standardised assessment items. Qualitative data were collected through interviews and lesson observation initially in thirty case study schools. There were three data collection points: pre- and post-intervention, and one year later.

[Guskey, T. (1986). Staff development and the process of teacher change. Educational Researcher, 15(5), 5–12.] Levels of Professional Development Evaluation model was used as the analysis framework. The quantitative data from the teachers’ subject knowledge assessment indicated neither the full nor the partial training programmes had a statistically significant impact on teachers’ performance. In contrast, the qualitative data suggested that many teachers in the full treatment group believed that their subject knowledge had improved and reported increased confidence in their teaching of science. Lesson observations provided corroborating evidence of change in teachers’ practice, and some modest evidence of wider change in schools. There was no statistically significant improvement in pupil performance in subject knowledge assessments when teachers had participated in the intervention. In the context of research methods, the study suggests that a mixed-methods approach to evaluation is likely to yield a more rounded and nuanced picture of the overall impact of an intervention.     

Relationships between families’ use of Sure Start Children’s Centres,changes in home learning environments,and preschool behavioural disorders

UK Sure Start Children’s Centres (SSCCs) aim to lessen behavioural disorders yet we lack evidence concerning how this is achieved. This study evaluates one possible mechanism: improved home learning environments (HLEs). Data come from a longitudinal study of 2568 families and children recruited at a mean age of 14 months from 117 SSCCs in England in 2012. Behavioural disorders were measured at 38 months via the Strengths and Difficulties Questionnaire. HLEs were measured at 14 and 38 months via parental interview. Families’ use of SSCCs was measured via parental interview at 14, 22, and 38 months. This study suggests that the use of SSCCs is associated with fewer preschool behavioural disorders via intermediate changes to the quality of HLE’s. Implications are discussed for social policies and for early years’ professionals.