Reassessing RCTs as the ‘gold standard’: synergy not separatism in evaluation designs

ABSTRACT

Randomised controlled trials (RCTs) are increasingly used to evaluate educational interventions in the UK. However, RCTs remain controversial for some elements of the research community. This paper argues that the widespread use of the term ‘gold standard’ to describe RCTs is problematic, as it implies that other research methods are inferior. The usefulness of RCTs can be greatly enhanced when used in conjunction with implementation-specific measures (e.g. observation tools, attitude/engagement surveys and interviews). The proposal is advanced through case studies of two evaluations. One relates to the development of science subject leader skills and expertise at primary school level and the other to co-operative learning of primary maths. Both evaluations randomised schools to the intervention or the business-as-usual control, and compared impact using subject knowledge tests. Integral to each study was a process evaluation which looked at evidence from classroom practice along with feedback from the teachers and pupils themselves. We contend that this enabled much more holistic and richly interpretative pieces of research. The paper concludes that privilege for particular paradigms should be set aside when designing effective evaluations of educational interventions, and that it is insufficient to ask ‘what works?’ without also asking ‘why?’, ‘where?’ and ‘how?’.     

Simulating conditions of learned helplessness: the effects of interventions and attributions

We report 2 experiments that assess factors potentially responsible for a proactive interference with the sensitivity of a mother's response to infant signals. Using a version of the "learned helplessness" paradigm, mothers' performance on a solvable task was assessed following pretreatments that involved exposure to an infant cry but that differed in the mothers' ability to exert control over the termination of the cry. Each experiment explored the role of varying attributions made by a mother in the development of, or the reversal of, the helplessness phenomenon. The first experiment addressed the question of whether a specific intervention (i.e., providing the mother an attribution for failure) can reduce the debilitating effect of prior experience with failure. The results indicated that the debilitating effects associated with previous failure were reversed for the 16 mothers assigned to the intervention group. 40 mothers participated in the second experiment, which varied attributions assigned to an identical cry stimulus (i.e., the cry was produced by an "easy" vs. a "difficult" infant). This experiment assessed the effect of varying attributions on the mothers' ability to terminate the cry. We found that mothers pretreated with inescapable cries and those receiving the experimental manipulation of attributing the cry to a "difficult" infant showed debilitated performance in stopping the cry when given the opportunity. We propose that models based on learned helplessness theory have value in the study of caregiver-infant relationships, in particular, caregivers' perceived and objective effectiveness in responding to an infant's signals.     

Career plateauing: Implications for career development specialists

Conclusion While career plateauing is a major challenge facing contemporary organizations, possible solutions are available. The career development specialist possessing a variety of skills geared toward both proaction and reaction can be instrumental in designing interventions which minimize the negative impact of plateauing. While career plateauing can be a devastating problem for many high achieving, self-sacrificing baby boomers, for others it may present an opportunity to escape the internal pressures for success. Much of what happens to a plateaued employee is determined by the resources possessed by the employee as well as the organization's response to the employee. The activities of the career development specialist constitute much of what is considered the organization's response. A career plateau may mark the beginning of a period characterized by frustration and depression or it may present the opportunity for a new challenge or reappraisal of life direction and goals.     

Politics,Poverty and Education in Rural South Africa

This article examines the relationship between poverty and education in South Africa, and how its conceptualisation has changed historically. By analysing two major inquiries into poverty conducted by the Carnegie Corporation (in 1929‐32 and 1982‐84), it highlights the political nature of poverty and also its racialisation in South Africa. Using material from a peri‐urban research study, it extends the analysis to include the underprovision of schooling, gender relationships of poverty and also child labour. The article illustrates how the relationship between poverty and education has been differently constructed in different discourses, and concludes by considering the challenges of developing policies to address the education/poverty nexus in the rural areas of post‐apartheid South Africa.     

The Role of Drawing in Graphic Design and the Implications for Curriculum Planning

Over a number of years a programme of research has been carried out into the role of drawing in design processes, primarily the graphic design process, and on the possible implications for curriculum planing for design students. From the findings of an original in-depth analysis of the use of drawing in the commercial environment of the design consultant, various initiatives in educational planning and assessment in England and Scotland have been fostered and a contribution has been made to the ongoing debate in drawing studies     

Engaging New Faculty in the Scholarship of Teaching

This article describes how scholarly teaching projects were conceived, implemented, and evaluated by junior faculty from a variety of disciplines at a medium-sized midwestern university. The authors explore: considerations in designing scholarly teaching projects, methods used to evaluate teaching effectiveness, and outcomes of the teaching projects. Finally, two of the teaching projects illustrate how junior faculty with diverse courses, class sizes, and teaching concerns approached the task of improving their teaching.     

Mixed methods,mixed outcomes? Combining an RCT and case studies to research the impact of a training programme for primary school science teachers

A randomised controlled trial (RCT) and a series of case studies were used to determine the impact of two variants of an intervention (a professional development programme) aimed at improving primary school science teachers’ subject and pedagogic content knowledge, and enhancing their subject leadership ability. Ninety-six schools were randomly assigned to full or partial treatment groups or a ‘business-as-usual’ control group. Quantitative data were collected from teachers and pupils through an assessment of scientific knowledge based on standardised assessment items. Qualitative data were collected through interviews and lesson observation initially in thirty case study schools. There were three data collection points: pre- and post-intervention, and one year later.

[Guskey, T. (1986). Staff development and the process of teacher change. Educational Researcher, 15(5), 5–12.] Levels of Professional Development Evaluation model was used as the analysis framework. The quantitative data from the teachers’ subject knowledge assessment indicated neither the full nor the partial training programmes had a statistically significant impact on teachers’ performance. In contrast, the qualitative data suggested that many teachers in the full treatment group believed that their subject knowledge had improved and reported increased confidence in their teaching of science. Lesson observations provided corroborating evidence of change in teachers’ practice, and some modest evidence of wider change in schools. There was no statistically significant improvement in pupil performance in subject knowledge assessments when teachers had participated in the intervention. In the context of research methods, the study suggests that a mixed-methods approach to evaluation is likely to yield a more rounded and nuanced picture of the overall impact of an intervention.