Reassessing RCTs as the ‘gold standard’: synergy not separatism in evaluation designs

ABSTRACT

Randomised controlled trials (RCTs) are increasingly used to evaluate educational interventions in the UK. However, RCTs remain controversial for some elements of the research community. This paper argues that the widespread use of the term ‘gold standard’ to describe RCTs is problematic, as it implies that other research methods are inferior. The usefulness of RCTs can be greatly enhanced when used in conjunction with implementation-specific measures (e.g. observation tools, attitude/engagement surveys and interviews). The proposal is advanced through case studies of two evaluations. One relates to the development of science subject leader skills and expertise at primary school level and the other to co-operative learning of primary maths. Both evaluations randomised schools to the intervention or the business-as-usual control, and compared impact using subject knowledge tests. Integral to each study was a process evaluation which looked at evidence from classroom practice along with feedback from the teachers and pupils themselves. We contend that this enabled much more holistic and richly interpretative pieces of research. The paper concludes that privilege for particular paradigms should be set aside when designing effective evaluations of educational interventions, and that it is insufficient to ask ‘what works?’ without also asking ‘why?’, ‘where?’ and ‘how?’.     

‘Educating rita’: Is an academic science course for prospective infant and primary teachers appropriate

Conclusion Our decision to choose the Open University S101 Science: A Foundation Course represents a direct attempt on our part to provide a background in science for our prospective infant and primary teachers. Our objective that this course achieve parity with first year Faculty of Science courses may appear to be an unattainable goal, given the methodological problems that have to be faced for such a general purpose course. It would certainly not be feasible for a lecture staff of four persons were it not for the printed readers, texts, assignments, examinations, and audio-visual software purchased, with approval, from the Open University in the United Kingdom. Our pilot study in 1983 gave us some encouragement and, most interestingly, 10 out of the 17 students continued with Science study into Year 2 in 1984, thus committing them to the major study in science in Years 3 and 4. Given that three students failed, 10 out of 14 is a far higher retention rate than anything we have experienced previously. Clearly, the evaluation of our efforts to influence the teaching of science in Tasmanian primary schools must wait, but these evaluative studies should make a contribution to the research data on curriculum issues in Australia. This course could play an important in-service function for present primary teachers. We are hopeful that such a content-based approach to science teacher education may help to convince teachers to take the extensive range of available curriculum materials seriously.     

Development of reasoning associated with pictographs: Representing, interpreting, and predicting

A developmental model involving four response levels is proposed concerning how students arrange pictures to represent data in a pictograph, how they interpret these pictographs, and how they make predictions based on these pictographs. The model is exemplified by responses from three related interview-based studies. In Study 1, examples of each response level are provided from 48 preparatory- to tenth-grade students. Students from higher grades were more likely to respond at higher levels. In Study 2, 22 students were interviewed longitudinally after a three-year interval; many improved in response level over time, although a few responded at lower levels. In Study 3, 20 third-grade students were interviewed and then prompted with conflicting responses of other students on video; many improved their initial responses to higher levels after exposure to the conflicting prompts. Associations among levels of representing, interpreting, and predicting were explored. Educational implications are discussed concerning reasonable expectations of students and suggestions to develop these skills in students at different grades. This revised version was published online in July 2006 with corrections to the Cover Date.     

The Influence of First‐Hand Testimony and Hearsay on Children's Belief in the Improbable

Children (3.5–8.5 years; n = 105) heard claims about the occurrence of improbable or impossible events, then were asked whether the events could really happen. Some claims were based on informants' first‐hand observations and others were hearsay. A baseline group (n = 56) reported their beliefs about these events without hearing testimony. Neither first‐hand claims nor hearsay influenced beliefs about impossible events, which remained low across the age range. Hearsay (but not first‐hand claims) did influence beliefs about improbable events. Preschoolers expressed greater belief following hearsay, compared to their beliefs following first‐hand claims and compared to the baseline group's beliefs. By contrast, older children expressed less belief following hearsay, compared to their beliefs following first‐hand claims and compared to the baseline group's beliefs.     

Trends in Performance and Growth by Students With and Without Disabilities on Five State Summative Assessments

Indicators of student academic growth are desired in state accountability systems in order to approximate student learning over time and attribute observed growth to schooling inputs. Through an extant analysis of five states’ assessment data, this study offers evidence about whether longitudinal match rates and measures of growth differ at the state level for students with disabilities, relative to students without disabilities. There were three main findings: 1) In states in which a modified assessment was offered, students with disabilities were more likely to have missing prior year scores, and consequently missing growth scores; 2) Low scoring students, many of whom had a disability, were more likely to have missing prior scores on the state general assessment, and consequently missing growth scores; 3) Students with and without disabilities showed similar growth using transition and gain score definitions of growth, but students with disabilities had lower growth when estimated via a regression-based model. Measurement and policy considerations are discussed.     

A preliminary study of the effect of computer-assisted practice on reading attainment

A computer system which gives prerecorded speech prompts on request was developed and a study was carried out to evaluate its effectiveness in enhancing the reading attainment of young children. Sixty children aged 61 to 84 months were tested on three measures of sight vocabulary (BAS Word Recognition, Frequently Occurring Words, words from the books read) before and after the intervention period of one month, during which 30 of the children had access to the system. The intervention group made significantly higher gains on the tests than the control group. The amount of practice undertaken by subjects was highly positively correlated with their gains. The amount of help which they requested showed no relationship to the gains. These results suggest that some effective practice in reading can be given by a computer.     

What predicts health students’ self-reported preparedness to work in Indigenous health settings?

Australian undergraduate programs are implementing curriculum aimed at better preparing graduates to work in culturally diverse settings, but there remains uncertainty over the role of extant student attitudes towards Indigenous Australians. To begin to address this, we obtained baseline data on student attitudes upon entry to tertiary education. 1175 health science first-year students (275 males, 897 females) completed an anonymous in-class paper questionnaire. On average, students reported positive attitudes towards Indigenous Australians, with female students reporting more positive attitudes than male and domestic students more positive than international. Hierarchical multiple regression analyses indicated that after controlling for demographic variables, interactional diversity experiences and attitudes accounted for significant variance in a range of measures of preparedness to work in Indigenous health contexts. The knowledge gained will inform the development of, and alignment between, curriculum and pedagogical approaches, leading to an improved facilitation model for educators in the Indigenous Studies context.     

Disentangling life events, stress and social support: Implications for the primary prevention of child abuse and neglect

There is a growing body of literature linking stress and child abuse and neglect, but the relationship is not unambiguously supported by empirical data. Two considerations regarding Garbarino's ecological model of child abuse and neglect may explain this research problem. First, any of the predisposing factors, which are grouped into four levels called individual, familial, social, and cultural, may either positively or negatively affect the potential for child abuse and neglect depending upon the quality of social networks and social supports available to families. Second, these factors operate most importantly, not between the perception of stress and the act of abuse or neglect, but through the interpretation of whether a given life event is stressful or not. This clarification of the ecological model points the way to redefining interventions for the primary prevention of child abuse and neglect. Existing support systems can be strengthened in order to increase a family's ability to cope with untoward events before these become stressful. In addition, advocacy activities which support children and families in general can be major components in the primary prevention of child abuse and neglect.     

"Birds of a Feather…": Behavioral Concordances and Preferential Personal Attraction in Children

The purpose of this study was to assess the extent to which preferential personal attraction was associated with behavioral similarity among newly acquinted children. Participants included 69 focal children, selected from a sample of 236 7-year-old children who met, for the first time, in same-sex quartets ( n = 59) for a free-play session. Within each of these quartets, a "discriminating child" was identified; this child expressed a clear preference for one of his or her playmates over one other of her or his quartet playmates. Preference was determined sociometrically after the children became acquainted during free play. The results indicated that "discriminating" children were more behaviorally similar to preferred playmates than to nonpreferred playmates both in terms of cognitive play style and social participation. Implications of the findings are discussed in terms of the relation to the acquaintanceship process.