Predicting item difficulty of science national curriculum tests: the case of key stage 2 assessments

Predicting item difficulty is highly important in education for both teachers and item writers. Despite identifying a large number of explanatory variables, predicting item difficulty remains a challenge in educational assessment with empirical attempts rarely exceeding 25% of variance explained.

This paper analyses 216 science items of key stage 2 tests which are national sampling assessments administered to 11 year olds in England. Potential predictors (topic, subtopic, concept, question type, nature of stimulus, depth of knowledge and linguistic variables) were considered in the analysis. Coding frameworks employed in similar studies were adapted and employed by two coders to independently rate items. Linguistic demands were gauged using a computational linguistic facility. The stepwise regression models predicted 23% of the variance with extended constructed questions and photos being the main predictors of item difficulty.

While a substantial part of unexplained variance could be attributed to the unpredictable interaction of variables, we argue that progress in this area requires improvement in the theories and the methods employed. Future research needs to be centred on improving coding frameworks as well as developing systematic training protocols for coders. These technical advances would pave the way to improved task design and reduced development costs of assessments.     

Communicating comparative findings from meta-analysis in educational research: some examples and suggestions

ABSTRACT

This article reviews some of the strengths and limitations of the comparative use of meta-analysis findings, using examples from the Sutton Trust-Education Endowment Foundation Teaching and Learning ‘Toolkit’ which summarizes a range of educational approaches to improve pupil attainment in schools. This comparative use of quantitative findings has similar characteristics to umbrella reviews which provide a succinct but applicable summary of the current state of evidence to inform practice or policy. Meta-analysis helps to identify which approaches have, on average, made the most difference to tested learning outcomes, in terms of effect size. We suggest that any comparative inferences made between meta-analyses should be treated cautiously, but taken seriously. Additionally, we present alternative ways of interpreting effect sizes, security ratings and cost-estimates to make research findings accessible, whilst retaining appropriate accuracy which is discussed using the ‘Toolkit’ as an example. We conclude by arguing that we should consider the available information not as ‘what works’, but ‘what has worked’ to understand its value and limits in terms of supporting the development of research-based practice.     

The bright and dark side of writing motivation: Effects of explicit instruction and peer assistance

The authors investigated the impact of explicit instruction and peer-assisted writing on students' writing motivation and self-efficacy for writing. Eleven teachers and their 206 fifth- and sixth-grade students participated in a 2 (explicit instruction vs. writing opportunities without explicit instruction) × 2 (peer-assisted writing vs. writing individually) experimental intervention study with a pretest-posttest design. The four experimental conditions were compared with a business-as-usual (BAU) condition. The five-week interventions were implemented in authentic classes by regular class teachers, who received a prior professional development training. Multilevel analyses showed that students who wrote with a peer were more autonomously motivated at posttest than BAU students. Additionally, BAU students and students receiving explicit instruction were more controlled motivated than students who were offered ample writing opportunities while practicing individually. Theoretical and educational implications are discussed in view of realizing a bright pathway towards autonomous writing motivation.     

How do tutors use data to support their students?

ABSTRACT

Despite widespread enthusiasm, evidence of the effectiveness of learning analytics remains mixed. One possible explanation for this is that insufficient attention has been paid to the contexts in which it is introduced. We report here on a small-scale study into the prior use of data and communications technologies by tutors, who comprise a key user group in The Open University’s tuition model. Tutors interviewed reported using a complex set of data sources and information tools, and creating local/personal tools and methods for keeping track of students and their interactions with them.     

CONSOLIDATING LEARNING FROM EXPERIENCE

The article describes an experiential learning programme at the University School of Education, Newcastle upon Tyne, in which teams of postgraduate trainee teachers undertook problem solving placements in local businesses and public services. The evaluation revealed a contrast between high general satisfaction levels and low perceived relevance to teaching. This is analysed and discussed in relation to the perceptions student teachers develop during their training. The vital importance of briefing and debriefing in experiential programmes is emphasised, as an aid to learning when much of value is easily lost.     

Changes in Korean Science Teachers' Perceptions of Creativity and Science Teaching After Participating in an Overseas Professional Development Program

This study investigated changes in Korean science teachers' perceptions of creativity and science teaching after participating in an overseas professional development program. Participants were 35 secondary science teachers. Data were collected from open-ended questionnaires and interviews. Results indicated that participants showed a growing awareness that creativity can be expressed by every student; creativity can be enhanced; science has a much wider range of activities that foster creativity; and creativity-centered science teaching can be implemented in Korea. The major elements of the professional program that promoted these perceptual changes included hands-on creativity activities, observation of creativity-centered classrooms, and discussion with other teachers. Follow-up study revealed that their perceptual changes have been reflected in their teaching practices.     

Silent versus oral reading comprehension and efficiency

Seventy‐four students read passages from an individually administered test of reading comprehension (a subtest from the Test of Dyslexia, a test of reading and related abilities currently in development; McCallum & Bell, 2001), and then answered literal and inferential questions. Students were randomly assigned to one of two conditions; 39 students read the passages silently and 35 read orally, with time recorded for each passage read. Comprehension and time were dependent measures for a Multivariate Analysis of Covariance (MANCOVA) and two follow‐up Analyses of Covariance (ANCOVA). After controlling for reading ability, results from the MANCOVA showed a significant combined effect ( p < .05); however, a comparison of mean reading comprehension scores showed no significant difference between silent readers and oral readers ( p > .05). On the other hand, with reading ability controlled, silent readers took significantly less time to complete passages compared to those who read orally ( p < .02). In fact, students took 30% longer to read orally than silently, on average. When test directions do not specify either oral or silent reading and error analysis is not a goal, testing will be more efficient via silent responding with no loss of comprehension. © 2004 Wiley Periodicals, Inc. Psychol Schs 41: 241–246, 2004.