DO INTERVENTION‐EMBEDDED ASSESSMENT PROCEDURES SUCCESSFULLY MEASURE STUDENT GROWTH IN READING?

Effective intervention delivery requires ongoing assessment to determine whether students are learning at the desired rate. Intervention programs with embedded assessment procedures (i.e., assessment that occurs naturally during the process of delivering intervention) can potentially enhance instructional decisions. However, there is almost no psychometric research on this type of assessment procedure. This study was designed to examine the psychometric characteristics of three types of progress measures that are embedded within a commonly used reading intervention program. Results indicated that generalized gains across different oral reading fluency passages predict concurrent gains on common and comprehensive tests of reading fluency, and that immediate instructional gains measured during instruction were significantly different from zero and thus sensitive to intervention effects. Overall findings suggest that at least some embedded assessment procedures demonstrate predictive validity and that these types of procedures have the potential to assist educators with data‐driven instructional decisions about students’ responsiveness to intervention.     

Statistical Power in Evaluations That Investigate Effects on Multiple Outcomes: A Guide for Researchers

Researchers are often interested in testing the effectiveness of an intervention on multiple outcomes, for multiple subgroups, at multiple points in time, or across multiple treatment groups. The resulting multiplicity of statistical hypothesis tests can lead to spurious findings of effects. Multiple testing procedures (MTPs) are statistical procedures that counteract this problem by adjusting p values for effect estimates upward. Although MTPs are increasingly used in impact evaluations in education and other areas, an important consequence of their use is a change in statistical power that can be substantial. Unfortunately, researchers frequently ignore the power implications of MTPs when designing studies. Consequently, in some cases, sample sizes may be too small, and studies may be underpowered to detect effects as small as a desired size. In other cases, sample sizes may be larger than needed, or studies may be powered to detect smaller effects than anticipated. This paper presents methods for estimating statistical power for multiple definitions of statistical power and presents empirical findings on how power is affected by the use of MTPs.     

A primer of research on cognitive strategy instruction: The important issues and how to address them

Five types of strategy research are reviewed. (1) We argue it makes sense first to determine whether there is a need for strategy instruction. If there is, (2) development of a treatment with preliminary evaluations can follow, as can (3) formal evaluation of the resultant intervention in true experiments. As instructional need research, strategy development, and experimental evaluation proceed, two other types of research should be conducted. (4) It is important to study the acceptability of strategy interventions to educators and students. (5) Research on material modifications can provide information about how strategy benefits can be made available to students when strategy instruction is not effective or unlikely to occur. Very little strategy instruction has been evaluated in all five types of research covered here, making obvious the need for more systematic research on strategies. Observational, ethnographic, and experimental methods are required in order to address the many issues comprising comprehensive empirical analysis of any type of strategy instruction, with many recommendations made here about how to conduct informative studies.     

Customizing Instruction to Maximize Functional Outcomes for Students With Profound Multiple Disabilities

This article describes a process for customizing instruction for students with profound multiple disabilities that has been used to design instructional programs and maximize the attainment of functional outcomes for students. The process focuses on collaborative teamwork and problem solving to design and implement instructional programs by ensuring that the 5 components of the process are included. Team members determine (a) prepositioning handling procedures; (b) overall body positioning for instruction; (c) hand, arm, and head positioning; (d) instructional adaptations and materials; and (e) handling procedures to combine with systematic instructional strategies.     

A meta-analysis of single-subject-design intervention research for students with LD

This article summarizes single-subject-design intervention studies that include students with learning disabilities. Effect sizes of 85 studies were analyzed across instructional domains (e.g., reading, mathematics); sample characteristics (e.g., age, intelligence); intervention parameters (e.g., number of instructional sessions, instructional components); and methodological procedures (e.g., internal validity, treatment integrity, sample representation). The major findings were as follows: (a) All domain areas except handwriting yielded effect sizes at or above Cohen's .80 threshold for a substantial finding; (b) instructional components related to drill-repetition-practice-review, segmentation, small interactive groups, and the implementation of cues to use strategies contributed significant variance (15%) to estimates of effect size; (c) strategy instruction (SI) models better predicted effect size estimates than direct instruction (DI) models when the results were qualified by the reported intellectual and reading levels of the participants; (d) high-IQ discrepancy groups yielded lower effect sizes compared to low-IQ discrepancy groups in the domain of reading, whereas the reverse effect occurred when treatment outcomes were not reading measures; and (e) the low-IQ discrepancy groups yielded higher effect sizes for a Combined DI and SI Model when compared to competing models. The results are supportive of the pervasive influence of cognitive strategy and direct instruction models across treatment domains and of the notion that variations in sample definition moderate treatment outcomes.     

Learning the Cell Structures with Three-Dimensional Models: Students’ Achievement by Methods, Type of School and Questions’ Cognitive Level

The cell topic was taught to 9th-grade students in three modes of instruction: (a) students “hands-on,” who constructed three-dimensional cell organelles and macromolecules during the learning process; (b) teacher demonstration of the three-dimensional model of the cell structures; and (c) teaching the cell topic with the regular learning material in an expository mode (which use one- or two-dimensional cell structures as are presented in charts, textbooks and microscopic slides). The sample included 669, 9th-grade students from 25 classes who were taught by 22 Biology teachers. Students were randomly assigned to the three modes of instruction, and two tests in content knowledge in Biology were used. Data were treated with multiple analyses of variance. The results indicate that entry behavior in Biology was equal for all the study groups and types of schools. The “hands-on” learning group who build three-dimensional models through the learning process achieved significantly higher on academic achievements and on the high and low cognitive questions’ levels than the other two groups. The study indicates the advantages students may have being actively engaged in the learning process through the “hands-on” mode of instruction/learning.     

A Randomized Study of a Literacy-Integrated Science Intervention for Low-Socio-economic Status Middle School Students: Findings from first-year implementation

This paper presents the findings from a randomized control trial study of reading/literacy-integrated science inquiry intervention after 1 year of implementation and the treatment effect on 5th-grade low-socio-economic African-American and Hispanic students’ achievement in science and English reading. A total of 94 treatment students and 194 comparison students from four randomized intermediate schools participated in the current project. The intervention consisted of ongoing professional development and specific instructional science lessons with inquiry-based learning, direct and explicit vocabulary instruction, and integration of reading and writing. Results suggested that (a) there was a significantly positive treatment effect as reflected in students' higher performance in district-wide curriculum-based tests of science and reading and standardized tests of science, reading, and English reading fluency; (b) males and females did not differ significantly from participating in science inquiry instruction; (c) African-American students had lower chance of sufficiently mastering the science concepts and achieving above the state standards when compared with Hispanic students across gender and condition, and (d) below-poverty African-American females are the most vulnerable group in science learning. Our study confirmed that even a modest amount of literacy integration in inquiry-based science instruction can promote students' science and reading achievement. Therefore, we call for more experimental research that focus on the quality of literacy-integrated science instruction from which middle grade students, particularly low-socio-economic status students, can benefit.     

Measuring Complex Features of Science Instruction: Developing Tools to Investigate the Link Between Teaching and Learning

There is a growing national recognition that teachers and teaching are at the heart of successful educational reform. However, few tools exist for measuring classroom instruction. The primary purpose of this article is to describe methods we developed to measure and study teaching, specifically while teachers were using a multimedia intervention for promoting scientific problem solving. Lessons were videotaped, and coding schemes were developed to measure 2 aspects of teaching: (a) the lesson's organization, particularly whole-class instruction used to introduce problems and share students' work; and (b) the nature of tasks and questions given to students. Results showed that the coding schemes were reliable and that they detected differences in instruction across teachers. Qualitative analyses were consistent with the quantitative findings. The codes also captured features of teaching that would have been difficult to detect or verify with qualitative observations alone. Finally, we explored how these measures could be used with student outcome data to examine the relationship between teaching and learning in future studies. We argue that quantitative measures of instruction serve many purposes, not the least of which is allowing researchers to explore the relationship between teaching and student learning at a high degree of granularity.     

Selecting Students and Services for Compensatory Education: Lessons from Aptitude--Treatment Interaction Research

In this keynote review, we draw from recent theory and research on individual differences, learning, and instruction to answer questions about the selection of students and the provision of services in compensatory education. In addressing the question regarding provision of services to Chapter 1 students, we focus on five issues: (a) fragmentation versus integration of what is to be learned, (b) time as a variable, (c) the provision of direct instruction and possible alternatives to direct instruction, (d) compensatory education for aptitude processes, and (e) instructional grouping decisions. Current Chapter 1 services may promote fragmentation rather than integration of the academic content for the student. Moreover, Chapter 1 services focus more on teaching basic skills than on promoting higher-order thinking in reading and mathematics. We discuss problems associated with current selection procedures such as misclassification and "labeling" of Chapter 1 students. We conclude with recommendations for educational researchers and policymakers.     

Reciprocal Teaching of Comprehension‐fostering and Comprehension‐monitoring Activities with Ten Primary School Girls

Abstract This paper reports an experimental evaluation of a direct instruction procedure in which children are taught and practise activities to both foster and monitor their comprehension while reading. Three different class groups of low comprehenders, Standard 4, Form 1 and Form 2, were serially exposed to 21 days of 20‐25 minutes instruction in four study activities: summarising, questioning, clarifying and predicting. Instruction was by reciprocal teaching whereby the adult tutor and children took turns leading a dialogue centred around the important topics of the instructional text. The design of this study incorporated both with‐subjects multiple baseline and between‐subjects comparisons, the no treatment comparison groups being average and above average comprehending class peers. Daily comprehension assessment on a different text at the same difficulty level as that used for reciprocal teaching and pre and post intervention scores on a reading comprehension test served as the dependent measures for the study. Results show significant increases in accuracy on comprehension tests for all experimental groups upon introduction of the reciprocal teaching procedure. On completion of the intervention two of these groups scores matched those of the above average controls. Follow‐up probes eight weeks later showed that all groups had maintained their comprehension gains.     

THE RELIABILITY AND USER‐FEASIBILITY OF MATERIALS AND PROCEDURES FOR MONITORING THE IMPLEMENTATION INTEGRITY OF A READING INTERVENTION

Within the realm of school‐based interventions, implementation integrity is important for practical, legal, and ethical purposes. Unfortunately, evidence suggests that proper monitoring of implementation integrity is often absent from both research and practice. School psychology practitioners and researchers have reported that a major barrier to monitoring integrity is a lack of procedural guidance, and currently there is little research that has examined the psychometric reliability of monitoring procedures and materials. Therefore, the purpose of this two‐part study was to examine (a) the extent to which relatively novice educators could self‐learn and successfully use an implementation integrity monitoring system designed to evaluate a structured reading intervention program, and (b) the inter‐observer reliability of two individuals using the system to evaluate the same interventionist. Overall findings suggested that it is feasible for most individuals to learn the implementation integrity monitoring system (and associated materials) and the system can be used reliably across multiple observers. Implications of these findings are discussed, including how the procedures and materials might be adapted for other intervention programs to assist researchers and practitioners with monitoring implementation integrity.     

Principles for the Prevention and Intervention of Mathematics Difficulties

Three levels of prevention and intervention in the area of mathematics are addressed: (a) primary prevention focusing on universal design, (b) secondary prevention focusing on adaptations, and (c) tertiary prevention focusing on intensive and explicit contextualization of skills‐based instruction. The purpose of this paper is to identify and discuss principles of prevention and intervention in the area of mathematics. First, we identify research‐based principles associated with primary prevention. Second, we turn our attention to secondary prevention, with a focus on prereferral intervention. We identify principles that serve to differentiate primary and secondary prevention and specify instructional variables that are promising for use within a secondary prevention mode. Finally, we discuss intervention. We identify principles of effective intervention, which include individually referenced decision making, instructional intensity, and deliberate contextualization of skills‐based instruction.     

Sensitivity of test items to teaching quality

Instructional sensitivity is the psychometric capacity of tests or single items of capturing effects of classroom instruction. Yet, current item sensitivity measures’ relationship to (a) actual instruction and (b) overall test sensitivity is rather unclear. The present study aims at closing these gaps by investigating test and item sensitivity to teaching quality, reanalyzing data from a quasi-experimental intervention study in primary school science education (1026 students, 53 classes, M_age = 8.79 years, SD_age = 0.49, 50% female). We examine (a) the correlation of item sensitivity measures and the potential for cognitive activation in class and (b) consequences for test score interpretation when assembling tests from items varying in their degree of sensitivity to cognitive activation. Our study (a) provides validity evidence that item sensitivity measures may be related to actual classroom instruction and (b) points out that inferences on teaching drawn from test scores may vary due to test composition.     

Motivational vulnerability as a challenge for educational interventions

The focus of this study was on the question, `Are there differences among young (third grade) students who are poor learners in their metacognitive (in)competence and motivational vulnerability that are crucial for better self-regulation?’ This question was studied in an intervention context, where an effort was made to mindfully implement teaching methods that have been found to be effective in promoting strategic learning. The subjects were reading comprehension (in environmental science) and mathematical problem solving. The results suggest there is a need for more coordinated, long-term analyses of transactional, strategy-focused instruction that (a) extends over a long period of time, (b) moves from innovative to more conventional settings as students’ cognitive and motivational competence increases, (c) is based on students’ own, personal and meaningful experiences and competencies, and (d) moves from individual to collaborative, shared and multiple transaction. There also is a need to train students in the use of cognitive and metacognitive strategies, emotional coping, and motivation strategies that promote self-regulation.     

Bringing Research‐Based Practice in Reading Intervention to Scale

Multiple consensus reports have provided converging evidence regarding effective instruction for students who have difficulty learning to read. Evidence‐based instruction in general education classrooms must be in place in order to implement response‐to‐intervention models. Despite the well‐developed knowledge base supporting the value of interventions that have been demonstrated to have positive outcomes, these interventions are not widely employed in typical classroom instruction, and models of service delivery for students with reading and learning disabilities implemented in schools are often ineffective. Recent research has demonstrated that this need not be the case, but there are many obstacles to change. Large‐scale implementation of effective educational practices for struggling readers depends on a research agenda that directly addresses questions related to scaling and sustaining educational innovations. We suggest that reform depends on collaboration among researchers, educational practitioners, teacher educators, and policymakers, with the common goal of improving outcomes for students who might otherwise experience reading failure.     

Teaching Students with Emotional and Behavioral Disorders to Self-advocate through Persuasive Writing

We examined the effectiveness of the Self-Regulated Strategy Development (SRSD) model of writing instruction with a self-determination training component for middle school-age students with emotional and behavioral disorders. We randomly assigned students to experimental or comparison treatments during which special education teachers provided the intervention. Students in the experimental groups received instruction on how to plan and write persuasive essays, were trained on self-determination skills, and were taught how to use persuasive writing to self-advocate. Students in the comparison condition received writing instruction with the established school writing curriculum. Instruction for both groups lasted 33 days, four days a week during 30-minute sessions. Experimental students significantly outperformed comparison students at posttest in all the persuasive essay-writing components assessed, in their ability to recall the parts of a persuasive essay, in the self-efficacy measure, and on self-determination knowledge. Experimental students were able to maintain gains in almost all writing measures and were able to generalize to content areas, although comparison students slightly increased in number of words. Student and teacher interviews revealed an overall satisfaction with SRSD procedures and the results.     

Longitudinal Studies of Alternative Types of Early Intervention: Rationale and Design

In 1985, the US. Department of Education awarded a competitively bid contract to Utah State University to conduct a series of longitudinal studies on the effects and costs of alternative types of early intervention programs for children having disabilities. Issues addressed by this set of studies included the effects and costs of varying (a) the intensity of intervention, (b) the age at which intervention is begun, and (c) the ways in which parents are involved in intervention programs. This article summarizes the historical context in which these studies were implemented, the rationale for conducting this type of study, and the procedures incorporated into the design of each study to ensure that the results would be useful to practitioners and other researchers.     

Improving instruction of future teachers: A multimedia approach that supports implementation of evidence-based vocabulary practices

Teacher education efforts must prepare future teachers to implement evidence-based practices that are effective for all students. A strand of practices necessary for reading competence is vocabulary instruction; however, teachers spend less time teaching vocabulary as opposed to other literacy skills. In this experimental study, researchers investigated a multimedia-based intervention, which pairs video with a Content Acquisition Podcast (i.e., video plus CAP) to teach preservice teachers (N = 49) to implement vocabulary practices with struggling students. Those who watched the video plus CAP used significantly more teaching behaviors associated with an evidence-based vocabulary practice during instruction than the comparison group.