Monitoring Reading Growth: Goal Setting,Measurement Frequency,and Methods of Evaluation

This study examined effects of goal setting, frequency of progress monitoring, and method of evaluating progress on the number of instructional change prompts generated by reading‐curriculum‐based measurements (CBM). Participants were 31 high‐incidence special education students in grades 2–6 selected from 8 schools. Following baseline measurements, researchers administered grade‐level CBM passages every 2 weeks for 8 weeks. Comparisons included 3 goal levels (0.5, 1.0, or 1.5 word growth per week), 3 monitoring/evaluation schedules (every 2, 4, or 8 weeks), and 2 methods for evaluating progress (points below the aim line or slope). Overall, more ambitious goals, less frequent monitoring, and using slope to evaluate progress generated more instructional change prompts. Results favored less frequent monitoring and using slope to evaluate progress.     

General education teacher judgments of their low‐performing students' short‐term reading progress

This study investigated the accuracy of classroom teachers' judgments of the reading progress of their low‐performing students. Participants were 36 second grade teachers and students in their lowest reading groups (n = 150). Student progress was monitored weekly using reading‐curriculum‐based measurement (R‐CBM) procedures. After 6 weeks, teachers were asked to rate their students' progress. Expert judges later reviewed the teachers' R‐CBM graphs and rated the individual and group progress based on the graphs. Teacher ratings did not correlate with expert ratings or the R‐CBM slope estimates. Expert ratings correlated highly with slope estimates. Teachers' estimates of progress were significantly higher than expert judges' ratings, indicating that teachers may overestimate student progress. Implications for practice and future research are discussed. © 2008 Wiley Periodicals, Inc.     

American Students’ Perspectives on Alternative Assessment: do they know it's different?

This study used the 1993 California Learning Assessment System (CLAS) Middle Grades Mathematics Performance Assessment as a platform to examine alternative assessment in actual practice in the United States. Reported here is information gathered using the CLAS regarding student attitudes and approaches toward this new type of assessment. At issue is whether students find alternative assessments to be more motivating and interesting than traditional types of tests, and whether they appreciate the difference between traditional and alternative tasks. Data were collected in 13 schools across the state of California, involving more than 800 students. Instrumentation used in data collection included student surveys as well as in‐depth student retrospective interviews. Findings suggest that students do indeed understand the differences in approaches necessitated by novel, open‐ended versus more familiar multiple‐choice tasks. In addition, student attitudes toward these two types of tasks are discussed in detail.     

Parent‐implemented reading interventions within a response to intervention framework

This study tested the effects of parent‐implemented reading interventions on four elementary students’ reading fluency. Student participants had been receiving a Tier 2 reading intervention, but they were not responding favorably to the Tier 2 intervention. A consultant conducted brief experimental analyses of reading interventions and identified a Tier 3 intervention for each student. Then, the consultant trained the students’ mothers to implement the interventions at home. The interventions were tested via a multiple baseline design across students. During the intervention phase, the consultant monitored parents’ treatment integrity and assessed students’ oral reading fluency for novel, grade‐level progress monitoring passages. Visual analysis indicates that all four students demonstrated increases in oral reading fluency for instructional passages and novel progress monitoring passages after intervention implementation. In addition, single‐case design effect sizes indicate strong effects for instructional passages for all four students, moderate effects for novel progress monitoring passages for two students, and strong effects for novel progress monitoring passages for two students. Finally, parents implemented interventions with moderate to high integrity, and parents rated the interventions as acceptable. Results are discussed in terms of implications for research and practice as they relate to parent‐implemented interventions within a response to intervention framework.     

Students Demonstrating Persistent Low Response to Reading Intervention: Three Case Studies

Three students with severe reading disabilities participated in a 3‐year intervention provided by trained tutors from first grade through third grade (175–231 hours). All three students demonstrated very low response to initial levels of intervention and were provided adjusted intensive intervention in an attempt to remedy their low response to instruction. Findings revealed that two of the three students realized limited growth and did not accelerate progress toward reducing the gap between their performance and the expected grade level performance. One student accelerated growth considerably during third grade and made accelerated progress toward grade level expectations. Information on intervention implementation and student progress as well as implications for practice and further research are discussed.     

Reading-growth estimates for elementary-school students using curriculum-based measurement

In this study, we examine grade-level growth rates for general education students and students with learning disabilities in grades two to six. In conducting the study, we demonstrate how schools, districts, and state educational agencies can use a combination of Curriculum-Based Measurement and Hierarchical Linear Modeling (HLM) methods to develop growth-rate norms in reading. The participants were made up of 273 general education students and 430 students with learning disabilities. The growth rates for these two groups of students in each grade were estimated using HLM. Within each grade, separate growth rates for subgroups of general education students (i.e., high, average, and low achievers) were estimated. The uses of estimated growth rates for setting year-end goals, monitoring student progress, and evaluating the effectiveness of instructional programs are also discussed.     

Student‐directed learning strategies to promote the progress of students with intellectual disability in inclusive classrooms

Student‐directed learning strategies involve teaching students to modify and regulate their own behaviour. Teaching students to self‐regulate behaviour has been identified as one means of promoting self‐determination, and promoting and enhancing self‐determination has become a valued outcome internationally. In addition, however, student‐directed learning strategies could prove important for including students with disabilities in general education classrooms by reducing the student's dependency upon others in the classroom setting. Four students with intellectual disability were supported to select an educational goal on which they wanted to work and were taught to self‐monitor their progress toward that goal in general education classrooms. Their progress toward the goal was evaluated through data collection activities and by using the Goal Attainment Scaling process. In general, students achieved their self‐selected goals. The potential contribution of student‐directed learning strategies to inclusion and promoting self‐determination are discussed.     

Dynamics of undergraduate student generic problem-solving skills captured by a campus-wide study

The ability to effectively problem solve is a highly valued competency expected of university graduates, independent of their area of study. Evaluation of problem-solving skill (PSS) development is hindered by a shortage of available tools for monitoring student progress and by lack of defined instructional strategies for development of these skills. Our research is aimed at addressing these problems. We have developed an evaluation tool, which we applied to study the dynamics of undergraduate student PSS. We tested first- and upper-year students from 26 different courses (total enrollment of 2229 students). Overall improvement of PSS was detected for the first-year students over their first term of study. There were no significant differences between the PSS of first- and upper-year students, and no improvement was detected over a single term by measuring PSS in upper-year students. Only three courses were effective at facilitating PSS. Our data indicate that most of the standard lecture approaches do not develop undergraduate student PSS and that universities and individual instructors must take active steps to advance this critical skill set in university students.     

Sample Exclusion and Student Attrition Effects in the Longitudinal Study of Middle School Mathematics Performance

Achievement data from a longitudinally matched student cohort from a large school district in the southwestern United States were analyzed to investigate sample exclusion and student attrition effects on estimates of student, school, and district mathematics performance. Use of 2- and 3-level longitudinal growth models to estimate the growth trajectories of middle school students revealed that mathematics performance differed across 2 sample conditions. Relative to the achievement outcomes associated with a sample that included all students from the longitudinal cohort, district and school achievement were generally higher and student group performance more similar in the smaller, more advantaged student sample used for district accountability reporting. Further investigation of the school performance estimates showed that cross-sample changes in student achievement outcomes were closely related to the proportion of students from special student populations who were excluded from the district accountability sample. The achievement differences and the differential patterns of association demonstrated in this study suggest that conclusions drawn about district and school performance and relationships between student characteristics and student achievement outcomes may depend to some degree on which students are included in an analytic sample. Investigators seeking to take advantage of longitudinal designs in school effectiveness research are cautioned to closely examine their data for nonrandom student attrition and document the impact of sample exclusion and student attrition effects in the research and accountability reports that are produced from longitudinal data sets.     

Observations of the middle school environment: The context for student behavior beyond the classroom

This article describes the use of an observation system to measure middle school staff practices, environment characteristics, and student behavior in the school common areas. Data were collected at baseline from 18 middle schools participating in a randomized controlled trial of school‐wide Positive Behavior Support. The observations were reliable and showed sensitivity to differences between school settings and between schools. Multilevel models with students nested in schools were used to examine the associations of staff practices and the school environment with student behavior. Less effective behavior management and more staff criticism, graffiti, and percentage of low‐income students were associated with student problem behaviors. Greater use of effective behavior management and positive attention, and fewer low‐income students were associated with positive student behavior. The use of data‐based feedback to schools for intervention planning and monitoring is illustrated. Implications for school‐wide efforts to improve student behavior in middle schools are discussed. © 2011 Wiley Periodicals, Inc.     

Probabilities and Predictions: Modeling the Development of Scientific Problem-Solving Skills

The IMMEX (Interactive Multi-Media Exercises) Web-based problem set platform enables the online delivery of complex, multimedia simulations, the rapid collection of student performance data, and has already been used in several genetic simulations. The next step is the use of these data to understand and improve student learning in a formative manner. This article describes the development of probabilistic models of undergraduate student problem solving in molecular genetics that detailed the spectrum of strategies students used when problem solving, and how the strategic approaches evolved with experience. The actions of 776 university sophomore biology majors from three molecular biology lecture courses were recorded and analyzed. Each of six simulations were first grouped by artificial neural network clustering to provide individual performance measures, and then sequences of these performances were probabilistically modeled by hidden Markov modeling to provide measures of progress. The models showed that students with different initial problem-solving abilities choose different strategies. Initial and final strategies varied across different sections of the same course and were not strongly correlated with other achievement measures. In contrast to previous studies, we observed no significant gender differences. We suggest that instructor interventions based on early student performances with these simulations may assist students to recognize effective and efficient problem-solving strategies and enhance learning.     

The influence of test‐based accountability policies on early elementary teachers: School climate,environmental stress,and teacher stress

This study examined the potential influence of test‐based accountability policies on school environment and teacher stress among early elementary teachers. Structural equation modeling of data from 541 kindergarten through second grade teachers across three states found that use of student performance on high‐stakes tests to evaluate teachers indirectly was related to teachers’ professional investment via test stress in the environment. Although students in kindergarten through second grade do not take high‐stakes assessments, early elementary teachers reported high levels of stress associated with test‐based accountability policies. This study provides data across multiple states that test‐based accountability policies may have negative influences on school environment and teacher stress among early elementary teachers. Implications for practice and research are discussed.     

It's not which school but which set you're in that matters: the influence of ability grouping practices on student progress in mathematics

The mathematics achievement of a cohort of 955 students in 42 classes in six schools in London was followed over a 4‐year period, until they took their General Certificate of Secondary Education examinations (GCSEs) in the summer of 2000. All six schools were regarded by the Office for Standards in Education (Ofsted) as providing a good standard of education, and all were involved in teacher training partnerships with universities. Matched data on Key Stage 3 test scores and GCSE grades were available for 709 students, and these data were analysed in terms of the progress from Key Stage 3 test scores to GCSE grades. Although there were wide differences between schools in terms of overall GCSE grades, the average progress made by students was similar in all six schools. However, within each school, the progress made during Key Stage 4 varied greatly from set to set. Comparing students with the same Key Stage 3 scores, students placed in top sets averaged nearly half a GCSE grade higher than those in the other upper sets, who in turn averaged a third of a grade higher than those in lower sets, who in turn averaged around a third of a grade higher than those students placed in bottom sets. In the four schools that used formal whole‐class teaching, the difference in GCSE grades between top and bottom sets, taking Key Stage 3 scores into account, ranged from just over one grade at GCSE to nearly three grades. At the schools using small‐group and individualized teaching, the differences in value‐added between sets were not significant. In two of the schools, a significant proportion of working‐class students were placed into lower sets than would be indicated by their Key Stage 3 test scores.     

CONSISTENCY AND MAGNITUDE OF DIFFERENCES IN READING CURRICULUM‐BASED MEASUREMENT SLOPES IN BENCHMARK VERSUS STRATEGIC MONITORING

Differences in oral reading curriculum‐based measurement (R‐CBM) slopes based on two commonly used progress monitoring practices in field‐based data were compared in this study. Semester‐specific R‐CBM slopes were calculated for 150 Grade 1 and 2 students who completed benchmark (i.e., 3 R‐CBM probes collected 3 times per year) and strategic (i.e., one R‐CBM probe collected monthly) assessments. Slopes based on two adjacent benchmark assessments were positively correlated with slopes based on three monthly strategic assessments in the spring semester of Grade 1 but not in either Grade 2 semester, and significant differences were found between the slopes in all semesters. Consistent with another study showing that slopes are overestimated when single probes are administered per occasion, slopes were larger when based on strategic versus benchmark data in the current study, and the average discrepancies between slopes were greater‐than‐expected growth rates in all semesters. The current findings, based on field‐based data, illustrate the impact of variations in commonly used progress monitoring procedures on the precision of calculated slope estimates.     

Development and calibration of the student opportunities for deeper learning instrument

This article describes the development and calibration of the student opportunities for deeper learning instrument (SODLI). The SODLI is designed to measure the amount of learning opportunities intended to support higher‐order thinking that students are exposed to through student self‐report, and is expected to be widely useful in psychological research in schools. Here, the psychometric functioning of the SODLI is investigated in a relatively large sample of U.S. high‐school students (N = 963). Using multidimensional item‐response theory methods, a nine‐factor correlated model is determined to fit the data best, and to produce reliable estimates of SODLI dimensions. The SODLI is also shown to exhibit scalar‐invariant measurement properties across sex, race/ethnicity, and language‐background groups, although latent mean differences on some of the SODLI dimensions across those groups were identified.     

Monitoring elementary students’ writing progress using curriculum-based measures: grade and gender differences

The purpose of this study was to examine slopes from curriculum-based measures of writing (CBM-W) as indicators of growth in writing. Responses to story prompts administered for 5 min to 89 students in Grades 2–5 were collected across 12 weeks and scored for correct word sequences (CWS) and correct minus incorrect sequences (CIWS). Linear mixed modeling revealed that, for students in Grades 2–3, a linear model with random effects on both intercept and slope fit the data best. For students in Grades 4–5, growth trends varied depending on number of weeks and scoring procedure used. The time point at which slopes were significantly different from zero varied by scoring procedure and grade. Gender was related to intercept and slope for CWS and CIWS in Grades 2–3 and to intercept and linear slope for CWS and CIWS in Grades 4–5. Findings suggest that CBM-W may be appropriate for monitoring student progress, and that gender should be considered in data-based decision making.     

An Evaluation of the Linearity of Curriculum‐Based Measurement of Oral Reading (CBM‐R) Progress Monitoring Data: Idiographic Considerations

Educators use curriculum‐based measurement of oral reading (CBM‐R) to measure student response to instruction. Current decision‐making frameworks assume students demonstrate linear growth across a school year. However, growth appears nonlinear for at least a subgroup of students. We assessed the degree to which grade two (n = 800) and grade three (n = 800) students receiving intensive interventions experienced discontinuous growth. We also explored when discontinuous growth tended to occur, and whether students improved or regressed afterward. Results indicate linear patterns were adequate for most students (80 percent). Students who showed discontinuous growth early tended to improve afterward. Conversely, students who showed discontinuous growth later tended to plateau. Findings suggest multilevel models may obscure variability in growth patterns. Practice and research implications are discussed.     

A Framework for Measuring the Amount of Adaptation of Rasch‐based Computerized Adaptive Tests

A key consideration when giving any computerized adaptive test (CAT) is how much adaptation is present when the test is used in practice. This study introduces a new framework to measure the amount of adaptation of Rasch‐based CATs based on looking at the differences between the selected item locations (Rasch item difficulty parameters) of the administered items and target item locations determined from provisional ability estimates at the start of each item. Several new indices based on this framework are introduced and compared to previously suggested measures of adaptation using simulated and real test data. Results from the simulation indicate that some previously suggested indices are not as sensitive to changes in item pool size and the use of constraints as the new indices and may not work as well under different item selection rules. The simulation study and real data example also illustrate the utility of using the new indices to measure adaptation at both a group and individual level. Discussion is provided on how one may use several of the indices to measure adaptation of Rasch‐based CATs in practice.     

Inquiry‐based science in the middle grades: Assessment of learning in urban systemic reform

Science education standards established by American Association for the Advancement of Science (AAAS) and the National Research Council (NRC) urge less emphasis on memorizing scientific facts and more emphasis on students investigating the everyday world and developing deep understanding from their inquiries. These approaches to instruction challenge teachers and students, particularly urban students who often have additional challenges related to poverty. We report data on student learning spanning 3 years from a science education reform collaboration with the Detroit Public Schools. Data were collected from nearly 8,000 students who participated in inquiry‐based and technology‐infused curriculum units that were collaboratively developed by district personnel and staff from the University of Michigan as part of a larger, district‐wide systemic reform effort in science education. The results show statistically significant increases on curriculum‐based test scores for each year of participation. Moreover, the strength of the effects grew over the years, as evidenced by increasing effect size estimates across the years. The findings indicate that students who historically are low achievers in science can succeed in standards‐based, inquiry science when curriculum is carefully developed and aligned with professional development and district policies. Additional longitudinal research on the development of student understanding over multiple inquiry projects, the progress of teacher enactment over time, and the effect of changes in the policy and administrative environment would further contribute to the intellectual and practical tools necessary to implement meaningful standards‐based systemic reform in science. © 2004 Wiley Periodicals, Inc. J Res Sci Teach 41: 1063–1080, 2004     

Differences Between Self-Adapted and Computerized Adaptive Tests: A Meta-Analysis

Self-adapted testing has been described as a variation of computerized adaptive testing that reduces test anxiety and thereby enhances test performance. The purpose of this study was to gain a better understanding of these proposed effects of self-adapted tests (SATs); meta-analysis procedures were used to estimate differences between SATs and computerized adaptive tests (CATs) in proficiency estimates and post-test anxiety levels across studies in which these two types of tests have been compared. After controlling for measurement error, the results showed that SATs yielded proficiency estimates that were 0.12 standard deviation units higher and post-test anxiety levels that were 0.19 standard deviation units lower than those yielded by CATs. We speculate about possible reasons for these differences and discuss advantages and disadvantages of using SATs in operational settings.