How Can We Improve the Accuracy of Screening Instruments?

Screening for early reading problems is a critical step in early intervention and prevention of later reading difficulties. Evaluative frameworks for determining the utility of a screening process are presented in the literature but have not been applied to many screening measures currently in use in numerous schools across the nation. In this study, the accuracy of several Dynamic Indicators of Basic Early Literacy Skills (DIBELS) subtests in predicting which students were at risk for reading failure in first grade was examined in a sample of 12,055 students in Florida. Findings indicate that the DIBELS Nonsense Word Fluency, Initial Sound Fluency, and Phoneme Segmentation Fluency measures show poor diagnostic utility in predicting end of Grade 1 reading performance. DIBELS Oral Reading Fluency in fall of Grade 1 had higher classification accuracy than other DIBELS measures, but when compared to the classification accuracy obtained by assuming that no student had a disability, suggests the need to reevaluate the use of classification accuracy as a way to evaluate screening measures without discussion of base rates. Additionally, when cut scores on the screening tools were set to capture 90 percent of all students at risk for reading problems, a high number of false positives were identified. Finally, different cut scores were needed for different subgroups, such as English Language Learners. Implications for research and practice are discussed.     

TAKING A “SIMPLE VIEW” OF THE DYNAMIC INDICATORS OF BASIC EARLY LITERACY SKILLS AS A PREDICTOR OF MULTIPLE MEASURES OF THIRD‐GRADE READING COMPREHENSION

The purpose of this study was to examine the validity evidence of first‐grade Dynamic Indicators of Basic Early Literacy Skills (DIBELS) scores for predicting third‐grade reading comprehension scores. We used the “simple view” of reading as the theoretical foundation for examining the extent to which DIBELS subtest scores predict comprehension through both word recognition and language comprehension. Scores from the DIBELS Oral Reading Fluency (ORF) subtest, a measure of word recognition speed and accuracy, strongly and significantly predicted multiple measures of reading comprehension. No other DIBELS subtest score explained additional variance beyond DIBELS ORF. Although experimental DIBELS Word Use Fluency (WUF) was significantly correlated with a language comprehension measure and measures of reading comprehension, WUF scores did not predict reading comprehension beyond ORF scores. Alternatively, first‐grade Peabody Picture Vocabulary Test scores did predict additional, significant variance in reading comprehension, beyond DIBELS ORF.     

Does the Dynamic Indicators of Basic Early Literacy Skills Next Assessment Take a “Simple View” of Reading?

This study involved examination of the validity evidence of the Dynamic Indicators of Basic Early Literacy Skills–Next Edition (DIBELS Next) for a sample of 85 third- and fifth-grade students, in reference to the “simple view” of reading. Tests administered included DIBELS Next, Peabody Picture Vocabulary Test–IV (PPVT–IV), Group Reading Assessment and Diagnostic Evaluation (GRADE), and the New York State English Language Arts (NYSELA) test. DIBELS Next Oral Reading Fluency (ORF) and Daze scores were significantly correlated with GRADE and NYSELA scores. Daze scores also explained significant variance in GRADE scores beyond ORF only at third grade and not NYSELA scores at either grade. PPVT–IV scores explained significant variance in GRADE scores beyond ORF at both grades but only NYSELA at third grade. GRADE Vocabulary explained variance in both measures of comprehension and at both grade levels. Research and educational implications of these findings are discussed.     

Predictive Validity of Curriculum-Based Measures for English Learners at Varying English Proficiency Levels

This study examined the predictive validity of curriculum-based measures in reading for Spanish-speaking English learners (ELs) at various levels of English proficiency. Third-grade Spanish-speaking EL students were screened during the fall using DIBELS Oral Reading Fluency (DORF) and Daze. Predictive validity was examined in relation to spring reading outcomes on a statewide assessment. Overall, results revealed that DORF was a stronger predictor of reading outcomes than was Daze. Although Daze was a significant predictor when examined individually, it did not explain significant additional variance beyond DORF. There was not a significant difference in the predictive validity of DORF or Daze for students of varying English proficiency levels. However, the predictive accuracy of DORF and Daze cut-scores varied by English proficiency levels. The results suggest that schools may consider minimizing assessment time by only using DORF when screening.     

First-Grade Spelling Scores Within the Dynamic Indicators of Basic Early Literacy Skills (DIBELS) Screening: An Exploratory Study

The purpose of this study was to examine the validity evidence of first-grade spelling scores from a standardized test of nonsense word spellings and their potential value within universal literacy screening. Spelling scores from the Test of Phonological Awareness: Second Edition PLUS for 47 first-grade children were scored using a standardized procedure and an alternative invented spelling procedure. Correlations were examined among spelling and diagnostic word reading and decoding scores, along with scores from the Dynamic Indicators of Basic Early Literacy Skills (DIBELS). Spelling scores were significantly correlated with word reading and decoding scores, as well as DIBELS scores, except Phoneme Segmentation Fluency. Hierarchical multiple regression analyses revealed that spelling scores reliably accounted for significant variance in decoding but not word reading scores, beyond DIBELS scores. Implications are discussed related to the potential value of including early spelling scores within universal literacy screening.     

COMPARISON OF PREDICTIVE VALIDITY AND DIAGNOSTIC ACCURACY OF SCREENING MEASURES OF READING SKILLS

Assessment data must be valid for the purpose for which educators use them. Establishing evidence of validity is an ongoing process that must be shared by test developers and test users. This study examined the predictive validity and the diagnostic accuracy of universal screening measures in reading. Scores on three different universal screening tools were compared for nearly 500 second‐ and third‐grade students attending four public schools in a large urban district. Hierarchical regression and receiver operating characteristic curves were used to examine the criterion‐related validity and diagnostic accuracy of students’ oral reading fluency (ORF), Fountas and Pinnell Benchmark Assessment System (BAS) scores, and fall scores from the Measures of Academic Progress for reading (MAP). Results indicated that a combination of all three measures accounted for 65% of the variance in spring MAP scores, whereas a reduced model of ORF and MAP scores predicted 60%. ORF and BAS scores did not meet standards for diagnostic accuracy. Combining the measures improved diagnostic accuracy, depending on how criterion scores were calculated. Implications for practice and future research are discussed.     

Is fluency‐based story retell a good indicator of reading comprehension?

This study examined the reliability and validity of scores on a fluency‐based measure of reading comprehension. The Dynamic Indicators of Basic Early Literacy Skills (DIBELS; 6th ed. revised) Retell Fluency (RTF), Oral Reading Fluency (DORF), and Woodcock Johnson III NU Tests of Achievement (WJ‐III NU ACH) Reading Comprehension measures were administered to fourth‐grade students. Results indicated a large difference between real time and recorded retell fluency scores for each passage. In addition, students' retell fluency scores had a low correlation with their reading comprehension scores. In light of these findings, practitioners may want to exercise caution in using fluency‐based story‐retell scores as a measure of reading comprehension. © 2011 Wiley Periodicals, Inc.     

Predicting ELL students’ beginning first grade English oral reading fluency from initial kindergarten vocabulary, letter naming, and phonological awareness skills

The precursors of early English reading success have been widely studied for native English-speaking students, and those findings have been generalized to the English language learner (ELL) student population. However, the development of English language acquisition may be different for ELL students. The purpose of this study was to investigate the predictive role of English letter naming fluency, initial sound fluency, and vocabulary skills at the time of kindergarten entry for first grade English oral reading fluency and to examine the variability in language and literacy skills of ELL students by their demographic characteristics. The data for this study came from the Progress Monitoring and Reporting Network (PMRN), and were collected from Florida's Reading First schools. Letter Naming Fluency, Initial Sound Fluency, and Oral Reading Fluency components of Dynamic Indicators of Basic Early Literacy Skills (DIBELS) and the Peabody Picture Vocabulary Test were used as measures. Hierarchical Linear Modeling was used to analyze the curvilinear growth of ELL students’ first grade oral reading fluency. The results of this study revealed that kindergarten English letter naming fluency was the best predictor and vocabulary skills were the second best predictor of oral reading fluency in the first grade, followed by initial sound fluency. On average, male ELL students compared to female ELL students, ELL students eligible for free or reduced price lunch eligibility (FRPL) compared to those not eligible for FRPL, and Hispanic ELL students compared to White ELL students read fewer words at the beginning of the first grade and showed a slower growth rate. English oral reading fluency scores of Asian ELL students were the highest.     

Examining the accuracy of teachers' judgments of DIBELS performance

The purpose of this investigation was to examine the accuracy of teachers' judgments of students' early literacy skills and to determine if students' achievement levels influenced teachers' judgments. Typical and lower‐achieving kindergarten and first‐grade students' scores on the Nonsense Word Fluency and Phoneme Segmentation Fluency measures of the Dynamic Indicators of Basic Early Literacy Skills (DIBELS) were compared to the predicted performance by their teachers. Results showed a moderately strong correlation between teachers' judgments and students' performance across all of the students, which is consistent with past research. Teachers' judgments, however, consistently and significantly overestimated the actual performance of students, particularly those who were typically performing students. The findings of the investigation suggest that relying on teachers' judgments of students' early literacy skills alone may be insufficient to accurately identify students at risk for reading difficulties. © 2011 Wiley Periodicals, Inc.     

EXAMINING POTENTIAL BIAS IN SCREENING MEASURES FOR MIDDLE SCHOOL STUDENTS BY SPECIAL EDUCATION AND LOW SOCIOECONOMIC STATUS SUBGROUPS

To provide timely and effective supports for students reading below grade level, schools require methods for quickly and accurately identifying those students in need. One method for identifying those students is through universal screening. Assessments such as oral reading fluency (ORF) and Maze reading comprehension are commonly used as screening assessments in middle grades. The current study examined ORF and Maze for evidence of bias across two subgroups known to be at increased risk for failure in reading: (a) students with learning disabilities and (b) students from low‐income households. Data from 4,215 students in the sixth (n = 1,126), seventh (n = 1,361), and eighth grades (n = 1,728) were analyzed. Results indicate no significant differences in predictive validity for students from low‐income households compared to students from middle and upper income households. For students with learning disabilities only 8th grade scores showed any evidence of bias compared to students without diagnosed disabilities. Implications and recommendations for future research are discussed.     

Oral reading fluency as a screening tool with English learners: A systematic review

Students learning to read in more than one language are a growing population in schools. Early screening and identification of reading difficulties may lead to better outcomes for all students. Oral reading fluency (ORF) is one tool shown to be both a reliable measure of reading and an accurate method to identify students at risk for poor reading outcomes. This study synthesized available validity evidence for ORF with biliterates. All included studies (n = 31) were conducted with English language learners in grades K–8. Results suggested that although ORF is correlated with reading outcomes, the accuracy of ORF to identify English learners at risk of poor reading outcomes does not meet criteria. The strength of validity evidence differed by language proficiency of participants. Finally, the quality of the reviewed studies was low in critical areas.     

The use of a dynamic screening of phonological awareness to predict risk for reading disabilities in kindergarten children

This study examined the usefulness and predictive validity of a dynamic screening of phonological awareness in two samples of kindergarten children. In one sample (n = 90), the predictive validity of the dynamic assessment was compared to a static version of the same screening measure. In the second sample (n = 96), the dynamic screening measure was compared to a commonly used screening tool, Dynamic Indicators of Basic Early Literacy Skills Initial Sound Fluency. Results showed that the dynamic screening measure uniquely predicted end-of-year reading achievement and outcomes in both samples. These results provide preliminary support for the usefulness of a dynamic screening measure of phonological awareness for kindergarten students.     

USING CURRICULUM‐BASED MEASUREMENT TO EXAMINE SUMMER LEARNING LOSS

Summer loss of reading is a potential factor in maintaining, and potentially widening, the achievement gap. This study used curriculum‐based measurement of reading (R‐CBM) to investigate the effect of the summer on reading. For this study, 317 students in Grades 2 to 5 were assessed in the spring and fall using Dynamic Indicators of Basic Early Literacy Skills Oral Reading Fluency (ORF) measure. Repeated measures analysis of variance modeled overall change in ORF scores by grade, family income, ethnic minority status, English language learner status, and special education (SPED) status. Students in Grades 2 and 3 evidenced overall summer loss, whereas students in Grades 4 and 5 did not exhibit summer loss. In addition, students in Grade 2 showed differential loss based on family income and SPED status. These results support the broadening application of R‐CBM and add to the summer loss literature. Findings are discussed relative to prevention and intervention.     

Relationship between second grade oral reading fluency and third grade reading

Oral reading fluency is a critical feature and outcome of early literacy instruction and it has amassed great attention as a powerful predictor of success at all levels of schooling. We examined relationships between second grade oral reading fluency scores and third grade end-of-grade reading achievement scores for students (N = 9562) in a large school district in the United States. Stakeholder interviews were also conducted. In our model, oral reading fluency and reading comprehension scores were moderately correlated; oral reading fluency was the strongest predictor of subsequent achievement, followed by ethnicity; growth on oral reading fluency was not strongly associated with end-of-grade performance; and greater than 90% of students classified as ‘at risk’ on benchmark assessments performed poorly on third grade assessments. Stakeholders believed that oral reading fluency testing was not necessary above selected levels of proficiency. We discuss implications of our findings for future research and practice.     

Predicting reading achievement for Kindergarten boys and girls

This study assessed the validity of the Kindergarten Teacher Rating Scale (KTRS) in predicting reading achievement for male and female students. The KTRS was a significant predictor of reading achievement for both boys and girls; differential predictive validity for boys and girls was not found. The KTRS explained about 30% of the variance in reading achievement both at the end of the 1st grade and the beginning of 2nd grade. The proportion of variance in reading achievement explained by variance in KTRS scores was significantly greater than the proportion of variance in reading achievement explained by variance in reading readiness scores. There were no significant differences in the mean KTRS scores for male and female students.     

Measuring the dimensions of alphabetic principle on the reading development of first graders: the role of automaticity and unitization

This article presents critical issues related to word reading development within Ehri's theoretical context by focusing primarily on the relation of decoding skill (of nonwords) to word reading and the development of unitization. Within this context, issues and considerations related to research, measurement, and reading development are presented from research and field-based perspectives. Analyses examining the relation between a measure of alphabetic principle, Nonsense Word Fluency (NWF), and fluency with connected text, Oral Reading Fluency (ORF), across first grade demonstrated a linear relation between the measures, which was attenuated for students initially above criterion on the NWF measure. A new scoring approach to the NWF measure is presented to capture initial unitization development and was found to account for unique variance in initial status and growth on ORF and provide instructionally relevant information on the nature of developing alphabetic principle skills. Considerations for future research and school-based applications are provided.     

STABILITY AND CHANGE OF BEHAVIORAL AND EMOTIONAL SCREENING SCORES

Universal screening for behavioral and emotional difficulties is integral to the identification of students needing early intervention and prevention efforts. However, unanswered questions regarding the stability of screening scores impede the ability to determine optimal strategies for subsequent screening. This study examined the 2‐year stability of behavioral and emotional risk screening scores and investigated whether change could be predicted based on student characteristics or initial risk scores. As part of a district‐wide screening effort, 863 middle and high school students completed the Behavioral and Emotional Screening System at two time points. Stability coefficients were moderate, with the majority of students remaining in a similar risk category across time. Gender, race/ethnicity, socioeconomic status, grade, school transition, and special education status were not predictive of movement across time. Initial risk score was predictive of movement from normal to at‐risk categorization, with the internalizing domain being the most predictive of change.     

Identifying Writing Difficulties in First Grade: An Investigation of Writing and Reading Measures

Early identification of students who are at risk for writing difficulties is an important first step in improving writing performance. First grade students (N = 150) were administered a set of early writing measures and reading measures in January. Sentence Writing Quality and Oral Reading Fluency demonstrated strong classification accuracy when a Teacher Rating was used to identify which students had writing difficulties (AUC > .90), and the combined measures yielded sensitivity and specificity indices exceeding 0.90. The combined reading and writing measures yielded AUCs of a higher magnitude than those associated with individual measures when norm‐referenced writing subtests were used to identify which students had writing difficulties (AUC in .80 range). The findings suggest that it may be possible to accurately identify which students may be at risk for writing difficulties within first grade.     

Practical issues in field based testing of oral reading fluency at upper elementary grades

In this series of studies, we explore the ideal frequency, duration, and relative effectiveness of measuring oral reading fluency. In study one, a sample of 389 fifth graders read out loud for 1 min and then took a traditional state-level standardized reading test. Results suggest administering three passages and using the median yields the highest predictive validity. Study two compared oral reading fluency rates at 30 and 60 s for 815 elementary and middle school students on the same passage. Results indicate that the 30 s measures yield a comparable score. Study three found relatively similar predictive validity of oral reading fluency for 67 fourth- and 125 sixth-grade students on Aimsweb, EasyCBM, and DIBELS. Implications for practice are discussed.     

CAN READABILITY FORMULAS BE USED TO SUCCESSFULLY GAUGE DIFFICULTY OF READING MATERIALS?

A grade level of reading material is commonly estimated using one or more readability formulas, which purport to measure text difficulty based on specified text characteristics. However, there is limited direction for teachers and publishers regarding which readability formulas (if any) are appropriate indicators of actual text difficulty. Because oral reading fluency (ORF) is considered one primary indicator of an elementary aged student's overall reading ability, the purpose of this study was to assess the link between leveled reading passages and students’ actual ORF rates. ORF rates of 360 elementary‐aged students were used to determine whether reading passages at varying grade levels are, as would be predicted by readability levels, more or less difficult for students to read. Results showed that a small number of readability formulas were fairly good indicators of text, but this was only true at particular grade levels. Additionally, most of the readability formulas were more accurate for higher ability readers. One implication of the findings suggests that teachers should be cautious when making instructional decisions based on purported “grade‐leveled” text, and educational researchers and practitioners should strive to assess difficulty of text materials beyond simply using a readability formula.