Investigating the validity of two widely used quantitative text tools

In recent years, readability formulas have gained new prominence as a basis for selecting texts for learning and assessment. Variables that quantitative tools count (e.g., word frequency, sentence length) provide valid measures of text complexity insofar as they accurately predict representative and high-quality criteria. The longstanding consensus of text researchers has been that such criteria will measure readers’ comprehension of sample texts. This study used Bormuth’s (1969) rigorously developed criterion measure to investigate two of today’s most widely used quantitative text tools—the Lexile Framework and the Flesch–Kincaid Grade-Level formula. Correlations between the two tools’ complexity scores and Bormuth’s measured difficulties of criterion passages were only moderately high in light of the literature and new high stakes uses for such tools. These correlations declined a small amount when passages from the University grade band of use were removed. The ability of these tools to predict measured text difficulties within any single grade band below University was low. Analyses showed that word complexity made a larger contribution relative to sentence complexity when each tool’s predictors were regressed on the Bormuth criterion rather than their original criteria. When the criterion was texts’ grade band of use instead of mean cloze scores, neither tool classified texts well and errors disproportionally placed texts from higher grade bands into lower ones. Results suggest these two text tools may lack adequate validity for their current uses in educational settings.     

CAN READABILITY FORMULAS BE USED TO SUCCESSFULLY GAUGE DIFFICULTY OF READING MATERIALS?

A grade level of reading material is commonly estimated using one or more readability formulas, which purport to measure text difficulty based on specified text characteristics. However, there is limited direction for teachers and publishers regarding which readability formulas (if any) are appropriate indicators of actual text difficulty. Because oral reading fluency (ORF) is considered one primary indicator of an elementary aged student's overall reading ability, the purpose of this study was to assess the link between leveled reading passages and students’ actual ORF rates. ORF rates of 360 elementary‐aged students were used to determine whether reading passages at varying grade levels are, as would be predicted by readability levels, more or less difficult for students to read. Results showed that a small number of readability formulas were fairly good indicators of text, but this was only true at particular grade levels. Additionally, most of the readability formulas were more accurate for higher ability readers. One implication of the findings suggests that teachers should be cautious when making instructional decisions based on purported “grade‐leveled” text, and educational researchers and practitioners should strive to assess difficulty of text materials beyond simply using a readability formula.     

反馈方式对二语写作语言复杂度的影响

本研究以自动评分系统反馈和范文反馈为例,探讨纠错反馈与非纠错反馈对大学生英语作文语言复杂度的影响。借助LCA、L2SCA和Coh-Metrix 3.0等在线分析器,对词汇复杂度和句法复杂度共10个代表性指标进行测量和分析。发现系统反馈能提高词汇复杂度但降低句法复杂度,而范文反馈对学生作文的词汇和句法复杂度均有显著的提升作用。由此可见,不同反馈方式对语言复杂度产生错综复杂的影响,相比而言范文反馈能促进文本质量在更多维度上的提升。     

Who Did What to Whom? The Relationship Between Syntactic Aspects of Sentence Comprehension and Text Comprehension

This study investigated the relationship between syntactic comprehension at the sentence level and text-level comprehension. The study isolated the specific contribution of syntax by asking whether sentence comprehension efficiency of difficult syntactic constructions explained variance in text comprehension after controlling for sentence comprehension efficiency of basic constructions with similar semantic complexity. Seventy-three Grade 5 students completed assessments of text comprehension; basic and difficult written sentence comprehension efficiency; and control measures of decoding fluency, vocabulary, and verbal memory. Efficiency measures were used to assess individual differences in basic sentence comprehension with accuracy near ceiling. Difficult sentence comprehension efficiency explained 6% unique variance in text comprehension after controlling for basic sentence comprehension efficiency and other controls. Thus, the results show that individual differences in the ability to establish sentence meaning from syntactic information are related to text comprehension.     

The effect of emphasising the realistic modelling complexity in the text or picture on pupils’ realistic solutions of P-items

We investigated the effect of emphasising the realistic modelling complexity in text or in the accompanying picture on the solution of P-items, in relation to pupils’ grade. 290 pupils from the 5th and 6th grade of various elementary schools in Flanders (Belgium) made a paper-and-pencil task with 7 word problems that are problematic from a realistic modelling perspective (so-called P-items). Pupils were divided in four conditions, involving the four combinations of the two above-mentioned task variables. Their reactions to the P-items were coded as realistic or non-realistic. We found a relatively small but significant effect of emphasising the realistic modelling complexity in the text of the P-items on the number of realistic reactions, whereas no effect was observed for the accentuation of that complexity in the picture nor for the interaction of the two manipulated task variables. Theoretical and educational implications of these results are discussed.     

Syntactic complexity as a predictor of adolescent writing quality: Which measures? Which genre?

This study examined the relationship of different measures of syntactic complexity with rated quality for two genres of text produced by middle school students. It was hypothesized that different measures would be associated with distinct aspects of syntactic complexity; words per clause with greater use of structures more typical of expository texts, and clauses per T-unit with structures more typical of conversational or narrative registers. A sample of 41 seventh and eighth grade students from suburban middle schools composed a narrative and persuasive essays. Texts were rated for quality and coded for syntactic features including words per clause and clauses per T-unit. Syntactic complexity as measured in words per clause was positively correlated with quality for essays but not for narratives. Clauses per T-unit was positively correlated with quality for narratives, but negatively correlated with quality for essays. The relationships between syntactic complexity and text quality were thus found to be dependent both on the genre of the text, and the measure of syntactic complexity used.     

An Examination of Text Complexity as Characterized by Readability and Cohesion

To better understand dimensions of text complexity and their effect on the comprehension of adolescents, 103 high school seniors were randomly assigned to 4 groups. Each group read versions of the same 2 informational passages and answered comprehension test items targeting factual recall and inferences of causal content. Group A passages had a challenging readability level and high cohesion; Group B passages had an easier readability and low cohesion; Group C passages had a challenging readability level and low cohesion; and Group D passages had an easier readability and high cohesion. Students in Group D significantly outperformed students in Group C (g = 0.78). Although the effect sizes of comparisons among all groups ranged from g = 0.13 to 0.73, no other comparisons were statistically significant. Results indicate that adolescents’ reading comprehension is dually influenced by a text's readability and cohesion. Implications for matching readers to instructional text are discussed.     

Developing phonological awareness and word recognition skills: A two-year intervention with low-income,inner-city children

Low-income, inner-city children were involved in a two-year intervention delivered in the regular classroom by regular classroom teachers to develop phonological awareness and word recognition skills. For the treatment children, an 11-week phoneme awareness program in kindergarten was followed by a first grade reading program (extended to grade 2 for some children) that emphasized explicit, systematic instruction in the alphabetic code. Control children participated in the school district's regular basal reading program. Both groups participated in a phonetically-based spelling program mandated by the district. At the end of grade 1, treatment children (n = 66) significantly outperformed control children (n = 62) on measures of phonological awareness, letter name and letter sound knowledge, and three measures of word recognition, and reached marginal significance (0.056) on a fourth. They also significantly outperformed the control children on two measures of spelling. One year later, at the end of grade 2, the treatment children (n = 58) significantly outperformed the control children (n = 48) on all four measures of word recognition. For the groups as a whole, there were no differences on the one measure of spelling readministered at the end of grade 2. However, there were significant differences in spelling between the treatment (n = 16) and control children (n = 13) who remained in the bottom quartile of spellers at the end of grade 2 when partial credit was given for phonetically correct spelling, and significant differences in reading favoring these treatment children on all four measures of word recognition.     

Sources of variance in curriculum‐based measures of silent reading

Curriculum‐Based Measurement silent reading (CBM‐SR) items have been found to be reliable and valid for measuring reading comprehension skills This generalizability study reports the findings from administration of three CBM‐SR passages to fifth through eighth grade students in one school district. Using Repeated Measures Analyses of Variance (RMANOVA) procedures, the statistical probability of performance on the CBM‐SR task as a differential indicator of reading comprehension skill was found to be significant among students in different grade levels and between students who did and did not receive special education services. Follow‐up analyses were conducted using generalizability theory to estimate the amount of variance in CBM‐SR scores from individual score differences, grade levels, and special education status. The results indicated that on two of the passages, variability in CBM‐SR scores came primarily from grade level differences in scores on the tasks, while on the third passage, the differences were most attributable to individual differences in scores, regardless of grade level or special education services. Implications for the use of CBM‐SR items for routine assessment of students' reading skills are discussed. © 2003 Wiley Periodicals, Inc. Psychol Schs 40: 363–377, 2003.     

The Efficacy of an Early Literacy Tutoring Program Implemented by College Students

Absctract. This article describes a two‐year study addressing the effectiveness of a highly structured, systematic tutoring intervention implemented by minimally trained college students with two cohorts of at‐risk first‐grade readers. Participants were 61 first‐grade children in Cohort 1 and 76 first‐grade children in Cohort 2. Tutors participated in three one‐hour training sessions and received occasional on‐site assistance. Individual tutoring sessions were scheduled for three to four times each week for one school year, with each cohort receiving approximately 10–14 hours of instruction across 44 sessions. The curriculum included a game to teach phonemic awareness and letter‐sound correspondence, structured word‐study activities, reading of leveled books, and simple comprehension strategies. Significant differences were found on measures of phonemic awareness and nonsense word reading for both cohorts. For Cohort 1, but not Cohort 2, significant differences were also detected for real‐word identification. Our results support using tutors to provide additional assistance and instruction in early reading, even when tutors are not professionally trained teachers.     

The limited benefits of rereading educational texts

Though rereading is a study method commonly used by students, theoretical disagreement exists regarding whether rereading a text significantly enhances the representation and retention of the text’s contents. In four experiments, we evaluated the effectiveness of rereading relative to a single reading in a context paralleling that faced by students in the classroom. Participants read educational texts (textbook chapters or a Scientific American article) under intentional learning instructions. Learning and memory were tested with educationally relevant summative assessments (multiple choice, short-answer questions, and text summaries). With only several exceptions, rereading did not significantly increase performance on the assessments. We also found that reading comprehension ability did not alter this pattern. It appears that when using ecologically valid materials such as a textbook chapter, immediate rereading may have little or no benefit for improving performance on educationally relevant summative assessments.     

Online Training: An Evaluation of the Effectiveness and Efficiency of Training Law Enforcement Personnel over the Internet

Evaluation of online training is a necessity to its future as a viable alternative method to classroom training. The purpose of Study 1 was to evaluate the effectiveness and efficiencies of online training as it compared to current classroom training with Nebraska Law Enforcement Training Center's (NLETC's) Jail Management trainees. Trainees were randomly assigned to receive either online or classroom training. Learning, motivation, and attitudes were measured for instructional effectiveness. Instructional time and cost/benefit calculations were used as measures of efficiencies. Results indicated, as predicted, that online training is as effective an instructional method as classroom training, and more efficient than classroom training. No meaningful learning differences occurred between the two groups, but online training was completed in almost half the time of classroom instruction and at a lesser cost. However, the classroom group reported higher motivation and positive feelings concerning their instruction than did the online group. Study 2 experimentally examined learning, instructional time, motivation, and attitude advantages of multimedia included in the Jail Management online training courses. Although hypothesized that video would enhance learning, motivation, and attitude, there were no differences in UNL student participants' test scores or surveys scores whether they received training with text only, audio with the text, or video with the text. Only instructional time differed among the groups.     

Experienced Teachers’ Strategies for Assessing Nature of Science Conceptions in the Elementary Classroom

This study explored the nature of science (NOS) assessments K-4 classroom teachers developed for measuring students’ understandings of NOS elements. We used the Views of Nature of Science Questionnaire-Form VNOS-D2 (Views of Nature of Science Elementary School Version 2) and interviews to verify that teachers’ conceptions of NOS were sufficient to enable them to teach and assess NOS. We collected copies of teachers’ action research designs, lesson plans, and assessment tools, conducted classroom observations and made field notes of their science instruction and assessments. We videotaped conversations at monthly workshops to note discussion surrounding teaching and assessing NOS in K-4 classrooms. We found that experienced teachers designed a variety of strategies for assessing NOS conceptions that differed by grade level.     

Beyond first grade: examining word,sentence, and discourse text factors associated with oral reading rate in informational text in second grade

Text complexity in elementary classrooms is typically measured by traditional readability tools, which rely on surface-level measures of word and sentence complexity. Theoretical and empirical work on text complexity, however, indicates that additional measures of semantics, syntax, and discourse structure may be equally important for understanding how young children process texts. This study evaluated the components of two available text analysis tools representing different generations of readability: the Lexile Framework and the Coh-Metrix Text Easability Assessor. Using multilevel modeling, informational passage readings were nested in 5133 second-grade children in 459 schools to identify text variables that predicted changes in oral reading rate. Coh-Metrix dimensions of word concreteness, referential cohesion, and deep cohesion were uniquely associated with oral reading rate after controlling for traditional readability components. A full model including both the Lexile components and three Coh-Metrix dimensions provided the best fit to the data. Results suggest that measures of semantic complexity and text cohesion are needed to improve prediction of text difficulty for young readers and better inform text selection in the early grades. Implications for policy and practice under the Common Core State Standards, which emphasize reading complex texts across the gradespan, are discussed.

     

The effects of dyad reading and text difficulty on third-graders’ reading achievement

This study replicated, with modifications, previous research of dyad reading using texts at various levels of difficulty (Morgan, 1997). The current project measured the effects of using above–grade-level texts on reading achievement and sought to determine the influences of dyad reading on both lead and assisted readers. Results indicate that weaker readers, using texts at two, three, and four grade levels above their instructional levels with the assistance of lead readers, outscored both proficient and less proficient students in the control group across multiple measures of reading achievement. However, the gains made by assisted readers were not significantly different relative to the various text levels. When all assessments were considered, assisted readers reading texts two grade levels above their instructional levels showed the most robust gains in oral reading fluency and comprehension. Lead readers also benefited from dyad reading and continued their respective reading developmental trajectories across measures.     

Depressive symptomatology in learning-disabled and nonlearning-disabled students

The present study examined the level of depressive symptomatology in a sample of 100 learning-disabled and nonlearning-disabled fourth-, fifth-, and sixth-grade students. Depressive symptomatology was assessed using the Children's Depression Inventory (CDI). In addition, students were rated by their classroom teachers on behaviors related to depression based on the DSM-III criteria. It was found that students identified as learning disabled obtained significantly higher scores on the CDI and the behavior rating scale than did children in the regular education classroom. There were no significant differences among grade levels for CDI scores, but a trend was noted. A significant difference was found among grade levels for the behavior rating scale scores. There was a significant positive correlation between CDI scores and teacher ratings of depressive characteristics.     

The Effects of Text Familiarity and Cohesion on Retrieval of Information Learned from Text

This study examined the effects of familiarity of passage concepts and passage cohesion on retrieval of text information. In order to distinguish between comprehension and retrieval processes, subjects were equated on comprehension; therefore, any differences found could be attributed to differences in retrieval processes. The passages used varied on familiarity and cohesion, but syntax was held constant. Middle school students learned information in passages to an 85% correct criterion using a study-test procedure. The subjects were not told that they would be tested on the material. Two weeks later, they took a free recall test and reported their rehearsal of the passage information during the intervening time period. The results showed that although the original amount learned was equated, recall of propositions from passages with more familiar concepts was about 35% greater than the recall from passages with less familiar concepts. There were no significant differences due to cohesion or the interaction of cohesion and familiarity. The explanation presented is that familiarity stimulates elaboration of passage material and elaborations provide alternate retrieval pathways at recall. Educational implications are discussed.