Measuring college students' reading comprehension ability using cloze tests

Recent investigations challenge the construct validity of sustained silent reading tests. Performance of two groups of post‐secondary students (e.g. struggling and non‐struggling) on a sustained silent reading test and two types of cloze test (i.e. maze and open‐ended) was compared in order to identify the test format that contributes greater variance in reading comprehension. One hundred participants were recruited from students enrolled in a preparatory course for a high‐stakes statewide reading examination. Our results suggest that all three measures have good concurrent validity. There was no evidence that open‐ended cloze performance was more related to verbal ability than any other reading measure. Maze performance did the best job at discriminating between our struggling and non‐struggling readers. Implications for reading comprehension assessment in post secondary‐aged adults are discussed.     

Matching and Multiple-Choice Cloze Tests

Abstract

Few reliable and valid measures of reading achievement are available to evaluate programs for elementary English-as-a-second-language (ESL) pupils. Four variations on the cloze procedure, which has been previously used with disadvantaged and ESL elementary pupils, were evaluated using randomly assigned groups of fourth and fifth grade students. Matching and multiple- choice variations were selected for comparison because they are in greater consonance with current psycho- linguistic theories of the reading process than are other types of reading comprehension measures. Although the overall results were quite similar for the four cloze variations examined, the matching cloze procedure seems to be preferable for elementary ESL students since these tests produced better item characteristics and were more easily constructed.     

Imagery-Based Learning: Improving Elementary Students' Reading Comprehension With Drama Techniques

Abstract

The causal relationship between drama-based reading instruction and reading comprehension among 4th-grade students was examined. Cognitive theories related to the role of imagery in memory were used to develop a drama-based reading comprehension program. A randomized pretest-posttest control-group design was then employed to assess the impact of the drama-based instruction on students' test scores on the reading comprehension portion of the Iowa Tests of Basic Skills. Results suggest a direct causal link between drama-based instruction and improved reading comprehension.     

英语测试中影响完形填空难度因素的实证研究

完形填空被认为是一种测试综合语言能力、阅读理解能力的快捷经济的方式。本研究就可能影响完形填空难度的几个变量进行实证探讨, 其中包括语篇类型、删词类型及答题方法。本研究以98 名高三学生为实验对象,完成3 篇填空式完形测试和3 篇选择式测试 测试完成后,笔者对实验数据进行收集、分析, 探究这些变量对完形填空测试难度的影响,并试图在命题难度的把握上找到一种更为合理、科学的测试方法。     

Cloze procedure and comprehension: an exploratory study across three languages

This cross-cultural study of cloze procedure and comprehension involved samples of 10- to 11-year-old schoolchildren in Canada, Japan, Sweden and the United States. The aim of the study was to explore the nature of what might be called‘cloze comprehension’in relation to overall or‘global comprehension’of a passage; in particular to establish (a) whether cloze procedure measures the same facets of comprehension regardless of what language is being read; and (b) to what extent cloze procedure, in different linguistic areas, measures‘global comprehension', or comprehension of the general ideas contained in a passage, as distinct from literal comprehension. The results of the study indicate that cloze procedure is a valid and reliable measure of certain aspects of reading comprehension in all the linguistic and cultural areas sampled. Furthermore, comprehension as measured by cloze procedure seems to be a necessary, albeit not sufficient, condition for overall or global understanding of the meaning of a passage. The study also shows that the ability measured by cloze procedure is more generalized (i.e., less text specific) than the ability measured by our global comprehension task. One implication of this seems to be that the higher-order skills necessary for global understanding do not always develop automatically once children have mastered the skills necessary for literal comprehension of simple texts. On the contrary, the higher-order skills may have to be taught systematically at an appropriate stage in the children's reading development.     

Monitoring Student Achievement for Accountability: The Demonstration of a Model

Abstract

This study demonstrated a procedural model that can be applied by any school to assess, guide, and account for the progress of its students as well as to analyze its own effectiveness. The model uses equivalent achievement tests to monitor student achievement in subject areas at grade levels, between grade levels, and across subgroups of students. Multiple regression analyses of test scores between grades identify factors associated with achievement Using sixth and eighth grade Comprehensive Tests of Basic Skills scores in a matched longitudinal sample of 208 students, the study found small differences in average achievement between boys and girls. Differences between corresponding sixth and eighth grade test means were higher in mathematics than in language. From the sixth grade to the eighth, there was a widening gap in average achievement between high and low I.Q. groups. In multiple regressions of eighth grade test scores on sixth grade measures, I.Q., study skills, and reading were prevalent in the regression equations, but clusters of measures associated with achievement differed between high and low’ LQ. groups. The results of the study have implications for developing and evaluating the achievement of students with varying mental abilities.     

Further Progress Towards a Standardised Curriculum-based Measure of Reading: Calibrating a new passage reading test against the New South Wales Basic Skills Test

Results are presented from a pilot study and a larger scale study aligning a curriculum-based passage reading test, the Wheldall Assessment of Reading Passages (WARP), with the literacy components of the New South Wales Basic Skills Test (BST). A strong relationship between the WARP and BST Literacy is demonstrated, particularly for Year Three students. Representative (approximate) norms for students in Years One to Five on the WARP are reported, based on the results from a school with a BST literacy profile that is very similar to that for the state of New South Wales as a whole.     

CLOZE MEASURES AS INDICES OF ACHIEVEMENT COMPREHENSION WHEN LEARNING FROM EXTENDED PROSE

The hypothesis that cloze measures are a function of content achievement among adult learners and, consequently, should be sensitive to instructional treatments was tested in two experimental studies. College juniors and seniors took tests immediately before (pre), immediately after (post) and four weeks after (delay) studying a prose passage. The types of tests administered in each session were: (I) a 20-item multiple-choice test, (2) a reproduction passage cloze test, (3) a recognition passage cloze test, (4) a reproduction summary cloze test, and (5) a recognition summary cloze test. All tests show significant differences between pre- and posteonditions, and between recognition and reproduction modes. The reproduction summary cloze test was found to be the most sensitive to the instructional treatment, as indicated by an oJ 2 statistic on pre-post measures. The summary cloze tests were resistant to forgetting while the cloze passage and multiple-choice tests show significant decreases in performance over the four week delay interval.     

语篇分析理论在CET4完形填空中的应用

CET4考试中,学生普遍反映在所有题型中,完形填空属于难度较大、失分较多的题型。经过调查分析,发现造成失误主要是因为学生缺乏相应的语篇分析能力。教师在教学过程中要有意识地培养学生的语篇意识,并引导学生运用于学习过程中,能提高解题能力,减少错误率。     

Development of an Early Hebrew Oral Reading Fluency Measure

Currently, there are no Hebrew (L2) reading assessments that have been tested to obtain evidence for reliability and validity on which to base decisions about Hebrew instruction. The authors developed a Hebrew benchmark assessment tool for first grade students modeled after Dynamic Indicators of Basic Early Literacy Skills, a standardized test of accuracy and fluency used to identify at-risk students and to monitor student progress. Results of pilot data collection (N=53) provide evidence for strong alternate form reliability for this measure, as well as evidence for content, face and criterion-related validity. Future directions for research and development are discussed.     

Is fluency‐based story retell a good indicator of reading comprehension?

This study examined the reliability and validity of scores on a fluency‐based measure of reading comprehension. The Dynamic Indicators of Basic Early Literacy Skills (DIBELS; 6th ed. revised) Retell Fluency (RTF), Oral Reading Fluency (DORF), and Woodcock Johnson III NU Tests of Achievement (WJ‐III NU ACH) Reading Comprehension measures were administered to fourth‐grade students. Results indicated a large difference between real time and recorded retell fluency scores for each passage. In addition, students' retell fluency scores had a low correlation with their reading comprehension scores. In light of these findings, practitioners may want to exercise caution in using fluency‐based story‐retell scores as a measure of reading comprehension. © 2011 Wiley Periodicals, Inc.     

USING CURRICULUM‐BASED MEASUREMENT TO EXAMINE SUMMER LEARNING LOSS

Summer loss of reading is a potential factor in maintaining, and potentially widening, the achievement gap. This study used curriculum‐based measurement of reading (R‐CBM) to investigate the effect of the summer on reading. For this study, 317 students in Grades 2 to 5 were assessed in the spring and fall using Dynamic Indicators of Basic Early Literacy Skills Oral Reading Fluency (ORF) measure. Repeated measures analysis of variance modeled overall change in ORF scores by grade, family income, ethnic minority status, English language learner status, and special education (SPED) status. Students in Grades 2 and 3 evidenced overall summer loss, whereas students in Grades 4 and 5 did not exhibit summer loss. In addition, students in Grade 2 showed differential loss based on family income and SPED status. These results support the broadening application of R‐CBM and add to the summer loss literature. Findings are discussed relative to prevention and intervention.     

Evaluating the cost-effectiveness of the Dyslexia Pilot Project: A multitiered system of supports for early literacy

The Dyslexia Pilot Project provided funding to school districts to implement a multitiered system of support (MTSS) framework for the prevention, early identification, and early intervention of reading difficulties. This article describes the evaluation of the multiyear Dyslexia Pilot Project for students in kindergarten through Grade 2. The evaluation extended a conceptual model for evaluating the systems effects of an MTSS for reading by including a cost-effectiveness analysis. The results indicate that mean rates of improvement on Dynamic Indicators for Basic Early Literacy Skills Next curriculum-based measures for students in participating schools exceeded the expected rates of improvement based on benchmark norms. This reduction in risk precluded the need for more intensive, individualized, and costly interventions and specialized educational services. Implications of the findings and future directions are discussed.     

Testing reading comprehension of theoretical discourse with cloze

The ability of cloze tests containing frequent, every n‐th word deletions to measure comprehension of macropropositions has been challenged on both theoretical and empirical grounds, calling into question the validity of such tests for assessing comprehension of much of the discourse encountered by university‐level students. To evaluate the comprehension of a writer’s reasoning, it is recommended that cloze tests position gaps so as to target recognition of cohesive devices and the ability to draw inferences from other sentences. To test the validity of such a design, a large sample of scores on discourse cloze tests administered in introductory college economics is compared to scores on true–false comprehension tests designed to target recognition of connective propositions. The two distributions of scores do not differ significantly in terms of mean value, dispersion or frequency distribution, suggesting that appropriately designed cloze tests can provide a valid assessment of the reader’s integration of theoretical text. In addition, the usefulness of readability formulas based on surface characteristics of text is challenged when readability is defined in terms of the difficulty of constructing a coherent representation of theoretical text.     

The Dynamic Indicators of Basic Early Literacy Skills (DIBELS) used in an Australian context

Abstract

This study explored use of the Dynamic Indicators of Basic Early Literacy Skills (DIBELS) subtests in an Australian context, as part of a larger study of reading accuracy achievement. Subjects were 398 Queensland (Qld) students in Years 1, 2, and 3, who were tested on DIBELS reading accuracy subtests at mid‐year and end‐year test points in 2005. Alignment with DIBELS USA benchmark cutpoints was poor, probably due to schooling differences, as Qld starts formal reading instruction a year later than the USA. In discussion with the DIBELS authors, it was decided to develop interim local Qld benchmarks from the data, to be used by schools and researchers until longitudinal research establishes more valid benchmark cutpoints. The characteristics and advantages of DIBELS are discussed.     

A comparison of the Visual Aural Digit Span and the Bender Gestalt as discriminators of low achievement in the primary grades

The Visual Aural Digit Span (VADS) and the Bender Visual Motor Gestalt Test (Bender) were studied with regard to their ability to discriminate low from average achievers in reading and arithmetic skills, as identified by the Iowa Test of Basic Skills. A sample of 78 normal children aged 6 through 9 were administered a battery of tests, including the verbal section of the WISC-R. Analysis of covariance with IQ controlled showed that the Bender and the VADS were able to discriminate between achievement groups for vocabulary and math concepts. The Bender discriminated between math problem-solving groups, but neither test could discriminate between reading comprehension groups. Age was a significant variable for the Bender and all VADS subtests except Aural-Written. Correlational analysis indicated that although the VADS was related to Verbal IQ, it is related only minimally to the Bender when age is controlled.     

Damaging confusions in England's KS2 reading tests: a response to Anne Kispal

This article is written in response to the article published in issue 39.3 of this journal, in November 2005, on the nature of the Key Stage 2 National Curriculum reading tests: ‘Examining England's National Curriculum assessments: an analysis of the KS2 reading test questions’ by Anne Kispal of the National Foundation for Educational Research. It argues that, far from providing a valid and rewarding assessment experience for pupils as Kispal suggests, the primary English tests at the end of KS2 are invalid as a measuring instrument and are having a damaging effect on pedagogy. The tests and the information on them provided by the Qualifications and Curriculum Agency are based on a misleading unidimensional conception of reading literacy attainment. Because the test assessment simply adds together marks achieved for very different cognitive skills, it propagates a dysfunctional model of literacy pedagogy that conflates and confuses two separate developmental trajectories – word reading and text comprehension. The article goes on to argue that the unidimensionality of the national tests and their pedagogic apparatus has constricted the primary English curriculum in ways that are damaging for young pupils and for the national need for creativity and enterprise.     

Examining the role of vocabulary depth, cross-linguistic transfer, and types of reading measures on the reading comprehension of Latino bilinguals in elementary school

Given the increase of bilingual students in the K-12 public school system, understanding reading comprehension performance, especially among this population, has been a major focal point in the research literature. This study explores the nature of reading comprehension among a sample of 123 Spanish–English bilingual elementary students. We add to the existing knowledge base regarding reading comprehension in two significant ways: (1) augmenting the Simple View of Reading by testing the role of both vocabulary depth contribution and dual-linguistic ability in English reading comprehension; and (2) questioning the manner through which reading comprehension is understood through measurement and conceptualization. Specifically, we build a comprehensive model of reading comprehension that tests the effects for vocabulary depth, Spanish oral language, and biliteracy. In line with previous research that suggests different reading measures tap different abilities, we test our model for three different measures of reading comprehension: a cloze exercise, a passage and multiple choice based test, and a timed silent sentence reading judgment task. Our findings converge with previous research on the role of vocabulary depth in reading comprehension and also challenge prior work which has compared different reading measures. Implications for theoretical and empirical approaches to understanding reading comprehension, specifically among Spanish–English bilingual students, are discussed.     

Is retell a valid measure of reading comprehension?

Retell is used widely as a measure of reading comprehension. In this meta-analysis, we evaluated the relation between retell and other measures of reading comprehension among students in Grades 1–12. Data from 23 studies (82 effect sizes; N = 4705 participants) showed a moderate relation between retell and other measures of reading comprehension, r = 0.46. Moderation analyses revealed that the relation was stronger when reading comprehension was measured by cloze or maze tasks than when measured using a multiple-choice format. In addition, the relation was weaker in higher grades, but this was largely explained by text genre or the number of prompts in retell. The relation between ‘oral’ retell and reading comprehension was stronger with a greater number of prompts provided during retell tests. In contrast, results did not differ by other features of retell such as reading mode (oral or silent), text genres of retell (narrative or informational), or use of different oral retell evaluation methods (e.g., number of words or ideas, overall quality). Overall, the moderate magnitude of the relation between retell and other measures of reading comprehension indicates caution for using retell as the sole measure of reading comprehension. The results also indicate a need for a better understanding about more systematic approaches to retell assessment (e.g., number and kind of prompts in the case of oral retell) as a measure of reading comprehension.     

Exploratory and confirmatory factor analyses of the Woodcock reading mastery tests–revised with special education students

The factor structure of the Woodcock Reading Mastery Tests–Revised (WRMT‐R) was analyzed using data from a sample of 140 special education students with learning disabilities, mild mental retardation, and serious emotional disturbance. Woodcock asserted that the WRMT‐R measured two important aspects of reading ability: Basic Skills and Reading Comprehension. According to Woodcock, the Word Identification and Word Attack Tests measure two elements of Basic Skills, and the Word Comprehension and Passage Comprehension Tests measure two elements of Reading Comprehension. Together, Basic Skills and Reading Comprehension produce the Total Reading Full‐Scale score. Principal axis factor analysis with a promax rotation and confirmatory factor analysis were used to evaluate evidence that the four tests of the WRMT‐R combine to form the two factors: Basic Skills and Reading Comprehension. The results of the analyses indicated a robust single factor (Total Reading Full‐Scale), and provided little support for Woodcock's hypothesized two‐factor structure. © 2001 John Wiley & Sons, Inc.