Balancing varied assessment functions to attain systemic validity: Three is the magic number

Accountability-oriented reforms demand immediate and continual gains on achievement test, for all students, and without diminishing other outcomes or undermining instruction. This paper describes a framework for aligning classroom assessment and external testing with the aim of negotiating these seemingly contradictory goals. The framework varies the sensitivity to instruction and the representations of knowledge across approaches to assessment. Cycles of design- based studies successively refine relationships between a curriculum and the frame that each assessment provides. Doing so, we argue, leverages the unique formative and summative balance across assessments in order to scaffold learning and demonstrate the “consequential” validity of our strategy without compromising curricula, instruction, or the “evidential” validity that warrants their continued use.     

Resistance to Confounding Style and Content in Scoring Constructed-Response Items

An assumption that is fundamental to the scoring of student-constructed responses (e.g., essays) is the ability of raters to focus on the response characteristics of interest rather than on other features. A common example, and the focus of this study, is the ability of raters to score a response based on the content achievement it demonstrates independent of the quality with which it is expressed. Previously scored responses from a large-scale assessment in which trained scorers rated exclusively constructed-response formats were altered to enhance or degrade the quality of the writing, and scores that resulted from the altered responses were compared with the original scores. Statistically significant differences in favor of the better-writing condition were found in all six content areas. However, the effect sizes were very small in mathematics, reading, science, and social studies items. They were relatively large for items in writing and language usage (mechanics). It was concluded from the last two content areas that the manipulation was successful and from the first four that trained scorers are reasonably well able to differentiate writing quality from other achievement constructs in rating student responses.     

Relationships between students' meaningful learning orientation and their understanding of genetics topics

This study explored factors predicting the extent to which high school students (N = 140) acquired meaningful understanding of the biological topics of meiosis, the Punnett-square method, and the relationships between these topics. This study (a) examined mental modeling as a technique for measuring students' meaningful understanding of the topics, (b) measured students' predisposed, generalized tendency to learn meaningfully (meaningful learning orientation), (c) determined the extent to which students' meaningful learning orientation predicted meaningful understanding beyond that predicted by aptitude and achievement motivation, (d) experimentally tested two instructional treatments (relationships presented to students, relationships generated by students), (e) explored the relationships of meaningful learning orientation, prior knowledge, instructional treatment, and all interactions of these variables in predicting meaningful understanding. The results of correlations and multiple regressions indicated that meaningful learning orientation contributed to students' attainment of meaningful understanding independent of aptitude and achievement motivation. Meaningful learning orientation and prior knowledge interacted in unique ways for each topic to predict students' attainment of meaningful understanding. Instructional treatment had relatively little relationship to students' acquisition of meaningful understanding, except for learners midrange between meaningful and rote. These findings imply that a meaningful learning approach among students may be important, perhaps as much or more than aptitude and achievement motivation, for their acquisition of interrelated, meaningful understandings of science.     

The Challenge and Attainability of Goals for Adequate Yearly Progress

For standards-referenced assessment systems, there is a tension between setting high goals versus attainable ones for adequate yearly progress. The more relevant information that policymakers can consider in setting these goals, the more likely it will be that productive goals are set. Some strategies are presented here that can be used either singly or in combination to evaluate the level of rigor implied by these expectations. These strategies relate to understanding how likely it is, based on past experience, that a particular goal can be met.     

Essential Assessment Skills in Professional Education of Teachers

What content areas should be included in a testing and measurement course? What skills should teachers learn in each content area? How can this content and these skills be rationalized?     

Continuity and discontinuity in competing models of criminal justice education: Evidence from illinois

Competing models of criminal justice education have been conceptualized to exist for many years. Relevant empirical data to support this notion can be identified in the literature, especially data derived from early surveys of criminal justice educators and content analyses of criminal justice curricula as found across institutions of higher education. Using contemporary survey data from criminal justice faculty members in the state of Illinois, the authors attempt to assess whether the evolution of criminal justice education has resulted in contemporary demarcations and models of criminal justice education that differ from those identified in the earlier literature. The survey data suggest strong patterns, some of which differ from those identified previously. Four models of criminal justice education emerging from the data are discussed, and their implications for criminal justice education are highlighted.     

Doctoral Comprehensive Exams: Standardization,Customization, and Everywhere in Between

Comprehensive examinations mark one of the final student assessments within doctoral programs. Despite the presumed importance of this process, little is known about how it is carried out within the field of criminology and criminal justice. This study examined the structure of comprehensive exams in relevant doctoral programs. The findings reveal considerable variation in how programs organize comprehensive exams. This variation might suggest that programs vary in the objectives they associate with the comprehensive exam process, though other explanations are possible. The discipline may need to open a dialogue on the goals and objectives driving these processes to better understand their fit within doctoral education.     

Increasing the Homogeneity of CAT's Item-Exposure Rates by Minimizing or Maximizing Varied Target Functions While Assembling Shadow Tests

A computerized adaptive testing (CAT) algorithm that has the potential to increase the homogeneity of CAT's item-exposure rates without significantly sacrificing the precision of ability estimates was proposed and assessed in the shadow-test ( van der Linden & Reese, 1998 ) CAT context. This CAT algorithm was formed by a combination of maximizing or minimizing varied target functions while assembling shadow tests. There were four target functions to be separately used in the first, second, third, and fourth quarter test of CAT. The elements to be used in the four functions were associated with (a) a random number assigned to each item, (b) the absolute difference between an examinee's current ability estimate and an item difficulty, (c) the absolute difference between an examinee's current ability estimate and an optimum item difficulty, and (d) item information. The results indicated that this combined CAT fully utilized all the items in the pool, reduced the maximum exposure rates, and achieved more homogeneous exposure rates. Moreover, its precision in recovering ability estimates was similar to that of the maximum item-information method. The combined CAT method resulted in the best overall results compared with the other individual CAT item-selection methods. The findings from the combined CAT are encouraging. Future uses are discussed.