Shifting the focus of validity for test use

The conventional focus of validity in educational measurement has been on intended interpretations and uses of test scores. Empirical studies of test use by teachers, administrators and policy-makers show that actual interpretations and uses of test scores in context are invariably shaped by local users’ questions, which frequently require attention to multiple sources of evidence about students’ learning and the factors that shape it, and depend on local capacity to use such information well. This requires a more complex theory of validity that can shift focus as needed from the intended interpretations and uses of test scores that guide test developers to local capacity to support the actual interpretations, decisions and actions that routinely serve local users’ purposes. I draw on the growing empirical literature on data use to illustrate the need for an expanded theory of validity, point to theoretical resources that might guide such an expansion, and suggest a research agenda towards these ends.     

Standard Errors of Proportions Used in Reporting Changes in School Performance With Achievement Levels

The primary purpose of this study was to estimate the amount of variability in the proportions of students in a school district, scoring within each of three achievement levels that could be attributed to factors other than random sampling error. The approach taken is based on a general conceptual framework that collectively incorporates five sources of variability: instructional intervention, random sampling error, measurement error, equating error, and systematic error. Statewide school-level assessment data for reading and mathematics in grades four and eight from four consecutive years were used to examine annual grade-group change. The intent was to assess the impact of random sampling error in grade-group change estimates when either single-year proportions or 2-year average proportions are used to report school improvement with achievement levels. Observed variability in change was compared with theoretically-derived estimates of change due to random sampling error to determine the relative influence of sampling error and the aggregate of the other four sources of variability. Results indicate that the error variance of estimates of change at the school level is large enough to interfere with interpretations of annual change estimates. Recommendations are offered for establishing annual improvement goals and for reporting results with achievement levels-all in the context of adequate yearly progress (AYP)-while taking error estimates into account.     

英语习得错误分析

错误在总体语言习得研究 ,尤其是二语和外语习得研究中起着举足轻重的作用。错误的概念并非总是客观、泾渭分明的 ,在错误和非错误之间并不总是有明显的界限。此篇关注对错误概念的分析。错误易被语言教师识别 ,但并不始终以同一方式被解释。本文试图从不同的角度阐明出现错误时的种种现象     

Intertextuality/Intervisuality in Translation: The Jolly Postman's Intercultural Journey from Britain to the Netherlands

Texts that are intertextually and intervisually rich and highly culture-specific are often deemed to be untranslatable as they are too specifically set in a particular locale. Yet, some intertextual texts, such as Janet and Allan Ahlberg's Jolly Postman books, are rendered quite successfully in translation, gaining a new lease on life in another language with a different audience. Translators are first of all readers of the original text, and it is their interpretation which guides and colours their translation. To deal with the intertextual and culture-specific elements in the source text, the translator of the Jolly Postman books into Dutch uses a variety of translation strategies, such as literal translation for shared intertexts, substitution for intertexts likely to be unknown to the intended target audience, and addition or compensation. In this way he has created Dutch target texts with plenty of opportunities for the readers to forge their own interpretations and play their own intertextual games.     

Uncovering Multivariate Structure in Classroom Observations in the Presence of Rater Errors

We examine the factor structure of scores from the CLASS‐S protocol obtained from observations of middle school classroom teaching. Factor analysis has been used to support both interpretations of scores from classroom observation protocols, like CLASS‐S, and the theories about teaching that underlie them. However, classroom observations contain multiple sources of error, most predominantly rater errors. We demonstrate that errors in scores made by two raters on the same lesson have a factor structure that is distinct from the factor structure at the teacher level. Consequently, the “standard” approach of analyzing on teacher‐level average dimension scores can yield incorrect inferences about the factor structure at the teacher level and possibly misleading evidence about the validity of scores and theories of teaching. We consider alternative hierarchical estimation approaches designed to prevent the contamination of estimated teacher‐level factors. These alternative approaches find a teacher‐level factor structure for CLASS‐S that consists of strongly correlated support and classroom management factors. Our results have implications for future studies using factor analysis on classroom observation data to develop validity evidence and test theories of teaching and for practitioners who rely on the results of such studies to support their use and interpretation of the classroom observation scores.     

试论课堂因“错误”而精彩纷呈

新课程改革倡导新的课程观,课程资源不再是一成不变的范本,教师要充分挖掘身边有效的课程资源,灵活应用于教学。“错误”可以作为有效的课程资源,只要教师巧妙地利用,就可以为课堂服务。数学课堂会因错误而妙趣横生。教师在课堂教学中巧妙地利用“错误”,可以激发学生的学习兴趣,提高学生的反思能力。     

A Comparison of Method Effects in Two Confirmatory Factor Models for Structurally Different Methods

It is well known that measurement error in observable variables induces bias in estimates in standard regression analysis and that structural equation models are a typical solution to this problem. Often, multiple indicator equations are subsumed as part of the structural equation model, allowing for consistent estimation of the relevant regression parameters. In many instances, however, embedding the measurement model into structural equation models is not possible because the model would not be identified. To correct for measurement error one has no other recourse than to provide the exact values of the variances of the measurement error terms of the model, although in practice such variances cannot be ascertained exactly, but only estimated from an independent study. The usual approach so far has been to treat the estimated values of error variances as if they were known exact population values in the subsequent structural equation modeling (SEM) analysis. In this article we show that fixing measurement error variance estimates as if they were true values can make the reported standard errors of the structural parameters of the model smaller than they should be. Inferences about the parameters of interest will be incorrect if the estimated nature of the variances is not taken into account. For general SEM, we derive an explicit expression that provides the terms to be added to the standard errors provided by the standard SEM software that treats the estimated variances as exact population values. Interestingly, we find there is a differential impact of the corrections to be added to the standard errors depending on which parameter of the model is estimated. The theoretical results are illustrated with simulations and also with empirical data on a typical SEM model.     

用掷骰子的方法模拟不同类型的测量误差

利用掷骰子的方法模拟了不同类型的测量误差,且通过实例,对均匀分布、三角分布、正态分布的随机误差进行模拟,对其数字特征与对应不确定度进行印证,并提出了用掷骰子的方法产生给定数学期望与方差的正态随机数的模拟法。     

外语学习者的错误及纠错策略研究

外语学习是一种不断发展的心理、认知过程。学习中出现的错误具有积极意义。按其本质，错误表现为谬误（error),错误(mistake)和失误(1apse)。在教学中，教师必须坚持认知的、系统的、情感的和启发的原则，对学生的错误进行识别和科学的分类。在此基础上，教师要掌握错误的阶段性和原因，从而才能对错误进行具体分析，采取有效的策略纠正错误和提高教学。     

Assessing historical literacy among 12-year-old Finns

The new national core curriculum has been put into operation in Finland. Teaching and learning is intended to focus on historical literacy. In this paper, we study how it can be assessed with a pen-and-paper test, and what that reveals in regards to the mastering of historical literacy among elementary pupils. To study this, we designed a test where pupils analysed sources, answered open-ended questions, and filled out weighted multiple-choice questions. Furthermore, we implemented the test with the think-aloud method. The results from this indicate the way 12-year-olds read historical documents and formulate their own interpretations. The results demonstrate that most of the pupils succeeded in acquiring historical information from different sources, as well as understanding the interpretative nature of historical knowledge. The test also revealed that pupils can identify some intentions of the sources’ producers, as well as reasoning through their own interpretations using a single source. Some pupils, however, could not process the cognitive noise inherent to the test (i.e. conflicting sources). Some of them also considered an authoritative source to be principally more reliable than other sources. Our study sheds light on which kind of task can be used at the elementary level to assess historical literacy.     

Actual Interpretations and Use of Scores as Aspects of Validity

Validity is the most fundamental consideration in test development. Understandably, much time, effort, and money is spent in its pursuit. Central to the modern conception of validity are the interpretations made, and uses planned, on the basis of test scores. There is, unfortunately, however, evidence that test users have difficulty understanding scores as intended. That is, although the proposed interpretations and use of test scores might be theoretically valid they might never come to be because the meaning of the message is lost in translation. This necessitates pause. It is almost absurd to think that the intended interpretations and uses of test scores might fail because there is a lack of alignment with the actual interpretations made and uses enacted by the audience. Despite this, there has only recently been contributions to the literature regarding the interpretability of score reports, the mechanisms by which scores are communicated to their audience, and their relevance to validity. These contributions have focused upon linking, through evidence, the intended interpretation and use with the actual interpretations being made and actions being planned by score users. This article reviews the current conception of validity, validation, and validity evidence with the goal of positioning the emerging notion of validity of usage within the current paradigm.     

Note on Sources of Sampling Variability in Science Performance Assessments

In 1993, we reported in Journal of Educational Measurement that task-sampling variability was the Achilles' heel of science performance assessment. To reduce measurement error, tasks needed to be stratified before sampling, sampled in large number, or possibly both. However, Cronbach, Linn, Brennan, & Haertel (1997) pointed out that a task-sampling interpretation of a large person x task variance component might be incorrect. Task and occasion sampling are confounded because tasks are typically given on only a single occasion. The person x task source of measurement error is then confounded with the pt x occasion source. If pto variability accounts for a substantial part of the commonly observed pt interaction, stratifying tasks into homogenous subsets—a cost-effective way of addressing task sampling variability—might not increase accuracy. Stratification would not address the pro source of error. Another conclusion reported in JEM was that only direct observation (DO) and notebook (NB) methods of collecting performance assessment data were exchangeable; computer simulation, short-answer, and multiple-choice methods were not. However, if Cronbach et al. were right, our exchangeability conclusion might be incorrect. After re-examining and re-analyzing data, we found support for Conbach et al. We concluded that large task-sampling variability was due to both the person x task interaction and person x task x occasion interaction. Moreover, we found that direct observation, notebook and computer simulation methods were equally exchangeable, but their exchangeability was limited by the volatility of student performances across tasks and occasions.     

Prevention–Permission–Promotion: A Review of Approaches to Errors in Learning

Errors are often perceived as undesirable events to be avoided at all costs. However, a growing body of research suggests that making errors is, in fact, beneficial for learning. Building on human resource development literature, the present review proposes a 3P framework of approaches to errors during learning: prevention (avoiding or observing errors), permission (allowing errors), and promotion (inducing or guiding errors). This framework is applied to examine and integrate the empirical evidence on errors that have been commonly investigated in cognitive, educational, and applied psychology research. The psychological mechanisms of each error approach are discussed, and implications for education are considered. This review then concludes by highlighting the ways in which the various error approaches interact with learner characteristics and learning contexts, as well as discussing the role of feedback in error correction and proposing directions for future research toward understanding how errors can be optimized in learning.     

Positive error climate promotes learning outcomes through students’ adaptive reactions towards errors

Errors are an integral part of the learning process and an opportunity to increase skills and knowledge, but they are often discouraged, sanctioned and derided in the classroom. This study tests whether students' perceptions of being part of an error-friendly classroom context (i.e., a positive classroom error climate) is positively related to students' learning outcomes via students' adaptive reactions towards errors. A total of 563 Italian middle school students from 32 mathematics classes completed a questionnaire on their perceptions of classroom error climate and their reactions towards errors. Students' math grades were used as indicators of their level of learning outcomes. A multilevel model showed that perceived classroom error climate was positively related to math grades via increased adaptive reactions towards errors. Our findings revealed that an error-friendly classroom context is associated with students’ adaptive adjustment to errors and to better learning outcomes in mathematics.     

Assessment Validation in the Context of High-Stakes Assessment

Including the perspectives of stakeholder groups (e.g., teachers, parents) can improve the validity of high-stakes assessment interpretations and uses. How stakeholder groups view high-stakes assessments and their uses may differ significantly from state-level policy officials. The views of these stakeholders can contribute to identifying the strengths and weaknesses of the intended assessment interpretations and uses. This article proposes a process approach to validity that addresses assessment validation in the context of high-stakes assessment. The process approach includes a test evaluator or validator who considers the perspectives of five stakeholder groups at four different stages of assessment maturity in relationship to six aspects of construct validity. The tasks of the test evaluator and how stakeholders' views might be incorporated are illustrated at each stage of assessment maturity. How the test evaluator might make judgments about the merit of high-stakes assessment interpretations and uses is discussed.     

Measurement,Sampling, and Equating Errors in Large-Scale Assessments

In large-scale assessments, such as state-wide testing programs, national sample-based assessments, and international comparative studies, there are many steps involved in the measurement and reporting of student achievement. There are always sources of inaccuracies in each of the steps. It is of interest to identify the source and magnitude of the errors in the measurement process that may threaten the validity of the final results. Assessment designers can then improve the assessment quality by focusing on areas that pose the highest threats to the results. This paper discusses the relative magnitudes of three main sources of error with reference to the objectives of assessment programs: measurement error, sampling error, and equating error. A number of examples from large-scale assessments are used to illustrate these errors and their impact on the results. The paper concludes by making a number of recommendations that could lead to an improvement of the accuracies of large-scale assessment results.     

“设错教学法”下会计电算化实验如何“设错”

"设错教学法"在会计电算化实验中能够有效培养学生对会计软件的应用能力,在设错、识错、纠错和防错四个环节中"设错"是关键环节。提出了科学"设错"的三个原则,详细阐述了在会计电算化实验中"设错"的方法。     

Is dealing with errors in the classroom specific for school subjects? A study of the error climate in mathematics,German, and English

A frequent observation in the school context is that opportunities to learn from errors are often missed. However, a positive error climate may support learning from errors. For the school subject of mathematics, some findings about characteristics of the error climate already exist. But, a comparison of the error climate between different school subjects is still pending. In the present study, it is analyzed whether the error climate differs in different school subjects and whether the same interrelations between the ways in which individuals deal with errors can be found in these different school subjects. In a study with 937 students from 48 classrooms from grades 5 to 7, in different secondary schools in Germany and Austria, we assessed the error climate and individual reactions following errors in mathematics, German, and English. Small mean differences between mathematics and the two language subjects were yielded. In addition, we found medium-sized correlations between the error climate measures in the three school subjects. However, the same pattern of interrelations between error climate and the way individuals deal with errors for all three school subjects could be shown. The results suggest that the perception of the error climate is rather similar in different school subjects. This has implications, for instance, for interventions that aim at fostering the error climate.

     

The Accuracy of Aggregate Student Growth Percentiles as Indicators of Educator Performance

Mean or median student growth percentiles (MGPs) are a popular measure of educator performance, but they lack rigorous evaluation. This study investigates the error in MGP due to test score measurement error (ME). Using analytic derivations, we find that errors in the commonly used MGP are correlated with average prior latent achievement: Teachers with low prior achieving students have MGPs that underestimate true teacher performance and vice versa for teachers with high achieving students. We evaluate alternative MGP estimators, showing that aggregates of SGP that correct for ME only contain errors independent of prior achievement. The alternatives are thus more fair because they are not biased by prior mean achievement and have smaller overall variance and larger marginal reliability than the Standard MGP approach. In addition, the mean estimators always outperform their median counterparts.     

Twenty questions about student errors

Errors in science learning (errors in expression of organized, purposeful thought within the domain of science) provide a window through which glimpses of mental functioning can be obtained. Errors are valuable and normal occurrences in the process of learning science. A student can use his/her errors to develop a deeper understanding of a concept as long as the error can be recognized and appropriate, informative feedback can be obtained. A safe, non-threatening, and nonpunitive environment which encourages dialogue helps students to express their conceptions and to risk making errors. Pedagogical methods that systematically address common student errors produce significant gains in student learning. Just as the nature-nurture interaction is integral to the development of living things, so the individual-environment interaction is basic to thought processes. At a minimum, four systems interact: (1) the individual problem solver (who has a worldview, relatively stable cognitive characteristics, relatively malleable mental states and conditions, and aims or intentions), (2) task to be performed (including relative importance and nature of the task), (3) knowledge domain in which task is contained, and (4) the environment (including orienting conditions and the social and physical context). Several basic assumptions underlie research on errors and alternative conceptions. Among these are: Knowledge and thought involve active, constructive processes; there are many ways to acquire, organize, store, retrieve, and think about a given concept or event; and understanding is achieved by successive approximations. Application of these ideas will require a fundamental change in how science is taught.