教师测评素养国外研究述评

教师测评素养是教师教育和教师发展的重要组成部分。通过对国外有关教师测评素养研究文献的梳理和分析,其研究内容主要包括教师测评素养的构成、现状及培训需求、测量工具的开发和验证、测评素养发展途径和影响因素等。这些研究对我国开展教师测评素养相关研究有一定的参考价值,同时也启示我们可以在研究对象、研究方法和研究内容上进一步拓展,要基于我国国情,开展教师测评素养及其测量测量工具的本土化研究。     

Educational and Employment Testing: Changing Concepts in Measurement and Policy

How will the expansion of the concept of construct validity affect validation practice in employment testing? How does the need for consequential validity differ in educational and employment testing? How do the research bases differ for performance assessment in these settings? Are there parallel trends in policies for test use in education and industry?     

A General Framework for the Validation of Embedded Formative Assessment

In educational practice, test results are used for several purposes. However, validity research is especially focused on the validity of summative assessment. This article aimed to provide a general framework for validating formative assessment. The authors applied the argument‐based approach to validation to the context of formative assessment. This resulted in a proposed interpretation and use argument consisting of a score interpretation and a score use. The former involves inferences linking specific task performance to an interpretation of a student's general performance. The latter involves inferences regarding decisions about actions and educational consequences. The validity argument should focus on critical claims regarding score interpretation and score use, since both are critical to the effectiveness of formative assessment. The proposed framework is illustrated by an operational example including a presentation of evidence that can be collected on the basis of the framework.     

Cross-cultural instrument translation: assessment, translation, and statistical applications

This article has four major sections: (a) general issues of assessment; (b) assessment of ethnic-group members, including those who are deaf; (c) translation of assessment tools, emphasizing translation into American Sign Language (ASL); and (d) statistical applications for translated instruments. The purpose of the article is to provide insight into and guidelines for translating instruments into another language and using the results for practice or research purposes. The use of translated instruments with deaf individuals is highlighted. A protocol for translating a scale into ASL is offered. Statistical applications for examining internal consistency and component structure are discussed. Finally, implications for practice and research are addressed.     

Interpretive validity theory: mapping a methodology for validating educational assessments

Background:?Validity theory has evolved significantly over the past 30 years in response to the increased use of assessments across scientific, social and educational settings. The overarching trajectory of this evolution reflects a shift from a purely quantitative, positivistic approach to a conception of validity reliant on the interpretation of multiple evidence sources integrated into validity arguments. Moreover, within contemporary validity, interpretation has been emphasised as a central process; however, despite this emphasis, there have been few explicit articulations of specific interpretive methodologies applicable to the practice of validation.

Purpose:?To link contemporary theoretical foundations in validity to practical methods and structures to help guide the collection and analysis of interpretive validity evidence. By building upon existing validity theory, this paper aims to provide greater clarity on the practice of validation and contribute toward the larger developing framework for the validation of educational assessments.

Source of evidence:?An interdisciplinary, integrative review of over 60 research articles and sources related to the theory and practice of educational validation and interpretive inquiry approaches. Sources include literature from the fields of educational assessment and more broadly social scientific research.

Main argument:?As assessments in education increasingly aim to measure complex constructs that are value-laden and socially dependant, validity theory must keep pace and evolve in ways that address the inherent complexities associated with contemporary educational assessment. Through this paper, I assert that a greater understanding of interpretive methodologies represents one of the most promising areas for development of validation theory and practice. Specifically, I argue that dialectic, hermeneutic and transgressive forms of inquiry can be integrated within current argument-based structures for the collection, analysis and representation of validity evidence in several useful ways.

Conclusions:?Interpretive inquiry processes, namely dialectic, hermeneutic and transgressive forms of interpretation, serve to expand validation practice to include diverse evidences for the generation of multiple-perspective validity arguments. The paper concludes with specific implications for future research and practice within the field of interpretive validity theory.     

Constructing validity arguments for test combinations

The argument-based approach to validation has been widely adopted in validation theory. However, this approach aims to validate the intended interpretation and use of a single test or assessment. This article proposes an extension of the argument-based approach for validation of multiple tests. This extension is illustrated with the validation of a competency assessment program (CAP). This CAP was validated in collaboration with a quality manager of an educational program. In this case study, it became apparent that this approach fosters an in-depth evaluation of the assessment program and that the approach appears suitable for validation efforts of competency assessment programs. The approach guides validation research from a more general perspective, but also guides more detailed validation efforts.     

Validating test score meaning and defending test score use: different aims,different methods

Advances in validity theory and alacrity in validation practice have suffered because the term validity has been used to refer to two incompatible concerns: (1) the degree of support for specified interpretations of test scores (i.e. intended score meaning) and (2) the degree of support for specified applications (i.e. intended test uses). This article provides a brief summary of current validity theory, explication of a critical flaw in the current conceptualisation of validity, and a framework that both accommodates and differentiates validation of test score inferences and justification of test use.     

Laying the Groundwork for a Licensure Assessment

Licensure is a strict form of practice regulation: People must first earn a license before they may legally engage in professional practice. Typically, the last hurdle in the licensure process is the passing of an assessment of skills (responsibilities) and knowledge important for competent beginning professional practice. A content-oriented approach to validation is often applied to licensure assessments, and implementing a job analysis is a critical component in such a validation strategy. The job analysis conducted to define the content domain for the School Leaders Licensure Assessment—a licensure assessment for school principals—is described. Content experts first defined responsibilities and knowledge areas believed to be important for competent, beginning professional practice. The survey responses of 2,460 school principals confirmed these judgments. The methods and results of the job analysis are discussed.     

The Faculty Self-Reported Assessment Survey (FRAS): Differentiating Faculty Knowledge and Experience in Assessment

Science, technology, engineering, and mathematics education reform efforts have called for widespread adoption of evidence-based teaching in which faculty members attend to student outcomes through assessment practice. Awareness about the importance of assessment has illuminated the need to understand what faculty members know and how they engage with assessment knowledge and practice. The Faculty Self-Reported Assessment Survey (FRAS) is a new instrument for evaluating science faculty assessment knowledge and experience. Instrument validation was composed of two distinct studies: an empirical evaluation of the psychometric properties of the FRAS and a comparative known-groups validation to explore the ability of the FRAS to differentiate levels of faculty assessment experience. The FRAS was found to be highly reliable (α = 0.96). The dimensionality of the instrument enabled distinction of assessment knowledge into categories of program design, instrumentation, and validation. In the known-groups validation, the FRAS distinguished between faculty groups with differing levels of assessment experience. Faculty members with formal assessment experience self-reported higher levels of familiarity with assessment terms, higher frequencies of assessment activity, increased confidence in conducting assessment, and more positive attitudes toward assessment than faculty members who were novices in assessment. These results suggest that the FRAS can reliably and validly differentiate levels of expertise in faculty knowledge of assessment.     

Using rasch measurement to score,evaluate, and improve examinations in an anatomy course

Any examination that involves moderate to high stakes implications for examinees should be psychometrically sound and legally defensible. Currently, there are two broad and competing families of test theories that are used to score examination data. The majority of instructors outside the high‐stakes testing arena rely on classical test theory (CTT) methods. However, advances in item response theory software have made the application of these techniques much more accessible to classroom instructors. The purpose of this research is to analyze a common medical school anatomy examination using both the traditional CTT scoring method and a Rasch measurement scoring method to determine which technique provides more robust findings, and which set of psychometric indicators will be more meaningful and useful for anatomists looking to improve the psychometric quality and functioning of their examinations. Results produced by the more robust and meaningful methodology will undergo a rigorous psychometric validation process to evaluate construct validity. Implications of these techniques and additional possibilities for advanced applications are also discussed. Anat Sci Educ 7: 450–460. © 2014 American Association of Anatomists.     

无纸化考试手段的实现——以湖南省职业院校职业能力考试为例

湖南省职业院校职业能力考试手段,经历了传统的纸笔考试、纸笔和无纸化相结合、无纸化考试三个阶段,通过多年的摸索,充分发挥计算机辅助评价技术在教育测量和学习评价方面的作用,实现了全省职业院校职业能力考试的无纸化。本文重点介绍湖南省职业院校职业能力无纸化考试实现的方法。     

Improving lecturers' assessment practice in higher education: a problem-based approach

This article reports on a project based on the authors' concerns about assessment practice in higher education in the United Kingdom. These concerns stem from their experience as university lecturers, and from knowledge of research evidence and policy issues relating to assessment in higher education. Given the complex nature of higher education, improvement rather than mere change in assessment procedures will require the full participation of people who possess intimate day-to-day contextual knowledge, including an understanding of those aspects of the situation which are not easily measurable or even readily observable. The article details (a) a Popperian problem-based approach to action research designed to facilitate improvements in higher education assessment practice, and (b) how this methodology was adopted, tested and developed at the University of Sunderland. The authors’ aim is to encourage colleagues in other higher education institutions to adopt and further test this methodology in subsequent research.[1]     

Validity Issues in Computer-Based Testing

Advances in technology are stimulating the development of complex, computerized assessments. The prevailing rationales for developing computer-based assessments are improved measurement and increased efficiency. In the midst of this measurement revolution, test developers and evaluators must revisit the notion of validity. In this article, we discuss the potential positive and negative effects computer-based testing could have on validity, review the literature regarding validation perspectives in computer-based testing, and provide suggestions regarding how to evaluate the contributions of computer-based testing to more valid measurement practices. We conclude that computer-based testing shows great promise for enhancing validity, but at this juncture, it remains equivocal whether technological innovations in assessment have led to more valid measurement.     

Historical view of the influences of measurement and writing theories on the practice of writing assessment in the United States

The purpose of this study is to examine the interactions among measurement theories, writing theories, and writing assessments in the United States from an historical perspective. The assessment of writing provides a useful framework for examining how theories influence, and in some cases fail to influence actual practice. Two research traditions are described to classify measurement theories (test-score and scaling), and three research traditions are proposed for classifying writing theories (form, idea and content, and sociocultural context). The results of this study trace the impact of measurement and writing traditions on writing assessment practices within selected time periods during the 20th century in the United States. One of the major findings of this historical analysis is that measurement theory has had a strong influence on writing assessments, while writing theory has had minimal influence on writing assessments. We also found support for the idea that a new discipline of writing assessment has emerged. This new discipline combines multiple fields including the writing, composition, and measurement communities of scholars, and it has the potential to set the stage for the future of writing assessment in the 21st century.     

Attitudes, Behaviors, and Reasons for Changing Responses Following Answer-Changing Instruction

Contrary to the adage warning against changing test answers, mean gain from changing has been an invariant research finding. Consistency of this gain was tested for students instructed about the research results, and composition of the gain was analyzed by examining the students' reasons for changing. Students in six graduate measurement classes instructed about the answer-changing literature responded to three exams and a questionnaire. Mean gain remained positive and consistent with gain for previously studied uninstructed groups; amount of change was also stable. "Rethinking the item and conceptualizing a better answer" was the most frequent reason given for changing. "Rereading the item and better understanding the question" was the second most cited reason, followed by "rereading/rethinking" combined, and "making a clerical error." For each frequently used reason, wrong-to-right (WR) changes were in the majority. Implications for research and practice are discussed.     

Cattell-Horn-Carroll (CHC) theory-based assessment with deaf and hard of hearing children in the school setting

The general purpose of the present article is to emphasize contemporary research-based and theory-based assessment, specifically Cattell-Horn-Carroll (CHC) theory-based assessment (Carroll, 1993, 1997; Horn & Noll, 1997), in work with deaf and hard of hearing students in the school setting. The article focuses on the history of cognitive ability theory and test development and interpretation, as well as contemporary perspectives, including recent applications of CHC-based assessment useful with deaf and hard of hearing students in school-based evaluations. Implications for future research and educational practice are discussed.     

基于马尔科夫链的外语教学动态评估模型

动态评估近年来成为教育测量领域的热点。在动态评估的定量测量研究中,传统心理测量模式在评估前提、评估过程及预测学习者潜能方面存在局限,阻碍了动态评估在外语教学实践中的应用。基于马尔科夫链构建的动态评估模型以学习者基础的差异性为前提,在评估对象的个体性、评估过程的动态性与评估结果的预测性方面独具优 势。本文以大学英语词汇教学为例,阐述模型在外语教学动态评估中的应用,为教学实践中动态评估的定量测量研究 提供了科学的评估工具。     

Test Design with Cognition in Mind

One of the primary themes of the National Research Council's 2001 book Knowing What Students Know was the importance of cognition as a component of assessment design and measurement theory ( NRC, 2001 ). One reaction to the book has been an increased use of sophisticated statistical methods to model cognitive information available in test data. However, the application of these cognitive-psychometric methods is fruitless if the tests to which they are applied lack a formal cognitive structure. If assessments are to provide meaningful information about student ability, then cognition must be incorporated into the test development process much earlier than in data analysis. This paper reviews recent advancements in cognitively-based test development and validation, and suggests various ways practitioners can incorporate similar methods into their own work.     

A statewide writing assessment model: Student proficiency and future implications

This paper is an examination of statewide district writing achievement gain data from the Nebraska Statewide Writing Assessment system and implications for statewide assessment writing models. The writing assessment program is used to gain compliance with the United States No Child Left Behind Law (NCLB), a federal effort to influence school district assessment and accountability. While the research literature is mixed with some viewing statewide writing assessment as narrowing writing curriculum and “standardizing” writing, others point out that it facilitates sharing between teachers to improve writing and teachers are empowered by their participation in the process.Nebraska's Statewide Writing Assessment, based on the six-trait model, has strong support from teachers regarding their perception of the assessment in supporting teaching practices and student success. Examination of achievement gain data revealed gains in 4th, 8th, and 11th grades in the average number of students rated proficient from one administration of the assessment to the next.Implications for educational practice, recommendations for future research, and relevant related issues (NCLB concerns and measurement issues) are discussed.