Assessing Writing Portfolios: issues in the Validity and Meaning of Scores

This article discusses an approach to analyzing performance assessments that identifies potential reasons for misfitting items and uses this information to improve on items and rubrics for these assessments. Specifically, the approach involves identifying psychometric features and qualitative features of items and rubrics that may possibly influence misfit; examining relations between these features and the fit statistic; conducting an analysis of student responses to a sample of misfitting items; and finally, based on the results of the previous analyses, modifying characteristics of the items or rubrics and reexamining fit. A mathematics performance assessment containing 53 constructed-response items scored on a holistic scale from 0 to 4 is used to illustrate the approach. The 2-parameter graded response model (Samejima, 1969) is used to calibrate the data. Implications of this method of data analysis for improving performance assessment items and rubrics are discussed as well as issues and limitations related to the use of the approach.     

Validity issues in Meta‐Analysis: Suggestions for Research and Policy

The technique of meta‐analysis has been credited with resolving research issues in the areas of class size and the efficacy of psychotherapy, yet meta‐analyses have failed to form a consensus regarding the effects of student ratings feedback to college teachers. This failure can be traced to three types of problems: (a) methodological problems with the technique of meta‐analysis; (b) the quality of the research base; and (c) the social and professional context in which research is conducted. Meta‐analysis is an important addition to what may become a rigorous review methodology, but it is not a substitute for the qualitative review. Comprehensive meta‐analyses should include (a) a discursive review of each study, (b) a report of how each effect‐size was calculated, (c) the location of the statistics on which the calculation was based, and (d) a discussion of the factors that limit the validity of the statistic. Individual authors can contribute to the validity of future quantitative syntheses by reporting their results more thoroughly and clearly.     

Criteria for performance-based teacher assessments: Validity, standards, and issues

Conclusion Validity theory, together with currently available and emerging standards for performance assessments, provides guidance for the developers of high-stakes performance assessments. It is imperative, however, that important aspects of validity and standards for quality and fairness of performance assessments be built into such assessments from their very inception. Specifying the target performance in terms legitimate to all of the assessment participants and creating an explicit methodology for integrating diverse points of view provide the foundation for defensible assessments. It is only through painstaking analyses and field work, however, that many validity-related aspects of the assessments can be satisfactorily resolved. Perhaps, with the passage of time, a cycle can be established in which these experiences from the field can inform further development of standards of performance assessment, which can then be used to raise the standard of assessment development practice. Only then can the full promise of modern validity theory be fulfilled.     

Measurement issues in designing and implementing longitudinal evaluation studies

Evaluation is required for almost all educational activities or programs, particularly for federal- and state-funded programs. Such a requirement is quite understandable given that we need to know what has been done actually produces its intended purposes. However, it is almost impossible that bad data can provide justified conclusions, and there are many evaluation studies that are circumvented by the fact that evaluation is a post-hoc test of educational activities. Thus, it is hard to produce the rigor and relevance of an evaluation study that may need good design and analysis to even provide partial answers to the original questions. The goal of this paper is to provide researchers and practitioners with some of the insights that may serve as a useful guide for designing and implementing longitudinal evaluation in a way that can improve the likelihood of high quality data for future evaluation studies.

高职毕业生职业能力结构模型研究与标准制定

通过对人才定义的内涵分析,提出了成为人才的三个基本阶段。通常情况下,高职毕业生还不能成为人才,只能是人材。为此,在充分调研基础上,研究设计了高职毕业生(人材)职业能力的三维度四层次结构模型。设计开发了《GX/SDWZ 011-2015高职毕业生(人材)职业能力要求》标准,以此作为高职院校对毕业生的职业能力进行评价的标准。     

Gender studies: some methodological and theoretical issues

In the first part of this paper, studies are reviewed which suggest that gender differences in science achievement can be explained entirely in terms of social factors, and which also suggest that these differences could be eliminated, or even reversed, by changing the social environment. For the studies examined, it is suggested that research workers may not have taken sufficient account of confounding variables in their work, and that the apparent changes in gender differential reported may be explained in terms of these confounding variables.

In the second part of the paper, studies are reviewed which seek explanation of gender differences in science achievement in terms of unequal treatment of boys and girls by science teachers in the classroom. It is concluded that the quality of the research work which has been undertaken is disappointing, and that the research evidence provided by these studies for the differential treatment of boys and girls by science teachers is far from convincing.

It is concluded that real gender differences in science achievement do exist, and that they cannot be explained fully in terms of the social environment. It is suggested that both biological and sociological theoretical perspectives on gender differences in science achievement are inadequate, and that further progress in understanding gender differences in science achievement will require a new theoretical synthesis. It is suggested that sociobiology may provide the required new theoretical synthesis by allowing for the recognition of both biological and environmental influences on science achievement.     

Canvassing conversations: obstinate issues in studies of elites and elite education

Studies of elites and elite education have largely not involved rigorous debate, either with regard to the conceptual resources deployed or methodologies adopted. Even the concepts elite and elite schools have not been problematized much. Further, there is a tendency for people to cite, rather than engage or dispute each other. So while the number of published studies increases and the field grows in size, and expands in focus, it is not necessarily growing through spirited dialog and critique. The first part of this paper considers the ways in which the methodological scope in the study of elites and elite education has been restrained and limited through the repetition of particular methodological frameworks and practices. Drawing on the work included in this special issue and other recent research, we then suggest some methodological possibilities for expanding this scope. In the second main part, we offer some provocations about the theoretical resources that are conventionally deployed. We argue that scholars need to be much more critically self-conscious about their uses of elite theories and class theories and much more aware of the ideological implications of their research.     

Senior secondary school assessment and standard-setting in Queensland,Australia: social context and paradigmatic change

ABSTRACT

This article draws on three assessment paradigms – psychometrics, outcomes-based and curriculum-based assessment – to discuss paradigmatic changes in senior school assessment and achievement standard-setting in Queensland, Australia, over the last 50 years. These include radical reforms in 1970 from university-controlled examinations to school-based assessments applying normative standard-setting, to subsequent reforms in 1978 introducing competence(curriculum)-based assessment and standards. From 2019, a new reform introduces a combination of school-based and external assessment with procedures for establishing standards still in progress.

Changes to Queensland assessment and standard-setting are discussed in terms of three preconditions for paradigm change – dissatisfaction, an alternative acceptable paradigm, and majority acceptance of change. Influence of paradigmatic origins of reformers is discussed. The amalgam of curriculum-based assessment and psychometric paradigms in the new Queensland system is considered in terms of theoretical compatibility and potential impact on the new standards.     

Standards of Validity and the Validity of Standards in Performance Asessment

What are six distinct aspects of construct validation? How do these aspects apply to performance assessment? Are the consequences of performance assessment on teaching and learning relevant to construct validation?     

Social studies teacher candidates' views on the controversial issues incorporated into their courses in Turkey

Incorporating controversial issues into pre-service teacher education provides teacher candidates with opportunities not only to gain effective civic competence but also to develop skills and experience in their teaching of controversial issues. The purpose of this study was to ascertain social studies teacher candidates' views on the controversial issues involved in their courses in Turkey. The study was based on semi-structured interviews with 15 teacher candidates. It was found that controversial issues were not sufficiently included in teacher candidates' courses and that teacher candidates have difficulty in discussing critical issues in class. This study also introduced the idea that teacher candidates experiencing controversial issues in pre-service teacher education can gain positive attitudes towards teaching such issues in their professional lives and can form their own teaching strategies. Teacher candidates came to believe that good teachers should include controversial issues related to the course content and both teachers and teacher candidates should prepare for their discussion phase.     

Validity Issues in Test Speededness

Speededness refers to the situation where the time limits on a standardized test do not allow substantial numbers of examinees to fully consider all test items. When tests are not intended to measure speed of responding, speededness introduces a severe threat to the validity of interpretations based on test scores. In this article, we describe test speededness, its potential threats to validity, and traditional and modern methods that can be used to assess the presence of speededness. We argue that more attention must be paid to this issue and that more research must be done to set appropriate time limits on power tests so that speed of responding does not interfere with the construct measured.     

Current Concerns in Validity Theory

We are at the end of the first century of work on models of educational and psychological measurement and into a new millennium. This certainly seems like an appropriate time for looking backward and looking forward in assessment. Furthermore, a new edition of the Standards for Educational and Psychological Testing (AERA, APA, & NCME, 1999) has been published, and the previous editions of the Standards have served as benchmarks in the development of measurement theory.
This backward glance will be just that, a glance. After a brief historical review focusing mainly on construct validity, the current state of validity theory will be summarized, with an emphasis on the role of arguments in validation. Then how an argument-based approach might be applied will be examined in regards to two issues in validity theory: the distinction between performance-based and theory-based interpretations, and the role of consequences in validation.     

Validity Issues in Computer-Based Testing

Advances in technology are stimulating the development of complex, computerized assessments. The prevailing rationales for developing computer-based assessments are improved measurement and increased efficiency. In the midst of this measurement revolution, test developers and evaluators must revisit the notion of validity. In this article, we discuss the potential positive and negative effects computer-based testing could have on validity, review the literature regarding validation perspectives in computer-based testing, and provide suggestions regarding how to evaluate the contributions of computer-based testing to more valid measurement practices. We conclude that computer-based testing shows great promise for enhancing validity, but at this juncture, it remains equivocal whether technological innovations in assessment have led to more valid measurement.     

Validity in quantitative content analysis

Over the past 15 years, educational technologists have been dabbling with a research technique known as quantitative content analysis (QCA). Although it is characterized as a systematic and objective procedure for describing communication, readers find insufficient evidence of either quality in published reports. In this paper, it is argued that QCA should be conceived of as a form of testing and measurement. If this argument is successful, it becomes possible to frame many of the problems associated with QCA studies under the well-articulated rubric of test validity. Two sets of procedures for developing the validity of a QCA coding protocol are provided, (a) one for developing a protocol that is theoretically valid and (b) one for establishing its validity empirically. The paper is concerned specifically with the use of QCA to study educational applications of computer-mediated communication.     

测验效度概念的新发展

1985年《教育与心理测验标准》(第5版)出版之前,效度研究的核心概念是"效标(criterion)",效度研究被视为一种用"效标"对测验的效度进行证明(verify)、对测验分数做出有效(valid)解释的过程。1985年以后,效度研究的核心概念是"证据(evidence)",效度研究被视为一种通过积累证据对测验的效度进行支持(support)、对测验分数做出合理(reasonable)解释的过程。关于效度的这种理解,突出体现在1999年出版的《教育与心理测验标准》(第6版)中。美国教育协会和美国国家教育测量学会共同组织编写的《教育测量》在业内被称为"教育测量领域的《圣经》"。2006年《教育测量》(第4版)出版以后,效度研究的核心概念演变为"理由(warrant)",效度研究被视为一种通过构造"理由系统"和"理由网络"对效度进行"论证(argument)"、对测验分数做出可接受的(plausible)解释的过程。本文结合笔者的考试实践,介绍了效度概念的新发展。     

Addressing the crisis in curriculum studies: curriculum integration that bridges issues of identity and knowledge

Many people from non-dominant backgrounds or believers from various religions want their children to acquire the best modern knowledge and to remain open to their home cultures and beliefs. However, this double aspiration poses complex challenges, and most scholars have either stressed the importance of addressing identity (and diversity) issues, or claimed that the key is to give everyone access to powerful knowledge. Beginning from curriculum studies’ alleged crisis and its relation to this dichotomy, this paper suggests that bridging concern for diverse identities and access to powerful knowledge implies devising curricula that allow for issues that are transversal to the disciplines without collapsing the boundaries between them. Since this has been generally difficult to develop, the paper reflects on the kind of curriculum integration that is needed, arriving at the idea of interstitial curriculum or connective tissue amid the disciplines. Subsequently, unique features of the International Baccalaureate Diploma Program (IBDP) are presented to exemplify these curricular interstices, and how they help to deal with the epistemological challenges posed by the initially mentioned double aspiration of many families at present.