Threats to the validity of science performance assessments for English language learners

In recent years, at the same time that performance assessments in science have become more popular, the number of English language learners (ELLs) (i.e., students whose native language is other than English) served by the U.S. educational system has also increased rapidly. While the research base is growing in each of these areas independently, little attention has been paid to their intersection. This case study of the use of a science performance assessment with 96 ELLs in five high school science classes investigated the face, construct, and consequential validity of this intersection. Qualitative and quantitative data analyses showed that both teachers and students had an overall favorable response to the assessment, although students' English comprehension and expression skills were determining factors for certain items. While most responses were reliably scored, ELL spelling and syntax on certain responses were significant sources of error. The degree of specificity of teachers' guidance also significantly affected students' scores. Recommendations from this study include increasing the clarity of an assessment's design, allowing ELLs more time to complete assessments, and scoring by raters who are knowledgeable about typical patterns in written English for this student population. Furthermore, it is recommended that the use of performance assessments with ELLs be exploratory until such time as their validity and reliability with this population can be more adequately established. J Res Sci Teach 34: 721–743, 1997.     

The validity of assessments

The paper reviews the concept of validity, and adopts the formulation that validity is the extent to which the results of an assessment can be generalized. The evidence about the validity of paper-and-pencil tests is then examined, leading to the conclusion that the relationship between performance on such tests and criteria of occupational performance is generally very modest. Research about the factors influencing performance on assessment tests is reviewed, and the three main categories of factor found to be motivation, the conditions of assessment and the way in which the task is presented. In conclusion, the responsibility of the assessor to elicit the condidate’s best performance is contrasted with the responsibility to ensure the validity of the results of the assessment, and hence to ensure that the sample of behaviour assessed is representative of the universe of interest.     

明代前期理学走向与文化精神的变革

明代学术文化的盛衰转换,是理学践履中不断反思批判、整合开新的结果.这一结果的产生都是以明代前期理学走向为基础的,如理学的政治化所带来的双重效应,理学践履中由穷理致知转向涵养主敬,由虚玄之论向致用之学的变革等,都为明代中晚期文化精神的嬗变与哲学的转换起到了奠基作用.     

Issues of validity in science attitude measurement

This article explores the work that Moore and Foy (1997) did in revising and field testing the Scientific Attitude Instrument (SAI). I begin by showing that the empirical work with the revised instrument, the SAI II, raises doubts about the instrument's validity, and I suggest that these doubts seem to coincide rather remarkably with ones raised in an extensive study of science attitude instruments published in the early 1980s. I then consider the field testing of the SAI II in light of alternative approaches to validating instruments, and I show that the revisions made to the SAI leave its validity in question. I end by considering how we might develop guidelines for validating the instruments we use in science education research. J Res Sci Teach 34: 337–341, 1997.     

Developing and evaluating instructionally sensitive assessments in science

The purpose of this article is to address a major gap in the instructional sensitivity literature on how to develop instructionally sensitive assessments. We propose an approach to developing and evaluating instructionally sensitive assessments in science and test this approach with one elementary life‐science module. The assessment we developed was administered to 125 students in seven classrooms. The development approach considered three dimensions of instructional sensitivity; that is, assessment items should: represent the curriculum content, reflect the quality of instruction, and have formative value for teaching. Focusing solely on the first dimension, representation of the curriculum content, this study was guided by the following research questions: (1) What science module characteristics can be systematically manipulated to develop items that prove to be instructionally sensitive? and (2) Are the instructionally sensitive assessments developed sufficiently valid to make inferences about the impact of instruction on students' performance? In this article, we describe our item development approach and provide empirical evidence to support validity arguments about the developed instructionally sensitive items. Results indicated that: (1) manipulations of the items at different proximities to vary their sensitivity were aligned with the rules for item development and also corresponded with pre‐to‐post gains; and (2) the items developed at different distances from the science module showed a pattern of pre‐to‐post gain consistent with their instructional sensitivity, that is, the closer the items were to the science module, the larger the observed gains and effect sizes. © 2012 Wiley Periodicals, Inc. J Res Sci Teach 49: 691–712, 2012     

On negotiating White science: a call for cultural relevance and critical reflexivity

This article is a response to Randy Yerrick and Joseph Johnson’s article “Negotiating White Science in Rural Black America: A Case for Navigating the Landscape of Teacher Knowledge Domains”. They write about research conducted by Yerrick in which videos of his teaching practice as a White educator in a predominately Black rural classroom were examined. Their analysis is framed through Shulman’s (1986) work on “domains of teacher knowledge” and Ladson-Billings’ (1999) critical race theory (CRT). Although we appreciate a framework that attends to issues of power, such as CRT, we see a heavier emphasis on Shulman’s work in their analysis. We argue that a culturally relevant pedagogy (CRP) framework has the potential to provide a more nuanced analysis of what occurred in Yerrick’s classroom from a critical lens. Thus we examine Yerrick and Johnson’s work through the five main CRP components (as defined by Brown-Jeffy and Cooper 2011) and ultimately argue that science educators who want to promote equity in their classrooms should engage in continuous critical reflexivity, aid students in claiming voice, and encourage students to become not only producers of scientific knowledge but also users and critics of such knowledge.     

Articulating the validity evidence for a science alternate assessment

Students with the most significant cognitive disabilities (SCD) are the 1% of the total student population who have a disability or multiple disabilities that significantly impact intellectual functioning and adaptive behaviors and who require individualized instruction and substantial supports. Historically, these students have received little instruction in science and the science assessments they have participated in have not included age‐appropriate science content. Guided by a theory of action for a new assessment system, an eight‐state consortium developed multidimensional alternate content standards and alternate assessments in science for students in three grade bands (3–5, 6–8, 9–12) that are linked to the Next Generation Science Standards (NGSS Lead States, 2013 ) and A Framework for K‐12 Science Education (Framework; National Research Council, 2012 ). The great variability within the population of students with SCD necessitates variability in the assessment content, which creates inherent challenges in establishing technical quality. To address this issue, a primary feature of this assessment system is the use of hypothetical cognitive models to provide a structure for variability in assessed content. System features and subsequent validity studies were guided by a theory of action that explains how the proposed claims about score interpretation and use depend on specific assumptions about the assessment, as well as precursors to the assessment. This paper describes evidence for the main claim that test scores represent what students know and can do. We present validity evidence for the assumptions about the assessment and its precursors, related to this main claim. The assessment was administered to over 21,000 students in eight states in 2015–2016. We present selected evidence from system components, procedural evidence, and validity studies. We evaluate the validity argument and demonstrate how it supports the claim about score interpretation and use.     

Explanations as cultural tools in science education

Cultural Studies of Science Education - After reviewing the main discussions on scientific explanation in the literature, this paper outlines a particular approach to explanations based on...     

Mediating cultural borders during science field trips

In this paper, I analyze teacher and student roles in the teaching and learning of science at an informal learning center, The Outdoor Classroom. As a white middle class informal learning science teacher, I examine my struggles to teach science to students across boundaries of race, class, gender, and experience with the outdoors during field trips. Through the field trip I did not have the time or face-to-face experience to make sense of the students’ culture, see their culture in terms of capital, and align my enactment to benefit their learning. Likewise, the students did not have the time or face-to-face experience with me in order to adapt their cultural capital and build the essential stocks of symbolic and social capital. This research demonstrates how the classroom teacher draws upon previous transactions and emotions to successfully engage her students in practices that promote the participation and learning of science. Through creating culturally adaptive ways of transacting, teachers can provide opportunities for their students to generate positive emotional energy and group solidarity in the learning of science at an informal science center.     

Science assessments for all: Integrating science simulations into balanced state science assessment systems

This article reports on the collaboration of six states to study how simulation‐based science assessments can become transformative components of multi‐level, balanced state science assessment systems. The project studied the psychometric quality, feasibility, and utility of simulation‐based science assessments designed to serve formative purposes during a unit and to provide summative evidence of end‐of‐unit proficiencies. The frameworks of evidence‐centered assessment design and model‐based learning shaped the specifications for the assessments. The simulations provided the three most common forms of accommodations in state testing programs: audio recording of text, screen magnification, and support for extended time. The SimScientists program at WestEd developed simulation‐based, curriculum‐embedded, and unit benchmark assessments for two middle school topics, Ecosystems and Force & Motion. These were field‐tested in three states. Data included student characteristics, responses to the assessments, cognitive labs, classroom observations, and teacher surveys and interviews. UCLA CRESST conducted an evaluation of the implementation. Feasibility and utility were examined in classroom observations, teacher surveys and interviews, and by the six‐state Design Panel. Technical quality data included AAAS reviews of the items' alignment with standards and quality of the science, cognitive labs, and assessment data. Student data were analyzed using multidimensional Item Response Theory (IRT) methods. IRT analyses demonstrated the high psychometric quality (reliability and validity) of the assessments and their discrimination between content knowledge and inquiry practices. Students performed better on the interactive, simulation‐based assessments than on the static, conventional items in the posttest. Importantly, gaps between performance of the general population and English language learners and students with disabilities were considerably smaller on the simulation‐based assessments than on the posttests. The Design Panel participated in development of two models for integrating science simulations into a balanced state science assessment system. © 2012 Wiley Periodicals, Inc. J Res Sci Teach 49: 363–393, 2012     

Interpretive validity theory: mapping a methodology for validating educational assessments

Background:?Validity theory has evolved significantly over the past 30 years in response to the increased use of assessments across scientific, social and educational settings. The overarching trajectory of this evolution reflects a shift from a purely quantitative, positivistic approach to a conception of validity reliant on the interpretation of multiple evidence sources integrated into validity arguments. Moreover, within contemporary validity, interpretation has been emphasised as a central process; however, despite this emphasis, there have been few explicit articulations of specific interpretive methodologies applicable to the practice of validation.

Purpose:?To link contemporary theoretical foundations in validity to practical methods and structures to help guide the collection and analysis of interpretive validity evidence. By building upon existing validity theory, this paper aims to provide greater clarity on the practice of validation and contribute toward the larger developing framework for the validation of educational assessments.

Source of evidence:?An interdisciplinary, integrative review of over 60 research articles and sources related to the theory and practice of educational validation and interpretive inquiry approaches. Sources include literature from the fields of educational assessment and more broadly social scientific research.

Main argument:?As assessments in education increasingly aim to measure complex constructs that are value-laden and socially dependant, validity theory must keep pace and evolve in ways that address the inherent complexities associated with contemporary educational assessment. Through this paper, I assert that a greater understanding of interpretive methodologies represents one of the most promising areas for development of validation theory and practice. Specifically, I argue that dialectic, hermeneutic and transgressive forms of inquiry can be integrated within current argument-based structures for the collection, analysis and representation of validity evidence in several useful ways.

Conclusions:?Interpretive inquiry processes, namely dialectic, hermeneutic and transgressive forms of interpretation, serve to expand validation practice to include diverse evidences for the generation of multiple-perspective validity arguments. The paper concludes with specific implications for future research and practice within the field of interpretive validity theory.     

The validity of performance assessments for teacher licensure: Connecticut''s ongoing research

This article was based on a paper presented at the annual meeting of the American Educational Research Association, San Francisco, March 27 to March 31, 1989.     

Reframing understandings of cultural influences on learning science

Cultural Studies of Science Education - This review essay addresses issues raised in Valerie Frède’s paper entitled: Comprehension of the nightandday cycle among French and Cameroonian...     

The alignment between the fourth grade Lebanese science curriculum and classroom assessments

The purpose of this study was to investigate the alignment between the fourth-grade summative classroom assessments and the ‘Matter and Energy’ unit in the Lebanese science curriculum using: content and cognitive levels. Summative assessments were collected from a sample of 17 schools. The alignment between the classroom assessments and the Lebanese science curriculum were examined using three instruments: Curriculum Coding Sheet, Items Coding Sheet, and Item Analysis Sheet. The curricular learning objectives were used for matching the test items to the learning objectives in terms of content. Findings showed moderate content alignment; further analysis revealed that public schools had higher content alignment, and they covered a wider range of topics. In terms of cognitive level, there was a low alignment between the assessments collected and the curriculum; however, there were items that had higher cognitive level. Recommendations for curriculum developers and future research are discussed in light of these findings.     

文学解读的有效，陡

以往的文学解读,希望通过艺术分析为文学文本找到一个永恒不变的终极结论,但在具体的文学活动中已被证明是不可能的.今天的文学解读,则希望通过创造性的解读,找寻他人所没有发现过的文学意义,以此来证明自己的文学解读是有效的.     

On the validity of useless tests

A misconception exists that validity may refer only to the interpretation of test scores and not to the uses of those scores. The development and evolution of validity theory illustrate test score interpretation was a primary focus in the earliest days of modern testing, and that validating interpretations derived from test scores remains essential today. However, test scores are not interpreted and then ignored; rather, their interpretations lead to actions. Thus, a modern definition of validity needs to describe the validation of test score interpretations as a necessary, but insufficient, step en route to validating the uses of test scores for their intended purposes. To ignore test use in defining validity is tantamount to defining validity for ‘useless’ tests. The current definition of validity stipulated in the 2014 version of the Standards for Educational and Psychological Testing properly describes validity in terms of both interpretations and uses, and provides a sufficient starting point for validation.     

Cultural diversity in science education through Novelization: Against the Epicization of science and cultural centralization

Science educators are confronted with the challenge to accommodate in their classes an increasing cultural and linguistic diversity that results from globalization. Challenged by the call to work towards valuing and keeping this diversity in the face of the canonical nature of school science discourse, we propose a new way of thinking about and investigating these problems. Drawing on the work of Mikhail Bakhtin, we articulate epicization and novelization as concepts that allow us to understand, respectively, the processes of (a) centralizing and homogenizing culture and language and (b) pluralizing culture and language. We present and analyze three examples that exhibit how existing mundane science education practices tend, by means of epicization, towards a unitary language and to cultural centralization. We then propose novelization as a way for thinking the opening up of science education by interacting with and incorporating alternative forms of knowing that arise from cultural diversity. © 2011 Wiley Periodicals, Inc. J Res Sci Teach 48: 824–847, 2011