Evaluating Instrument Quality in Science Education: Rasch‐based analyses of a Nature of Science test

Given the central importance of the Nature of Science (NOS) and Scientific Inquiry (SI) in national and international science standards and science learning, empirical support for the theoretical delineation of these constructs is of considerable significance. Furthermore, tests of the effects of varying magnitudes of NOS knowledge on domain‐specific science understanding and belief require the application of instruments validated in accordance with AERA, APA, and NCME assessment standards. Our study explores three interrelated aspects of a recently developed NOS instrument: (1) validity and reliability; (2) instrument dimensionality; and (3) item scales, properties, and qualities within the context of Classical Test Theory and Item Response Theory (Rasch modeling). A construct analysis revealed that the instrument did not match published operationalizations of NOS concepts. Rasch analysis of the original instrument—as well as a reduced item set—indicated that a two‐dimensional Rasch model fit significantly better than a one‐dimensional model in both cases. Thus, our study revealed that NOS and SI are supported as two separate dimensions, corroborating theoretical distinctions in the literature. To identify items with unacceptable fit values, item quality analyses were used. A Wright Map revealed that few items sufficiently distinguished high performers in the sample and excessive numbers of items were present at the low end of the performance scale. Overall, our study outlines an approach for how Rasch modeling may be used to evaluate and improve Likert‐type instruments in science education.     

My Attitudes Toward Science (MATS): the development of a multidimensional instrument measuring students’ science attitudes

The number of students in the United States choosing science, technology, engineering or mathematics careers is declining at a time when demand for these occupations is rapidly increasing. Numerous efforts have been undertaken to reverse this trend, yet results are uncertain. One’s attitude is key to many choices one makes, and this includes, for many, what career is pursued. Hence, teachers, informal science educators and researchers often wish to measure children’s attitudes towards science using a pretest and a posttest to determine the effects of a curriculum, an activity or an intervention. However, measuring children’s attitudes toward science has been problematic because of both the limited use of basic psychometrics in checking reliability and validity of instruments and the lack of a single construct of students’ attitudes towards science being surveyed. This article reports the development and testing of an instrument for measuring students’ science attitudes across several dimensions. Thirty-two scientists and teachers from the northeastern and south central United States participated in content validity trials. The instrument was field tested with 549 children (92 elementary-school students, 327 middle-school students and 130 high-school students) from 6 rural and suburban school systems located in the northeastern United States to determine inter-item reliability for each dimension. The resulting instrument, entitled My Attitudes Toward Science (MATS), has 40 items that measure four dimensions: (1) Attitude towards the subject of science; (2) Desire to become a scientist; (3) Value of science to society; and (4) Perception of scientists. The MATS, as a multidimensional instrument, can measure several facets of students’ attitude toward science and is designed to be used across grades levels and to be scored easily.     

Assessment of the psychosocial environment of university science laboratory classrooms: a cross-national study

A strong tradition in educational research has involved several widely-used instruments assessing student or teacher perceptions of characteristics of actual or preferred classroom psychosocial environment. Existing instruments, however, are unsuitable for one of the most important settings in science teaching, namely, the science laboratory class. Consequently, the present research aimed to develop and validate a new instrument, the Science Laboratory Environment Inventory (SLEI), which is specifically suited to science laboratory environments at the higher education level. The SLEI assesses students' or teachers' perceptions of five dimensions of actual or preferred classroom environment, namely, Student Cohesiveness, Open-Endedness, Integration, Rule Clarity, and Material Environment. A distinctive feature of the design of the study was that the instrument was field tested and validated cross-nationally in six different countries, namely, the USA, Canada, Australia, England, Israel, and Nigeria. The total sample consisted of 1720 students in 71 university laboratory classes. Various item and factor analyses guided the evolution of a refined version, and attested to each SLEI scale's internal consistency reliability, discriminant validity, factorial validity, predictive validity (i.e., ability to predict student outcomes), and ability to differentiate between the perceptions of students in different classes. An important finding was that the SLEI is equally valid for use in its actual and preferred versions, and for the individual or the class mean as the unit of analysis. As well, separate within-country analyses confirmed the validity and usefulness of the SLEI in each of the six countries involved in the field testing. Overall, the study attested to the general advantages of employing cross-national designs in learning environment research.     

In Pursuit of Validity: A comprehensive review of science attitude instruments 1935–2005

The purpose of this review is to present a comprehensive evaluation of science attitude instruments based on published psychometric evidence. A multitude of instruments have been used through the years and some have been linked to career choice and school performance. Substantiating such associations is of paramount importance if researchers wish to influence educational policy. However, associations are reduced, or hard to discover, if instruments have weak psychometric properties. Several databases were searched for peer‐reviewed articles that discussed the development and use of science attitude instruments. Instruments were grouped into the following categories: attitudes toward science, scientific attitudes, nature of science, scientific career interests, and other. A data abstraction and scoring rubric was used to summarize and evaluate 150 published articles that spanned 66 instruments. Most instruments had single study usage and showed an absence of psychometric evidence. This review demonstrated that there are few instruments available with the necessary psychometric data to merit recommendation. The review quantifies the current state of the research regarding the measurement of science attitude in students; the results should elicit further discussion and encourage more rigorous analyses of instruments. The findings may assist other researchers to select an instrument and alert them to its strengths and weaknesses. This review points the way forward for research in this field. Instruments already in existence should be used in repeat studies, and reliability and validity evidence should be collected and shared.     

Validation of an Instrument for Measuring Students’ Understanding of Interdisciplinary Science in Grades 4-8 over Multiple Semesters: a Rasch Measurement Study

So far, not enough effort has been invested in developing reliable, valid, and engaging assessments in school science, especially assessment of interdisciplinary science based on the new Next Generation Science Standards (NGSS). Furthermore, previous tools rely mostly on multiple-choice items and evaluation of student outcome is linked only to their raw scores in standardized tests. In addition, educational research on student science achievement always limits in certain grade/subject according to application of raw scores. This study provides a way to design, validate, and further improve an instrument to assess student understanding in interdisciplinary science across grade and time. Meanwhile, student learning growth is investigated. The data are collected from elementary/middle school student survey from an Interdisciplinary Science and Engineering Partnership (ISEP) in Northeastern part of the USA. The results show a generally good quality of instrument through various aspects of empirical evidence, such as dimensionality, model-data-fit, and validity. Finally, student learning growth in understanding interdisciplinary science shows a sharp increase between elementary and middle school. The study sheds light on developing more reliable and valid instruments in assessing student science understanding based on the new standards and it also provides suggestions of implementation in both educational research and practice.     

Measuring Primary Teachers' Attitudes Toward Teaching Science: Development of the Dimensions of Attitude Toward Science (DAS) Instrument

In this article, we present a valid and reliable instrument which measures the attitude of in-service and pre-service primary teachers toward teaching science, called the Dimensions of Attitude Toward Science (DAS) Instrument. Attention to the attitudes of primary teachers toward teaching science is of fundamental importance to the professionalization of these teachers in the field of primary science education. With the development of this instrument, we sought to fulfill the need for a statistically and theoretically valid and reliable instrument to measure pre-service and in-service teachers' attitudes. The DAS Instrument is based on a comprehensive theoretical framework for attitude toward (teaching) science. After pilot testing, the DAS was revised and subsequently validated using a large group of respondents (pre-service and in-service primary teachers) (N = 556). The theoretical underpinning of the DAS combined with the statistical data indicate that the DAS possesses good construct validity and that it proves to be a promising instrument that can be utilized for research purposes, and also as a teacher training and coaching tool. This instrument can therefore make a valuable contribution to progress within the field of science education.     

Assessing culturally sensitive factors in the learning environment of science classrooms

As schools are becoming increasingly diverse in their scope and clientele, any examination of the interaction of culturally sensitive factors of students' learning environments with learning science assumes critical importance. The purpose of this exploratory study was to develop an instrument to assess learning environment factors that are culturally sensitive, to provide initial validation information on the instrument and to examine associations between students' perceptions of their learning environments and their attitudes towards science and achievement of enquiry skills. A measure of these factors of science student's learning environment, namely the Cultural Learning Environment Questionnaire (CLEQ), was developed from past learning environment instruments and influenced by Hofstede's four dimensions of culture (Power Distance, Uncertainty Avoidance, Individualism, and Masculinity/Femininity). The reliability and discriminant validity for each scale were obtained and associations between learning environment, attitude to science and enquiry skills achievement were found.     

Comparison of personal and class forms of the science laboratory environment inventory

Existing instruments in classroom environment research have limitations when subgroups are investigated or case studies of individual students conducted. This study reports the validation and development of a personal form of the Science Laboratory Environment Inventory which is better suited to such studies. Further, systematic differences between scores on the class and personal forms of the instrument are reported along with comparisons of their associations with inquiry skill and attitudinal outcomes. Specializations: Science education, Preservice science teacher education. Specializations: Learning environments, science education, educational evaluation, curriculum. Specializations: Curriculum, science education, science laboratory teaching.     

The science support scale: Its appropriateness with high school students

Implementing the idea that more emphasis should be placed on student achievement in the affective domain is contingent upon the concurrent development of suitable instruments for the assessment of prescribed criteria. One such instrument, the Schwirian Science Support Scale (Tri-S scale), was reported in a recent NSTA publication as a promising tool for measuring student science support. Recent research using the Tri-S scale with high school pupils showed that scores on this instrument did not increase after the students had taken a tenth grade introductory course in biology. Further analysis indicated students of teachers scoring “high” in science support did not produce higher scores on the Tri-S scale than students studying biology from teachers “low” in science support. Reliability estimates using high school student scores were well below previous estimates using scores from college undergraduates. Factor analysis of inter-item correlations indicated that student interpretation of item meaning did not correspond to the five subtest structure of the Tri-S scale. Findings from this study demonstrate that the Tri-S scale is not an appropriate instrument for measuring attitudinal changes of tenth grade high school students. This study is suggestive of the fact that went and future instruments that purport to measure achievement in noncognitive areas should be carefully analyzed before they are recommended for use with specific populations.     

维柯与当代文化诗学

自20世纪80年代以来，化诗学渐渐进入中国当代学术话语体系。从理论渊源看，它源自意大利历史哲学家维柯的《新科学》。它的核心概念是诗性思维和诗性智慧，诗与科学之间永恒的张力是其立论基础。目前从事这项研究的学人，其学术贡献主要表现在对中国传统化的创造性阐释，但在工具理性日益扩张其描述疆域的当代，诗性则成为一个关于现实批判的寓言。     

Issues of validity in science attitude measurement

This article explores the work that Moore and Foy (1997) did in revising and field testing the Scientific Attitude Instrument (SAI). I begin by showing that the empirical work with the revised instrument, the SAI II, raises doubts about the instrument's validity, and I suggest that these doubts seem to coincide rather remarkably with ones raised in an extensive study of science attitude instruments published in the early 1980s. I then consider the field testing of the SAI II in light of alternative approaches to validating instruments, and I show that the revisions made to the SAI leave its validity in question. I end by considering how we might develop guidelines for validating the instruments we use in science education research. J Res Sci Teach 34: 337–341, 1997.     

Measuring specific interests in biological,physical and earth sciences in intermediate grade levels

The Science Activities Checklist which was revised from the Reed activities checklist shows promise of providing science teachers with an instrument to measure interest in biological, earth, and physical science. If teachers are sincere in their efforts to foster interest in their specific science field, this instrument, or the revision the writers are currently committed to, should help them measure behavioral objectives involving science interests.     

“The approaching storm.” Ideology power and control: The national science teachers association curriculum development in the USA

This paper examines the ideology of one the best known figures in science education in the USA, and draws attention to the relationship between the political climate and curriculum in national curriculum developments. We are mindful of the forces shaping the schooling of science in Australia, and we present this analysis as an example of the social forces that dominate education both here and overseas. Paramount is our desire to open the door for a socially responsible Australian school science experience. Social Responsibility of Science in Science Education Group.Specializations: sociology of science education, the nature of science and the production of scientific knowledge, comparative science education and environmental education. Social Responsibility of Science in Science Education Group.Specializations: comparative education with particular reference to China, the nature of science and the production of scientific knowledge.     

Measuring instructional practice in science using classroom artifacts: lessons learned from two validation studies

With growing interest in the role of teachers as the key mediators between educational policies and outcomes, the importance of developing good measures of classroom processes has become increasingly apparent. Yet, collecting reliable and valid information about a construct as complex as instruction poses important conceptual and technical challenges. This article summarizes the results of two studies that investigated the properties of measures of instruction based on a teacher‐generated instrument (the Scoop Notebook) that combines features of portfolios and self‐report. Classroom artifacts and teacher reflections were collected from samples of middle school science classrooms and rated along 10 dimensions of science instruction derived from the National Science Education Standards; ratings based on direct classroom observations were used as comparison. The results suggest that instruments that combine artifacts and self‐reports hold promise for measuring science instruction with reliability similar to, and sizeable correlations with, measures based on classroom observation. We discuss the implications and lessons learned from this work for the conceptualization, design, and use of artifact‐based instruments for measuring instructional practice in different contexts and for different purposes. Artifact‐based instruments may illuminate features of instruction not apparent even through direct classroom observation; moreover, the process of structured collection and reflection on artifacts may have value for professional development. However, their potential value and applicability on a larger scale depends on careful consideration of the match between the instrument and the model of instruction, the intended uses of the measures, and the aspects of classroom practice most amenable to reliable scoring through artifacts. We outline a research agenda for addressing unresolved questions and advancing theoretical and practical knowledge around the measurement of instructional practice. © 2011 Wiley Periodicals, Inc. J Res Sci Teach 49: 38–67, 2012     

THE INQUIRY SCIENCE IMPLEMENTATION SCALE: DEVELOPMENT AND APPLICATIONS

Instruments for evaluating the implementation of inquiry science in K-12 classrooms are necessary if evaluators and researchers are to know the extent to which programs are implemented as intended and the extent to which inquiry science teaching accounts for student learning. For evaluators and researchers to be confident about the quality of these instruments, information about their development and validation—particularly about teacher self-report instruments—must be available. In this article, we present the Inquiry Science Implementation Scale, describe the instrument’s development and the results of analyses of the validity and reliability of data collected with it, and discuss the possible uses of the instrument.     

Psychometric properties of Science Motivation Questionnaire II-Chinese version in two waves of longitudinal data

Science motivation is an important factor that directly influences students’ science learning. Numerous studies have been undertaken to develop and validate questionnaire items for measuring students’ motivation in science learning. This study is the first longitudinal examination of the Chinese version of Science Motivation Questionnaire II (SMQ II-C) in a Chinese cultural context. Using two waves of surveys, we evaluated its internal structure validity and criterion-related validity. Results showed that at each time point, scores were internally consistent and the hypothesized five-factor model was confirmed as the best model fit for the data. Results of multigroup invariance revealed the structure of the SMQ II-C was equivalent within gender subgroups. Furthermore, the present study added longitudinal invariance evidence of the SMQ II-C by sampling two-time points. Overall, this study suggests the SMQ II-C is a robust instrument for evaluating Chinese high school students’ motivation to learn science. Furthermore, boys yielded higher scores than girls among all five subscales of science motivation and significant gender differences were observed in both waves. Implications and limitations of these results are discussed.     

Evolution and validation of a personal form of an instrument for assessing science laboratory classroom environments

The research reported in this article makes two distinctive contributions to the field of classroom environment research. First, because existing instruments are unsuitable for science laboratory classes, the Science Laboratory Environment Inventory (SLEI) was developed and validated. Second, a new Personal form of the SLEI (involving a student's perceptions of his or her own role within the class) was developed and validated in conjunction with the conventional Class form (involving a student's perceptions of the class as a whole), and its usefulness was investigated. The instrument was cross-nationally fieldtested with 5,447 students in 269 senior high school and university classes in six countries, and cross-validated with 1,594 senior high school students in 92 classes in Australia. Each SLEI scale exhibited satisfactory internal consistency reliability, discriminant validity, and factorial validity, and differentiated between the perceptions of students in different classes. A variety of applications with the new instrument furnished evidence about its usefulness and revealed that science laboratory classes are dominated by closed-ended activities; mean scores obtained on the Class form were consistently somewhat more favorable than on the corresponding Personal form; females generally held more favorable perceptions than males, but these differences were somewhat larger for the Personal form than the Class form; associations existed between attitudinal outcomes and laboratory environment dimensions; and the Class and Personal forms of the SLEI each accounted for unique variance in student outcomes which was independent of that accounted for by the other form.     

Measuring knowledge of natural selection: A comparison of the CINS,an open‐response instrument,and an oral interview

Growing recognition of the central importance of fostering an in‐depth understanding of natural selection has, surprisingly, failed to stimulate work on the development and rigorous evaluation of instruments that measure knowledge of it. We used three different methodological tools, the Conceptual Inventory of Natural Selection (CINS), a modified version of Bishop and Anderson's (Bishop and Anderson [1990] Journal of Research in Science Teaching 27: 415–427) open‐response test that we call the Open Response Instrument (ORI), and an oral interview derived from both instruments, to measure biology majors' understanding of and alternative conceptions about natural selection. We explored how these instruments differentially inform science educators about the knowledge and alternative conceptions their students harbor. Overall, both the CINS and ORI provided excellent replacements for the time‐consuming process of oral interviews and produced comparable measures of key concept diversity and, to a lesser extent, key concept frequency. In contrast, the ORI and CINS produced significantly different measures of both alternative conception diversity and frequency, with the ORI results completely concordant with oral interview results. Our study indicated that revisions of both the CINS and ORI are necessary because of numerous instrument items characterized by low discriminability, high and/or overlapping difficulty, and mismatches with the sample. While our results revealed that both instruments are valid and generally reliable measures of knowledge and alternative conceptions about natural selection, a test combining particular components of both instruments—a modified version of the CINS to test for key concepts, and a modified version of the ORI to assess student alternative conceptions—should be used until a more approprite instrument is developed and rigorously evaluated. © 2008 Wiley Periodicals, Inc. J Res Sci Teach 45: 1131–1160, 2008