A systematic review of trends and findings in research employing drawing assessment in science education

ABSTRACT

In this study, we reviewed 76 journal articles on employing drawing assessment as a research tool in science education. Findings from the systematic review suggest four justifications for using drawing as a type of research tool, including assessment via drawing as (a) an alternative method considering young participants’ verbal or writing abilities, and affective or economic reasons, (b) a unique method that can reveal aspects not easily measured by other methods, (c) a major method that reflects characteristics of science subjects, and (d) a formative assessment to diagnose students’ ideas to benefit their learning. Furthermore, five research trends of studies using drawing as assessment tools are identified, including: (a) students’ conceptions of scientists from the Draw-a-Scientist-Test (DAST) and evolving studies, (b) students’ understanding or mental models of science concepts, (c) participants’ conceptions of science learning or teaching, (d) students’ inquiry abilities and modelling skills via drawing, and (e) technology to support drawing. For each trend, we synthesised and commented on the current findings. A framework conceptualising phases and issues when designing research and instruments employing drawing assessments is proposed. The review provides insights into the design and future direction of research employing drawing assessments in science education.     

Discursive Modes and Their Pedagogical Functions in Model-Based Inquiry (MBI) Classrooms

This research investigated the emergent discursive modes and their pedagogical functions found in model-based inquiry (MBI) science classrooms. A sample of four high school physics classrooms was video-recorded and analysed using a newly established discourse mode analysis framework. Qualitative methods were employed to identify the most salient types and sequences of discursive modes and to interpret their pedagogical functions in the context of MBI. Results indicated that exploring was one of the most frequently used discursive modes in the MBI classrooms. Sometimes two different types of exploring discourse were connected sequentially so that the pattern of exploring (scientific phenomena)–exploring (student ideas) appeared. The discursive sequences of exploring–retrieving and exploring–negotiating also emerged whereby the teacher guided students to align their models with the canonical knowledge of science. The teacher also helped students in modelling using elaborating and reformulating discourse modes as well as metadiscourse. Although the MBI classes were implemented in a manner similar to teacher-guided inquiry, it was also observed that students participated actively in several discursive practices. Implications for model- or modelling-based science instruction and science education research were discussed.     

Using Educative Assessments to Support Science Teaching for Middle School English-language Learners

Grounded in Hallidayan perspectives on academic language, we report on our development of an educative science assessment as one component of the language-rich inquiry science for English-language learners teacher professional learning project for middle school science teachers. The project emphasizes the role of content-area writing to support teachers in diagnosing their students’ emergent understandings of science inquiry practices, science content knowledge, and the academic language of science, with a particular focus on the needs of English-language learners. In our current school policy context, writing for meaningful purposes has received decreased attention as teachers struggle to cover large numbers of discrete content standards. Additionally, high-stakes assessments presented in multiple-choice format have become the definitive measure of student science learning, further de-emphasizing the value of academic writing for developing and expressing understanding. To counter these trends, we examine the implementation of educative assessment materials—writing-rich assessments designed to support teachers’ instructional decision making. We report on the qualities of our educative assessment that supported teachers in diagnosing their students’ emergent understandings, and how teacher–researcher collaborative scoring sessions and interpretation of assessment results led to changes in teachers’ instructional decision making to better support students in expressing their scientific understandings. We conclude with implications of this work for theory, research, and practice.     

Large‐scale interventions in science education: The road to utopia?

This article provides commentary on the five articles in this issue on large‐scale interventions in science education for diverse student groups in varied educational settings. Using these articles as a point of departure, I discuss three challenges to science education research and practice. The first concerns the changed meaning of science education reform over the past two decades. The science education research and development community must be aware of the shifting policy and public awareness context of their work and react accordingly if it wishes for its work to make a difference in the lives of teachers and students. The second addresses the importance of teacher professional development and the role that well‐designed and internally valid research plays in developing knowledge in this area. The third discusses the chaotic and contradictory nature of educational policy in this country and argues that our field needs to align our research work with the ways in which real and impactful decisions are made about education in general and science education in particular. © 2012 Wiley Periodicals, Inc. J Res Sci Teach 49: 420–427, 2012     

Assessment of science learning: Living in interesting times

Beginning with a reference to living in a time of both uncertainty and opportunity, this article presents a discussion of key areas where shared understanding is needed if we are to successfully realize the design and use of high quality, valid assessments of science. The key areas discussed are: (1) assessment purpose and use, (2) the nature of assessment and the importance of research on learning, (3) assessment design processes, (4) validity arguments, (5) measurement and statistical inference, (6) affordances of technology, and (7) systems of assessment. After introducing each vital area, the article discusses how each of the five articles in the special issue is connected to the areas. Concluding comments emphasize the reminder that despite the large amount of work to be done, we are well positioned to realize the high quality, valid science education assessments that we need for K‐16 science education. © 2012 Wiley Periodicals, Inc. J Res Sci Teach 49: 831–841, 2012     

Moving PISA Results into the Policy Arena: Perspectives on Knowledge Transfer for Future Considerations and Preparations

Evidence-based policies, decisions, and practices are highly valued and underachieved in the international mathematics and science education reforms. Many in the mathematics and science education research communities lament the lack of influence that research results have on the education profession, schools, and teaching. Academic research done in isolation of end-users—with the faint hope that teachers, politicians, and bureaucrats will access and utilise these results to inform curriculum, assessment, and instruction and to influence public policy—has not worked. Some funding agencies require dissemination of research and development results to the broader political and education communities; therefore, applicants agree to these requirements without fully realizing the breadth of these demands. However, to achieve such knowledge transfer requirements, researchers need to become more (a) aware of the needs, players, and processes of ‘speaking truth to power’; (b) active in knowledge transfer and influencing public policy; and (c) alert to values and normative premises of the policy makers. This article outlines the essential principles, barriers within the academic community, international efforts, and future considerations for knowledge transfer regarding international assessments. Specific articles on PISA 2000, 2003, and 2006 included in this special issue are used to illustrate these insights into verification of curricular influences, educational opportunity and equality, regional comparisons, and direct influence on policy.     

Responsible research and innovation indicators for science education assessment: how to measure the impact?

ABSTRACT

The emerging paradigm of responsible research and innovation (RRI) in the European Commission policy discourse identifies science education as a key agenda for better equipping students with skills and knowledge to tackle complex societal challenges and foster active citizenship in democratic societies. The operationalisation of this broad approach in science education demands, however, the identification of assessment frameworks able to grasp the complexity of RRI process requirements and learning outcomes within science education practice. This article aims to shed light over the application of the RRI approach in science education by proposing a RRI-based analytical framework for science education assessment. We use such framework to review a sample of empirical studies of science education assessments and critically analyse it under the lenses of RRI criteria. As a result, we identify a set of 86 key RRI assessment indicators in science education related to RRI values, transversal competences and experiential and cognitive aspects of learning. We argue that looking at science education through the lenses of RRI can potentially contribute to the integration of metacognitive skills, emotional aspects and procedural dimensions within impact assessments so as to address the complexity of learning.     

Classroom action research on formative assessment in a context-based chemistry course

Context-based science courses stimulate students to reconstruct the information presented by connecting to their prior knowledge and experiences. However, students need support. Formative assessments inform both teacher and students about students’ knowledge deficiencies and misconceptions and how students can be supported. Research on formative assessments suggests a positive impact on students’ science achievement, although its success depends on how the formative assessment is implemented in class. The aim of this study was to provide insights into the effects of formative assessments on achievement during a context-based chemistry course on lactic acid. In a classroom action research setting, a pre-test/post-test control group design with switching replications was applied. Student achievement was measured in two pre-tests, two post-tests and a retention test. Participants were Grade 9 students from one secondary school in the Netherlands. Repeated-measures analysis showed a significant effect of formative assessments on students’ achievement. During the implementation of the formative assessments, intriguing discussions emerged between students, between students and teacher, and between teachers. Adding formative assessments to context-based approaches reinforces their strength to meet with the current challenges of chemistry education. Formative assessments affect students’ achievement positively and stimulate feedback between students and teacher(s).     

Measuring Knowledge Integration Learning of Energy Topics: A two-year longitudinal study

Although researchers call for inquiry learning in science, science assessments rarely capture the impact of inquiry instruction. This paper reports on the development and validation of assessments designed to measure middle-school students’ progress in gaining integrated understanding of energy while studying an inquiry-oriented curriculum. The assessment development was guided by the knowledge integration framework. Over 2 years of implementation, more than 4,000 students from 4 schools participated in the study, including a cross-sectional and a longitudinal cohort. Results from item response modeling analyses revealed that: (a) the assessments demonstrated satisfactory psychometric properties in terms of reliability and validity; (b) both the cross-sectional and longitudinal cohorts made progress on integrating their understanding energy concepts; and (c) among many factors (e.g. gender, grade, school, and home language) associated with students’ science performance, unit implementation was the strongest predictor.     

Science teachers’ mission impossible?: a qualitative study of obstacles in assessing students’ practical abilities

Science teachers regard practical work as important and many claim that it helps students to learn science. Besides theoretical knowledge, such as concepts and formulas, practical work is considered to be an integral and basic part of science education. As practical work is perceived and understood in different ways, comparing the results between classes and schools is difficult. One way of making the results comparable is to develop systematic inquiries to be assessed in national large-scale tests. However, introducing similar testing conditions in a laboratory environment is not always possible. Although the instructions and assessment guides for such tests are detailed, many obstacles need to be overcome if equality in the overall test situation is to be achieved. This empirical case study investigates two secondary school science teachers’ assessments of 15–16 years old students in three separate groups in the practical part of a Swedish national test in chemistry. Data are gathered using two video cameras and three pairs of spy camera glasses. The results show that individual and independent assessments are difficult due to the social interactions that take place and the physical sources of errors that occur in this type of setting.     

Multi‐level Assessment of Scientific Content Knowledge Gains Associated with Socioscientific Issues‐based Instruction

This study explored the impact of using a socioscientific issue (SSI) based curriculum on developing science content knowledge. Using a multi‐level assessment design, student content knowledge gains were measured before and after implementation of a three‐week unit on global warming (a prominent SSI) that explored both the relevant science content and the controversy surrounding global warming. Measures of student content knowledge were made using a standards‐aligned content knowledge exam (distal assessment) and a curriculum‐aligned exam (proximal assessment). Data were collected from 108 students enrolled from two schools. Quantitative analysis of the distal assessment indicated that student post‐test scores were statistically significantly different than their pre‐test scores (F = 15.31, p<0.001). Qualitative analyses of student responses from the proximal assessment indicated that students, on average, expressed more accurate, more detailed, and more sophisticated understandings of global warming, the greenhouse effect, and the controversy and challenges associated with these issues following the three‐week unit. Combined results from the proximal and distal assessments explored in this study offer important evidence in supporting the efficacy of using SSI as contexts for science education. In addition to a discussion of the components of an SSI‐based curriculum, this study provides support for the use of SSI as a context for learning science content.     

Science and Language Teachers’ Assessment of Upper Secondary Students’ Socioscientific Argumentation

Researchers and policy-makers have recognized the importance of including and promoting socioscientific argumentation in science education worldwide. The Swedish curriculum focuses more than ever on socioscientific issues (SSI) as well. However, teaching socioscientific argumentation is not an easy task for science teachers and one of the more distinguished difficulties is the assessment of students’ performance. In this study, we investigate and compare how science and Swedish language teachers, participating in an SSI-driven project, assessed students’ written argumentation about global warming. Swedish language teachers have a long history of teaching and assessing argumentation and therefore it was of interest to identify possible gaps between the two groups of teachers’ assessment practices. The results showed that the science teachers focused on students’ content knowledge within their subjects, whereas the Swedish language teachers included students’ abilities to select and use content knowledge from reliable reference resources, the structure of the argumentation and the form of language used. Since the Swedish language teachers’ assessment correlated more with previous research about quality in socioscientific argumentation, we suggest that a closer co-operation between the two groups could be beneficial in terms of enhancing the quality of assessment. Moreover, SSI teaching and learning as well as assessment of socioscientific argumentation ought to be included in teacher training programs for both pre- and in-service science teachers.     

Science assessments for all: Integrating science simulations into balanced state science assessment systems

This article reports on the collaboration of six states to study how simulation‐based science assessments can become transformative components of multi‐level, balanced state science assessment systems. The project studied the psychometric quality, feasibility, and utility of simulation‐based science assessments designed to serve formative purposes during a unit and to provide summative evidence of end‐of‐unit proficiencies. The frameworks of evidence‐centered assessment design and model‐based learning shaped the specifications for the assessments. The simulations provided the three most common forms of accommodations in state testing programs: audio recording of text, screen magnification, and support for extended time. The SimScientists program at WestEd developed simulation‐based, curriculum‐embedded, and unit benchmark assessments for two middle school topics, Ecosystems and Force & Motion. These were field‐tested in three states. Data included student characteristics, responses to the assessments, cognitive labs, classroom observations, and teacher surveys and interviews. UCLA CRESST conducted an evaluation of the implementation. Feasibility and utility were examined in classroom observations, teacher surveys and interviews, and by the six‐state Design Panel. Technical quality data included AAAS reviews of the items' alignment with standards and quality of the science, cognitive labs, and assessment data. Student data were analyzed using multidimensional Item Response Theory (IRT) methods. IRT analyses demonstrated the high psychometric quality (reliability and validity) of the assessments and their discrimination between content knowledge and inquiry practices. Students performed better on the interactive, simulation‐based assessments than on the static, conventional items in the posttest. Importantly, gaps between performance of the general population and English language learners and students with disabilities were considerably smaller on the simulation‐based assessments than on the posttests. The Design Panel participated in development of two models for integrating science simulations into a balanced state science assessment system. © 2012 Wiley Periodicals, Inc. J Res Sci Teach 49: 363–393, 2012     

Measuring knowledge integration: Validation of four‐year assessments

Science education needs valid, authentic, and efficient assessments. Many typical science assessments primarily measure recall of isolated information. This paper reports on the validation of assessments that measure knowledge integration ability among middle school and high school students. The assessments were administered to 18,729 students in five states. Rasch analyses of the assessments demonstrated satisfactory item fit, item difficulty, test reliability, and person reliability. The study showed that, when appropriately designed, knowledge integration assessments can be balanced between validity and reliability, authenticity and generalizability, and instructional sensitivity and technical quality. Results also showed that, when paired with multiple‐choice items and scored with an effective scoring rubric, constructed‐response items can achieve high reliabilities. Analyses showed that English language learner status and computer use significantly impacted students' science knowledge integration abilities. Students who took the assessment online, which matched the format of content delivery, performed significantly better than students who took the paper‐and‐pencil version. Implications and future directions of research are noted, including refining curriculum materials to meet the needs of diverse students and expanding the range of topics measured by knowledge integration assessments. © 2011 Wiley Periodicals, Inc. J Res Sci Teach 48: 1079–1107, 2011     

Selection of Authentic Modelling Practices as Contexts for Chemistry Education

In science education, students should come to understand the nature and significance of models. In the case of chemistry education it is argued that the present use of models is often not meaningful from the students' perspective. A strategy to overcome this problem is to use an authentic chemical modelling practice as a context for a curriculum unit. The theoretical framework for this strategy is activity theory rooted in socio‐cultural theories on learning. An authentic chemical modelling practice is characterized by a set of motives for model development through a well‐defined modelling procedure using only relevant issue knowledge. The aim of this study was to explore, analyse, and select authentic chemical modelling practices for use in chemistry education. The suitability of the practices was reviewed by applying a stepwise procedure focused on criteria such as students' interest and ownership, modelling procedure, issue knowledge, and feasibility of the laboratory work in the classroom. It was concluded that modelling drinking‐water treatment and human exposure assessment are both suitable to serve as contexts, because both practices exhibit clear motives for model construction and the applied modelling procedures are in line with students' pre‐existing procedural modelling knowledge. The issue knowledge involved is consistent with present Dutch science curriculum, and it is possible to carry out experimental work in the classroom for model calibration and validation. The method described here to select and evaluate practices for use as contexts in chemistry education can also be used in other science domains.     

Student explanations of their science teachers’ assessments,grading practices and how they learn science

The current paper draws on data generated through group interviews with students who were involved in a larger ethnographic research project performed in three science classrooms. The purpose of the study from which this data was generated, was to understand science teachers’ assessment practices in an upper-secondary school in Sweden. During group interviews students were asked about their conceptions of what were the assessment priority of teachers, why the students were silent during lecturing and their experiences regarding peer- and self-assessments. The research design and analysis of the findings derives from what students told us about their assessments and learning sciences experiences. Students related that besides the results of the written test, they do not know what else teachers assessed and used to determine their grades. It was also found that students did not participate in the discussion on science because of peer-pressure and a fear of disappointing their peers. Student silence is also linked with student conceptions of science learning and student experiences with methodologies of teaching and learning sciences.     

Analyzing language demands facing english learners on science performance assessments: The sald framework

This article report on the development and use of an analytical framework intended to map the language demands encountered by English learners as they engage in science performance assessments. Grounded in functional and interactional views of language and language use, the authors—two science education researchers and a language scholar—developed the framework via an inductive, iterative, and systematic review of written assessment materials associated with three fifth grade science performance tasks. The resulting Science Assessment Language Demands (SALD) framework is organized around three dimensions: participant structures, communicative modes, and written texts and genres that students are called upon to read and produce. The authors used textual analysis to conduct an expert review of the written documents associated with the three assessment tasks. The results indicate that the framework can be used to document a wide range of functional and interactional language demands involved in science performance assessments. The demands revealed by the SALD framework highlight both potential challenges facing English learners during science performance assessments as well as opportunities afforded by such assessments for demonstrating their knowledge and skills and further developing language proficiency. A major implication of the study is the potential use of the framework to evaluate the language demands and opportunities of science assessments used in classrooms with English learners. © 2010 Wiley Periodicals, Inc. J Res Sci Teach 47: 909–928, 2010     

Elementary teachers’ use of content knowledge to evaluate students’ thinking in the life sciences

Science learning environments should provide opportunities for students to make sense of and enhance their understanding of disciplinary concepts. Teachers can support students’ sense-making by engaging and responding to their ideas through high-leverage instructional practices such as formative assessment (FA). However, past research has shown that teachers may not understand FA, how to implement it, or have sufficient content knowledge to use it effectively. Few studies have investigated how teachers gather information to evaluate students’ ideas or how content knowledge factors into those decisions, particularly within the life science discipline. We designed a study embedded in a multi-year professional development program that supported elementary teachers’ development of disciplinary knowledge and FA practices within science instruction. Study findings illustrate how elementary teachers’ life science content knowledge influences their evaluation of students’ ideas. Teachers with higher levels of life science content knowledge more effectively evaluated students’ ideas than teachers with lower levels of content knowledge. Teachers with higher content exam scores discussed both content and student understanding to a greater extent, and their analyses of students’ ideas were more scientifically accurate compared to teachers with lower scores. These findings contribute to theory and practice around science teacher education, professional development, and curriculum development.