Designing and Validating Assessments of Complex Thinking in Science

Typical assessment systems often measure isolated ideas rather than the coherent understanding valued in current science classrooms. Such assessments may motivate students to memorize, rather than to use new ideas to solve complex problems. To meet the requirements of the Next Generation Science Standards, instruction needs to emphasize sustained investigations, and assessments need to create a detailed picture of students’ conceptual understanding and reasoning processes.

This article describes the design process and potential for automated scoring of 2 forms of inquiry assessment: Energy Stories and MySystem. To design these assessments, we formed a partnership of teachers, discipline experts, researchers, technologists, and psychometricians to align curriculum, assessments, and rubrics. We illustrate how these items document middle school students’ reasoning about energy flow in life science. We used evidence from review by science teachers and experts in the discipline; classroom experiments; and psychometric analysis to validate the assessments, rubrics, and automated scoring.     

Evaluating Comparative Judgment as an Approach to Essay Scoring

ABSTRACT

As an alternative to rubric scoring, comparative judgment generates essay scores by aggregating decisions about the relative quality of the essays. Comparative judgment eliminates certain scorer biases and potentially reduces training requirements, thereby allowing a large number of judges, including teachers, to participate in essay evaluation. The purpose of this study was to assess the validity, labor costs, and efficiency of comparative judgments as a potential substitute for rubric scoring. An analysis of two essay prompts revealed that comparative judgment measures were comparable to rubric scores at a level similar to that expected of two professional scorers. The comparative judgment measures correlated slightly higher than rubric scores with a multiple-choice writing test. Score reliability exceeding .80 was achieved with approximately nine judgments per response. The average judgment time was 94 seconds, which compared favorably to 119 seconds per rubric score. Practical challenges to future implementation are discussed.     

Accountability: A Search for a Solution

The work students create should help them develop the knowledge, skills, and attitudes needed in the adult world. In this article, the authors identify adult roles essential for young people to develop and outline ways teachers can use curriculum standards to create assignments and assessments that prepare youths for those adult roles in society. They also provide guidelines for scoring these assessments and a sample rubric.     

The Quality of Local District Assessments Used in Nebraska's School-Based Teacher-Led Assessment and Reporting System (STARS)

A sample of 293 local district assessments used in the Nebraska STARS (School-based Teacher-led Assessment and Reporting System), 147 from 2004 district mathematics assessment portfolios and 146 from 2003 reading assessment portfolios, was scored with a rubric evaluating their quality. Scorers were Nebraska educators with background and training in assessment. Raters reached an agreement criterion during a training session; however, analysis of a set of 30 assessments double-scored during the main scoring session indicated that the math ratings remained reliable during scoring, while the reading ratings did not. Therefore, this article presents results for the 147 mathematics assessments only. The quality of local mathematics assessments used in the Nebraska STARS was good overall. The majority were of high quality on characteristics that go to validity (alignment with standards, clarity to students, appropriateness of content). Professional development for Nebraska teachers is recommended on aspects of assessment related to reliability (sufficiency of information and scoring procedures).     

"Do I need to know this for the exam?" Using popular media, inquiry-based laboratories, and a community of scientific practice to motivate students to learn developmental biology

One of the greatest challenges instructors face is getting students to connect with the subject in a manner that encourages them to learn. In this essay, we describe the redesign of our Developmental Biology course to foster a deeper connection between students and the field of developmental biology. In our approach, we created a community of scientific practice focused on the investigation of environmental impacts on embryonic development and informed by popular and scientific media, the students' own questions, and the instructor. Our goals were to engage students in meaningful ways with the material, to develop students' science process skills, and to enhance students' understanding of broad principles of developmental biology. Though significant challenges arose during implementation, assessments indicate using this approach to teach undergraduate developmental biology was successful.     

师生满意度：中小学学校评估的核心指标

师生满意度是教师和学生对所处学校整体情况的切身体验并基于个人感知和需要而做出的评判。师生满意度包括教师满意度和学生满意度,教师的满意度直接影响学校办学水平和教育质量,学生的满意度则是学校办学水平和教育质量的体现。由于中小学学校办学绩效难以量化、教育质量标准的多样化以及教育效果的滞后性,因此,可以将师生满意度设定为对中小学学校评估的核心指标。此外,师生满意度调查的实施也可以进一步提高中小学学校评估工作的科学性和有效性。做好师生满意度调查工作应当转变评估理念、明确调查指标、优化问卷设计、运用好师生满意度调查结果等改进学校工作。     

Assessment rubrics: towards clearer and more replicable design,research and practice

‘Rubric’ is a term with a variety of meanings. As the use of rubrics has increased both in research and practice, the term has come to represent divergent practices. These range from secret scoring sheets held by teachers to holistic student-developed articulations of quality. Rubrics are evaluated, mandated, embraced and resisted based on often imprecise and inconsistent understandings of the term. This paper provides a synthesis of the diversity of rubrics, and a framework for researchers and practitioners to be clearer about what they mean when they say ‘rubric’. Fourteen design elements or decision points are identified that make one rubric different from another. This framework subsumes previous attempts to categorise rubrics, and should provide more precision to rubric discussions and debate, as well as supporting more replicable research and practice.     

Unpacking the data: an analysis of the use of Danielson’s (2007) <Emphasis Type="Italic">Framework for Professional Practice</Emphasis> in a teaching residency program

This mixed methods study examines one teacher preparation program’s use of Danielson’s 2007 Framework for Professional Practice, with an emphasis on how different stakeholders in the traditional student teaching triad rated student teachers, called residents, and justified their ratings. Data sources include biannual self-assessments of each resident as well as assessments by the residents’ cooperating teachers and university supervisors based on the Framework, including both a numerical score for each of the 22 indicators and a written justification for the highest and lowest scores in each of the four domains. Findings show significant differences in terms of how stakeholders are rating residents’ teaching practice. The variation in scores and rationales raises questions about the reliability and validity of the results of the Framework for use as a tool to evaluate student teachers. Implications for practice include the need to consider multiple and potentially conflicting roles, such as that of providing feedback while also evaluating student teachers. In addition, we consider the costs and benefits of more extensive training around the Framework within teacher preparation, if a lack of expertise with the rubric was the cause for the variation. Finally, we consider implications for student teachers around the different messages they may be receiving about what it means to learn to teach.     

Using Student Video Cases to Assess Pre-service Elementary Teachers’ Engineering Teaching Responsiveness

Our study addresses the need for new approaches to prepare novice elementary teachers to teach both science and engineering, and for new tools to measure how well those approaches are working. This in particular would inform the teacher educators of the extent to which novice teachers are developing expertise in facilitating their students’ engineering design work. One important dimension to measure is novice teachers’ abilities to notice the substance of student thinking and to respond in productive ways. This teacher noticing is particularly important in science and engineering education, where students’ initial, idiosyncratic ideas and practices influence the likelihood that particular instructional strategies will help them learn. This paper describes evidence of validity and reliability for the Video Case Diagnosis (VCD) task, a new instrument for measuring pre-service elementary teachers’ engineering teaching responsiveness. To complete the VCD, participants view a 6-min video episode of children solving an engineering design problem, describe in writing what they notice about the students’ science ideas and engineering practices, and propose how a teacher could productively respond to the students. The rubric for scoring VCD responses allowed two independent scorers to achieve inter-rater reliability. Content analysis of the video episode, systematic review of literature on science and engineering practices, and solicitation of external expert educator responses establish content validity for VCD. Field test results with three different participant groups who have different levels of engineering education experience offer evidence of construct validity.     

Authentically assessing graduate teaching: outside and beyond neo-liberal constructs

In this paper, we challenge the current focus on ‘best practice’, graduate teacher tests, and student test scores as the panacea for ensuring teaching quality and argue for ways of thinking about evidence of quality beginning teaching outside and beyond the current neoliberal accountability discourses circulating in Australia and other countries. We suggest that teacher educators need to reinsert themselves as key players in the debates around quality beginning teaching, rather than being viewed as a source of the problem. To enable teacher educators to assume accountability for quality beginning teachers, we propose the framework of a capstone teacher performance assessment—a structured portfolio called the Authentic Teacher Assessment (ATA)—and examine examples of these assessments through the lens of critical discourse analysis. As a measure of ‘readiness to teach’, the ATA is compared with supervising teachers’ assessments of preservice teachers. We argue that structured portfolios that include artefacts derived from preservice teachers’ practice in classrooms along with graduate teacher self assessments provide a stronger accountability measure of effective beginning teaching and demonstrably address the current anxiety regarding ‘evidence’. We suggest that such an approach should be reliable enough to be ‘read’ by external assessors (and moderated across other teacher education institutions). Rigorous research on a national basis is called for in order to develop and implement a structured portfolio as rich evidence of graduates’ quality and readiness to teach.     

Reliability and Preliminary Use of a Rubric to Assess Pre-Kindergarten Teachers’ Video Uses

The presence and use of new technologies in early childhood settings are rapidly increasing. One technology tool used in early childhood settings is monthly DVD classroom newsletters, yet there is a lack of assessments to support pre-kindergarten teachers’ uses of such DVD newsletter technology—in general and in specific. The present study helps to fill this gap by developing and testing a revised rubric to evaluate the quality of monthly DVD classroom newsletters. Results indicate that the revised Monthly DVD Classroom Newsletter-Rubric exhibited good overall reliability. We suggest that the use of a rubric to assess pre-kindergarten teacher-created monthly DVD classroom newsletters supports teachers’ decision making about technology uses and professional development.     

Analysing the integration of engineering in science lessons with the Engineering-Infused Lesson Rubric

ABSTRACT

Science teachers are being called on to incorporate engineering practices into their classrooms. This study explores whether the Engineering-Infused Lesson Rubric, a new rubric designed to target best practices in engineering education, could be used to evaluate the extent to which engineering is infused into online science lessons. Eighty lessons were selected at random from three online repositories, and coded with the rubric. Overall results documented the strengths of existing lessons, as well as many components that teachers might strengthen. In addition, a subset of characteristics was found to distinguish lessons with the highest level of engineering infusion. Findings are discussed in relation to the potential of the rubric to help teachers use research evidence-informed practice generally, and in relation to the new content demands of the U.S. Next Generation Science Standards, in particular.     

PSYCHOMETRIC ANALYSIS OF A 5E LEARNING CYCLE LESSON PLAN ASSESSMENT INSTRUMENT

The purpose of this paper is to describe the procedures and the analysis of an instrument designed to measure preservice teachers’ ability to develop appropriate 5E learning cycle lesson plans. The 5E inquiry lesson plan (ILP) rubric is comprised of 12 items with a scoring range of zero to four points per item. Content validity was determined through the expertise of a panel of five science educators. Sixty six preservice teachers enrolled in elementary science methods at three universities prepared lesson plans, which were scored by their instructors using the ILP rubric. Using a Pearson two-tailed correlation, inter-rater reliability was established at a value of 0.83. An exploratory factor analysis provided evidence of construct validity, with three factors. The factors included (1) explore, (2) engage/explain/elaborate, and (3) evaluate. In addition, a secondary analysis revealed the means and standard deviations of the students' performance on each of the phases of the 5E that include: engage, explore, explain, elaborate, and evaluate. The engage item held the highest mean rating, and the evaluation items had the lowest mean ratings. Examination of the instrument's structure in light of the 5E phases is discussed and provides directions for future revisions and research.     

The use of scoring rubrics: Reliability, validity and educational consequences

Several benefits of using scoring rubrics in performance assessments have been proposed, such as increased consistency of scoring, the possibility to facilitate valid judgment of complex competencies, and promotion of learning. This paper investigates whether evidence for these claims can be found in the research literature. Several databases were searched for empirical research on rubrics, resulting in a total of 75 studies relevant for this review. Conclusions are that: (1) the reliable scoring of performance assessments can be enhanced by the use of rubrics, especially if they are analytic, topic-specific, and complemented with exemplars and/or rater training; (2) rubrics do not facilitate valid judgment of performance assessments per se. However, valid assessment could be facilitated by using a more comprehensive framework of validity when validating the rubric; (3) rubrics seem to have the potential of promoting learning and/or improve instruction. The main reason for this potential lies in the fact that rubrics make expectations and criteria explicit, which also facilitates feedback and self-assessment.     

创设问题情境 促进智能发展

在教学过程中恰当地创设问题情境，有利于调动学生学习的积极性，激发学生积极思想，培养学生的创造能力，提高学生的自身素质，教师创设问题情境时应考虑到学生原有的认知水平，要善于“愤”、“悱”的情境，并引导学生积极思考，另外，还应从“巧”字上下功夫。     

Teachers' use of rubrics to score non‐traditional tasks: factors related to discrepancies in scoring

This study considered middle school mathematics teachers’ use of rubrics to score non‐traditional tasks. A group of eighth‐grade teachers attended a two‐day workshop where they evaluated assessment tasks and discussed the use an associated scoring rubric. Scored samples of student work submitted by the teachers indicated that they had difficulty using the rubrics for scoring. When compared to expert ratings, all except one teacher had discrepancies in scoring and some discrepancies indicated major problems. These discrepancies appear to be related to whether the task contained familiar or unfamiliar content and the mix of procedure and explanation the task required. Several other factors related to discrepancies, such as leniency errors, teacher knowledge, and the halo effect are also discussed. With the expanded use of rubrics in many arenas, these results show the need for more professional development related to rubric use.     

Science Teachers’ Representations of Classroom Practice in the Process of Formative Assessment Design

Formative assessment has been recognized as an essential element of effective classroom practice; as a result, teachers are increasingly required to create formative assessments for their classrooms. This study examines data drawn from a long-term, site-based professional development program that supported a department of biology teachers in the iterative design and enactment of common formative assessment tools. We analyze teacher conversations to understand how teachers collaborated to design formative assessments. Results indicate that when teachers attended to problems of practice related to teaching evolution, increased transparency in their talk helped build consensus about the design of formative assessment tools. These results highlight the importance of encouraging transparency in teacher dialog when they are engaged in collaborative design of formative assessments.     

Prediction assessments: Using video-based predictions to assess prospective teachers’ knowledge of students’ mathematical thinking

In order to evaluate the effectiveness of an experimental elementary mathematics field experience course, we have designed a new assessment instrument. These video-based prediction assessments engage prospective teachers in a video analysis of a child solving mathematical tasks. The prospective teachers build a model of that child’s mathematics and then use that model to predict how the child will respond to a subsequent task. In this paper, we share data concerning the evolution and effectiveness of the instrument. Results from implementation indicate moderate to high degrees of inter-rater reliability in using the rubric to assess prospective teachers’ models and predictions. They also indicate strong correlation between participation in the experimental course and prospective teachers’ performances on the video-based prediction assessments. Such findings suggest that prediction assessments effectively evaluate the pedagogical content knowledge that we are seeking to foster among the prospective teachers.     

The Use and Effectiveness of an Argumentation and Evaluation Intervention in Science Classes

This study explored teachers’ use of the Argumentation and Evaluation Intervention (AEI) and associated graphic organizer to enhance the performance of students in middle and secondary science classrooms. The results reported here are from the third year of a design study during which the procedures were developed in collaboration with teachers. A quasi-experimental pretest–posttest design with 8 experimental and 8 control teachers was used with a total of 282 students. An open-ended test assessed students’ abilities to evaluate a scientific argument made in an article. The students were asked to identify the claim and its qualifiers, identify and evaluate the evidence given for the claim, examine the reasoning in support of the claim, consider counterarguments, and construct and explain a conclusion about the claim. The quality of students’ responses was assessed using a scoring rubric for each step of the argumentation process. Findings indicated a significantly higher overall score and large effect size in favor of students who were instructed using the AEI compared to students who received traditional lecture–discussion instruction. Subgroup and subscale scores are also presented. Teacher satisfaction and student satisfaction and confidence levels are reported.