The Critical Thinking Analytic Rubric (CTAR): Investigating intra-rater and inter-rater reliability of a scoring mechanism for critical thinking performance assessments

The purpose of this study was to investigate the intra-rater and inter-rater reliability of the Critical Thinking Analytic Rubric (CTAR). The CTAR is composed of 6 rubric categories: interpretation, analysis, evaluation, inference, explanation, and disposition. To investigate inter-rater reliability, two trained raters scored four sets of performance-based student work samples derived from a pilot study and subsequent larger study. The two raters also blindly scored a subset of student work samples a second time to investigate intra-rater reliability. Participants in this study were high school seniors enrolled in a college preparation course. Both raters showed acceptable levels of intra-rater reliability (α ≥ 0.70) in five of the six rubric categories. One rater showed poor consistency (α = 0.56) for the analysis category of the rubric, while the other rater showed excellent consistency (α = 0.91) for the same category suggesting the need for further training of the former rater. The results of the inter-rater reliability investigation demonstrate acceptable levels of consistency (α ≥ 0.70) in all rubric categories. This investigation demonstrated that the CTAR can be used by raters to score student work samples in a consistent manner.     

The reading–science learning–writing connection: Breakthroughs,barriers, and promises

This article establishes a broad framework from which to interpret and evaluate the reading–science learning–writing connection. The presentation of breakthroughs, barriers, and promises is intended to outline the established links between, to identify current bottlenecks in thinking about, and to highlight productive inquiries into, print-based languages and scientific understanding. The ideas presented come from various disciplines connected to science education. The ideas are meant to be informative, provocative, integrative, supportive, and without hidden agenda.     

Assessing personnel practices in higher education: A misleading note of caution

Williams and I provided a model of the ideal or random flow of people from a pre-selection pool of applicants to a post-selection work force. Dometrius and Sigelman claim that our test of the ideal versus actual work force composition lacks statistical power. They also claim that our assumption of random terminations is unreasonable. I show that their two claims are wrong, because they did not use relative measures of female representation in the work force, and because they base their modification of our model on unsupported supposition.     

Development and validation of a group test of integrated science processes

The paper describes the development and validation of a group test of integrated process skills. The test assesses student performance on a set of twelve objectives related to the generic objective: planning and conducting an investigation. Evidence of content validity, construct validity, and reliability are presented in the paper. A range of generalizability coefficients from 0.77 to 0.98 is reported for specific uses of the 24-item test. Since the items measure performance on objectives that can be readily translated into classroom activity, the test has direct applicability to classroom based research, and evaluation of instruction. In addition to sound psychometric properties, the Test of Integrated Science Processes is distincitve because it includes a set of interrelated, cumulative objectives which reflect autonomous problem solving.     

Setting theoretical and empirical foundations for assessing scientific inquiry and discovery in educational programs

This study was designed to develop measures of student competence in conducting scientific inquiry. Two assessment techniques were developed. The first measures Scientific Inquiry Capabilities, variables which are indicators of diverse aspects of competence in conducting scientific inquiry. The second measures Scientific Discovery, an indicator of success in attaining scientific concepts as a result of direct investigations into natural phenomena. Thirty‐two high school students were presented with tasks requiring the building and testing of logical–mathematical models of natural phenomena. The relationship between each Scientific Inquiry Capability and success in making discoveries was tested. Several Inquiry Capabilities were identified as strongly correlated with success in Discovery, notably: Proportional Reasoning, the Coordination of Theory with Evidence, and the Disposition to Search for Necessary Underlying Principles. © 2000 John Wiley & Sons, Inc. J Res Sci Teach 37: 938–962, 2000