Role of comprehension on performance at higher levels of Bloom's taxonomy: Findings from assessments of healthcare professional students

The first four levels of Bloom's taxonomy were used to create quiz questions designed to assess student learning of the gross anatomy, histology, and physiology of the gastrointestinal (GI) system. Information on GI histology and physiology was presented to separate samples of medical, dental, and podiatry students in computer based tutorials where the information from the two disciplines was presented either separately or in an integrated fashion. All students were taught GI gross anatomy prior to this study by course faculty as part of the required curriculum of their respective program. Student responses to the quiz questions were analyzed to assess both the validity of Bloom's cumulative hierarchy and the effectiveness of an integrated curriculum. No statistically significant differences were found between quiz scores from students who received the integrated tutorial and from those who received the separate tutorials. Multiple regression analyses provided partial support for a cumulative hierarchy where scores on the lower levels of Bloom's taxonomy predicted scores on higher levels. Notably, in the majority of regression analyses, the comprehension score was the key foundational predictor for application and analysis scores. This study supports the suggestion that educators increase the number of comprehension level questions, even at the expense of knowledge level questions, in course assessments both to evaluate lower order cognitive skills and also as a predictor of success on questions requiring application and analysis levels of the higher order cognitive skills of Bloom's taxonomy. Anat Sci Educ 11: 433–444. © 2018 American Association of Anatomists.     

Alignment of Assessment Objectives with Instructional Objectives Using Revised Bloom's Taxonomy—The Case for Food Science and Technology Education

Abstract: Nine food science and technology (FST) subjects were assessed for alignment between the learning outcomes and assessment using revised Bloom's taxonomy (RBT) of cognitive knowledge. Conjoint analysis was used to estimate the utilities of the levels of cognitive, knowledge, and the attribute importance (cognitive process and knowledge dimension) for learning outcomes and assessments. Lecturers for these subjects produced learning outcomes for ability of students to Understand (4.935) Procedural (3.316) as well as Apply (4.491) Conceptual (3.083) knowledge. Lecturers’ expected students’ to move beyond mere recall and recognition of knowledge to higher order cognitive knowledge of apply, analyze, evaluate, and create. However, the assessments tested students’ ability to Understand (4.791) Conceptual (4.168) as well as Remember (3.217) Procedural (0.581) knowledge resulting in a misaligned teaching and learning exercise. For all the subjects, emphasis was more (52.9% to 72.9%) on the cognitive dimension than on forms of knowledge in formulating the learning outcomes, whereas emphasis placed on the cognitive (33.3% to 62.5%) dimension and the knowledge (37.5% to 66.7%) forms in the questions differed from subject to subject. The cognitive weight in the assessment was more for Understand (1.781)/Remember (0.787) Conceptual (1.416) knowledge. RBT provides an assessment framework that can be used to assist instructors in going beyond factual knowledge and comprehension to include academic skills such as application, analysis, evaluation, and creation.     

Instructor perspectives of multiple-choice questions in summative assessment for novice programmers

Learning to program is known to be difficult for novices. High attrition and high failure rates in foundation-level programming courses undertaken at tertiary level in Computer Science programs, are commonly reported. A common approach to evaluating novice programming ability is through a combination of formative and summative assessments, with the latter typically represented by a final examination. Preparation of such assessment is driven by instructor perceptions of student learning of programming concepts. This in turn may yield instructor perspectives of summative assessment that do not necessarily correlate with student expectations or abilities. In this article, we present results of our study around instructor perspectives of summative assessment for novice programmers. Both quantitative and qualitative data have been obtained via survey responses from programming instructors with varying teaching experience, and from novice student responses to targeted examination questions. Our findings highlight that most of the instructors believed that summative assessment is, and is meant to be, a valid measure of a student's ability to program. Most instructors further believed that Multiple-choice Questions (MCQs) provide a means of testing a low level of understanding, and a few added qualitative comments to suggest that MCQs are easy questions, and others refused to use them at all. There was no agreement around the proposition that if a question was designed to test a low level of skill, or a low level in a hierarchy of a body of knowledge, that such a question should or would be found to be easy by the student. To aid our analysis of assessment questions, we introduced four measures: Syntax Knowledge; Semantic Knowledge; Problem Solving Skill and the Level of Difficulty of the Problem. We applied these measures to selected examination questions, and have identified gaps between the instructor perspectives of what is considered to be an easy question and also in what is required to be assessed to determine whether students have achieved the goals of their course.     

Student‐Generated Pre‐Exam Questions is an Effective Tool for Participatory Learning: A Case Study from Ecology of Waterborne Pathogens Course

This multiyear study helps elucidate how the instructional practice of student‐generated questions support learning in a blended classroom in science, technology, engineering, and mathematics subjects. Students designed multiple‐choice pre‐exam questions aimed at higher levels of learning, according to Bloom's taxonomy. Student‐generated questions were edited by the instructor and then discussed by the students in the classroom and in an online forum. We tested the hypothesis that this intervention improves student learning, measured as student achievement on the exam following the intervention, and compared to student achievement on the traditional exam (prior to which a review session focused on instructor‐led recitation of the key concepts). Following the intervention in all years, average grade on the post‐intervention exam increased by 7.44%. It is important to point out that not all students benefited equally from this activity. Students who were in the 4th quintile (60% to 80%) based on the results of the 1st exam demonstrated the highest achievement improving their performance on average by 12.37% percentage points (measured as a score on the 2nd exam). Gains were not observed in the semesters when the intervention was not implemented. In this study we provided students detailed instructions on how to design questions that focus on testing higher levels of learning.     

Teachers' Instructional Use of Summative Student Assessment Data

This study examines the extent to which classroom teachers self-report using summative assessment data in formative ways to shape instruction. A Web-based survey was administered to elementary, middle, and high school teachers in a large, suburban school district in central Virginia. Teachers reported administering a variety of summative assessments with varying frequency, and analyzing data at the aggregate level, most often using central tendency statistics. Useful methods for disaggregating data by content standards or student subgroups were not as frequently reported. Regardless of the methods of data analysis, a majority of teachers reported using assessment results to evaluate their instructional practice and make adjustments to support student learning. The results suggest, however, that teachers engaged in a cursory analysis of student performance fairly regularly but conduct more in-depth analyses less often. The study raises questions about how teachers can effectively use summative data for instructional purposes.     

Teacher Professional Growth on Assessment Literacy: A Case Study of Prevocational Geography Education in the Netherlands

Abstract

Teachers should have the necessary assessment knowledge and skills to contribute to students’ learning. This study provides insight into how a professional development program for teachers contributed to the achievement of higher mastery levels of assessment literacy. This study used Xu and Brown's TALiP (Teacher Assessment Literacy in Practice) framework to achieve these higher levels. To evaluate the extent to which teachers demonstrated professional growth regarding their assessment literacy, teachers’ practices were evaluated through a content analysis of summative assessments. Potential changes in teachers’ knowledge, skills, and beliefs were measured through questionnaires and interviews. The outcomes showed that reflection on educational goals and on teachers’ beliefs, collaborative practice, and peer feedback played an important role in fostering higher mastery levels in assessment literacy. The outcomes of this study support the value of the TALiP framework and support the idea that professional growth could be fostered via diverse growth pathways.     

Measuring Longitudinal Gains in Student Learning: A Comparison of Rasch Scoring and Summative Scoring Approaches

This study pioneers a Rasch scoring approach and compares it to a conventional summative approach for measuring longitudinal gains in student learning. In this methodological note, our proposed methodology is demonstrated using an example of rating scales in a student survey as part of a higher education outcome assessment. Such assessments have become increasingly important worldwide for purposes of institutional accreditation and accountability to stakeholders. Data were collected from a longitudinal study by tracking self-reported learning outcomes of individual students in the same cohort who completed the student learning experience questionnaire (SLEQ) in their first and final years. Rasch model was employed for item calibration and latent trait estimation, together with a scaling procedure of concurrent calibration incorporating a randomly equivalent group design and a single group design to measure the gains in self-reported learning outcomes as yielded by repeated measures. The extent to which Rasch scoring compared to the conventional summative scoring method in its sensitivity to change was quantified by a statistical index namely relative performance (RP). Findings indicated greater ability to capture learning outcomes gains from Rasch scoring over the conventional summative scoring method, with RP values ranging from 3 to 17% in the cognitive, social, and value domains of the SLEQ. The Rasch scoring approach and the scaling procedure presented in the study can be readily generalised to studies using rating scales to measure change in student learning in the higher education context. The methodological innovations and contributions of this study are discussed.     

Improving representational competence using molecular simulations embedded in inquiry activities

The present article discusses the design and impact of computer‐based visualization tools for supporting student learning and representational competence in science. Specifically, learning outcomes and student representation use are compared between eight secondary classrooms utilizing The Connected Chemistry Curriculum and eight secondary chemistry using lecture‐based methods. Results from the quasi‐experimental intervention indicate that the curriculum and accompanying visualization tool yield only small to modest gains in student achievement on summative assessments. Analysis of student representation use on pre‐ and post‐assessments, however, indicate the students in Connected Chemistry classrooms are significantly more likely to use submicroscopic representations of chemical systems that are consistent with teacher and expert representation use. The affordances of visualization tools in inquiry activities to improve students' representational competence and conceptual understanding of content in the science classroom are discussed. © 2011 Wiley Periodicals, Inc. J Res Sci Teach 48: 1137–1158, 2011     

Does Learning-Centered Teaching Promote Grade Improvement?

When the grade distribution within a course shifts towards higher grades, it may be due to grade inflation or grade improvement. If the positive shift is accompanied by an increase in achievement or learning, it should be considered grade improvement, not grade inflation. Effective learning-centered teaching is designed to promote student learning due to increased responsibility for learning, engagement with course material, and opportunity for formative assessments prior to summative assessments of course learning outcomes, which leads to improved grades. We suggest ways that faculty members practicing learning-centered teaching can collect and analyze data to support increased learning and grade improvement.     

Climbing Bloom's taxonomy pyramid: Lessons from a graduate histology course

Bloom's taxonomy was adopted to create a subject‐specific scoring tool for histology multiple‐choice questions (MCQs). This Bloom's Taxonomy Histology Tool (BTHT) was used to analyze teacher‐ and student‐generated quiz and examination questions from a graduate level histology course. Multiple‐choice questions using histological images were generally assigned a higher BTHT level than simple text questions. The type of microscopy technique (light or electron microscopy) used for these image‐based questions did not result in any significant differences in their Bloom's taxonomy scores. The BTHT levels for teacher‐generated MCQs correlated positively with higher discrimination indices and inversely with the percent of students answering these questions correctly (difficulty index), suggesting that higher‐level Bloom's taxonomy questions differentiate well between higher‐ and lower‐performing students. When examining BTHT scores for MCQs that were written by students in a Multiple‐Choice Item Development Assignment (MCIDA) there was no significant correlation between these scores and the students' ability to answer teacher‐generated MCQs. This suggests that the ability to answer histology MCQs relies on a different skill set than the aptitude to construct higher‐level Bloom's taxonomy questions. However, students significantly improved their average BTHT scores from the midterm to the final MCIDA task, which indicates that practice, experience and feedback increased their MCQ writing proficiency. Anat Sci Educ 10: 456–464. © 2017 American Association of Anatomists.     

Integrative assessment: reframing assessment practice for current and future learning

The requirement to provide timely formative tasks that are designed to facilitate student learning and autonomy has provoked a wider examination of the role of assessment in higher education and encouraged further investigation of the alignment of learning, teaching and assessment in curriculum design frameworks. Many current authors have proposed that the primary purpose of assessment is to enhance current and future learning and that current practice tends to overemphasise the importance of assessment for progression and certification purposes. This paper proposes that a clearer distinction be made between assessment tasks designed to facilitate and test current learning through the use of formative and summative assessments, and those tasks primarily designed to enhance future learning, which could be better termed integrative assessments. This distinction would allow students and teachers to have greater clarity around the proposed outcomes and reward mechanisms associated with assessment tasks and feedback. This paper proposes that teachers should strive to incorporate four different types of assessment tasks throughout a programme of study, namely diagnostic, formative, integrative and summative tasks, and that the outcomes and reward mechanisms for different assessment types be explained more clearly to students.     

Exams as Learning Tools: A Comparison of Traditional and Collaborative Assessment in Higher Education

Exams are increasingly being used as learning tools in the form of collaborative assessments as opposed to their traditional use as a summative assessment tool for verifying individual student learning. Despite the growing popularity of collaborative assessments, few studies test the differential effects of collaborative assessments (versus traditional assessments) on student learning. This paper analyzes 16 empirical studies from various disciplines and investigates the extent to which collaborative assessments improve student learning. The paper further provides recommendations and improved practices for instructors interested in using collaborative assessment in their classroom.     

Feedback,feedforward: evaluating the effectiveness of an oral peer review exercise amongst postgraduate students

Assessment for learning approaches, such as peer review exercises may improve student performance in summative assessments and increase their satisfaction with assessment practices. We conducted a mixed methods study to evaluate the effectiveness of an oral peer review exercise among post-graduate students. We examined: (1) final assessment grades among students who did and did not take part in the peer review exercise; (2) student perceptions of the impact of the peer review exercise; and (3) student understanding of, and satisfaction with, this new assessment practice. We found that students who took part in the exercise had a significantly higher mean grade in a subsequent summative oral presentation assessment than students who did not take part in the exercise. Students gained a better understanding of assessment and marking criteria and expressed increased confidence and decreased anxiety about completing the subsequent summative assessment. Assessment for learning improves academic attainment and the learning experience in postgraduate students.     

Large-scale student assessment studies measure the results of processes of knowledge acquisition: Evidence in support of the distinction between intelligence and student achievement

Given the relatively high intercorrelations observed between mathematics achievement, reading achievement, and cognitive ability, it has recently been claimed that student assessment studies (e.g., TIMSS, PISA) and intelligence tests measure a single cognitive ability that is practically identical to general intelligence. The present article uses three lines of reasoning to show that the outcomes of schooling can and must be conceptually distinguished from the intelligence construct. First, the conceptual differences between student assessments and tests of cognitive ability are delineated. Second, results from construct validation studies providing strong empirical support for the multidimensionality of the achievement measures applied in large-scale educational assessments are reported. Third, data supporting the differential development of educational outcomes in different domains are presented.     

普通高中学生学业评价体系的重构

新课程的评价理念、结构特点和内容都和现行高中的学生学业评价不吻合,有必要重构普通高中学生学业评价体系。新的高中学生学业评价体系分模块水平和毕业水平两个层次、采用包括过程性评价与终结性评价两种方式,简称为2×2的评价体系。过程性评价的主要作用在于促进学生及时地进行反思和展示学生学习的过程。终结性评价可根据模块的内容和类型的不同,组织书面笔试或开放式、表现式的测试。     

The value of adaptivity based on cognitive style: an empirical study

Adaptive Hypermedia Systems can be developed to accommodate a variety of individual differences, including learning style and cognitive style. This study investigates the hypothesis that adaptive hypermedia accommodating cognitive styles can be beneficial for the observed learning outcomes. A prototype system, designed to be adapted to individual cognitive styles, was developed as a case study. In order to evaluate the effectiveness of the prototype system, an empirical study was conducted. This paper presents the results of the summative evaluation of the system. Statistical analyses indicated that students in the experimental group performed significantly better than students in a control group. These findings indicate that student performance is mainly affected by adaptivity based on individual cognitive styles.     

Exploring the Use of Audience Response Systems in Secondary School Science Classrooms

An audience response systems (ARS) allows students to respond to multiple choice questions using remote control devices. Once the feedback is collected and displayed, the teacher and students discuss misconceptions and difficulties experienced. ARSs have been extremely popular and effective in higher education science classrooms, although almost no research has been done at the secondary school level. The purpose of this study was to conduct a detailed formative analysis of the benefits, challenges, and use of ARSs from the perspective of 213 secondary school science students. Perceived benefits were increased student involvement (engagement, participation, and attention) and effective formative assessment of student understanding. Perceived challenges included decreased student involvement and learning when ARSs were used for summative assessment, occasional technological malfunctions, resistance to using a new method of learning, and increased stress due to time constraints when responding to questions. Finally, students rated the use of ARSs significantly higher when it was used for formative as opposed to summative assessment.