Multiple-choice exams: an obstacle for higher-level thinking in introductory science classes

Learning science requires higher-level (critical) thinking skills that need to be practiced in science classes. This study tested the effect of exam format on critical-thinking skills. Multiple-choice (MC) testing is common in introductory science courses, and students in these classes tend to associate memorization with MC questions and may not see the need to modify their study strategies for critical thinking, because the MC exam format has not changed. To test the effect of exam format, I used two sections of an introductory biology class. One section was assessed with exams in the traditional MC format, the other section was assessed with both MC and constructed-response (CR) questions. The mixed exam format was correlated with significantly more cognitively active study behaviors and a significantly better performance on the cumulative final exam (after accounting for grade point average and gender). There was also less gender-bias in the CR answers. This suggests that the MC-only exam format indeed hinders critical thinking in introductory science classes. Introducing CR questions encouraged students to learn more and to be better critical thinkers and reduced gender bias. However, student resistance increased as students adjusted their perceptions of their own critical-thinking abilities.     

Score Gains, Attitudes, and Behavior Changes Due to Answer-Changing Instruction

Six undergraduate and three graduate classes were given multiple-choice tests with subsequent evaluation of answer changes. The 300 students were tested twice, once before and once after instruction on answer changing. After each test, students were asked to complete two forms. The forms evaluated attitude toward answer changing, reasons for changing, and confidence in final answers. Students showed a significant increase in favorability toward answer changing after instruction. No significant change was found in number of answers changed. Psychology students were found to change significantly more items than were business students. Mean gain score did not change significantly after instruction. It was concluded that although instruction does lead to a change in attitude in answer changing, the number of changes and overall gain due to answer changing do not change. It was also determined that students continue to make significant gains even when their confidence in the final answer is less than 50 on a 100-point scale.     

The effect of distributed questioning with varied examples on exam performance on inference questions

The purpose of this study was to investigate whether the distributed presentation of different versions of a question would produce better performance on a new version of the question than distributed presentation of the same version of the question. A total of 48 four question sets of five alternative multiple‐choice questions were presented during a college lecture course. The answers to all the four questions in each set required an inference from the same fact statement. One question in each set appeared on both a block exam and the final. Each of the other three questions was presented either on an online quiz before class, during class or on an online quiz a week after class. Answering different versions of a question before class, during class and after class resulted in better performance on a fourth version of the question on the exam and final than answering the same question before class, during class and after class. On the final exam, 44 questions were answered twice, first under a time constraint of 90 seconds and then when response time was unlimited. The two responses were virtually identical.     

Answer Changing on Multiple-Choice Test Items Among Eighth-Grade Readers

This study was done to examine the effect of answer changing on multiple-choice test performance among good and poor readers in the eighth grade. Although the gains of poor readers were higher than those of good readers, all subjects profited significantly from changing their answers on items. For all subjects, when a single response was changed, there was a two-to-one chance that the new response would raise rather than lower the final score. Gains from answer changing on test items were slightly higher for poor readers as a group than were those for good readers. However, the result was determined not to be significant. More important, this hypothesis is strengthened by the fact that all subjects profited from answer changing. Therefore, the results were interpreted as lending support to the notion that answer-changing response among young examinees should be encouraged if there is a reasonable doubt about their “first impression.”     

Student‐Generated Pre‐Exam Questions is an Effective Tool for Participatory Learning: A Case Study from Ecology of Waterborne Pathogens Course

This multiyear study helps elucidate how the instructional practice of student‐generated questions support learning in a blended classroom in science, technology, engineering, and mathematics subjects. Students designed multiple‐choice pre‐exam questions aimed at higher levels of learning, according to Bloom's taxonomy. Student‐generated questions were edited by the instructor and then discussed by the students in the classroom and in an online forum. We tested the hypothesis that this intervention improves student learning, measured as student achievement on the exam following the intervention, and compared to student achievement on the traditional exam (prior to which a review session focused on instructor‐led recitation of the key concepts). Following the intervention in all years, average grade on the post‐intervention exam increased by 7.44%. It is important to point out that not all students benefited equally from this activity. Students who were in the 4th quintile (60% to 80%) based on the results of the 1st exam demonstrated the highest achievement improving their performance on average by 12.37% percentage points (measured as a score on the 2nd exam). Gains were not observed in the semesters when the intervention was not implemented. In this study we provided students detailed instructions on how to design questions that focus on testing higher levels of learning.     

Gender Differences in Introductory Atmospheric and Oceanic Science Exams: Multiple Choice Versus Constructed Response Questions

An analysis of 295 male and 194 female examinations from introductory atmospheric and oceanic science courses is conducted to determine whether or not there exists gender differences in the performance on multiple choice versus constructed response sections of the exams. The difference in the mean performance of males and females on constructed response relative to multiple choice sections of final exams, even in years where the females performed better than or worse than the males on both sections, is on average 5% that is significant at the 0.1% level. Gender differences on time-limited midterm exams are not significant. It is further shown that final exam performance is not significantly related to whether or not the exam starts with a multiple choice versus constructed response set of questions. While our analysis is unable to differentiate between the possibilities that multiple choice questions favor male students and the competing hypothesis that constructed response questions favor female students, existing literature is reviewed to suggest that a combination of both is possible. Nevertheless, from the analysis of our examination results, we can conclude that an exam of introductory atmospheric or oceanic science curricula, which is made up of 60% multiple choice questions and 40% constructed response questions, would not be skewed to favor any particular gender.     

高职“计算机辅助设计”课程考核方法的改革与实践

对高职专业技能类课程而言,采用以期末考试为主的传统考核方法是不科学的。笔者对柳州职业技术学院机械制造类专业的核心课程"计算机辅助设计"的考核方法进行分析与探索,设计实行"24X"新考核方法,结合学校竞赛开展课程教学,利用第三方变参化测试平台进行期末考试后,每位学生的试题考核点相同但答案不同,批改试卷工作量小,取得良好的教学效果。     

A Systematic Replication Comparing Interteaching and Lecture in the Community College Classroom

Researchers demonstrated the effectiveness of interteaching relative to lecture in 4-year university classrooms, but exploration in other settings is deficient. This systematic replication examines the extent to which interteaching leads to increased exam scores compared to traditional lecture in the community college classroom. Participants in two introductory psychology sections took identical exams following counterbalanced alternating lecture and interteaching conditions. Most student exam scores following interteaching were slightly higher than exams following the lecture condition, with statistically significant differences in two of the six exams. Students in both sections correctly answered more interteaching-based than lecture-based questions on the cumulative final exam, although these differences were not significant. During interteaching, students earned significantly more points throughout the semester compared to the lecture condition. Also, more students reportedly preferred interteaching relative to lecture. The results of this study comparing interteaching to lecture in a community college setting are consistent with the results in the original study within a 4-year university classroom. Students’ exam score gains and significant cumulative point differences suggest interteaching may be an effective alternative to traditional lecture, potentially producing meaningful differences in student performance within the community college setting.     

Combined online and in‐class pretesting improves exam performance in general psychology

This study examined the effect of distributed questioning on learning and retention in a college lecture course. A total of 48 question pairs were presented over four exams. The 16 question pairs associated with each of the three blocks of the course appeared on the block exams, and all 48 appeared on the final exam. The two questions in each pair were related to each other, so that knowing the answer to one question usually implied knowing the answer to the other. One question in each pair was included in an experimental condition, in which questions were presented online, in class, or both online and in class, before appearing in exams. These conditions were counter‐balanced across the sample. The control questions appeared only in exams. Providing a question online in advance of class, as well as in class, had a significant long‐term effect on the probability of knowing the answers to both experimental and control questions when they appeared in exams. These results demonstrate that coordinated online and in‐class instruction can significantly improve exam performance. These results are consistent with the hypothesis that distributed instruction creates more robust memory traces, rather than the hypothesis that it creates additional memory traces.     

Early intervention in college classes and improved student outcomes

This research investigates the effectiveness of an early academic intervention in Principles of Economics courses at a large public university. After the end of the fourth week of classes, students who fell below a 70% threshold on a performance measure, or had an attendance rate below 75%, were referred to the university's Student Success Center for additional academic support. A referral consisted of students being informed of their status and being given optional assistance in course specific skills through tutoring, as well as training in general skills like time management and study skills. Using a regression discontinuity framework at the referral threshold, we find that the performance intervention improved student scores on common questions on the final exam by 6.5 to 7.5 percentage points for students at or near the performance threshold. The gains are particularly large for students who entered college with below average math placement scores. These results indicate that low-cost light-touch interventions may significantly affect student academic performance.     

The effects of flipped instruction on out-of-class study time,exam performance,and student perceptions

This study assessed the impact of flipped instruction on students’ out-of-class study time, exam performance, preference, motivation, and perceptions in two sections of a large undergraduate chemistry course. Flipped instruction caused a shift in student workload without appreciably changing the overall study time. The treatment impact on student performance gradually diminished over time, showing a small but statistically significant effect with the final exam. No marked interaction was identified, indicating that flipped instruction benefited students of diverse backgrounds uniformly. Students in the flipped section showed mixed feelings with about one fifth of them displaying polarized attitudes. Open-ended student survey responses revealed non-compliance with pre-class studying as a serious implementation issue: By slowing down the overall pace of the class, it negatively affected students with different study behaviors and characteristics in ways that partly explained the small, diminishing treatment effect and absence of marked interaction.     

Learn before lecture: A strategy that improves learning outcomes in a large introductory biology class

Actively engaging students in lecture has been shown to increase learning gains. To create time for active learning without displacing content we used two strategies for introducing material before class in a large introductory biology course. Four to five slides from 2007/8 were removed from each of three lectures in 2009 and the information introduced in preclass worksheets or narrated PowerPoint videos. In class, time created by shifting lecture material to learn before lecture (LBL) assignments was used to engage students in application of their new knowledge. Learning was evaluated by comparing student performance in 2009 versus 2007/8 on LBL-related question pairs, matched by level and format. The percentage of students who correctly answered five of six LBL-related exam questions was significantly higher (p < 0.001) in 2009 versus 2007/8. The mean increase in performance was 21% across the six LBL-related questions compared with <3% on all non-LBL exam questions. The worksheet and video LBL formats were equally effective based on a cross-over experimental design. These results demonstrate that LBLs combined with interactive exercises can be implemented incrementally and result in significant increases in learning gains in large introductory biology classes.     

Assessment of learning gains associated with independent exam analysis in introductory biology

This study evaluates the impact of an independent postmidterm question analysis exercise on the ability of students to answer subsequent exam questions on the same topics. It was conducted in three sections (～400 students/section) of introductory biology. Graded midterms were returned electronically, and each student was assigned a subset of questions answered incorrectly by more than 40% of the class to analyze as homework. The majority of questions were at Bloom's application/analysis level; this exercise therefore emphasized learning at these higher levels of cognition. Students in each section answered final exam questions matched by topic to all homework questions, providing a within-class control group for each question. The percentage of students who correctly answered the matched final exam question was significantly higher (p < 0.05) in the Topic Analysis versus Control Analysis group for seven of 19 questions. We identified two factors that influenced activity effectiveness: 1) similarity in topic emphasis of the midterm-final exam question pair and 2) quality of the completed analysis homework. Our data suggest that this easy-to-implement exercise will be useful in large-enrollment classes to help students develop self-regulated learning skills. Additional strategies to help introductory students gain a broader understanding of topic areas are discussed.     

How Far Have We Moved Toward the Integration of Theory and Practice in Self-Regulation?

In this article, we address four main questions, including: What is self-regulated learning for? What key strategies do students need to guide and direct their own learning process? What cues in the learning environment trigger self-regulation strategies? What can teachers do to help student to self-regulate their learning, motivation, and effort in the classroom? We illustrate that answers to these questions have changed over time and that changing conceptualizations of the self-regulation process have influenced the assessment tools that were used. We also point to changing classroom conditions as a factor that has affected the assessment of self-regulation. Finally, we formulate some questions that need to be tackled in research on self-regulation and introduce the articles and commentaries in the special issue that provide some cutting-edge work on the use of assessment to register self-regulation over time.     

Teaching to the Test…or Testing to Teach: Exams Requiring Higher Order Thinking Skills Encourage Greater Conceptual Understanding

In order to test the effect of exam-question level on fostering student conceptual understanding, low-level and high-level quizzes and exams were administered in two sections of an introductory biology course. Each section was taught in a high-level inquiry based style but was assigned either low-level questions (memory oriented) on the quizzes and exams, or high-level questions (application, evaluation, and analysis) on the quizzes and exams for the entirety of the semester. A final exam consisting of 20 low-level and 21 high-level questions was given to both sections. We considered several theoretical perspectives based on testing effect, test expectancy, and transfer-appropriate processing literature as well as the theoretical underpinnings of Bloom’s taxonomy. Reasoning from these theoretical perspectives, we predicted that high-level exams would encourage not only deeper processing of the information by students in preparation for the exam but also better memory for the core information (learned in the service of preparing for high-level questions). Results confirmed this prediction, with students in the high-level exam condition demonstrating higher performance on both the low-level final-exam items and the high-level final exam items. This pattern suggests that students who are tested throughout the semester with high-level questions acquire deep conceptual understanding of the material and better memory for the course information, and lends support to the proposed hierarchical nature of Bloom’s taxonomy.     

Improving teacher questioning in science using ICAP theory

Abstract

The purpose of this research was to investigate the efficacy of professional development in changing two middle school science teachers’ questioning to include more questions that require deeper student responses. The professional development was based on ICAP theory which proposes a framework for identifying cognitive engagement based on what is required of students. ICAP hypothesizes that Interactive and Constructive questions, such as those requiring inferential thought and collaboration, lead to deeper thinking and therefore stronger learning gains than questions that are more Active, Passive. Teacher questioning before and after the PD showed a marked improvement in increasing the number of questions that required students to make inferences about the materials. Student gain scores from tests on the lesson’s content suggests a positive relationship between constructive questions and student learning. This paper also details a coding scheme based on ICAP that can be used to provide feedback on teachers’ questioning.     

The ‘Power Test’: its impact on student learning in a materials science course for engineering students

This paper explores the impact on students’ learning of a change in assessment in a materials science course for engineers. The theory behind the move is discussed with reference to previous work on developing deeper approaches to learning in students at university. The new concept adopted was a ‘power test‘ — an open book format final exam, with extended time and opportunities for colleague interaction. Student response was evaluated by interviews and structured group discussions using a nominal group technique and outcomes were assessed by classification of student exam answers using the SOLO taxonomy of Biggs & Collis (1982). Results were compared with those of students taking the same course in the previous year but who were assessed by standard closed book exam. Problems associated with the introduction of such a novel course assessment to an otherwise traditional degree are discussed.     

Approaches to studying and academic performance in short-essay exams

Previous research has generallyfailed to find a relation between the waystudents approach the task of studying andtheir exam grades. The present studyinvestigated why it is that a deep approach tostudying, which has been shown to result in ahigher quality of learning, does notconsistently result in higher exam grades. Theparticipants in the study were 49 third-yearpsychology students. They completedquestionnaires that assessed approaches tostudying, motivation, and intelligence. Theirfinal exam marks, along with indicators of thequality of their final exam responses and thequantity of information reproduced in the finalexam were used as indicators of academicperformance. The results showed that the deepstudy approach was linearly related to thequality of exam responses, but quadraticallyrelated to the quantity of informationreproduced. The use of the deep approach was nomore effective in facilitating high exam marksfor questions that emphasised understanding ofthe study material than for questions thatemphasised reproduction of it. Taken together,these findings suggest that students who usehigh levels of the deep approach fail toconsistently achieve higher exam grades becauseof deficiencies in the quantity of theirresponses, rather than because of theinsensitivity of exams to students'understanding of the study material.     

Increased learning observed in redesigned introductory biology course that employed web-enhanced, interactive pedagogy

Our Introduction to Biology course (BIOL 1010) changed in 2004 from a standard instructor-centered, lecture-homework-exam format to a student-centered format that used Web-enhanced, interactive pedagogy. To measure and compare conceptual learning gains in the traditional course in fall 2003 with a section of the interactive course in fall 2004, we created concept inventories for both evolution and ecology. Both classes were taught by the same instructor who had taught BIOL 1010 since 1976, and each had a similar student composition with comparable biological knowledge. A significant increase in learning gain was observed with the Web-enhanced, interactive pedagogy in evolution (traditional, 0.10; interactive, 0.19; p = 0.024) and ecology (traditional, -0.05; interactive, 0.14; p = 0.000009) when assessment was made unannounced and for no credit in the last week of classes. These results strengthen the case for augmenting or replacing instructor-centered teaching with Web-enhanced, interactive, student-centered teaching. When assessment was made using the final exam in the interactive course, for credit and after studying, significantly greater learning gains were made in evolution (95%, 0.37, p = 0.0001) and ecology (143%, 0.34, p = 0.000003) when compared with learning gains measured without credit or study in the last week of classes.     

Verbal Final Exam in Introductory Biology Yields Gains in Student Content Knowledge and Longitudinal Performance

We studied gains in student learning over eight semesters in which an introductory biology course curriculum was changed to include optional verbal final exams (VFs). Students could opt to demonstrate their mastery of course material via structured oral exams with the professor. In a quantitative assessment of cell biology content knowledge, students who passed the VF outscored their peers on the medical assessment test (MAT), an exam built with 40 Medical College Admissions Test (MCAT) questions (66.4% [n = 160] and 62% [n = 285], respectively; p < 0.001);. The higher-achieving students performed better on MCAT questions in all topic categories tested; the greatest gain occurred on the topic of cellular respiration. Because the VF focused on a conceptually parallel topic, photosynthesis, there may have been authentic knowledge transfer. In longitudinal tracking studies, passing the VF also correlated with higher performance in a range of upper-level science courses, with greatest significance in physiology, biochemistry, and organic chemistry. Participation had a wide range but not equal representation in academic standing, gender, and ethnicity. Yet students nearly unanimously (92%) valued the option. Our findings suggest oral exams at the introductory level may allow instructors to assess and aid students striving to achieve higher-level learning.