The impact of an online tool for monitoring and regulating learning at university: overconfidence,learning strategy,and personality

Being overconfident when estimating scores for an upcoming exam is a widespread phenomenon in higher education and presents threats to self-regulated learning and academic performance. The present study sought to investigate how overconfidence and poor monitoring accuracy vary over the length of a college course, and how an intervention consisting of (1) a monitoring exercise and (2) a monitoring and regulation strategy, improves students’ monitoring accuracy and academic performance. Moreover, we investigated how personality factors (i.e., grandiose and vulnerable narcissism, optimism) influence monitoring accuracy. We found that the Monitoring and Regulation Strategy positively influenced monitoring accuracy and exam scores, whereas the Monitoring Exercise that confronted students with their overconfidence protected students against overconfidence in the second exam score prediction but did not affect exam score. The results further revealed that exam score predictions lowered from the start to the end of the course for both poor and high performing students, but still leaving poor performers overconfident and high performers underconfident. Topic knowledge gained in the course did not wash out the Dunning Kruger effect, and results indicate that poor and high performers use different cues when predicting exam scores. Both grandiose and vulnerable narcissism contributed to overconfidence on exam score predictions but not on the Monitoring Exercise. These findings underline the potential of the Monitoring and Regulation Strategy intervention and ask for upscaling it to include measurements of self-regulated learning activities.     

Improving metacognition in the classroom through instruction,training, and feedback

Accurately judging one’s performance in the classroom can be challenging considering most students tend to be overconfident and overestimate their actual performance. The current work draws upon the metacognition and decision making literatures to examine improving metacognition in the classroom. Using historical data from several semesters of an upper-level undergraduate course (N?=?127), we analyzed students’ judgments of their performance and their actual performance for two exams. Students were instructed on the concepts of overconfidence, received feedback on exams, and were given incentives for accurate calibration. We found results consistent with the “unskilled and unaware” effect Kruger & Dunning (Journal of Personality and Social Psychology, 77(6), 1121–1134, 1999) where lower performing students initially displayed overconfidence and the highest performing students initially displayed underconfidence. Importantly, students were able to change both judgments and performance such that metacognitive accuracy improved significantly from the first to the second exam. In a second study, two additional semesters for the same course used in Study 1 were examined (N?=?90). For one of the semesters feedback was not provided, allowing us to determine whether feedback can improve both metacognitive judgments and performance. Our findings revealed significant improvements in performance paired with decreases in overconfidence on Exam 2, but only for students who received feedback about their performance and judgments. We postulate that feedback may be an important component in improvement metacognitive judgments.     

Training metacognition in the classroom: the influence of incentives and feedback on exam predictions

In two semester-long studies, we examined whether college students could improve their ability to accurately predict their own exam performance across multiple exams. We tested whether providing concrete feedback and incentives (i.e., extra credit) for accuracy would improve predictions by improving students’ metacognition, or awareness of their own knowledge. Students’ predictions were almost always higher than the grade they earned and this was particularly true for low-performing students. Experiment 1 demonstrated that providing incentives but minimal feedback failed to show improvement in students’ metacognition or performance. However, Experiment 2 showed that when feedback was made more concrete, metacognition improved for low performing students although exam scores did not improve across exams, suggesting that feedback and incentives influenced metacognitive monitoring but not control.     

Undergraduate Students’ Preferences for Constructed Versus Multiple-Choice Assessment of Learning

Students (N = 161) in seven sections of an undergraduate educational psychology course rated ten performance-assessment options in collegiate courses. They rated in-class essay exams as their most preferred assessment and multiple-choice exams (in-class and out-of-class) as their least preferred. Also, student ratings of multiple papers and a term paper did not differ significantly from the rating for in-class essay exams. Overall, students preferred constructed forms of assessment over more objective assessment. With minor exceptions, student ratings of assessment preferences were generally consistent across gender and academic levels. In the main, student ratings of assessment options did not significantly correlate with exam performance in the course.     

High School Exit Exam Scores and University Performance

Many U.S. students must pass a standards-based exit exam to earn a high school diploma. The degree to which exit exams and state standards properly signal to students their preparedness for postsecondary schooling has been questioned. The alignment of test scores with college grades for students at the University of Arizona (n = 2,667) who took the Arizona high school exams was ascertained in this study. The pass/fail signal accuracy of test scores varied depending on subject: The writing cut score was well aligned with collegiate performance, the reading cut score was below expectations, and the mathematics cut score was set quite rigorously. High school content and performance standards might not be as diluted as prior research has suggested.     

The double curse of misconceptions: misconceptions impair not only text comprehension but also metacomprehension in the domain of statistics

Research shows that misconceptions are usually detrimental to text comprehension. However, whether misconceptions also impair metacomprehension accuracy, that is, the accuracy with which one self-assesses one’s text comprehension, has received far less attention. We conducted a study in which we examined students’ (N = 47) comprehension and metacomprehension accuracy (prediction accuracy and postdiction accuracy) of a statistics text as a function of their statistical misconceptions. Text comprehension and metacomprehension accuracy referred to both conceptual and procedural aspects of statistics. The results showed that students who had more misconceptions achieved poorer conceptual text comprehension and, at the same time, provided more overconfident predictions of their conceptual and procedural text comprehension than students who had fewer misconceptions. In contrast, postdiction accuracy of conceptual and procedural text comprehension was not affected by misconceptions.     

Exploring Flipped Classroom Instruction in Calculus III

In an undergraduate Calculus III class, we explore the effect of “flipping” the instructional delivery of content on both student performance and student perceptions. Two instructors collaborated to determine daily lecture notes, assigned the same homework problems, and gave identical exams; however, compared to a more traditional instructional approach, the flipped instructor utilized videos to communicate more procedural course content to students out-of-class, with time in-class spent on more conceptual activities and homework problems. Findings from two semesters indicate similar performance on more procedural problems and small to moderate gains for the flipped students (N = 74) over their traditional counterparts (N = 77) on more conceptual exam problems. However, student perceptions remain mixed, with flipped students reporting increased communication during class but traditional students perceiving more effective use of class time, despite the gains in performance for flipped students.     

College students' ratings of student effort,student ability and teacher input as correlates of student performance on multiple-choice exams

In the class session following feedback regarding their scores on multiple-choice exams, undergraduate students in a large human development course rated the strength of possible contributors to their exam performance. Students rated items related to their personal effort in preparing for the exam (identified as student effort in the paper), their ability to perform well on the exams (identified as student ability), and teacher input that might have affected their exam performance. Students rated most student effort items higher than teacher input and student ability items. Notwithstanding, across all exams, ratings of student ability and teacher input correlated more strongly with exam performance than did student effort ratings. High and low performers on the exams differed significantly on ratings of student ability and teacher input, but were more similar on ratings of student effort.     

University student’s goal profiles and metacomprehension accuracy

In this study, undergraduate students provided confidence ratings to predict future performance in answering questions drawn from the text before reading the text, after reading the text and after rereading the text. Self-reports of achievement goal orientations during reading and posttest scores were also collected. Student’s calibration index was the comparison between their predicted posttest performance and actual performance in the posttest. Correlational analyses did not reveal any statistically detectable relationships between self-reported goal orientations and monitoring accuracy, except that bias scores were marginally related to goal orientations. Further cluster analyses and analyses of variance (ANOVA) also showed that student’s multiple goal profiles failed to clearly differentiate the groups in terms of their calibration accuracy, yet performance-approach goals did distinguish overconfident from underconfident students. Plausible reasons for the finding were provided and implications for future research were also discussed.     

Category learning judgments in the classroom: Can students judge how well they know course topics?

A category learning judgment (CLJ) involves judging one’s learning or performance for a given topic or category. The present study was the first to investigate CLJs in a classroom, where students’ judgments of how well they have learned topics may be particularly relevant for guiding their study decisions. In an undergraduate statistics class, students predicted their performance on six different exam topics, as well as predicting their global exam performance, for each exam during the semester. Regarding the absolute accuracy of CLJs, we observed slight overestimation (bias), substantial deviation from accuracy (absolute bias), and little improvement across exams. Students’ CLJs varied among topics, but they were less variable than actual topic performance and were poor at discriminating well-learned from poorly-learned topics (i.e., low relative accuracy). We examined two factors predictive of CLJ accuracy: topic difficulty and student mastery of the topics. Regarding topic difficulty, a hard-easy effect was observed, such that more difficult topics produced greater overestimation and easier topics produced more underestimation. A hard-easy effect also extended to absolute bias: difficult topics produced larger deviations from accuracy than easy topics did. Regarding student mastery of topics, we found that lower mastery predicted CLJ overestimation and higher mastery predicted CLJ underestimation. Lower mastery was also associated with larger absolute bias. Compared to global judgments, CLJs were less accurate, although students were more confident in their CLJs. In sum, developing methods to improve the accuracy of CLJs in classrooms is an important direction for future research.     

Collaborative Testing Improves Performance but Not Content Retention in a Large-Enrollment Introductory Biology Class

Collaborative testing has been shown to improve performance but not always content retention. In this study, we investigated whether collaborative testing could improve both performance and content retention in a large, introductory biology course. Students were semirandomly divided into two groups based on their performances on exam 1. Each group contained equal numbers of students scoring in each grade category (“A”–“F”) on exam 1. All students completed each of the four exams of the semester as individuals. For exam 2, one group took the exam a second time in small groups immediately following the individually administered test. The other group followed this same format for exam 3. Individual and group exam scores were compared to determine differences in performance. All but exam 1 contained a subset of cumulative questions from the previous exam. Performances on the cumulative questions for exams 3 and 4 were compared for the two groups to determine whether there were significant differences in content retention. Even though group test scores were significantly higher than individual test scores, students who participated in collaborative testing performed no differently on cumulative questions than students who took the previous exam as individuals.     

Examining high-school students’ overconfidence bias in biology exam: a focus on the effects of country and gender

Accurate, rational, and scientific decision making is now considered to be the most important skill in science education. Many studies have found that overconfidence bias is one of the cognitive biases hindering people from achieving such decision making. Gender and country play crucial roles in overconfidence bias. For instance, some particular cultures and genders tend to be more overconfident than others. However, whether or not the two variables interact to influence overconfidence bias also indirectly influences decision making, especially in the context of science education. The purpose of this study is to identify the effects of country and gender on performance, confidence, and overconfidence bias in the samples of Indonesian and Korean high-school students while doing on a biology exam. The twenty-one American Association for the Advancement of Science (AAAS) questions on the topics of genetics and evolution were administered to 297 Indonesian and 235 Korean high-school students, in their first and second years. Every question was featured with a question asking students how confident they are in answering the question correctly. The two-way Analysis of Variances (2-way ANOVA) test was used to answer the research questions. Based on the analyses, we found no significance interactional effects of gender and country in test scores. In contrast, we found a significant interactional effects in both confidence in genetics and evolution. Regarding overconfidence bias, for which that we merged both concepts, we found that country had a higher influence on students’ overconfidence bias than did gender. Additionally, we found the hard-easy effect phenomenon followed overconfidence bias phenomenon. The relationships between country, gender, science education, cognitive bias, and overconfidence bias are discussed. Suggestions for reducing overconfidence bias are also provided.     

The instinct fallacy: the metacognition of answering and revising during college exams

Students often gauge their performance before and after an exam, usually in the form of rough grade estimates or general feelings. Are these estimates accurate? Should they form the basis for decisions about study time, test-taking strategies, revisions, subject mastery, or even general competence? In two studies, undergraduates took a real multiple-choice exam, described their general beliefs and feelings, tracked their performance for each question, and noted any revisions or possible revisions. Beliefs formed after the exams were poor predictors of performance. In contrast, real-time metacognitive monitoring – measured by confidence ratings for each individual question – accurately predicted performance and were a much better decisional guide. Measuring metacognitive monitoring also allowed us to examine the process of revising an answer. Should a test-taker rely on their first choice or revise in the face of uncertainty? Experience seems to show that first instincts are correct. The decision-making literature calls this the first-instinct fallacy, based on extensive analysis of revisions, and recommends revising more. However, whereas revisions have been analyzed in great detail, previous studies did not analyze the efficacy of sticking with an original choice. We found that both revising and sticking resulted in significantly more correct than incorrect outcomes, with real-time metacognition predicting when each was most appropriate.     

The Grass Isn’t Always Greener: Perceptions of and Performance on Open-Note Exams

Undergraduate biology education is often viewed as being focused on memorization rather than development of students’ critical-thinking abilities. We speculated that open-note testing would be an easily implemented change that would emphasize higher-order thinking. As open-note testing is not commonly used in the biological sciences and the literature on its effects in biology education is sparse, we performed a comprehensive analysis of this intervention on a primary literature–based exam across three large-enrollment laboratory courses. Although students believed open-note testing would impact exam scores, we found no effect on performance, either overall or on questions of nearly all Bloom’s levels. Open-note testing also produced no advantage when examined under a variety of parameters, including research experience, grade point average, course grade, prior exposure to primary literature–focused laboratory courses, or gender. Interestingly, we did observe small differences in open- and closed-note exam performance and perception for students who experienced open-note exams for an entire quarter. This implies that student preparation or in-test behavior can be altered by exposure to open-note testing conditions in a single course and that ­increased experience may be necessary to truly understand the impact of this intervention.     

Use of Exam Wrappers to Enhance Students’ Metacognitive Skills in a Large Introductory Food Science and Human Nutrition Course

Research shows that students struggle to develop higher order thinking skills and effective study strategies during the transition from high school to college. Therefore, in addition to teaching course content, effective instructors should assist students in developing metacognitive skills, that is, the practice of thinking about their thinking. An effective assignment that assists students in thinking about their exam performance is the exam wrapper. The objectives of this study were to examine students’ metacognitive skills, evaluate the correlation between study behaviors and student performance, and assess student perception of exam wrappers. Exam wrapper assignments were offered as extra credit after the first 3 exams in a large introductory Food Science and Human Nutrition course, and student responses and exam performance were analyzed. Many students with poor exam performance overestimated their exam scores, indicating students’ self‐assessment skills could be sharpened. However, students demonstrated the ability to make and implement goals to improve study strategies throughout the semester. A modest relationship between use of study strategies and improved exam performance was observed, particularly for students with a B exam average, suggesting that students in the middle of the grade distribution may benefit most from this type of intervention. Finally, most students expressed a belief that exam wrappers helped them improve their study habits and exam scores, and that they planned to use the exam wrapper process in future classes. In summary, this study shows that the exam wrapper is a valued and effective postexam reflection tool for improving students’ self‐reported study habits.     

Overconfidence in children's multi-trial judgments of learning

The underconfidence with practice effect (UWP) refers to the finding that people's judgments of learning shift from overconfidence to underconfidence on and after a first study-test trial (Koriat, Ma'ayan, & Sheffer, 2002). Finn and Metcalfe (2007, 2008) proposed that people show UWP because they use their memory of prior test performance as a cue to make subsequent judgments of learning and inadequately account for new learning (i.e. the Memory for Past Test (MPT) heuristic). In contrast to adults, 3rd and 5th graders' judgments showed persistent overconfidence on and after a first study-test trial. A second experiment tested children's ability to remember their prior test performance. Children's prior performance discriminations were accurate for items that they answered correctly on the prior trial, but were overconfident for items they had answered incorrectly indicating that their continued overconfidence was a result of faulty memory, rather than a failure to use the MPT heuristic.     

An Examination of the Outcomes of a Brief and Innovative Partnership: SJSU and Udacity

In an examination of a brief and innovative partnership, we compared outcomes for two disciplines, Elementary Statistics and General Psychology, across three formats: online as part of the San José State University-Udacity partnership (termed SJSU Plus), face-to-face (FTF), and online in a redesigned course offering. We also examine predictors of student performance in the SJSU Plus courses. The first offerings of the SJSU Plus courses showed poorer performance compared to their FTF and redesigned online equivalents. Redesigned online courses and FTF courses had similar pass rates. SJSU Plus course performance was significantly improved in the second offering of the Elementary Statistics course. More completed assignments in the SJSU Plus courses were associated with higher exam scores and final grades. We conclude that mode of delivery did not contribute significantly to variations in pass rates.     

Activation of inaccurate prior knowledge affects primary-school students’ metacognitive judgments and calibration

The study investigated whether activation of inaccurate prior knowledge before study contributes to primary-school children’s commission errors and overconfidence in these errors when learning new concepts. Findings indicate that inaccurate prior knowledge affects children’s learning and calibration. The level of children’s judgments of learning for recall responses for which they would not receive credit was inappropriately high after activation of inaccurate prior knowledge.Moreover, results showed that activation of inaccurate prior knowledge was not only detrimental for monitoring judgments during learning, but also for calibration accuracy after test taking. When judging the quality of their recall responses on the posttest, children were more overconfident when they had activated inaccurate prior knowledge. Also, the children often discarded concepts from further study after activation of inaccurate prior knowledge. These results suggest that in order to improve self-regulated learning, it may be important to detect inaccuracies in children’s prior knowledge.