Programming Pluralism: Using Learning Analytics to Detect Patterns in the Learning of Computer Programming

New high-frequency, automated data collection and analysis algorithms could offer new insights into complex learning processes, especially for tasks in which students have opportunities to generate unique open-ended artifacts such as computer programs. These approaches should be particularly useful because the need for scalable project-based and student-centered learning is growing considerably. In this article, we present studies focused on how students learn computer programming, based on data drawn from 154,000 code snapshots of computer programs under development by approximately 370 students enrolled in an introductory undergraduate programming course. We use methods from machine learning to discover patterns in the data and try to predict final exam grades. We begin with a set of exploratory experiments that use fully automated techniques to investigate how much students change their programming behavior throughout all assignments in the course. The results show that students’ change in programming patterns is only weakly predictive of course performance. We subsequently hone in on 1 single assignment, trying to map students’ learning process and trajectories and automatically identify productive and unproductive (sink) states within these trajectories. Results show that our process-based metric has better predictive power for final exams than the midterm grades. We conclude with recommendations about the use of such methods for assessment, real-time feedback, and course improvement.     

Skipping class in college and exam performance: Evidence from a regression discontinuity classroom experiment

In this paper we estimate the effect of class attendance on exam performance by implementing a policy in three large economics classes that required students scoring below the median on the midterm exam to attend class. This policy generated a large discontinuity in the rate of post-midterm attendance at the median of the midterm score. We estimate that near the policy threshold, the post-midterm attendance rate was 36 percentage points higher for those students facing compulsory attendance. The discontinuous attendance policy is also associated with a significant difference in performance on the final exam. We estimate that a 10 percentage point increase in a student's overall attendance rate results in a 0.17 standard deviation increase in the final exam score without adversely affecting performance on other classes taken concurrently.     

Crib sheets and exam performance in a data structures course

In this paper, we study the relationship between the use of “crib sheets” or “cheat sheets” and performance on in-class exams. Our extensive survey of the existing literature shows that it is not decisive on the questions of when or whether crib sheets actually help students to either perform better on an exam or better learn the material. We report on our own detailed analysis for a body of crib sheets created for the final exam in a junior-level Data Structures and Algorithms course. We wanted to determine whether there is any feature of the crib sheets that correlates to good exam scores. Exam performance was compared against a number of potential indicators for quality in a crib sheet. We have found that students performed significantly better on questions at the comprehension level of Bloom’s taxonomy when their crib sheet contained good information on the topic, while performance on questions at higher levels of the taxonomy did not show correlation to crib sheet contents. We have also seen that students at certain levels of performance on the final exam (specifically, medium-to-high performance) did relatively better on certain questions than other students at that performance level when they had good coverage of that question’s topic on their crib sheet.     

Student performance in computing education: an empirical analysis of online learning in programming education environments

The benefits of using online exercises have been analysed in terms of distance learning, automatic assessment and self-regulated learning. In this study, we have not found a direct proportional relationship between student performance in the course exercises that use online technologies and the exam grades. We see that the average submission rate to these online exercises is not positively correlated with the exercise points. Yet, our results confirm that doing exercises along supports student learning and skill accumulation equipping them with the knowledge of programming. While the student performance in programming courses is affected by factors such as prior background in programming, cognitive skills and the quality of teaching, completing the course exercises via learning-by-doing is an indispensable part of teaching. Based on the student feedback from the course survey, the students are highly satisfied with using online technologies as part of learning.     

“Java程序设计”课程教学改革与作业批改系统的构建

与单一的一次期末考试定胜负相比,过程性学习评价能更全面客观的考核学生的学习活动。我们对过程性学习评价中的作业环节进行研究,构建了“Java程序设计”课程的作业批改系统,该系统在学生中使用验证后,显著提高了学生的学习兴趣和Java编程水平。     

Use of Exam Wrappers to Enhance Students’ Metacognitive Skills in a Large Introductory Food Science and Human Nutrition Course

Research shows that students struggle to develop higher order thinking skills and effective study strategies during the transition from high school to college. Therefore, in addition to teaching course content, effective instructors should assist students in developing metacognitive skills, that is, the practice of thinking about their thinking. An effective assignment that assists students in thinking about their exam performance is the exam wrapper. The objectives of this study were to examine students’ metacognitive skills, evaluate the correlation between study behaviors and student performance, and assess student perception of exam wrappers. Exam wrapper assignments were offered as extra credit after the first 3 exams in a large introductory Food Science and Human Nutrition course, and student responses and exam performance were analyzed. Many students with poor exam performance overestimated their exam scores, indicating students’ self‐assessment skills could be sharpened. However, students demonstrated the ability to make and implement goals to improve study strategies throughout the semester. A modest relationship between use of study strategies and improved exam performance was observed, particularly for students with a B exam average, suggesting that students in the middle of the grade distribution may benefit most from this type of intervention. Finally, most students expressed a belief that exam wrappers helped them improve their study habits and exam scores, and that they planned to use the exam wrapper process in future classes. In summary, this study shows that the exam wrapper is a valued and effective postexam reflection tool for improving students’ self‐reported study habits.     

Explaining calibration accuracy in classroom contexts: the effects of incentives, reflection, and explanatory style

A 2 × 2 quasi-experimental design was used to investigate the impact of extrinsic incentives and reflection on students’ calibration of exam performance. We further examined the relationships among attributional style, performance, and calibration judgments. Participants were 137 college students enrolled in an educational psychology course. Results differed as a function of exam performance. Higher-performing students were very accurate in their calibration and did not show significant improvements across a semester-length course. Attributional style did not significantly contribute to their calibration judgments. Lower-performing students, however, were less accurate in their calibration, and students in the incentives condition showed significant increases in calibration. Beyond exam scores, attributional style constructs were significant predictors of calibration judgments for these students. The constructs targeting study and social variables accounted for most of the additional explained variance. The qualitative data also revealed differences by performance level in open-ended explanations for calibration judgments.     

Students’ continuity norms in the university and exam calendar: do they affect university academic performance? / Normas de permanencia y calendario de exámenes: ¿afectan al rendimiento académico universitario?

Students’ academic performance is a key factor in evaluating the efficiency and quality of university degree programmes. Not only do personal characteristics and the students’ own performance affect these results, but so does the university’s academic organization. Based on a database that contains the overall results by course and exam period in three different degree programmes from 2007–15, we estimate what external factors influenced the percentages of students sitting for and passing exams. The analysis of the data panel shows that the order of the exams and the students’ continuity norms established by the university to penalize failing grades are two incentives that improve student performance only in the ordinary exam periods. These results also imply first keeping the incentive system at the university and secondly strategically using the exam calendar for the courses with the worst overall results.     

GCSE exam stress: student perceptions of the effects on wellbeing and performance

ABSTRACT

In the UK education system, an ‘audit culture’ has led to pressures being placed on students to achieve high grades in their GCSEs. It has been suggested that schools are required to achieve good academic results and look after their students’ wellbeing, causing a conflict in relation to public examinations, such as GCSEs. School staff support both performance and wellbeing by preparing students for exams. However, research suggests that there is a danger that many underperform, or are negatively affected (emotionally) by exam stress, or both. This paper describes a research project within the UK. The aims of which were to explore the views of students who had recently taken GCSE exams, to gain an understanding of how they felt their GCSE experiences affected their wellbeing and performance, to find out what factors contributed to or alleviated their levels of exam stress, and whether theories relating to exam stress such as Achievement Goal Theory could be used to explain the individual differences in levels of exam stress. Findings and conclusions provided ways to improve the support for students during their GCSEs, improving academic performance and wellbeing.     

The contribution of classroom exams to formative evaluation of concept-level knowledge

Are students able to learn from exam experience about their level of knowledge for tested topics? Prior to taking an exam, undergraduates made predictive category learning judgments (CLJs) by estimating the percentage of questions they expected to answer correctly for six topics. After the exam, they made postdictive CLJs for the same topics. Supporting the postdiction superiority hypothesis, postdictive CLJs were slightly more accurate than predictive CLJs, indicating students could make better formative evaluations after taking an exam. However, postdiction accuracy was low, and in a second study, accuracy did not differ for predictive and postdictive CLJs. We also investigated two processes required to make accurate postdictive CLJs: monitoring performance for individual questions and accurately classifying each question with respect to the topic that it assessed. Although students performed these tasks adequately, their less-than-perfect performance would constrain their ability to accurately judge their topical knowledge from exam experience.     

Does relative grading help male students? Evidence from a field experiment in the classroom

We conduct a framed field experiment at a Dutch university to compare student effort provision and exam performance under the two most prevalent evaluation practices: absolute (criterion-referenced) and relative (norm-referenced) grading. We hypothesize that the rank-order tournament created by relative grading will increase effort provision and performance among students with competitive preferences. We use student gender and survey measures (self-reported as well as incentivized) as proxies for competitiveness. Contrary to our expectations, we find no significant impact of relative grading on preparation behavior or exam scores, neither among men nor among students with higher measures of competitiveness. We discuss several potential explanations for this finding, and argue that it is likely attributable to the low value that students in our sample attach to academic excellence.     

Cognitive ability,exam-related emotions and exam performance: A field study in a college setting

Using lab and field data, this paper investigates (a) the degree to which pre- and post-exam positive and negative achievement emotions and cognitive ability are uniquely and jointly associated with exam performance and (b) the degree to which exam performance influences subsequent post-exam emotions. Based on a sample of 102 students, results show that ability influences exam performance both directly and indirectly via pre-exam emotions. Also, ability and distraction interact to influence performance such that ability has a buffering effect. Distraction has a significant deleterious effect on performance for low ability students, but does not disrupt performance for high ability students. Moreover, positive emotions facilitate performance by decreasing distraction whereas negative emotions hinder performance by increasing it. Finally, results show that the exam itself has a significant impact on post-exam positive and negative emotions, even when controlling for pre-exam emotions. Results are discussed in terms of the emerging research on achievement emotions beyond test anxiety.     

The Effect of Subconscious Performance Goals on Academic Performance

We investigated the impact of subconscious goals on academic performance in two field experiments. We show that unobtrusive priming of goals with regard to achievement motivation by means of a photograph improves performance in different educational contexts. High-school students who were exposed to an achievement-related photograph achieved higher grades than students in two control conditions. This effect was not affected by students’ prior performance. University students exposed to a photograph representing a specific, difficult goal reached even higher performance than students taking the exam with a general achievement photograph. For practice, subconscious goals may form a powerful, cost-effective tool to enhance academic performance. However, varying results across the experiments also prompt the need for further investigations of such effects.     

Studying computer programming externally: Who succeeds?

Learning computer programming is difficult for many students. Distance education students in particular often have problems as the difficulty of the course content is compounded by the problems of isolation from other students and their tutor. This paper reports an investigation of distance education student performance in an introductory computer programming course. The aim of the study was to develop a better understanding of the factors associated with academic success, withdrawal or failure and to attempt to identify students at risk of non‐completion. Some implications for teaching computer programming externally are discussed.     

Even after thirteen class exams,students are still overconfident: the role of memory for past exam performance in student predictions

Students often are overconfident when they predict their performance on classroom examinations, and their accuracy often does not improve across exams. One contributor to overconfidence may be that students did not have enough experience, and another is that students may under-use their knowledge of prior exam performance to predict performance on their upcoming exams. To evaluate the former, we examined student prediction accuracy across 13 exams in an introductory course on educational psychology. For the latter, we computed measures that estimate the extent to which students use the prior exam score when predicting performance and whether students should use the prior exam scores. Several outcomes are noteworthy. First, students were overconfident, and contrary to expectations, this overconfidence did not decline across exams. Second, students’ prior exam scores were not related to subsequent predictions, even though prior exam performance showed little bias with respect to predicting future performance. Thus, students appear to under-use prior performance despite its utility for improving prediction accuracy about future exam performance.     

Applying the DT-CDIO engineering design model in a flipped learning programming course

A web design course has complex and diverse skills, which may attract students with an interest in technology and art fields to learn to program. It makes a need to have a flexible learning framework to develop all students to learn in a programming course. This study was designed to develop students’ learning achievement and computational thinking (CT) abilities by using a Design Thinking (DT)—Conceive-Design-Implement-Operate (CDIO) engineering design framework in a flipped web programming course. The participants were 41 students (males?=?17, females?=?24) coming from a Taiwan University. All of the students (20–21 years old) had e-learning-related backgrounds in a teacher’s college. The experiment was conducted for 14 weeks. The flipped learning and flipped DT-CDIO course each had a total teaching time of 6 weeks, and the midterm exam and final exam each took one week. We used a questionnaire and formative assessment to examine the students’ computational thinking ability and learning achievement before and after the course was applied. The results showed the students significantly improved their learning achievement and computational thinking ability. There were no significant gender differences in learning achievement. Some gender differences could be seen in some dimensions of CT ability. This study shows that the DT-CDIO framework brings many benefits to promote interdisciplinary learning by attracting STEAM talent and providing evidence to support the importance of flipped web programming courses.

     

The impact of an online tool for monitoring and regulating learning at university: overconfidence,learning strategy,and personality

Being overconfident when estimating scores for an upcoming exam is a widespread phenomenon in higher education and presents threats to self-regulated learning and academic performance. The present study sought to investigate how overconfidence and poor monitoring accuracy vary over the length of a college course, and how an intervention consisting of (1) a monitoring exercise and (2) a monitoring and regulation strategy, improves students’ monitoring accuracy and academic performance. Moreover, we investigated how personality factors (i.e., grandiose and vulnerable narcissism, optimism) influence monitoring accuracy. We found that the Monitoring and Regulation Strategy positively influenced monitoring accuracy and exam scores, whereas the Monitoring Exercise that confronted students with their overconfidence protected students against overconfidence in the second exam score prediction but did not affect exam score. The results further revealed that exam score predictions lowered from the start to the end of the course for both poor and high performing students, but still leaving poor performers overconfident and high performers underconfident. Topic knowledge gained in the course did not wash out the Dunning Kruger effect, and results indicate that poor and high performers use different cues when predicting exam scores. Both grandiose and vulnerable narcissism contributed to overconfidence on exam score predictions but not on the Monitoring Exercise. These findings underline the potential of the Monitoring and Regulation Strategy intervention and ask for upscaling it to include measurements of self-regulated learning activities.     

Excel操作题自动阅卷系统的设计与实现

介绍了用学生提交的文档与样卷比对进行自动阅卷的设计思路及方法,详细说明了题库设置和自动阅卷的实现方法,给出了其中的关键代码,并对系统的效果进行了测试及评估．     

Providing the answers does not improve performance on a college final exam

In the context of an upper-level psychology course, even when students were given an opportunity to refer to text containing the answers and change their exam responses in order to improve their exam scores, their performance on these questions improved slightly or not at all. Four experiments evaluated competing explanations for the students’ failure to correct their answers. Experiments 1–3 ruled out ceiling effects, cognitive bias from a previous response and item selection effects, respectively, as explanations of the result. Experiment 4 showed that no more than 41% of the students comprehended the paragraphs well enough to find the answer. Furthermore, even this 41% of the students did not put sufficient effort into finding the answer, regardless of the impact on their grade, when they were not coerced to do so.