Evaluating the quality of peer and self evaluations as measures of student contributions to group projects

ABSTRACT

Educators in higher education commonly use peer and self evaluations to help assess student performance on group projects. Although these evaluations provide multiple benefits, many educators are wary of using them due to concerns about their quality. This study addresses three questions debated in the literature regarding the quality of these assessments. How much do students differentiate among peer contributions through their ratings? How reliable are peer ratings? How much agreement exists between peer and self ratings? Although these questions have been addressed to varying degrees in past work, their answers have been far from settled. While many studies focus on just one of the questions, this study’s data make it possible to address all three questions for the same group of students as well as examine each question by student performance level. The evaluations assessed in this study were completed by a large number of students under conditions associated with obtaining more valid and reliable ratings. Overall, the results provide support for using peer and self evaluations to help assess student contributions to group projects. Peer ratings were largely reliable as group members generally agreed on the scores given to their peers. In addition, most students differentiated among group member contributions through their ratings. Students also tended to rate themselves higher than their peers rated them. This study has implications for how peer and self evaluations can be most effectively used by educators to measure student performance in group work.     

Affect and learning: An exploratory look into the role of affect in learning with AutoTutor

The role that affective states play in learning was investigated from the perspective of a constructivist learning framework. We observed six different affect states (frustration, boredom, flow, confusion, eureka and neutral) that potentially occur during the process of learning introductory computer literacy with AutoTutor, an intelligent tutoring system with tutorial dialogue in natural language. Observational analyses revealed significant relationships between learning and the affective states of boredom, flow and confusion. The positive correlation between confusion and learning is consistent with a model that assumes that cognitive disequilibrium is one precursor to deep learning. The findings that learning correlates negatively with boredom and positively with flow are consistent with predictions from Csikszentmihalyi's analysis of flow experiences.     

Mastering cognitive development theory in computer science education

To design an effective computer science curriculum, educators require a systematic method of classifying the difficulty level of learning activities and assessment tasks. This is important for curriculum design and implementation and for communication between educators. Different educators must be able to use the method consistently, so that classified activities and assessments are comparable across the subjects of a degree, and, ideally, comparable across institutions. One widespread approach to supporting this is to write learning objects in terms of Bloom’s Taxonomy. This, or other such classifications, is likely to be more effective if educators can use them consistently, in the way experts would use them. To this end, we present the design and evaluation of our online interactive web-based tutorial system, which can be configured and used to offer training in different classification schemes. We report on results from three evaluations. First, 17 computer science educators complete a tutorial on using Bloom’s Taxonomy to classify programming examination questions. Second, 20 computer science educators complete a Neo-Piagetian tutorial. Third evaluation was a comparison of inter-rater reliability scores of computer science educators classifying programming questions using Bloom’s Taxonomy, before and after taking our tutorial. Based on the results from these evaluations, we discuss the effectiveness of our tutorial system design for teaching computer science educators how to systematically and consistently classify programming examination questions. We also discuss the suitability of Bloom’s Taxonomy and Neo-Piagetian theory for achieving this goal. The Bloom’s and Neo-Piagetian tutorials are made available as a community resource. The contributions of this paper are the following: the tutorial system for learning classification schemes for the purpose of coding the difficulty of computing learning materials; its evaluation; new insights into the consistency that computing educators can achieve using Bloom; and first insights into the use of Neo-Piagetian theory by a group of classifiers.     

ASK Systems: An Approach to the Realization of Story-Based Teachers

Telling stories is an effective way to teach aspects of nearly every task and domain. However, to be effectively remembered, a story must be told in a context that enables the hearer to index it functionally in memory. This occurs naturally when stories are told to students while they are attempting to perform the task being taught. Unfortunately, it is not always possible to engage students in a task while teaching it, so some other context must be found that facilitates appropriate indexing. We argue that this context occurs naturally in a teaching dialog, called an Aesopic dialog, in which the student asks questions and the expert answers with stories. In this dialog, the coherence of the conversation itself provides a context that enables the stories to be usefully incorporated into the student's memory. The widespread application of teaching through Aesopic dialogs requires overcoming the hurdle that experts are scarce and access to them is limited. Our solution to this problem is to broaden access to expert stories through the development of hypermedia systems designed to provide an interaction that emulates, as much as possible, the cognitively relevant aspects of an Aesopic dialog with an expert. We have constructed a number of such story-based teachers, called ASK systems, in domains as diverse as trust bank consulting, Presidential decision-making, and the determinants of a nation's industrial success in global markets.     

The impact of electronic media on faculty evaluation

With the proliferation of computer networks and the increased use of Internet‐based applications, many forms of social interactions now take place in an on‐line context through Computer‐Mediated Communication (CMC). Many universities are now reaping the benefits of using CMC applications to collect data on student evaluations of faculty, rather than using paper‐based surveys in Face‐To‐Face (FTF) classroom settings. While the relative merits of CMC versus FTF student evaluations have been researched extensively, there is limited research published about the ways students respond to the questions from either mode of data collection. This paper reports on a research study to analyse the communication differences between student scores from FTF student evaluations and CMC evaluation questions from end of semester evaluations from a university in the Middle East region. In addition to the questions about communication mode differences between two evaluation questions, several demographic variables were measured to determine any interaction effects. The results of our study suggest that the type of communication channel mitigates the responses that students make on CMC evaluations vis‐à‐vis FTF evaluations of faculty. In particular, even though there were significant differences found at the aggregate level between CMC and FTF evaluations, when the course and instructor are controlled for, there were no significant differences reported. In addition, several differences were noted depending on the type and level of the course being studied. Also, we found that students are more likely to express more extreme responses to scale questions in CMC than FTF evaluations. Administrators should consider these potential differences when implementing on‐line evaluation systems.     

Using mid-semester course evaluation as a feedback tool for improving learning and teaching in higher education

The way in which mid-semester course evaluations are structured, administered and reported is important for generating rich and high-quality student feedback for the enhancement of learning and teaching. Mid-semester evaluations usually contain open-ended questions that trigger more elaborative feedback about what is going on in a class than that from end-of-semester evaluations with Likert scale-type questions. The anonymity of the process for students and the confidentiality of the process for instructors make the process more reflective for students and less stressful for instructors. This study describes how the mid-semester course evaluation process can be used as a feedback tool for improving the quality of teaching and learning at an institutional level. Through a longitudinal analysis of 341 mid-semester course evaluation reports, positive areas and areas of concern with respect to learning and teaching were identified, and changes in student evaluations over the years were examined meticulously to make an overall evaluation of the quality of learning and teaching at a non-profit Turkish university. This research showed that the value of mid-semester course evaluations can go beyond course-level if we use open-ended questions and are able to gather the reports together to make some comprehensive analysis at university level.     

Library Research Success: Designing an Online Tutorial to Teach Information Literacy Skills to First-Year Students

This article reports on a collaborative effort recently undertaken between librarians and faculty at Toronto's Seneca College to develop and implement an online, interactive tutorial for first-year business students enrolled in the College's School of Business Management. The concept for this tutorial arose when library management was approached by the Acting Chair from the College's School of Business Management to partner with them in developing a library tutorial that would be incorporated into the curriculum of this newly created course, and that would accommodate its over 600 first-year students. Library management agreed, and proposed developing a self-paced online, interactive tutorial that would utilize current Web-based technology. The primary pedagogical goals were to increase student knowledge of library resources in general, and business resources in particular, and to increase levels of information literacy and basic research skills. With the help of the College's Centre for New Technology and the Centre for Professional Development, the result was a five-module tutorial entitled Library Research Success that, in addition to its main content, also incorporated hands-on exercises at the end of each module, live connections to the Library's Web site and electronic databases, as well as two assignments, which were submitted electronically to professors at the end of the tutorial by each student, and which composed 25 percent of their final mark. Feedback from both faculty and students has been overwhelmingly positive, and efforts are currently underway to adapt this tutorial to meet the needs of other student groups within Seneca College.     

Flowing Toward Correct Contributions During Group Problem Solving: A Statistical Discourse Analysis

Groups that created more correct ideas (correct contributions or CCs) might be more likely to solve a problem, and students' recent actions (micro-time context) might aid CC creation. 80 high school students worked in groups of 4 on an algebra problem. Groups with higher mathematics grades or more CCs were more likely to solve the problem. Dynamic multilevel analysis statistically identified watersheds (breakpoints) that divided each group's conversation into distinct time periods with many CCs versus few CCs, and modeled the groups' 2,951 conversation turns. Wrong contributions, correct evaluations of one another's ideas, justifications, and polite disagreements increased the likelihood of a CC. In contrast, questions, rude disagreements, and agreements reduced it. Justifications had the largest effects, whereas the effects of correct evaluations lasted 3 speaker turns. Some effects differed across groups or time periods. In groups that solved the problem, justifications were more likely to yield CCs, and questions were more likely to elicit explanations. Meanwhile, the effects of agreements and correct evaluations on CCs differed across time periods. Applied to practice, teachers can encourage students to evaluate others' ideas carefully and politely, express and justify their own ideas, and explain their answers to group members' questions.     

The effectiveness of man-machine tutorial dialogues for teaching attribute blocks problem-solving skills with an Artificial Intelligence CAI System

Two prominent design features of an Artificial Intelligence CAI (AI CAI) System for teaching problem-solving skills were experimentally evaluated. Seventy-six subjects were randomly assigned to four treatment groups formed by factorially combining two values each of two system design variables. The variables were the ability/lack of ability of the system to answer specific student generated questions regarding the current problem state, and the inclusion/exclusion of computer generated tutorial dialogues aimed at correcting student misprocessing of information during the solution of the problem. Analysis of the data indicated that students receiving computer generated instructional dialogues performed significantly more poorly on a transfer task than students not receiving such tutorial dialogues. The variable of student initiated questioning and the interaction of the two variables yielded non-significant results. Goal-directed production systems are reported to formally model task performance and instructional decision-making activities. The usefulness of these formal modeling techniques to instructional system design and evaluation is illustrated in a detailed analysis of the system's performance. Implications for future instructional system design techniques using the AI CAI approach are discussed.This article is based upon the author's unpublished doctoral dissertation.     

Using Dialogic Writing Assessment to Support the Development of Historical Literacy

Though discipline-specific approaches to literacy instruction can support adolescents' academic literacy and identity development, scant attention has been paid to ways of targeting such instruction to address individual student needs. Dialogic writing assessment is an approach to conducting writing conferences that foregrounds students' composing process so that teachers can assess and support that process with instructional feedback. Because such feedback is immediate, teachers can observe how students take it up. While dialogic assessment has shown promise as an approach to revealing and supporting students' writing processes in English Language Arts classrooms, it remains to be explored how this approach can support developing writers in other subject areas. This paper offers an analytic narrative account of how a high school social studies teacher used this method to support the writing process of one student, exploring what the method revealed about the challenges the student faced in writing about history, the gaps and misconceptions in their understanding of history and the intersection between the two. We discuss how certain ‘mediational moves’ the teacher employed enabled the student to compose collaboratively with the teacher, and in this collaborative composing, to capture ideas that she later used in her independent writing.     

Innovative approaches to information literacy instruction for engineering undergraduates at Drexel University

As information professionals, engineering librarians have the primary responsibilities of providing access to engineering information resources and giving instruction in how to use these resources. In the case of undergraduate engineering students, this extends to building their information literacy skills, an important component in helping them become lifelong learners; to be curious and independent, and to take greater responsibility for their own learning. The challenge in building information literacy in engineering students is to acquaint the students with the array of library resources available to them and to help them intelligently navigate the systems that contain the information. Too often, information literacy instruction is presented as a set of procedures for locating a hypothetical resource in the library. However, students are not interested in finding some resource randomly chosen as an example; they want to find resources that they perceive as being important and useful. During the 2005–2006 academic year, the engineering librarians at Drexel University took this into consideration and employed a new methodology for information literacy instruction: combining an online tutorial covering basic library skills with face-to-face consultations between student design teams and the engineering librarians. By utilizing varied instruction techniques aimed at different learning styles, with a strong active learning component delivered at the student's point of need—when they have a concrete, perceived information need—information literacy instruction can be improved so that engineering students retain more and develop lifelong learning skills.     

Role of comprehension on performance at higher levels of Bloom's taxonomy: Findings from assessments of healthcare professional students

The first four levels of Bloom's taxonomy were used to create quiz questions designed to assess student learning of the gross anatomy, histology, and physiology of the gastrointestinal (GI) system. Information on GI histology and physiology was presented to separate samples of medical, dental, and podiatry students in computer based tutorials where the information from the two disciplines was presented either separately or in an integrated fashion. All students were taught GI gross anatomy prior to this study by course faculty as part of the required curriculum of their respective program. Student responses to the quiz questions were analyzed to assess both the validity of Bloom's cumulative hierarchy and the effectiveness of an integrated curriculum. No statistically significant differences were found between quiz scores from students who received the integrated tutorial and from those who received the separate tutorials. Multiple regression analyses provided partial support for a cumulative hierarchy where scores on the lower levels of Bloom's taxonomy predicted scores on higher levels. Notably, in the majority of regression analyses, the comprehension score was the key foundational predictor for application and analysis scores. This study supports the suggestion that educators increase the number of comprehension level questions, even at the expense of knowledge level questions, in course assessments both to evaluate lower order cognitive skills and also as a predictor of success on questions requiring application and analysis levels of the higher order cognitive skills of Bloom's taxonomy. Anat Sci Educ 11: 433–444. © 2018 American Association of Anatomists.     

Student Evaluation of Teaching Effectiveness: An assessment of student perception and motivation

Over the past century, student ratings have steadily continued to take precedence in faculty evaluation systems in North America and Australia, are increasingly reported in Asia and Europe and are attracting considerable attention in the Far East. Since student ratings are the most, if not the only, influential measure of teaching effectiveness, active participation by and meaningful input from students can be critical in the success of such teaching evaluation systems. Nevertheless, very few studies have looked into students' perception of the teaching evaluation system and their motivation to participate. This study employs expectancy theory to evaluate some key factors that motivate students to participate in the teaching evaluation process. The results show that students generally consider an improvement in teaching to be the most attractive outcome of a teaching evaluation system. The second most attractive outcome was using teaching evaluations to improve course content and format. Using teaching evaluations for a professor's tenure, promotion and salary rise decisions and making the results of evaluations available for students' decisions on course and instructor selection were less important from the students' standpoint. Students' motivation to participate in teaching evaluations is also impacted significantly by their expectation that they will be able to provide meaningful feedback. Since quality student input is an essential antecedent of meaningful student evaluations of teaching effectiveness, the results of this study should be considered thoughtfully as the evaluation system is designed, implemented and operated.     

Development of student–teacher relationships in rural early elementary classrooms

An increasing body of literature documents associations between student–teacher relationships, children's academic success, and children's social competence in school. Less is known about characteristics and processes involved in the quality of relationships between students and teachers, and little research has examined these issues with populations of young students and teachers living in rural communities. The current study examined the relationships between rural kindergarten and first-grade students and their teachers in spring of the school year, predicted by child demographic factors, child process factors, and teacher characteristics. Using a multi-level model to account for clustering of children in classrooms, children's behavior and literacy skills were examined as contributors to the teachers’ perceptions of the developing teacher–student relationship, focusing on their potential to mediate associations between more distal characteristics and teacher–student relationships. Controlling for relationship conflict in fall, boys and African American students were more likely to have relationships with teachers that were higher in conflict in spring. When behavior and literacy skills measures were added to the model, children's behavior mediated the effect of gender, such that behavior problems accounted for much of the variance in student–teacher conflict associated with gender. However, neither behavior problems nor literacy skills mediated the effects of minority status on conflict; African American students had poorer relationships with teachers regardless of behavior or literacy skills.     

地质专业本科生分级导师制探索与实践

我国传统的地质教育模式在创新型人才培养和大学生综合素质提高方面存在许多弊端,而本科生导师制作为一种很好的教育制度,在这方面具有显著的优势。本文系统地分析了本科生导师制在人才培养方面的积极作用,认为在当前阶段高等教育改革中,针对地质专业本科生实施分级导师制是提高教育质量的有效途径。分级导师制不仅有利于学生学习和成才,有利于优化师生关系,更有利于提高学生实践探索能力和创新能力。     

Editorial

The first article in this issue raises some fascinating issues that relate to my own background in research into student learning and experience of courses in conventional higher education. Richardson, Long and Woodley have administered the Academic Engagement Form', used widely in colleges in the USA, and the 'Course Experience Questionnaire', used widely in universities in Australia, to distance learning students. John Richardson and various colleagues have previously shown that these questionnaires, separately, work as well in distance learning contexts as they do in conventional contexts: that is, they identify the same factors as components of students' experience, and the same factors relating to overall perceptions of quality of experience, as in conventional contexts. Of the many findings reported in the study reported in this issue of Open Learning, two stand out for me. First, academic engagement is shown to play a key role in students' perceptions of academic quality: engaged students perceive their course to be of higher quality. This does not tell us if students who are happy with their courses become more engaged or if those who are engaged become happier with their courses, however, merely that they are related. 'Engagement' here encompasses both social and academic engagement as defined in Tinto's model of student retention. Second, students' overall perceptions of academic quality are mediated by their perceptions of their tutors. The authors conclude: '... the attitudes and behaviours of tutors are crucial to students' perceptions of the academic quality of courses in distance education'. In conventional contexts the item on the Course Experience Questionnaire that relates most closely to student performance concerns the quality of teacher feedback, not teaching, and this is easy to understand in a distance context. The methodology of this paper (relying on factor analysis of questionnaires and multivariate analysis of the relationship between questionnaire scale scores and background variables such as age, gender, educational qualifications, workload and hearing status) may be relatively unfamiliar to readers of Open Learning. What is perhaps more familiar is that such an analysis adds to similar conclusions about the centrality of the tutorial role in ODL students' learning from very different kinds of study (such as of the relationship between tutorial attendance and student performance). There is a growing body of evidence that the same variables are involved in student perceptions of courses and of academic quality in distance learning contexts as in conventional contexts.     

当前大学计算机信息技术教育存在的问题及其对策

大学计算机信息技术课程作为我国高校普遍开设的一门公共基础课程,直接关系着学生信息素养的培养,日益受到重视。但由于该课程受内容涉及面广,知识点与概念繁多抽象,信息量大而课时不足等因素影响,实际教学效果普遍不理想。面对教学中存在的这些问题,广大教师应结合自身教学实践,积极探索,采取针对性措施和教学方法,激发学生学习主动性,以有效提升教学质量与学生信息素养。为此,本文就大学计算机信息技术教育的定位,信息技术课程的现状,信息技术教育的提升途径进行了探讨。     

Hofstra's New College: The Student View

The authors examined student reactions to an activity in computer science, psychology, women's studies, and education courses. The reciprocal interview involves the instructor gathering information about the students, followed by students collectively asking questions of the instructor. The interview aims to make students more comfortable in interactions with the instructor by soliciting student input. Student responses indicated they enjoyed the activity and recommend faculty use it in various courses. Participants also reported the interview process helped create a more comfortable classroom environment and helped clarify instructors' expectations of students.