Peer assessment in a project-based engineering course: comparing between on-campus and online learning environments

As the number of participants in online distance learning courses increases, peer assessment is becoming a popular strategy for evaluating open assignments and for breaking the social isolation surrounding distance education. Yet, the quality and characteristics of peer assessment in massive online courses has received little attention. Hence, this study was set to examine peer feedback quality and grading accuracy in a project-based course. The study applied a sequential exploratory mixed methods design. It included 339 participants who studied the same engineering course, but in three different modes: on-campus (n = 77), small private online course (n = 110), and massive open online course (MOOC) (n = 152). Content analysis of feedback comments identified four categories: reinforcement, statement, verification and elaboration, arranged in an ascending scale of cognitive ability. The findings indicated that the MOOC participants provided more feedback comments and volunteered to assess more projects than their counterparts did. However, the on-campus students provided higher quality feedback and their peer grading was better correlated with the grades assigned by the teaching assistants.     

Individual Learning Vs. Interactive Learning: A Cognitive Diagnostic Analysis of MOOC Students’ Learning Behaviors

ABSTRACT

Researchers have been interested in classifying massive open online course (MOOC) students based on their learning behaviors. However, less attention has been paid to the cognitive attributes associated with various learning behaviors. In this study, we propose a conceptual model that links MOOC students’ observable learning behaviors to their latent attributes (i.e., individual learning versus interactive learning). Using students’ behavior data from a MOOC, we performed a cognitive diagnostic analysis to identify the students’ learning profiles and to determine how these profiles related to their course achievement. We found that a large portion of the students performed individual learning whereas only a very small portion of them overtly performed interactive learning. In addition, the students who performed interactive learning were more likely to pass the course with distinction than the students who did not show this attribute. The results of this study have important implications for improving students’ learning in MOOCs. Further, the study provides a good demonstration of how to use clickstream process data for psychometric analysis.     

Grudges and gratitude: the social-affective impacts of peer assessment

Abstract

Students’ dissatisfaction with peer assessment has been widely documented. While most relevant literature places focus on the cognitive (content and uptake of feedback) or structural (feedback design) dimensions, students’ emotions in peer assessment have received scant attention. This study investigates the social-affective impacts of peer assessment by analysing students’ appeal letters addressed to their tutors, reflective posts in the online discussion forum and responses to a survey. A thematic analysis of data indicated three main aspects of students’ (dis)satisfaction: content, scores and process of peer assessment. The most negative emotion that students expressed was related to ‘disrespectful’ behaviour and attitudes of peer reviewers, whereas the feeling of appreciation was triggered by the helpful feedback attributes which were perceived as reflecting reviewers’ respect to others’ works. Students generally held mixed feelings toward peer assessment, valuing learning in the process of providing and receiving feedback but showing resistance to using peer assessment for summative purposes. The findings highlight the significance of respect in peer assessment and argue that the perceived lack of mutual respect seems to underpin the nature of students’ dissatisfaction. This study carries implications for nurturing students’ respectful attitudes and behaviour in and through peer assessment.     

Factors associated with students’ attitude change in online peer assessment – a mixed methods study in a graduate-level course

Abstract

Positive student attitude towards peer assessment can be demonstrated through high-quality peer assessment activities. Research findings on students’ attitudes before and after such activities were mixed. This study was conducted using a mixed-methods approach to investigate the underlying factors influencing students’ attitude change. Participants were students enrolled in an online graduate-level assessment course in the college of education in a university in the southeast United States in the fall of 2017 (N?=?31). Surveys and interviews were used to collect quantitative and qualitative data separately, both of which indicated that perceived accurate and specific feedback tended to help students shift to more positive attitudes towards online peer assessment. In addition, final task score improvement after the activity was correlated with positive attitude change. Qualitative results also identified other factors, such as communication with the peers’ work and logistic concerns. The current study offers insights for researchers and instructors to promote positive attitude changes in online peer assessment activities.     

An empirical review of anonymity effects in peer assessment,peer feedback,peer review,peer evaluation and peer grading

Abstract

Peer assessment has proven to have positive learning outcomes. Importantly, peer assessment is a social process and some claim that the use of anonymity might have advantages. However, the findings have not always been in the same direction. Our aims were: (a) to review the effects of using anonymity in peer assessment on performance, peer feedback content, peer grading accuracy, social effects and students’ perspective on peer assessment; and (b) to investigate the effects of four moderating variables (educational level, peer grading, assessment aids, direction of anonymity) in relation to anonymity. A literature search was conducted including five different terms related to peer assessment (e.g., peer feedback) and anonymity. Fourteen studies that used a control group or a within group design were found. The narrative review revealed that anonymous peer assessment seems to provide advantages for students’ perceptions about the learning value of peer assessment, delivering more critical peer feedback, increased self-perceived social effects, a slight tendency for more performance, especially in higher education and with less peer assessment aids. Some conclusions are that: (a) when implementing anonymity in peer assessment the instructional context and goals need to be considered, (b) existent empirical research is still limited, and (c) future research should employ stronger and more complex research designs.     

Learner engagement,retention and success: why size matters in massive open online courses (MOOCs)

ABSTRACT

Massive open online courses (MOOCs) are usually six to ten weeks long. Participation tends to decrease as the courses progress, leading to low completion rates. This led to the question: Could shorter MOOCs contribute to learners’ engagement, retention and success? This paper compares two versions of Study Skills MOOC, which shared the same content but were delivered in different length formats. One was deployed as a single six-week course and the other as two three-week blocks. In total, 617 people registered for the two versions. Data sources included learning analytics, surveys and the Spanish version of the General Self-Efficacy Scale. Both versions of the Study Skills MOOC resulted in increased participants’ self-efficacy. However, learners enrolled in the version composed of two three-week blocks were also more engaged with course content, other students and the facilitators. Their retention and completion rates were higher than those in the longer version of the course. Reasons linked to goal proximity, motivation, interactions and social modelling are discussed     

Exploring the Dimensions of Medical Student Engagement with Technology-Enhanced Learning Resources and Assessing the Impact on Assessment Outcomes

Anatomy curricula are becoming increasingly populated with blended learning resources, which utilize the increasing availability of educational technology. The educational literature postulates that the use of technology can support students in achieving greater learning outcomes by increasing engagement. This study attempts to investigate the dimensions of student engagement with technology-enhanced learning (TEL) resources as part of a medical program’s anatomy curriculum using exploratory factor analysis. A 25-item five-point Likert-based survey was administered to 192 first-year medical students, with three emergent factors discerned: satisfaction, goal setting and planning, and physical interaction. The three factors closely aligned with the existing literature and therefore additional nonparametric analysis was conducted that explored the levels of engagement across three custom-made anatomy TEL resources, including: (1) anatomy drawing screencasts; (2) an eBook; and (3) a massive open online course (MOOC). Usage data indicated that the most popular resource to be accessed across the cohort was the anatomy drawing screencasts via YouTube, with the MOOC being used least. Moreover, some evidence suggests that those students who utilized the MOOC were more engaged. Generally, however, no correlations were observed between the levels of engagement and TEL resource usage or assessment outcomes. The results from this study provide a clear insight into how students engage with TEL resources, but do not reveal any relationship between levels of engagement, usage, and assessment outcomes.     

Learners’ Perceptions and Experiences of Two Chemistry MOOCs: Implications for Teaching and Design

ABSTRACT

Most Massive Open Online Courses (MOOC) research studies have focused on students’ demographics, surveys, and retention data, while little attention has been paid to the authentic voices of the learners: the expressions of experiences, perceptions, and emotions in the MOOC students’ own words. In this paper, we conducted and analyzed in-depth interviews with learners from two MOOCs on the same subject. Findings confirm that learners enrolled in MOOCs for career and/or personal purposes. Learners’ expressions of feeling a human connection to the instructor in videos had important implications for video editing decisions. Many of the reasons given for dropping out of active participation in MOOCs were related to the learners’ lack of time because of other commitments. We expect the results from this study to provide a new understanding of MOOC learners and their perceptions of the courses; key insights should guide video editing considerations and encourage the use of instructor communication pathways such as regular emails to students in future MOOC offerings.     

Effects of communities of reflecting peers on student‐teacher development – including in‐depth case studies

Despite continuing interest in teacher reflection and an extensive body of research on peer assessment, the interaction between these areas has not been sufficiently investigated. This study on reflection and peer feedback is part of an ongoing action research addressing the design and pedagogical model of a theoretically oriented teacher training course. The course comprised face‐to‐face lessons followed by written reflections of the student‐teachers concerning their learning experiences. These reflections were presented for peer feedback – thereby initiating a community of reflecting peers (‘co‐reflection’) – and were additionally subjected to instructor feedback under two conditions (full vs. diluted). Student‐teachers’ perception of their professional development that resulted from these co‐reflecting communities were quantitatively evaluated (using a self‐reported questionnaire) over a three‐year period. An enhanced positive effect over the three‐year period was found – regardless of the instructor feedback condition – as well as significant differences between parallel groups (higher for science vs. mathematics student‐teachers). These are described in the first part of the paper, followed by an in‐depth case study analysis in the second section. The case studies reported inclusive analysis of the students’ written reflections and feedback. It provides an insight into the processes of co‐reflection, identifies changes over time in the quality of reflections and feedback, as well as in the personal, professional and social development of the student‐teachers, attempting to identify the mechanisms leading to these changes. Examples are cited to illustrate the ways in which the students developed and what activated their motivation for development. The case studies reveal two developmental models, one of external orientation and the other internal. These are characterized by a willing receiver and willing donor, respectively. The importance of a ‘maturity interval’, when the student is ‘ripe’ for developmental change, was also demonstrated. Offering an appropriate trigger at this mature point may stimulate crucial development. Teacher development is further interpreted in terms of interpersonal variables. The current paper may encourage teacher educators and student‐teachers themselves to incorporate co‐reflecting in their course planning and community planning. Ways may be designed to train student‐teachers to offer and use feedback effectively. For feedback to be valuable, it should be reinforcing, challenging, thorough and sincere, radiate faith, and be of a friendly nature – it should be non‐insulting and non‐judgemental. They may be used for in‐text as well as summary remarks.     

The role of technology-facilitated peer assessment and supporting strategies: a meta-analysis

Abstract

Technology-facilitated peer assessment is gaining increasing attention. However, evidence for the contribution of technology-facilitated peer assessment to learning achievements has not been investigated. The present meta-analysis integrated findings on the effects of technology-facilitated peer assessment based on two main elements: (1) technology-facilitated peer assessment, (2) the use of extra supporting strategies in technology-facilitated peer assessment. A total of 37 empirical studies published from 1999 to 2018 were selected and analysed. Results indicated that technology-facilitated peer assessment had a significant and medium effect on learning achievements with an overall mean effect size of 0.576. The use of extra supporting strategies in technology-facilitated peer assessment also produced a positive and medium effect on students’ learning achievements with an overall mean effect size of 0.543. Different moderator variables, such as task types, assessment modes, training for assessors, durations, grouping types and assessment methods were related to different effect sizes. The results together with the implications for both practice and research are discussed.     

Can we engage postgraduates in active research methodology learning? Challenges,strategies and evaluation of learning

ABSTRACT

Due to the complex nature of research methodology courses, the current study focused on implementing the active teaching and learning approach to a postgraduate research method course in a Malaysian university over an academic semester. A qualitative analysis of observations, instructor-learner interactional exchanges, students’ drafts of tasks and pre-course and focus group discussion was performed. The findings revealed three types of challenges: student-oriented challenges, subject-matter-related challenges and instructor-oriented challenges. Three main pedagogical strategies: instructional scaffolding, peer scaffolding and engaging the postgraduates in drafting their tasks were employed as a response to these challenges. Although the active teaching and learning practices resulted into students’ enhancement of the assigned research methodology tasks and positive research learning experience, such practices were time and effort-consuming. Therefore, future research will need to examine the applicability of our active teaching and learning approach to research methodology courses in different contexts.     

Straddling rows and columns: Students’ (mis)conceptions of an assessment rubric

Abstract

Assessment rubrics intend to make criteria explicit to students. However, an understanding of assessment criteria requires tacit knowledge about which students may not be aware. The lack of such knowledge is notable in undergraduate health science students taking a compulsory course in the social sciences. This study performed a content analysis of how 1^st year health sciences students interpret a SOLO assessment rubric for a sociology ‘reflection piece’ essay. The findings suggest that empiricism, standardisation and lack of tacit knowledge limit students’ ways of reasoning about the rubric. These limitations manifest in a methodical difference between students’ expectations and lecturers’ intentions of using a rubric. This difference is illustrated by the students’ need for procedural knowledge and the lecturer’s expectation of conceptual knowledge. These differences have implications for teaching, learning and assessment in health sciences undergraduate education. The findings also justify a dialogical approach to assessments through facilitating students’ epistemological development.     

Relationship Between Peer Orientation and Achievement in Cooperative Learning-Based Research Methodology Courses

Abstract

The capability of the relationship between peer orientation and achievement to remain in research methodology courses when cooperative learning techniques are introduced was investigated. Participants comprised 159 students enrolled in 7 sections of a graduate-level research methodology course at a southern university over 2 semesters. The students, who were administered a learning-style instrument, were enrolled in sections in which cooperative-learning groups were formed to undertake the major course requirements. Findings revealed a small but statistically significant relationship between peer orientation and achievement; students who were more oriented toward cooperative learning attained lower levels of achievement than did those who did not have an orientation toward cooperative learning. Further research is warranted.     

Do defence styles and approaches to learning ‘fit together’ in students’ profiles? Differences between years of study

Abstract

The study aims at investigating students’ learning/defence profiles. It also explores students’ profiles during different years of study. Participants comprised of 425 undergraduates. They completed the ‘Approaches to Study and Skills Inventory’ and the ‘Defense Style Questionnaire’. The students’ academic achievement was measured through grade point average. Cluster analysis revealed three profiles: ‘restricted maturity and dissonant-unorganized’ students, ‘defensive and reproduction oriented’ students, and ‘mature and learning advanced’ students. The profiles correspond to adaptive and maladaptive learning patterns. The study also revealed few (a) differences across the years of study (on strategic approach) and (b) interactions between the three clusters and the years of study (on mature defence style and GPA). The findings are discussed in relation to recent literature. Limitations and future research suggestions are presented.     

Understanding MOOCs as an Emerging Online Learning Tool: Perspectives From the Students

This study examined participants’ learning experiences in the context of a six-week massive open online course (MOOC) in journalism with five thousand students from 137 countries. Three research questions were asked: (1) Who are the students and why are they enrolled in this MOOC?, (2) How much time have the students spent in taking this MOOC and have they completed all the assignments?, and, (3) What have they learned and what aspects of this MOOC do the students find most helpful? Four hundred and nine students responded to a survey and forty-four responded to interview questions. The main findings showed 84% of the participants were working professionals and only 28.9% were from a journalism background. Of those who did not complete the course, lack of time was the top reason. Most participants reported a positive learning experience, but lack of feedback and/or poor quality were reported as negative experiences. The discussion forum was the least liked aspect of the course.     

The development of student feedback literacy: the influences of teacher feedback on peer feedback

Abstract

Although the concept of student feedback literacy has drawn increasing attention in higher education, empirical research on this matter is still in its infancy. In the area of peer feedback, little research has investigated the role of teacher follow-up feedback on peer feedback in the development of student feedback literacy. To address the research gap, a multiple-case study of three Chinese master’s students enrolled in an academic writing course was conducted, drawing on the students’ drafts with peer feedback, teacher written feedback on that peer feedback, semi-structured interviews, retrospective verbal reports, observation field notes and class documents. Three students’ epistemological and practical knowledge about, attitudes towards, and self-efficacy beliefs in peer feedback were found to improve at different paces and to different degrees. However, considerable individual variations were observed with two high-achieving, highly motivated participants becoming more feedback-literate than their under-achieving, minimally motivated peer. Teacher feedback on peer feedback was found to have distinct impacts on individual students, depending on learner factors including language ability, beliefs and motivation. These findings suggest that teacher feedback on peer feedback, if consistently provided and compatible with learner factors, can scaffold both cognitive and social-affective aspects of student feedback literacy.     

Establishing the validity and reliability of a modified tool for assessing innovative thinking of engineering students

Abstract

The goal of this study was to establish the validity and reliability of a self-report tool for assessing individual differences in innovative thinking, based on a modification of the ‘innovative behavior scale’. A five-stage study was conducted among engineering students worldwide, who enrolled in a massive open online course in Nanotechnology and Nanosensors. Content validity was established by a panel of experts in engineering education. Construct validity was established through exploratory and confirmatory factor analysis, indicating a four-factor solution with 13 items loaded above critical level. Known-groups validity revealed differences among learners with distinct expertise. The scale’s stability across populations and over time was confirmed, providing there has been no intentional intervention. Finally, the scale’s concurrent validity was reinforced through data triangulation with actual practice - the fabrication of innovative nanosensors.     

Teacher students’ dilemmas when teaching science through inquiry

Background: Inquiry-based science education (IBSE) is suitable to teach scientific contents as well as to foster scientific skills. Similar conclusions are drawn by studies with respect to scientific literacy, motivational aspects, vocabulary knowledge, conceptual understandings, critical thinking, and attitudes toward science. Nevertheless, IBSE is rarely adopted in schools. Often barriers for teachers account for this lack, with the result that even good teachers struggle to teach science as inquiry. More importantly, studies indicate that several barriers and constraints could be ascribed to problems teacher students have at the university stage.

Purpose: The purpose of this explorative investigation is to examine the problems teacher students have when teaching science through inquiry. In order to draw a holistic picture of these problems, we identified problems from three different points of view leading to the research question: What problems regarding IBSE do teacher students have from an objective, a subjective, and a self-reflective perspective?

Design &; method: Using video analysis and observation tools as well as qualitative content analysis and open questionnaires we identified problems from each perspective.

Results: The objectively stated problems comprise the lack of essential features of IBSE especially concerning ‘Supporting pupils’ own investigations’ and ‘Guiding analysis and conclusions.’ The subjectively perceived problems comprise concerns about ‘Teachers’ abilities’ and ‘Pupils’ abilities,’ ‘Differentiated instruction’ and institutional frame ‘Conditions’ while the self-reflectively noticed problems mainly comprise concerns about ‘Allowing inquiry,’ ‘Instructional Aspects,’ and ‘Pupils’ behavior.’

Conclusions: Each of the three different perspectives provides plenty of problems, partially overlapping, partially complementing one another, and partially revealing completely new problems. Consequently, teacher educators have to consider these three perspectives.     

Enhancing the effectiveness of concept inventories using textual analysis: investigations in an electrical engineering subject

ABSTRACT

Concept inventories (CIs) are assessment instruments designed to measure students’ conceptual understanding of fundamental concepts in particular fields. CIs utilise multiple-choice questions (MCQs), and specifically designed response selections, to help identify misconceptions. One shortcoming of this assessment instrument is that it fails to provide evidence of the causes of the misconceptions, or the nature of students’ conceptual understanding. In this article, we present the results of conducting textual analysis on students’ written explanations in order to provide better judgements into their conceptual understanding. We compared students’ MCQ scores in Signals and Systems Concept Inventory questions, with the textual analysis utilising vector analysis approaches. Our analysis of the textual data provided the ability to detect answers that students identified as a ‘guessed’ response. However, the analysis was unable to detect if conceptually correct ideas existed within the ‘guessed’ responses. The presented approach can be used as a framework to analyse assessment instruments that utilise textual, short-answer responses. This analysis framework is best suited for the restricted conditions imposed by the short-answer structure.     

MOOC instructor designs and challenges: what can be learned from existing MOOCs in Indonesia and Malaysia?

The purpose of this study was to explore Indonesian and Malaysian instructors’ perceptions of massive open online course (MOOC) design and how they deal with the design challenges. Surveys, email interviews, and course reviews are the main data-collection methods employed in this sequential mixed methods study. Forty-six instructors participated in the survey, and nine of them voluntarily participated in an email interview. The findings revealed that half of the courses were delivered using a hybrid/blended type of MOOC. Personal motives, institutional encouragement, and altruism were among the main reasons for instructors to offer MOOCs. Preparation, attraction, participation, and assessment were the categories used to explain the design strategies used by these instructors in designing their courses. The survey also revealed that collaboration encouragement, participant engagement, video development, and time constraints were the primary design challenges that the instructors experienced during the design process. Furthermore, most instructors sought advice from other MOOC instructors, MOOC providers, their institutions, video tutorials, and open educational resources (OERs) to surmount their design challenges.