A modified peer rating system to recognise rating skill as a learning outcome

The peer rating system used here advances the quantitative literacy goals outlined in the social sciences. We instituted a mid-semester intervention to teach rating skills and used an index to track longitudinal changes of skill mastery over the course of the semester. Seventy-four students in five advanced research classes followed the procedure of the existing peer rating system by completing reading assignments, writing reflections online, engaging in class discussions, rating their peers’ reflections and receiving feedback on their group effort. Peer ratings were then compared with each other and also with the instructor ratings to derive individualised indices of reliability and validity. These technical indicators enabled two rounds of assessment before and after a class-wide intervention. An omnibus test across the five classes showed a significant improvement in rating quality due to the intervention. Our courses not only met a quantitative learning outcome but also promised vocational competence.     

Using mid-semester course evaluation as a feedback tool for improving learning and teaching in higher education

The way in which mid-semester course evaluations are structured, administered and reported is important for generating rich and high-quality student feedback for the enhancement of learning and teaching. Mid-semester evaluations usually contain open-ended questions that trigger more elaborative feedback about what is going on in a class than that from end-of-semester evaluations with Likert scale-type questions. The anonymity of the process for students and the confidentiality of the process for instructors make the process more reflective for students and less stressful for instructors. This study describes how the mid-semester course evaluation process can be used as a feedback tool for improving the quality of teaching and learning at an institutional level. Through a longitudinal analysis of 341 mid-semester course evaluation reports, positive areas and areas of concern with respect to learning and teaching were identified, and changes in student evaluations over the years were examined meticulously to make an overall evaluation of the quality of learning and teaching at a non-profit Turkish university. This research showed that the value of mid-semester course evaluations can go beyond course-level if we use open-ended questions and are able to gather the reports together to make some comprehensive analysis at university level.     

Effects of grading practices and time of rating on student ratings of faculty performance and student learning

The effects of grading practices (strict, lenient) and time of rating (after a lecture, after taking an examination, after receiving feedback on performance on the examination) on student ratings of faculty performance and student learning (i.e., performance on an examination) were assessed in two experimental studies. Results indicated that (1) student ratings were directly affected by grading practices and this effect was a function of the time at which ratings were cornpleted, and (2) student learning and students' ratings of faculty were positively correlated, and grading practices had no effect on student learning. Implications of these findings for the administrative use of student ratings are considered.     

The bilateral benefits of providing and receiving peer feedback in academic writing across varying L2 proficiency

Although the effectiveness of peer review has been examined, few have tested the joint benefits of providing and receiving feedback features and quality in L2 contexts. The present study investigated variation in key features and quality of feedback provided and received by high and low L2 proficiency students and its benefits on revision in the authentic setting of students experiencing both roles of providing and receiving peer feedback in English academic writing. Analyses on two drafts from 50 students, 1356 idea units of anonymous implementable peer feedback and back-evaluation ratings revealed that the combined effect of providing and receiving feedback on revision was prominent for all students. But the bilateral benefits of providing and receiving feedback features were more significant for high proficiency students, while providing feedback features and quality benefited low proficiency students more. The study implies that students need more bilateral training with both providing and receiving feedback.     

University instructors' ratings profiles: Stability over time,and disciplinary differences

This study investigated university instructors' teaching profiles (rank order of teaching behaviors) as they emerge from student ratings. The data consisted of student ratings of all faculty members in the physics and chemistry departments at Tel Aviv University, with no exception, who taught the same undergraduate course over two consecutive years. The ratings of these instructors were taken twice a semester (at mid- and end of semester), during the two academic years of the study, using a short rating instrument. A (4 × 5) × 2 ANOVA was conducted, with time of measurement and item serving as repeated measures, and department as a grouping factor. The same procedure was used for identifying differences between pairs of measurements for each item. These pairs compared the corresponding measures of the first year versus the second year, and separately those of mid-semester versus the end of semester. Findings indicate a very high degree of stability of instructors' profiles across time of measurement and activities to improve instructor performance on weak points; no clear-cut improvement from mid- to end-of-semester ratings except for teachers who undertake special improvement activities; and differences in teaching profiles for the two academic departments.     

Student self-reported use of standards-based grading resources and feedback

ABSTRACT

Standards-based grading (SBG) tightly links student assessment to course learning objectives, providing students with meaningful feedback. A well-designed SBG system should ease first-year engineering (FYE) students’ transition to tertiary education by setting clear expectations and enabling self-monitoring of learning progress. However, when students fail to access and use resources and feedback to guide their learning, the effort to construct an SBG system is not commensurate with the student learning benefits. FYE students in a course using SBG self-reported their access to and use of SBG resources and feedback to guide their work on assignments and prepare for exams. Access and use was low at the start or the semester but improved mid-semester around exam time. In a subsequent semester, strategies were employed to increase students’ engagement with the SBG system. Students’ access and use was significantly higher in the beginning of the semester. Insights on students’ engagement related to course management decisions are offered.     

Analysis of Feedback among Group Practicum Members

Feedback among practicum members, a neglected variable in practicum research, was assessed. Practicum members cited as giving meaningful feedback at or above the group median were recalled as producing significantly more verbal, more positive, and more owned feedback than practicum members cited as giving less meaningful feedback than the group median number of citations. The two groups did not differ significantly on the Myers-Briggs Type Indicator, Carkhuff's Communication Index, or tape ratings. Implications of the results are discussed.     

Non‐numeric Intrajudge Consistency Feedback in an Angoff Procedure

The credibility of standard‐setting cut scores depends in part on two sources of consistency evidence: intrajudge and interjudge consistency. Although intrajudge consistency feedback has often been provided to Angoff judges in practice, more evidence is needed to determine whether it achieves its intended effect. In this randomized experiment with 36 judges, non‐numeric item‐level intrajudge consistency feedback was provided to treatment‐group judges after the first and second rounds of Angoff ratings. Compared to the judges in the control condition, those receiving the feedback significantly improved their intrajudge consistency, with the effect being stronger after the first round than after the second round. To examine whether this feedback has deleterious effects on between‐judge consistency, I also examined interjudge consistency at the cut score level and the item level using generalizability theory. The results showed that without the feedback, cut score variability worsened; with the feedback, idiosyncratic item‐level variability improved. These results suggest that non‐numeric intrajudge consistency feedback achieves its intended effect and potentially improves interjudge consistency. The findings contribute to standard‐setting feedback research and provide empirical evidence for practitioners planning Angoff procedures.     

Effects of teacher feedback behavior on social acceptance in inclusive elementary classrooms: Exploring social referencing processes in a natural setting

Teacher feedback behavior is a key determinant of the social referencing processes that influence the social acceptance of pupils. The present longitudinal study explores how teacher feedback on academic performance and social behavior is related to social acceptance during classroom activities and recess in the natural setting of inclusive classrooms. Data come from a study with 32 teachers and their 546 first to third grade pupils in Switzerland. Teacher feedback behavior was videotaped and peer nominations and ratings were used to assess social acceptance. Multilevel regression analyses showed that feedback on incorrect social behavior was negatively correlated with feedback on correct academic performance. Teacher feedback on incorrect social behavior and on correct and incorrect academic performance predicted how pupils were accepted by their peers during classroom activities. However, teacher feedback did not affect social acceptance during recess. The effect of teacher feedback behavior on social acceptance appears to depend on context. Social acceptance during classroom activities is influenced by teacher feedback whereas social acceptance at recess is not.     

Effects of a Theory-Based Feedback and Consultation Process on Instruction and Learning in College Classrooms

This study examined how midterm student ratings feedback provided to teaching assistants via a theory-based ratings instrument, combined with consultation on instructional practices, would affect teaching practices, ratings of teaching effectiveness, and student learning and motivation. The student ratings instrument that was employed focused on a series of instructional activities derived from Gagné's theory of instruction and Reiser and Dick's instructional model. Thirty-seven teaching assistants in undergraduate computer science and chemistry courses were randomly assigned to either a feedback + consultation group or a no-feedback group. Results of this study indicated that the feedback and consultation process had a significant impact on instructional practices and ratings of teaching effectiveness. Student learning and student motivation were positively correlated with the frequency with which the various instructional activities examined in this study were utilized. Implications of these findings and suggestions for future research are discussed.     

Assessment training effects on student assessment skills and task performance in a technology-facilitated peer assessment

This study examines the impact of an assessment training module on student assessment skills and task performance in a technology-facilitated peer assessment. Seventy-eight undergraduate students participated in the study. The participants completed an assessment training exercise, prior to engaging in peer-assessment activities. During the training, students reviewed learning concepts, discussed marking criteria, graded example projects and compared their evaluations with the instructor’s evaluation. Data were collected in the form of initial and final versions of students’ projects, students’ scoring of example projects before and after the assessment training, and written feedback that students provided on peer projects. Results of data analysis indicate that the assessment training led to a significant decrease in the discrepancy between student ratings and instructor rating of example projects. In addition, the degree of student vs. instructor discrepancy was highly predictive of the quality of feedback that students provided to their peers and the effectiveness of revisions that they made to their own projects upon receiving peer feedback. Smaller discrepancies in ratings were associated with provision of higher quality peer feedback during peer assessment, as well as better revision of initial projects after peer assessment.     

Can we meet their expectations? Experiences and perceptions of feedback in first year undergraduate students

Student ratings of satisfaction with feedback are consistently lower than other teaching and learning elements within the UK higher education sector. However, reasons for this dissatisfaction are often unclear to teaching staff, who believe their students are receiving timely, extensive and informative feedback. This study explores possible explanations for this mismatch between staff and students’ perceptions of feedback quality. One hundred and sixty-six first year undergraduate students completed a questionnaire detailing their experiences of feedback on coursework before and throughout their first year at university. Results indicate that whilst procedural elements of feedback (timeliness and legibility) are considered satisfactory, past experiences (pre-university) may influence student expectations of feedback. Some students had a severe, negative emotional response to the feedback provided and few students engaged in self-help (independent learning) behaviours to improve their performance following feedback. We consider how changes in feedback practices could improve students’ use of, and satisfaction with, their feedback.     

The Effects of Modeling,Feedback, and Experiential Methods on Counselor Empathy

This study compares the effects of three supervisory techniques—modeling, feedback, and experiential intervention—in teaching counseling students to communicate empathic understanding. The students participated in three simulated counseling interviews during which their responses to standard clients were rated on the degree of empathic understanding conveyed. After each of the first two interviews students in the experimental groups were given one of the three supervisory treatments; students in the control group received no supervision. The ratings indicated that the modeling method of supervision was more effective than the feedback method and the feedback method was more effective than the experiential method in helping the counselors communicate empathy. No significant differences were found between students receiving experiential supervision and those in the control group. The interaction between method and sex was significant in that male students responded better than female students to the feedback method.     

The Cognitive Consequences of Collaborative Problem Solving With and Without Feedback

The goal of this research was to assess the impact of feedback, partner, and shared understanding in the course of problem solving. A sample of 180 6- to 9-year-olds was pretested to discover the children's "rule" for predicting the movement of a mathematical balance beam. For the treatment they either worked alone or with a partner who was equally, less, or more competent, with two-thirds receiving feedback from the materials. They subsequently participated in 2 individual posttests. The results revealed that children receiving feedback improved significantly more than those who did not, but that the presence of a partner was only beneficial when children received no feedback. Irrespective of feedback, those children whose partner exhibited higher-level reasoning were far more likely to benefit from collaboration than those whose partner did not, provided that the pair achieved shared understanding.     

Characterisation of pre-service teachers' attitude to feedback in a wiki-environment framework

This is a two-phase research process in a wiki environment. The first phase explored (1) the significance of peer-feedback on students’ academic performance on a specific task in a wiki environment, (2) three types of peer-feedback and (3) students’ constraints on offering meaningful feedback to their peers. The objective of phase two was to determine the reasons why students refrain from peer-feedback. Significant correlation exists between the willingness to give feedback and the willingness to receive feedback. Significant correlation also exists between the difficulty of giving feedback and the difficulty of receiving feedback. Female students tend to consider feedback as the teacher's responsibility more than male students. Religious beliefs did not have any significant effect on any of the parameters tested. The results indicate that traditional students tend to be more conservative regarding feedback. There are worrying implications to pre-service teachers’ refraining from giving feedback.     

How feedback can foster professional growth of teachers in the clinical workplace: A review of the literature

Student ratings of teaching have received much attention in both higher and medical education research. Paramount has been the attention to the robustness of the instruments used to capture these ratings as a source of feedback for teachers. However evidence is scarce with regard to what happens after ratings are fed back to the teachers. This paper will focus on feedback facilitation strategies needed for medical teachers in the workplace setting to improve their teaching. First, the attributes of clinical teaching will be introduced, followed by a disquisition on feedback uptake, and finally reflection as a tool to improve teaching. Second, several recently published strategies aimed at improving clinical teaching through the facilitation of feedback are discussed.     

Prediction of attributional consequences in an actual achievement setting

One hundred ninety-six undergraduates who received feedback on a midterm examination participated in this study. A random subgroup formed a cross-validation sample. Through the use of regression analysis, it was found that both objective and subjective performance ratings did not predict subjects' expectancy ratings but did substantially predict their ratings of pride/shame. Inclusion of attributions enhanced prediction of expectancy but not pride/shame ratings of subjects. On the basis of the findings from this study, a partial reformulation of Weiner's attributional model was proposed. In addition, numerous methodological concerns associated with attributional research were examined.     

Effects of social persuasion from parents and teachers on Chinese students’ self-efficacy: an exploratory study

This study compared the effects of social persuasion from parents and teachers on students’ self-efficacy in reading comprehension in English as a foreign language. Ninety-nine Grade 8 Chinese students in a secondary school in Hong Kong completed a questionnaire with six scenarios which tapped their self-efficacy after receiving positive and negative feedback that were described as coming from their teacher, mother and father. Students decreased their self-efficacy after receiving negative feedback and the source of feedback did not make a statistically significant difference. Conversely, positive feedback led to an increase in self-efficacy with the mother’s positive feedback leading to a higher increase in self-efficacy than feedback from the teacher and the father. No statistically significant difference was noted in the change in self-efficacy after receiving feedback from the father and the teacher. Parents and teachers should increase the use of positive feedback and decrease that of negative feedback. Implications for collaboration between teachers and parents in providing positive feedback on students’ academic performance were discussed.     

A Method for Detecting Regression of Hard and Easy Item Angoff Ratings

One common phenomenon in Angoff standard setting is that panelists regress their ratings in toward the middle of the probability scale. This study describes two indices based on taking ratios of standard deviations that can be utilized with a scatterplot of item ratings versus expected probabilities of success to identify whether ratings are regressed in toward the middle of the probability scale. Results from a simulation study show that the standard deviation ratio indices can successfully detect ratings for hard and easy items that are regressed in toward the middle of the probability scale in Angoff standard‐setting data, where previously proposed indices often do not work as well to detect these effects. Results from a real data set show that, while virtually all raters improve from Round 1 to Round 2 as measured by previously developed indices, the standard deviation ratios in conjunction with a scatterplot of item ratings versus expected probabilities of success can identify individuals who may still be regressing their ratings in toward the middle of the probability scale even after receiving feedback. The authors suggest using the scatterplot along with the standard deviation ratio indices and other statistics for measuring the quality of Angoff standard‐setting data.     

Faculty Perspectives on Course and Teacher Evaluations

Student ratings of instruction have been the subject of numerous studies with much of the research focusing on the validity and reliability of the ratings themselves. Comparatively little empirical investigation has been devoted to the perceptions of the individuals who are the subjects of the ratings, that is, the faculty. The current study explored faculty perspectives on the usefulness of student ratings for formative and summative purposes, and the actual use of student ratings for summative purposes. Contrary to what might have been deduced from the anecdotal literature, the results of this study do not portray a great deal of resistance to student ratings in general or to their use for formative and summative evaluation. It was also found that student ratings are actually being used for the latter purpose. The usefulness of the student feedback was viewed differentially by the faculty, with feedback on their interaction with students seen as most useful, followed by feedback on their grading practices, global ratings of instructor and course, and finally structural issues of the course.