Understanding student evaluations of teaching in online learning

In this paper, we have developed a classification model for online learning environments that relates the Instructors Overall Performance (IOP) rating (according to students’ perceptions) with the course characteristics, students’ demographics and the effectiveness of the instructor in his/her teaching roles. To that end, a comprehensive Student Evaluation of Teaching (SET) instrument is proposed, which includes not only conventional teaching elements, but also items that encourage twenty-first century skills. The goal of the study is twofold: (i) to quantify the extent to which the selected variables explain the IOP rating, and (ii) determine which teaching and non-teaching variables most affect the IOP rating. The best performing classifier achieved a competitive accuracy, highlighting that the selected variables mainly determine the IOP values. Other important findings include: (i) the IOP value is mainly influenced by the effectiveness of the instructor in his/her teaching roles; (ii) teaching strategies that involve the cooperation between the technical and pedagogical roles should be promoted; (iii) the pedagogical role has the highest impact on the final IOP value; and (iv) the most influential demographic variable is the student’s status (working commitments and family responsibilities).     

A phenomenological study of teamwork in online and face-to-face student teams

Team-based projects are widely used in both traditional face-to-face and online programs in higher education. To date, the teamwork experiences of students in each modality have been documented primarily through evaluative research conducted over short spans of time and limited by a priori frameworks. The literature also reflects a lack of agreement about what constitutes the phenomenon of teamwork in each modality. In order to address these limitations, we conducted a phenomenological study examining the lived experiences of teamwork among students in both face-to-face and online MBA programs in Iran. Our analysis revealed striking commonalities in the experiences of both groups, including a shared desire for effective leadership to alleviate the problem of free riders, as well as substantial time and effort invested in retaining reliable teammates from one team project to another. In other respects, face-to-face and online students’ experience differed strongly. For example, while face-to-face participants pursued teammates with similar beliefs about how teamwork should be accomplished, online participants found themselves pre-occupied with staying connected with their teammates and struggled to establish common communication channels with each and every team member. Overall, our findings suggest that while training and support for student teamwork can partly build on the shared needs among students in both modalities, the nature of the experience in each modality may be so different in vital respects that engaging in one mode of teamwork does not necessarily prepare students to participate well in the other mode. Other implications and limitations of the research are discussed.     

Applying the scholarship of teaching and learning: student perceptions,behaviours and success online and face-to-face

This study compares student perceptions, learning behaviours and success in online and face-to-face versions of a Principles of Microeconomics course. It follows a Scholarship of Teaching and Learning (SoTL) approach by using a cycle of empirical analysis, reflection and action to improve the learning experience for students. The online course design involves 58 interactive narrated online modules, interactive online quizzes and biweekly online meetings with the instructor via video and voice-over-IP technology. Findings indicate that schedule flexibility motivates students to choose the online course format. Students in both learning environments felt they had high-quality communication with the instructor, while online students indicated limited peer-to-peer communication. Online students report studying more at home than face-to-face students, but not enough to compensate for the time face-to-face students spend in class. Reflecting on the findings, the authors suggest actions to improve the online course experience.     

A study of teaching presence and student sense of learning community in fully online and web-enhanced college courses

This paper focuses on two components of a model for online teaching and learning—“teaching presence” and “community”. It is suggested that previous research points to the critical role that community plays in academic success and persistence in higher education. Through a review of recent literature it is proposed that teaching presence–viewed as the core roles of the online instructor–is a promising mechanism for developing learning community in online environments. This investigation presents a multi-institutional study of 1067 students across 32 different colleges that further substantiates this claim. An instrument to assess instructor teaching presence (“The Teaching Presence Scale”) is presented and validated. Factor and regression analysis indicate a significant link between students' sense of learning community and effective instructional design and “directed facilitation” on the part of course instructors, and highlights interesting differences between online and classroom environments. Alternative hypotheses regarding student demographics associated with variables such as age (the “net generation” effect) and gender are also examined. Despite recent assertions that younger students are or soon will be too sophisticated to “feel at home” in largely text-based asynchronous learning environments, no significant effects were found by demographic differences examined. Recommendations for online course design, pedagogy, and future research are included.     

Comparison of online and classroom-based student evaluations of instruction

In recent years, colleges have been moving from traditional, classroom‐based student evaluations of instruction to online evaluations. Because of the importance of these evaluations in decisions regarding retention, promotion and tenure, instructors are justifiably concerned about how this trend might affect their ratings. We recruited faculty members who were teaching two or more sections of the same course in a single semester and assigned at least one section to receive online evaluations and the other section(s) to receive classroom evaluations. We hypothesised that the online evaluations would yield a lower response rate than the classroom administration. We also predicted that there would be no significant differences in the overall ratings, the number of written comments, and the valence (positive/neutral/negative) of students’ comments. A total of 32 instructors participated in the study over two semesters, providing evaluation data from 2057 students. As expected, online evaluations had a significantly lower response rate than classroom evaluations. Additionally, there were no differences in the mean ratings, the percentage of students who provided written comments or the proportion of comments in the three valence categories. Thus, even with the lower response rate for online evaluations, the two administration formats seemed to produce comparable data.     

The product life-cycle of online courses and student engagement

ABSTRACT

With increased online course offerings at most higher education institutions, it is important to have a way to determine whether the courses continue to have value to students or outlived their usefulness. Courses are the central means of knowledge delivery for each institution and may be viewed through the lens of the product life-cycle, which is a common way for for-profit corporations to ensure they are aware of when it is time to make adjustments or completely overhaul what they provide to consumers. We leverage this concept to develop a conceptual mode of the life-span and cycle of curricular products; especially, those delivered online, which may have a shorter shelf life than in-classroom courses. To that end, we map a single course that has spanned nearly 20 years and shifted from face-to-face delivery, to a blended mode with learning technologies support, and then into a fully online format. This piece depicts the course life-cycle, which emerged from multi-strategy research analysis of that course. Understanding that online courses have a life-cycle can help department and university managers plan for the investment in the improvement of each as it comes to the end of its useful lifespan. This can ensure high-quality learning and a sustainable portfolio of programs and courses with which students are engaged.     

Expectancy theory outcomes and student evaluations of teaching

As student evaluation of teaching (SET) instruments are increasingly administered online, research has found that the response rates have dropped significantly. Validity concerns have necessitated research that explores student motivation for completing SETs. This study uses Vroom's [(1964). Work and motivation (3rd ed.). New York, NY: John Wiley & Sons] expectancy theory to frame student focus group responses regarding their motivations for completing and not completing paper and online SETs. Results show that students consider the following outcomes when deciding whether to complete SETs: (a) course improvement, (b) appropriate instructor tenure and promotion, (c) accurate instructor ratings are available to students, (d) spending reasonable amount of time on SETs, (e) retaining anonymity, (f) avoiding social scrutiny, (g) earning points and releasing grades, and (h) being a good university citizen. Results show that the lower online response rate is largely due to students’ differing feelings of obligation in the 2 formats. Students also noted that in certain situations, students often answer SETs insincerely.     

Computer-mediated instruction: a comparison of online and face-to-face collaboration

This study investigated the effects of collaboration mode and group composition during a computer-mediated collaborative (CMC) program. Six intact sections of a computer literacy course were assigned to either a face-to-face or a virtual, online collaboration treatment condition. Groups consisted of homogeneous lower-ability, homogeneous higher-ability, or heterogeneous-ability pairs. The study examined the effects of collaboration mode and group composition on individual posttest performance, group project performance, collaborative interaction behavior, and attitudes towards the instruction. Results indicated that virtual dyads exhibited significantly more questioning behaviors and significantly better project performance than those who collaborated face-to-face. By comparison, students in the face-to-face condition performed significantly better on the individual posttest than those in the virtual online condition. Findings suggest that both virtual and face-to-face collaboration can be effective in achieving learning goals. However, consideration should be given to the collaborative structure of the lesson and the type of task in the design of CMC environments.

Predicting student success by modeling student interaction in asynchronous online courses

Early-warning intervention for students at risk of failing their online courses is increasingly important for higher education institutions. Students who show high levels of engagement appear less likely to be at risk of failing, and how engaged a student is in their online experience can be characterized as factors contributing to their social presence. Social presence begins with teacher-student and student-student interaction in online courses. Fortunately, student interaction data can be gleaned from learning management systems, used to model and predict at-risk students at an early stage. This research addresses an existing model for predicting at-risk students to test a previous hypothesis that a holiday effect is a contributor for failure. A new analysis then presents an alternative approach, one that tests the frequency of student interaction rather than amount of interaction as a preferable indicator.     

Learning online,offline, and in-between: comparing student academic outcomes and course satisfaction in face-to-face,online, and blended teaching modalities

The purpose of this study was to conduct a three-way comparison of face-to-face, online, and blended teaching modalities in an undergraduate Child Development course to determine if there were differences in student academic outcomes and course satisfaction across modalities. Student academic outcomes were measured by three examinations, one research paper assignment, and the overall course total grade. Course satisfaction was measured by administering the Student Opinion Questionnaire (SOQ) across the three teaching modalities and the Constructivist On-Line Learning Environment Survey (COLLES) to online and blended modalities. Results indicated that students performed equally well on all three examinations, research paper, and the overall course total grade across three teaching modalities, allaying traditional reservations about online and blended teaching efficacy. The SOQ and COLLES analysis found students from the three modalities were equally satisfied with their learning experiences. A Two-Factor Model identifying Face-to-Face Interaction and Learn on Demand (Flexibility) as factors determining student academic outcomes was proposed. Implications, limitations, and future research direction were discussed.     

Statistical knowledge and the over-interpretation of student evaluations of teaching

Research shows that teachers interpret small differences in student evaluations of teaching as meaningful even when available statistical information indicates that the differences are not reliable. The current research explored the effect of statistical training on college teachers’ tendency to over-interpret student evaluation differences. A sample of 225 English, mathematics and psychology teachers evaluated statistical reports of student evaluation data corresponding to a fictional teacher’s courses. Teachers from the three disciplines varied significantly in their statistics training, but there were no differences in their tendency to rate small, non-significant differences in student evaluations as indicative of the need to make improvements in a course. These results illustrate that statistical knowledge alone does not prevent the over-interpretation of student evaluation means.     

The usefulness of student evaluations in improving college teaching

The present study was designed to assess the effects on faculty performance of a combination of feedback and personal consultations using college student evaluations. Student evaluation feedback and personal consultations were conducted at least a semester before any follow-up data were gathered. The results indicate that providing computerized results of college student evaluations along with individual faculty consulting sessions helped the instructors to significantly improve their student ratings on two instructional dimensions.This study was conducted while the author was Head of the Measurement and Research Division, Office of Instructional Resources, University of Illinois at Urbana-Champaign. The author is indebted to Nancy F. Halff for her valuable editorial and statistical assistance, and to Pamela Z. Hexner and Jeffrey A. Slinde in conducting the data analysis.University of Arizona     

Predicting student evaluations of teaching using decision tree analysis

Abstract

This study uses decision tree analysis to determine the most important variables that predict high overall teaching and course scores on a student evaluation of teaching (SET) instrument at a large public research university in the United States. Decision tree analysis is a more robust and intuitive approach for analysing and interpreting SET scores compared to more common parametric statistical approaches. Variables in this analysis included individual items on the SET instrument, self-reported student characteristics, course characteristics and instructor characteristics. The results show that items on the SET instrument that most directly address fundamental issues of teaching and learning, such as helping the student to better understand the course material, are most predictive of high overall teaching and course scores. SET items less directly related to student learning, such as those related to course grading policies, have little importance in predicting high overall teaching and course scores. Variables irrelevant to the construct, such as an instructor’s gender and race/ethnicity, were not predictive of high overall teaching and course scores. These findings provide evidence of criterion and discriminant validity, and show that high SET scores do not reflect student biases against an instructor’s gender or race/ethnicity.     

Measuring faculty learning in curriculum and teaching competence online courses

Online education is used for a variety of purposes in higher education. Two such purposes are improving one's performance over time and understanding one's professional development in the context of online teaching and learning. Relying on data from online staff development courses delivered in five Spanish universities, this article explores online faculty learning through the lens of staff development theory. This theoretical perspective emphasizes the universities' quality assurance contexts and offers an empirical examination of the ways in which faculty members learn curriculum and teaching competencies (CTCs) in online staff development programmes. At the core of this analysis is the contention that faculty members understand and respond to quality teaching lessons and activities. Finally, this study highlights the points deemed important when designing, implementing, and evaluating Internet CTC training courses.