Exploring graduate and undergraduate course evaluations administered on paper and online: a case study

The purpose of this study was to analyse the students’ evaluations of the course and instructor for all statistics courses offered during fall semester 2009 at a large university in the southern United States. Data were collected and analysed for course evaluations administered both online and on paper to students in both undergraduate and graduate courses. Unlike most previous studies on this subject, class section rather than student was treated as the unit of analysis. It was of specific interest to verify prior research findings that evaluation surveys administered online would not result in lower course and instructor ratings and lower response rates. The results showed that there is not sufficient evidence within the collected data to conclude that either course and instructor ratings or response rates are lower for evaluations administered online (online evaluations) than they are for evaluations administered on paper (paper evaluations). Of secondary interest was whether class ratings would be associated with students’ attendance and a comparison of variability among answers for undergraduate vs. graduate students. It was observed that class and teacher ratings were not related to students’ attendance and individual students did not tend to give the same answer for every question on their survey.     

Online Course Evaluations: Faculty Perspective and Strategies for Improved Response Rates

This article provides an overview of issues involved with traditional paper versus online course evaluations. Data were gathered from university faculty, who transitioned from traditional paper to online course evaluations. Faculty preferred traditional course evaluations versus online course evaluations by a small margin. However, faculty overwhelmingly believed that traditional course evaluations result in higher response rates from students. Incentives were also believed by faculty to increase student response rates. Suggestions from faculty on how to improve student response rates are also provided in this article.     

Understanding academics’ resistance towards (online) student evaluation

Many higher educational institutions and academic staff are still sceptical about the validity and reliability of student evaluation questionnaires, in particular when these evaluations are completed online. One month after a university-wide implementation from paper to online evaluation across 629 modules, (perceived) resistance and ambivalence amongst academic staff were unpacked. A mixed-method study was conducted amongst 104 academics using survey methods and follow-up semi-structured interviews. Despite a successful ‘technical’ transition (i.e. response rate of 60%, similar scores to previous evaluations), more than half of respondents reported a negative experience with this transition. The results indicate that the multidimensional nature of ambivalence towards change and the dual nature of student evaluations can influence the effectiveness of organisational transition processes.     

Do Technological and Course‐Related Variables Impact Undergraduates’ Perceived Favorability and Willingness to Recommend Online/Hybrid Business Courses?

Lower teaching evaluations can affect students’ willingness to recommend an online course. To maintain online course quality, it is important to keep the “integrity” of a course, that is, offer to the extent possible, the same content and learning outcomes in an online course as the face‐to‐face (F2F) equivalent. This study explored the impact of background, technological, and course‐related variables on perceived favorability of online (versus F2F) courses and willingness to recommend courses using two independent “mixed course format” samples of business undergraduates, that is, simultaneously taking online/hybrid and F2F courses. Participants were recruited based on their enrollment in at least one online or hybrid course. A complete data sample of 259 students filled out an online survey in the Fall 2015 semester, followed by a second independent sample of 269 students in the Spring 2016 semester. Hierarchical regression results showed that after controlling for background variables, one course‐related variable (instructor effectiveness) and one technological‐related (perceived ease of use) consistently explained perceived favorability of online (versus F2F) courses across both samples. For the willingness to recommend courses, after controlling for background variables, both course‐related variables (instructor effectiveness and student motivation) and the perceived favorability of the online course were each significant across both samples.     

Exploring the link among entry characteristics,participation behaviors and course outcomes of online learners: An examination of learner profile using cluster analysis

Online learner profiles have diversified such that now, they include all types of participants from a variety of backgrounds. This study analyzed online learner profiles in regard to their entry characteristics, participation behaviors and achievement of course outcomes. The sample consisted of 186 participants from an online course that required the employment of synchronous and asynchronous communication methods over the Internet. The data were collected through five online questionnaires and included 10 variables (gender, age, work status, self‐efficacy, online readiness, self‐regulation, participation in discussion list, participation in chat sessions, satisfaction and achievement). A two‐step cluster analysis, chi‐square analysis and multivariate analysis were used to analyze the collected data. The results indicated that online learners fell into three significantly different clusters and within each cluster, learners had a close combination of employment characteristics, gender type, age level, perception of self‐efficacy for online technologies and participation behavior level.     

Academic Performance,Age, Gender,and Ethnicity in Online Courses Delivered by Two-Year Colleges

This study investigated the effects the demographic variables age, gender, and ethnicity and their interactions had on academic performance in online courses delivered by public two-year colleges in Kentucky. The study controlled for previous academic performance measured by cumulative grade point average (GPA). The study used a random sample (N = 320) of all students who had enrolled in at least one online course delivered by the institutions of the Kentucky Community and Technical College System in the spring 2008 semester. A linear hierarchical multiple regression acting as ANCOVA served as the main analysis, with the order entry as follows: cumulative GPA; independent variables (age, gender, ethnicity); interaction vectors; and product vectors. Final course grade served as the dependent variable. The results of the analysis indicated that only cumulative GPA was a significant predictor, explaining approximately 40% of the variance of the final grade. Although differences in final grades were present among the variables age and ethnicity, these differences disappeared when controlling for cumulative GPA. Significance of the results and recommendations for future research are discussed.     

Gathering faculty teaching evaluations by in‐class and online surveys: their effects on response rates and evaluations

This study compares student evaluations of faculty teaching that were completed in‐class with those collected online. The two methods of evaluation were compared on response rates and on evaluation scores. In addition, this study investigates whether treatments or incentives can affect the response to online evaluations. It was found that the response rate to the online survey was generally lower than that to the in‐class survey. When a grade incentive was used to encourage response to the online survey, a response rate was achieved that was comparable with that to the in‐class survey. Additionally, the study found that online evaluations do not produce significantly different mean evaluation scores than traditional in‐class evaluations, even when different incentives are offered to students who are asked to complete online evaluations.     

Student evaluations of teaching: perceptions of faculty based on gender,position, and rank

The current study explores the feelings and thoughts that faculty have about their student evaluations of teaching (SET). To assess the perceptions of SETs, all teaching faculty in one college at a western Land Grant University were asked to complete an anonymous online survey. The survey included demographic questions (i.e. gender; rank such as assistant, associate, and full professor; and positions like non-tenure track, tenure track, and tenured) as well as questions related to faculty's feelings while reading their SETs. While minimal differences were found in responses based on rank or position, several differences were found based on faculty gender. Overall, female faculty appear to be more negatively impacted by student evaluations than male faculty. These gender differences support previous research that suggests males and females receive and react differently to personal evaluation. Resultant suggestions include modifying surveys from anonymous to confidential and offering professional development training for faculty.     

Examining focus,duration, and classroom impact of literacy and mathematics professional development

Over $18 billion is spent annually on professional development for in-service teachers. This study examined the focus, duration, and impact of professional development activities in which elementary (students ages 5–11) and middle school (students ages 11–14) teachers participated. Ninety-eight teacher-participants completed an online survey about their professional development experiences. In the analysis of literacy survey items and mathematics survey items, there were similarities in focus areas that received greater participation and areas that received less. There were also noticeable differences in frequency of participation in literacy and mathematics professional development. Comparisons between literacy and mathematics professional development indicated that more teachers reported participating in literacy experiences. This included workshops as well as ongoing support through planning meetings and mentorship. Overall, the consensus from teachers was that professional development was effective and has a beneficial impact on their classroom. Researchers share implications for the design and research of future professional development programs.     

Motivations,barriers, & understandings: how students at four universities perceive student–faculty partnership programs

ABSTRACT

Extracurricular student–staff partnership activities are increasingly common in higher education institutions, but concerns have been expressed about their relative inclusivity. In order to contribute to understanding of why students do and do not take part in such initiatives, this study draws on data from a survey of students at four institutions, in three different countries, with established extracurricular partnership programs. Though the findings indicate some preliminary differences between participants at the four institutions, they also demonstrate a number of shared perceptions, including similar understandings of key features of partnership and shared perceptions of barriers to participation. Implications for partnership practice and research are discussed.     

Correlates of monotonic response patterns in online ratings of a university course

This study examines the prevalence, contexts, and demographic correlates of monotonic response patterns (MRPs) in online student evaluations. Results of two-level hierarchical generalized linear models show evidence of careless monotonic responses to a survey administered to students enrolled in a university-level foreign language course in the Republic of Korea. All else being equal, freshmen and students in classes with fewer survey participants were more likely to choose monotonic response patterns in course evaluations. Possible factors at work in generating MRPs are identified and discussed. The severity of the MRP problem in online ratings underscores the importance for administrators to consider possible validity threats in student evaluations before using them as tools to inform instructional and administrative decisions. It is also important to design course evaluation surveys in such a way as to minimize careless responses and to identify means to induce more thoughtful responses from college students.     

Influences on peer evaluation in a group project: an exploration of leadership,demographics and course performance

Collaborative learning strategies are widely used in higher education to deepen learning, promote team-building skills and achieve course learning objectives. Using peer evaluation is an important strategy to ensure that engaged and active students are rewarded for their efforts, and to discourage loafing within groups. However, less is known about what biases may influence students’ peer evaluations. In this paper, we investigate what variables students may (consciously or unconsciously) use to evaluate their peers. We explore the role of sex, race, course performance and group leadership on peer evaluation. We also investigate whether these variables correlate with students’ final course grade. We found that students who reported being leaders in groups were evaluated higher than peers who reported being followers, and that course performance positively correlated with peer evaluations. White students received higher peer evaluations than students of colour. This difference reflects trends in group leadership and course performance, with more white students than students of colour reporting being leaders in groups and receiving higher grades.     

Course Design,Quality Matters Training,and Student Outcomes

ABSTRACT

North Carolina Central University (NCCU) recognized the need to address the increasing rates of Ds, Fs, and Withdrawal by students matriculating in online courses. Led by two science faculty, a faculty learning community in partnership with the NCCU Division of Extended Studies was created to assess online science course offerings and instruction. Faculty within this learning community completed the Quality Matters (QM) “Applying the Quality Matters Rubric” course. This online training course, coupled with services provided by the NCCU Division of Extended Studies, offered the faculty learning community a variety of pedagogical strategies to meet their instructional needs. All faculty participating in this learning community completed the online training course and all faculty indicated that they implemented changes in their subsequent online course offerings. This article describes the impact of that training and the application of quality course design standards in the Quality Matters Rubric on the design and student outcomes for an Introductory Biology course over four terms. As faculty learning communities are broadly utilized, it is anticipated that this article may present an effective strategy to increase the quality and quantity of online science, technology, engineering and mathematics (STEM) courses at similar institutions.     

Hail to Thee, Our Alma Mater: Alumni Role Identity and the Relationship to Institutional Support Behaviors

With the decline in state and federal support for higher education continuing to plague colleges and universities across the U.S., many institutions are looking to increase the levels of support annually received from alumni and other constituencies. Research on alumni relations in American colleges and universities has historically focused on different factors related to charitable giving. Although this study has resulted in some valuable information for institutions to use for alumni involvement purposes, most of the research has not been able to produce a meaningful look into how alumni identify with their college or university after graduation. The purpose of this study is to assess how college and university alumni view their role with these institutions after graduation and how that perception relates to behaviors of support. An online survey was constructed to assess three dimensions of alumni role identity based upon previous research on the identification process of blood donation. This study took place through the alumni association at a large, public research university in the Midwest and found that those who displayed increased alumni role identity were more likely to support the university through joining the alumni association, attending university sponsored events and charitable giving. This study breaks ground for a new method of measuring the role of alumni within colleges and universities in efforts to increase support and ease the financial pressures of today’s institutions.     

Motivation,Empowerment, and Innovation: Teachers' Beliefs About How Participating in the Edmodo Math Subject Community Shapes Teaching and Learning

Educators around the world participate in virtual communities, social media sites, and online networks in order to gain support and ideas for improving their practice. Many researchers have explored how and why teachers participate in these online spaces; however, there is limited research on how participation might impact teaching and learning. This study was designed to examine how K–12 teachers' participation in the Edmodo math subject community, an informal online space for educators, shaped their learning and practice. Data were collected through an online survey of 150 participants and in-depth, follow-up interviews with 10 participants. Three themes emerged from the participants' responses: motivation, empowerment, and innovation. The participants were motivated to spend time learning how to improve their practice, they felt empowered to make changes to their practice, and they believed they were able to adopt innovative, student-centered practices. Implications and ideas for further research are discussed.     

Undergraduate research experiences support science career decisions and active learning

The present study examined the reliability of student evaluations of summer undergraduate research experiences using the SURE (Survey of Undergraduate Research Experiences) and a follow-up survey disseminated 9 mo later. The survey further examines the hypothesis that undergraduate research enhances the educational experience of science undergraduates, attracts and retains talented students to careers in science, and acts as a pathway for minority students into science careers. Undergraduates participated in an online survey on the benefits of undergraduate research experiences. Participants indicated gains on 20 potential benefits and reported on career plans. Most of the participants began or continued to plan for postgraduate education in the sciences. A small group of students who discontinued their plans for postgraduate science education reported significantly lower gains than continuing students. Women and men reported similar levels of benefits and similar patterns of career plans. Undergraduate researchers from underrepresented groups reported higher learning gains than comparison students. The results replicated previously reported data from this survey. The follow-up survey indicated that students reported gains in independence, intrinsic motivation to learn, and active participation in courses taken after the summer undergraduate research experience.     

Comparison of online and classroom-based student evaluations of instruction

In recent years, colleges have been moving from traditional, classroom‐based student evaluations of instruction to online evaluations. Because of the importance of these evaluations in decisions regarding retention, promotion and tenure, instructors are justifiably concerned about how this trend might affect their ratings. We recruited faculty members who were teaching two or more sections of the same course in a single semester and assigned at least one section to receive online evaluations and the other section(s) to receive classroom evaluations. We hypothesised that the online evaluations would yield a lower response rate than the classroom administration. We also predicted that there would be no significant differences in the overall ratings, the number of written comments, and the valence (positive/neutral/negative) of students’ comments. A total of 32 instructors participated in the study over two semesters, providing evaluation data from 2057 students. As expected, online evaluations had a significantly lower response rate than classroom evaluations. Additionally, there were no differences in the mean ratings, the percentage of students who provided written comments or the proportion of comments in the three valence categories. Thus, even with the lower response rate for online evaluations, the two administration formats seemed to produce comparable data.     

The Development of a Discussion Rubric for Online Courses: Standardizing Expectations of Graduate Students in Online Scholarly Discussions

This paper reports on the design and evaluation of a rubric for assessing discussions in online graduate level education courses. The aims of the research were twofold. The first goal was to develop a discussion rubric that provides guidance to graduate students participating in online courses that are heavily discussion based. The second goal was to evaluate the efficacy of the rubric in student comfort with the expectations for the course and the quality of the discussions. Qualitative and quantitative course evaluations from four sections of an online graduate course are utilized to report on the effectiveness of the rubric upon implementation. Student average grades also show a statistically significant increase with the implementation of the rubric.     

The effect of switching to mandatory online course assessments on response rates and course ratings

Increasingly, student assessments of courses are being conducted online as opposed to administered in class. A growing body of research compares response rates and course ratings of courses evaluated online versus on paper. The present study extends this research by comparing student course assessments before and after the University of South Florida made online evaluations mandatory for all courses. This change only directly affected courses taught on-campus, as online courses were already being assessed online. However, we examine the effect of this change on courses taught on-campus and online, because we expect this change in policy to have differential effects. We hypothesise that by making online assessments mandatory for all courses, online assessments went from a novel method of evaluation to the norm; and, therefore, increased response rates for online courses, but had the opposite effect for on-campus courses. We find mixed support for our hypothesis.