Defending the Community College Equity Agenda By Thomas Bailey and Vanessa Smith Morest

The current rhetoric around using data to improve community college student outcomes with only limited research on data-driven decision-making (DDDM) within postsecondary education compels a more comprehensive understanding of colleges’ capacity for using data to inform decisions. Based on an analysis of faculty and administrators’ perceptions and behaviors at 41 community colleges that participated in an initiative to improve student success, an argument is presented to include social capital as an explicit component of the capacity of community colleges for using data on student outcomes to increase student success. Building on Newmann, King, and Rigdon’s (1997) conceptualization of schools’ organizational capacity to meet accountability expectations and Smylie and Evans’ (2006) exploration of the role of Coleman’s (1988) social capital in policy implementation, this study found a relationship between the presence of forms of social capital as part of the organizational capacity for DDDM and the frequency and extent of data use among faculty and administrators. In light of research on organizational learning that suggests that social capital creates opportunities for the creation of new knowledge—such as possible solutions for persistent problems of student success—and research on organizational routines as mechanisms for change and preservation in organizations, this article concludes with recommendations for community colleges undertaking data-driven educational reform.     

Academics' resistance to summative peer review of teaching: questionable rewards and the importance of student evaluations

This study draws from 30 semi-structured interviews with tenure-track faculty members in a research-intensive university to examine their lack of engagement in the summative peer review of teaching. Findings indicate that most academics in the study do not think peer review outcomes contribute meaningfully to decisions about career advancement and believe that, in comparison, student evaluation of teaching scores matter more. The findings suggest that faculty member resistance to summative peer reviews will persist unless academics are confident that the results will be seriously considered in decisions about tenure and promotion. This article also contends that senior administrators should provide further clarity about the purpose and use of peer review outcomes in high-stakes career decisions.     

Student Perspectives on Teaching and its Evaluation

The research on student ratings of instruction, while voluminous, has had minimal focus on the perceptions of the students who do the ratings. The current study explored student perspectives on course and teacher ratings as well as some issues related to teaching effectiveness and faculty roles. It was found that students are generally willing to do evaluations and to provide feedback, and have no particular fear of repercussions. However, they have little confidence that faculty or administrators pay attention to the results, and do not even consult the ratings themselves. The students view teaching and advising as the most important roles that should be played by faculty, yet project that faculty, while also viewing teaching as the most important, would rank research above the more student-interactive advising. Canonical correlations among various scales reveal a strong emphasis on such issues of the importance of faculty respect for student views.     

Student Evaluation of Faculty: Effects of Purpose on Pattern

Much of the existing research addressing student ratings of faculty has focused on psychometric properties of the rating process or on physical characteristics of courses and individuals. Less explored is the cognitive processes of students, as evaluators, and the administrative purposes for which outcomes from student evaluations are used. To determine if students used different cognitive sets as a function of the explicit purpose for which ratings are obtained, a field experiment was conducted. Explicit purposes for course evaluations were manipulated to include a control condition, a formative condition (instructional improvement), and a summative condition (salary increase). These conditions served as the classification variable in a discriminant analysis and six different dimensions of classroom performance, as assessed by students, served as the discriminating variables. Results indicate that the purpose of assessment fails to influence the pattern of faculty ratings provided by students. From an administrative perspective, these findings have practical implications both for the collection of student evaluations and for the use of student ratings.     

Assessing student exposure to and use of computer technologies through an examination of course syllabi

A syllabus analysis instrument was developed to assist program evaluators, administrators and faculty in the identification of skills that students use as they complete their college coursework. While this instrument can be tailored for use with a variety of learning domains, we used it to assess students' use of and exposure to computer technology skills. The reliability and validity of the instrument was examined through an analysis of 88 syllabi from courses within the teacher education program and the core curriculum at a private Midwest US university. Results indicate that the instrument has good inter‐rater reliability and ratings by and interviews with faculty and students provide evidence of construct validity. The use and limitations of the instrument in educational program evaluation are discussed.     

Ten Years of Puzzling about Audience Awareness

The decisions evaluators make regarding how to work with teachers through the annual evaluation process can have a positive impact on teacher growth and student learning. In this article, the author examines the quality of feedback and how administrators use and provide this feedback to teachers through the evaluation process. What constitutes quality feedback? How does quality feedback support teachers' instruction? How can the supervision and evaluation process offer both teachers and administrators an opportunity to work together to improve student learning? The evaluation process should support the goal of maintaining a high level of quality instruction. In the end, the goal is to increase student learning by improving the effectiveness of teaching.     

Faculty Perspectives on Course and Teacher Evaluations

Student ratings of instruction have been the subject of numerous studies with much of the research focusing on the validity and reliability of the ratings themselves. Comparatively little empirical investigation has been devoted to the perceptions of the individuals who are the subjects of the ratings, that is, the faculty. The current study explored faculty perspectives on the usefulness of student ratings for formative and summative purposes, and the actual use of student ratings for summative purposes. Contrary to what might have been deduced from the anecdotal literature, the results of this study do not portray a great deal of resistance to student ratings in general or to their use for formative and summative evaluation. It was also found that student ratings are actually being used for the latter purpose. The usefulness of the student feedback was viewed differentially by the faculty, with feedback on their interaction with students seen as most useful, followed by feedback on their grading practices, global ratings of instructor and course, and finally structural issues of the course.     

Instructors’ perspectives on the utility of student ratings of instruction

This study investigates instructors’ attitudes about student ratings of instruction. The sample consisted of 357 instructors at a major Canadian university, where student evaluation is carried out in all courses at the end of each term. Instructors tend to agree that the student rating practice is an acceptable means of assessing institutional integrity, and is useful to administrators in making summative decisions. However, they consider the student evaluations only marginally valuable in their impact on enhancing instruction. Implications for the development of student ratings are discussed.     

Student ratings of college teaching: a comparison of faculty and their students

Based on the review of student ratings myths by Aleamoni (1987, 1999), a survey research design was used to analyse differences between college students' (n = 968) and faculty's (n = 34) perceptions. Generally, students held stronger beliefs in these myths, in that they believed faculty with excellent publication records were better qualified to evaluate teaching and that student ratings on single general items are accurate measures of teaching effectiveness. On the other hand, faculty believed that student ratings were invalid and unreliable. Further examination of student characteristics revealed that male students held stronger beliefs in these myths. Finally, students' beliefs in these myths were correlated with their actual ratings of nine dimensions of the Student Evaluation of Educational Quality. A discussion as well as suggestions for using student ratings is provided.     

Faculty Views of Student Evaluation of College Teaching

The literature abounds with psychometric studies of course evaluation measures and articles debating the merits of student ratings of instruction, but little research has focused on faculty perceptions of this procedure. In the present study faculty perceptions are explored at a teachers' college where evaluation is carried out annually on a sample of courses. The sample includes 101 instructors who completed the research questionnaire. Faculty attitudes reflected a broad range of responses towards validity of student ratings, and their usefulness for improving instruction. Although overall attitudes were mildly positive, few instructors reported changing instruction as a result of student ratings. Moreover, few supported sending evaluation results directly to college administrators or publishing them for student consumption.     

Comparison of online and classroom-based student evaluations of instruction

In recent years, colleges have been moving from traditional, classroom‐based student evaluations of instruction to online evaluations. Because of the importance of these evaluations in decisions regarding retention, promotion and tenure, instructors are justifiably concerned about how this trend might affect their ratings. We recruited faculty members who were teaching two or more sections of the same course in a single semester and assigned at least one section to receive online evaluations and the other section(s) to receive classroom evaluations. We hypothesised that the online evaluations would yield a lower response rate than the classroom administration. We also predicted that there would be no significant differences in the overall ratings, the number of written comments, and the valence (positive/neutral/negative) of students’ comments. A total of 32 instructors participated in the study over two semesters, providing evaluation data from 2057 students. As expected, online evaluations had a significantly lower response rate than classroom evaluations. Additionally, there were no differences in the mean ratings, the percentage of students who provided written comments or the proportion of comments in the three valence categories. Thus, even with the lower response rate for online evaluations, the two administration formats seemed to produce comparable data.     

Bias and the intended use of student evaluations of university faculty

This study examines whether the intended use of data obtained from student evaluations of university faculty biases the ratings. The paper reviews and critiques previous research done in the area. Previous studies produced contradictory results, but the results were confounded by several methodological problems. The research reported here remedied some of the problems in previous studies. The findings indicated that, while students are aware of the intended use of ratings as stipulated in written directions, different uses do not result in statistically significant differences in the ratings. Implications for the use of student ratings in the evaluation of faculty are discussed.     

Class Level as a Possible Mediator of the Relationship between Grades and Student Ratings of Teaching

Although the research literature investigating the relationship between grade awarded to students and students’ evaluations of teaching performance is voluminous, very few studies have examined the grade‐rating relationship according to level of student. The present study examined correlations between mean instructor rating and mean class grade for all course evaluations (N = 625 classes) at Utah State University during an academic quarter. In lower‐division (courses 100–299) and upper‐division (courses 300–599) undergraduate classes, correlations between grades and ratings of faculty were of an expected direction and magnitude (0.29 and 0.28, respectively); however, the grade‐rating correlation for graduate classes (courses 600 +) was — 0.20. It is speculated that graduate students are both better students and more critical evaluators of instruction, but replication and extension with different samples are needed before this tentative explanation can be accepted with confidence.     

Effects of grading practices and time of rating on student ratings of faculty performance and student learning

The effects of grading practices (strict, lenient) and time of rating (after a lecture, after taking an examination, after receiving feedback on performance on the examination) on student ratings of faculty performance and student learning (i.e., performance on an examination) were assessed in two experimental studies. Results indicated that (1) student ratings were directly affected by grading practices and this effect was a function of the time at which ratings were cornpleted, and (2) student learning and students' ratings of faculty were positively correlated, and grading practices had no effect on student learning. Implications of these findings for the administrative use of student ratings are considered.     

Similarity of student ratings across instructors,courses, and time

This study raised three questions about the similarity or generalizability of student ratings of courses and instructors. First, how stable are student ratings of the same instructor giving the same course during twodifferent semesters? Second, how similar are student ratings of the same instructor in twodifferent courses? Third, how similar are student ratings of a given course being taught bydifferent instructors? Instances were identified in which student ratings on seven different factors were available for pairs of courses for each of these questions. For the case of the same instructor — same course—different semesters, student ratings were reasonably similar (median r for seven factors about 0.70). For the case of the same instructor—different courses, the median r was surprisingly low — about 0.40. For the case of the same course—different instructors, substantial correlations were obtained for some factors and insignificant correlations for other factors. Implications of these findings for practical use of student ratings and suggestions for further research in the area are discussed.     

An Assessment of Statistical Process Control-Based Approaches for Charting Student Evaluation Scores

We compare three control charts for monitoring data from student evaluations of teaching (SET) with the goal of improving student satisfaction with teaching performance. The two charts that we propose are a modified p chart and a z‐score chart. We show that these charts overcome some of the shortcomings of the more traditional charts for analyzing SET data. A comparison of three charts (an individuals chart, the modified p chart, and the z‐score chart) reveals that the modified p chart is the best approach for analyzing SET data because it utilizes distributions that are appropriate for categorical data, and its interpretation is more straightforward. We conclude that administrators and faculty alike can benefit by using the modified p chart to monitor and improve teaching performance as measured by student evaluations.     

On the use of student ratings of faculty in determination of pay,promotion and tenure

Student ratings of faculty have traditionally been obtained in a manner designed to be useful to the individual faculty member or to other students. It is now sometimes proposed that the resulting data be used to determine faculty pay, promotion, and tenure. Recent articles and the reviews of past literature on ratings are analyzed to determine whether student ratings are usually associated with teaching effectiveness, whether they are sometimes biased by irrelevant factors, whether faculty can effectively revise their behaviors to improve their ratings, and whether improved ratings result in improved teaching effectiveness. The results tend to indicate that student ratings of faculty as they are currently collected cannot be trusted for considerations of pay, promotion, and tenure.     

Student faculty evaluation (SFE) at Jordanian universities: a student perspective

This study is the first to address student evaluation of faculty members (SFE) from a student perspective at a major Jordanian public university using a comprehensive (71-item) questionnaire administered to 620 undergraduates. Addressed are students’ perceptions of the SFE process in terms of: (a) their paper-based vs. online-format preferences; (b) their beliefs about the process; (c) the standards they adopt in the evaluation; (d) the fruitfulness of SFE outcomes; and (e) their opinions about the evaluation instruments currently used at their institutions. It also explores whether students’ beliefs vary according to their gender, GPA and college (Educational Sciences vs. Sciences). The results reveal that most students prefer online evaluation, adopt academic rather than interpersonal criteria in their evaluation, have a low level of satisfaction with the impact of previous evaluations on faculty teaching behaviours and the evaluation instruments used at their institution. Additionally, the findings suggest that gender has a less influential role in shaping students’ beliefs about the SFE process compared to their college or GPA. The findings of this study and their implications are very useful pointers for faculty and higher education administrators, faculty members and students towards further improvement at higher education institutions.     

The localization,diagnostic, and monitoring functions of student ratings in a model for improving university teaching

A diversity of student questionnaires are used by colleges and universities to provide data on faculty teaching performance. Yet the purposes for collecting this data are frequently unclear, and at times superficial. Rarely is student rating data used as a tool to improve faculty teaching. A more relevant approach incorporates a variety of types of student ratings into a model for improving university teaching. One type of student rating data is used to identify broad instructional problem areas. Another type pinpoints probable causes and solutions for the instructional problems. Instructional improvement procedures are designed on the basis of this data. A third type of student rating data evaluates the instructional improvement procedures and indicates when modifications are needed. In addition to these three types of student ratings, and the generation of appropriate questionnaires, this paper presents an overview of the teaching improvement model and discusses its effectiveness.     

Comparing Faculty and Student Perceptions Regarding Factors That Affect Student Retention in Online Education

This qualitative study compared faculty and student perceptions regarding factors that affect student retention in online courses in an attempt to more effectively address the problem of attrition. A grounded study method was used to interview students taking online courses, analyze their responses related to the critical factors that affect student retention, and compare them with those given by expert online faculty documented by Gaytan (2013). Among the various findings, two are considered critical: online students would like to receive more instruction from their professors and more comprehensive feedback that would allow them to engage in corrective behaviors to improve performance. Comparing faculty and student responses related to the factors that affect student retention could give online program administrators and faculty advisors a better understanding of these critical factors to be able to respond to the student retention challenge more effectively.