Using a single score for summative teacher evaluation by students

In spite of the hundreds of studies done on teacher evaluation by student ratings, there are still several major controversial issues of how to construct evaluation instruments that best serve their purpose. The present study contributes to one of the ongoing debates of recent years that addresses the number of teaching dimensions to be considered in decision making, namely, whether to use a single score or multiple scores for teacher evaluation. The study demonstrates using a short form on which the global score overall teaching performance can almost perfectly predict the mean of all teacher-attribute items. The questionnaire used in this study was administered eight times—twice a semester for two years—for all faculty members and TAs in the departments of physics and chemistry at Tel Aviv University. The composition of teacher attribute items was different for the faculty members and TAs, reflecting their different teaching functions—lectures versus recitation problem solving. Results show that while for the faculty the global score can simply replace the mean of all instructor-attribute items and serve as a single score that faithfully represents all dimensions of teacher ratings, for the TAs, a linear transformation is needed.     

Student opinion of the value of student evaluations

This is an empirical investigation of student attitudes towards evaluation of instruction questionnaires. A 13-item attitude questionnaire was administered to 440 students in the Schools of Law and Engineering at Tel-Aviv University. Results indicated that students generally considered evaluation of instruction of value and were willing to participate in subsequent evaluations. They felt their evaluations were significant in influencing faculty promotion and selection; however, they were uncertain of their effect on improving the level of instruction. Criteria relevant for evaluation were the course work-load and instructor's grading policy; the instructor's academic status or talent for entertaining was of little importance. Students' academic expectations or self-definition made little significant difference either in attitude toward evaluations or in choice of rating criteria. However, students with high academic expectations were less ready to make evaluations, were unsure whether evaluation was part of the students' domain, and granted greater weight to the instructor's rank - all of which could indicate their identification with the norms of the academic world.     

Jo Handelsman

Note from the Editor

Educator Highlights for CBE-LSE show how professors at different kinds of institutions educate students in life sciences with inspiration and panache. If you have a particularly creative teaching portfolio yourself, or if you wish to nominate an inspiring colleague to be profiled, please e-mail Laura Hoopes at lhoopes@pomona.edu.LH: You are deeply involved with the HHMI Teaching Fellows Program at Wisconsin and the Wisconsin Program for Scientific Teaching (Pfund et al., 2009 ), and you''ve coauthored a book about scientific teaching (Handelsman et al., 2006 ). How do you teach people to teach in your summer institutes?Handelsman: The HHMI Graduate Teaching Fellows Program teaches graduate students and postdoctoral fellows to apply theories of learning to classroom practice. The fellows set learning goals and assess whether they''re achieved. It''s theory, then practice.LH: Can you explain a little more about how it works?Handelsman: The program starts with eight weeks of a course, “Teaching Biology” in which the fellows learn about education principles and then practice on each other applying those principles. Then they go on to design their own materials, and finally, in the second semester, use that material in teaching students. In our qualitative and quantitative analysis of their teaching philosophy, we see little change after the first semester. But there is radical improvement after they put their ideas into practice in the second part. People learn by doing.LH: How about a specific example of how the fellows develop materials.Handelsman: There''s a choice of venues, but let''s say one picks the honors biology course. They identify a technical problem, such as explaining Southern, Northern, and Western blotting. Our fellows then develop active-learning materials to address a challenging concept and test them in the classroom, often in multiple sections of a class. They refine and retest them. Another fellow might choose “Microbes Rule,” a course developed by fellows, which teaches about bacteria, viruses, and fungi. That fellow develops learning goals about antibiotic resistance, flu, or contaminated peanut butter, and designs classroom materials to achieve these goals.Open in a separate windowJo Handelsman, HHMI Professor, Department of Bacteriology, University of Wisconsin–Madison, Madison, WI.LH: Do the teaching fellows find the work difficult?Handelsman: It''s a challenge for them to narrow down to a workable subtopic. We work with them to focus on the learning goals, asking “The students will know and be able to do what at the end of this unit?”LH: Did you learn this method of focusing on goals when you were being trained?Handelsman: No, most of us were never taught to consider goals for learning. So in training our fellows, we direct them to focus on that over and over, and ask how their plans relate to the goals. It''s backward design—think about what you want to achieve, then think about how to get there.LH: Assessment is becoming more important at universities and colleges all over the country. How do you teach the fellows to use it?Handelsman: Students design their own instruments. They develop skills to determine whether their goals are being met. We go over the tools with them repeatedly, identify potential downfalls, let them implement, and then review the results to see if they obtained the information needed to determine whether their teaching worked.LH: What kind of questions do they tend to use for assessment?Handelsman: Exam-type questions are important, whether taken as an examination or in a questionnaire. Videos of student presentations with reviewers who score on effectiveness are also useful. We ask how the fellows know if the students understood the material, and how the evidence relates to each of their learning goals.LH: How do they evaluate and incorporate input from past assessment?Handelsman: Before using an instrument for assessment, the fellows develop a rubric to score the quality of the answers. Often they decide to share this rubric with the students. They want to show the students what goal the assessment is addressing, what is an adequate answer, what is an outstanding answer. Then they discuss with their peers how to use this feedback to improve their teaching.LH: I''ve heard faculty members at other places saying that they do lots of assessment but don''t know what to do with it after they are forced to collect the information.Handelsman: I''d suggest that they do less and use it more! Not using assessment results is like designing a new experiment but ignoring your earlier results. If we have the information to improve our teaching, we should use it.LH: A lot of interviews for faculty positions ask for a teaching philosophy. It sounds like your fellows are well-positioned to answer these questions.Handelsman: Yes, they have to write their teaching philosophy several times, discuss it with the other fellows, and rewrite. The fellows have been very successful in obtaining positions.LH: Have you had undergraduate research students?Handelsman: Yes, it''s one of the most important academic activities in which students take part—anything hands-on is good, but undergraduate research is the best because it incorporates inquiry, discovery, real scientific processes. It plays into curiosity. It''s such a rewarding process to watch a student in the research lab! It''s a powerful thing to see them learn and grow into scientists over the course of a semester or two.LH: What motivated you to take on undergraduate research students at the start?Handelsman: I started undergraduate research myself in my first year of college—I walked into a lab and asked to do experiments. The difference between doing research and reading about it is so dramatic. I''ve always assumed that part of the structure of an academic lab is undergraduate involvement. Interestingly, I sometimes give the undergraduates riskier projects than the graduate students, who have more to lose if their projects fail.LH: Thanks for sharing your insights into teaching with CBE-LSE.     

Faculty attitudes toward industrial research on campus

To gauge the attitudes of university faculty concerning the effect of corporate funding on campus research norms, we conducted a study of research faculty in Texas, employing the theoretical framework proposed by Alvin Gouldner. Gouldner theorized that the most privileged academics hold the most conservative social and academic views. In his view, the most highly research-oriented faculty—those who are senior, engaged in professional activities, and in important and secure positions—have careerist values that influence the reward system for other sociologists. These gatekeepers control professional dogma, and by means of editorships, board memberships, and other professional peer judgments establish normative behavior for the discipline. By analogy, natural scientists who compete successfully for research grants are in a position of having the most to gain by the status quo, and hence are very conservative; their own success convinces this elite tier of faculty that a meritocracy is at work. Several of our findings corroborated Gouldner's thesis, particularly in the convergence of ideologies between junior and senior faculty, and the higher propensity of scientists to support applied research. Finally, the impact of disciplinary orientation, as opposed to faculty rank or appointment, is discussed with a view to subsequent research.     

The academic department as environment for teaching and learning

This study reviews the literature on academic environments with particular reference to the academic department which is seen as the most important factor in the teaching and learning environment. Departmental environment characteristics as identified by faculty and by students are described. For students the most important are:-

Student-Faculty Relationships;

Interest and Engagement in Teaching; and

Satisfaction with Instruction.

Differences between teaching and between learning environments are explained, especially differences between social science and natural science departments. These differences reflect the interaction between discipline, personal styles of faculty and students, and faculty-student relationships. Student academic satisfaction seems to be heavily dependent upon the relationships between students and faculty. Student achievement in relation to students' perception of the academic department seems to be dependent on the degree of their adaptation to the department. The studies reviewed clearly show that there are differences between departments. They also show that these differences may be explained not only by differences with regard to the characteristics of the academic discipline concerned, but also by differences concerning student-faculty relationships, faculty interest in students and teaching and the interaction between these factors. Teacher and student satisfaction and student achievement are affected by these variables.     

The effects of autonomy support on student engagement in peer assessment

Although peer assessment is widely implemented in higher education, not all students are highly engaged in it. To enhance student engagement in peer assessment, we designed and developed a web-based tool, autonomy-supportive peer assessment (ASPA), to support students’ need for autonomy when they conducted peer assessment. Students’ sense of autonomy, and their behavioral, emotional, and cognitive engagement in peer assessment were examined via interviews and surveys. We also examined students’ academic performances, including their improvement from initial to revised essays and the quality of feedback they provided. Survey results indicated that the ASPA group (1) experienced a slightly higher sense of autonomy than the non-ASPA group, and (2) spent much more time on each evaluation criterion than the non-ASPA group. Interviews suggested that both the ASPA and non-ASPA groups were engaged in peer assessment. However, there were no statistically significant differences between the two groups in behavioral, emotional, cognitive engagement, and academic performances.     

Student attitudes that predict participation in peer assessment

Peer assessment has been widely applied to actively engage students in learning to write. However, sometimes students resist peer assessment. This study explores reviewers’ attitudes and other underlying factors that influence students’ participation in online peer assessment. Participants were 234 Chinese undergraduates from two different academic backgrounds: engineering majors (n = 168) and English majors (n = 66). Gender, academic background and prior experience with peer assessment were all related to participation levels. Moreover, factor analyses revealed three attitudinal factors: (1) positive attitude (a general endorsement of the benefits of peer assessment), (2) interpersonal negative (concerns about the negative effects on interpersonal relationships), and (3) procedural negative (doubts about the procedural rationality of peer assessment). Among the attitudinal factors, procedural negative was negatively associated with participation, as expected. Interestingly, interpersonal negative was associated with greater participation, and positive attitude was associated with lower participation, in part because students worked hard on each review rather than doing many reviews superficially. Implications for instruction are discussed.     

Hybrid structures: Faculty use and perception of web-based courseware as a supplement to face-to-face instruction

The researchers examined responses from 862 faculty members at 38 institutions nationwide using the blackboard Learning Management System (LMS) to supplement their face-to-face instruction. The four research questions addressed the primary uses that faculty make of blackboard, perceptions that faculty have of how certain blackboard features enhance or elevate (or might enhance or elevate) their assessment of student work and instructional capabilities, and how faculty use of blackboard might positively affect the psychosocial climate within the face-to-face classroom setting. Additional analysis sought to identify the factors that predict use and positive perception of blackboard as a supplement to face-to-face teaching activities. The results indicate that faculty primarily used blackboard as a course management/administration tool to make course documents available to students and manage course grades. Few faculty used blackboard for instructional or assessment purposes, and even fewer utilized blackboard to foster a more positive sense of community within their face-to-face classes. Faculty attitudes, on the whole, were positive when it came to the classroom management functions of blackboard, but neutral or otherwise undecided in terms of its instructional or psychosocial benefits. The main factor in determining blackboard usage—whether for course administration or instructional purposes—was experience with the tool. In addition, women had more positive attitudes than men did in terms of blackboard's potential to enhance classroom management and foster a positive relational climate. Limitations of the study and suggestions for future research are discussed before concluding.     

Dancing with the devil: formative peer assessment and academic performance

Peer assessment can be important in developing active and independent learners, as well as providing more and faster feedback in large classes, compared to marking done by tutors. In addition, the evaluative, critical stance required by students in order to assess their peers' work encourages the development of higher-order cognitive skills. Changing roles from being assessed to being an assessor can also improve students' ability to judge and improve on their own work. However, peer assessment does have potential problems and there is some debate as to the appropriate academic level at which to implement it, the kinds of feedback that are given and the ways in which students respond. In addition, there is little evidence that peer assessment has an impact on academic performance. This research reports the results of an online peer assessment exercise for a macroeconomics essay conducted in a large Economics 1 class at Rhodes University. Of the 800 students, about half participated in the peer assessment exercise. Data were collected from students via a formal course evaluation. In addition, a sample of 50 essays was evaluated in terms of the relationship between peer marks and final (tutor) marks received and the impact that peer assessment had on the quality of the final essay submitted. An Ordinary Least Squares regression was used to investigate the impact of peer assessment participation on marks. Results showed that peer marks tended to ‘bunch’ in the 60–68% range, indicating the reluctance of peers to give very high or low marks. In general, peers gave more useful feedback on technical aspects, such as presentation and referencing (which were also the categories in which students most often made improvements), than on content. Regression analysis showed that peer assessment participation was not a significant determinant of final essay mark, but that economics ability and English language proficiency were.     

The evaluation and improvement of teaching in higher education

Four procedures to evaluate teaching (by students, peers, video-recordings, and direct measurements of student learning) and three uses of the evaluation results (improving teaching, personnel decisions, course handbooks) are reviewed in the light of empirical evidence. Special emphasis is placed on the timing and validity of student ratings and the instruments used. Since none of the procedures appear sufficient in and by itself, a multiple indicator approach, especially for personnel decisions, would seem to be the most defensible one.While it is essential to take evidence of teaching effectiveness into account in considerations for tenure and promotion, faculty must also be given opportunities to become professionals as teachers. Higher education units, designed primarily for this purpose, appear to be effective as judged by their clients (the faculty they have served), but have failed to make an impact on the faculty as a whole. What is required now is an institutional commitment to quality instruction, i.e. a departmental policy on the evaluation of teaching and faculty development.Invited keynote address presented at the Third International Conference on Improving University Teaching, Newcastle-upon-Tyne, June 8–11, 1977. An annotated bibliography Evaluation des systèmes d'éducation supérieure produced by L. de Marchi in preparation for this address is available from the author.     

Emigration,internationalization, and “brain drain”: Propensities among British Academics

This research links two important higher education phenomena: potential brain drain among academics abroad and the U.S. academic labor market. The inquiry draws on the brain drain literature and is grounded primarily on a secondary analysis of a major survey (1989) of British university and polytechnic faculty members. The anlaysis shows that fully 40.0 percent of university faculty are seriously considering a move abroad with the substantial majority favoring the United States as a destination. Faculty characteristics—including academic field, research versus teaching orientation, rank, age, gender, and political identification—are correlated with faculty members' professed interest in emigration. The analysis also compares (former) polytechnic to university faculty as well as to a subset of Oxford and Cambridge faculty.     

Faculty-student engagement in teaching observation and assessment: a Hong Kong initiative

There is now a worldwide focus on the quality of university teaching and yet there is general dissatisfaction in universities with the student evaluation of teaching system. Peer observation of teaching seems to hold much promise in the assessment of teaching quality, but such observation pays little attention to the quality of teaching as perceived by students. One approach to overcome this deficiency is for faculty and students to also partner in the assessment of a faculty member’s teaching, with a student trained in observation and feedback techniques acting as a peer in the observation process. This paper describes and evaluates an ongoing student consultant initiative at Lingnan University in Hong Kong. It presents faculty and student observations in terms of the benefits to faculty regarding potential enhancement of university teaching, and the benefits to students especially in terms of close collaboration with faculty and training in consultation techniques. The paper notes that the student consultant initiative has been more popular with students than faculty, and recommends further investigation of the potential of such programmes in Hong Kong higher education.     

Examination involving students as peer examiners

The main interest in this article is students’ involvement in assessment as a part of growth towards self‐directedness in learning. In order to enhance students’ development of autonomy in learning, a project involving ‘older’ students as peer examiners for ‘younger’ students was designed and carried out. Students in the sixth semester in a PBL‐based Master’s program of Medical Biology participated, together with faculty, as examiners of fifth‐semester students. The examination and the assessment situation was carefully designed based on learning theories, empirical evidence and experiences underpinning student‐centred learning, especially in the form of PBL used at the faculty. The project was evaluated and analysed in order to understand students’ learning processes related to the responsibility for assessing peers. The situation of the peer examiners was interpreted based on their own experiences with statements from the students assessed and faculty involved in the assessment. Evaluations from six occasions, spring and fall, 2003–2005, were included in the study. The findings suggest that involving students in assessment as equal partners with faculty makes it is possible for students to apprehend the metacognitive competences needed to be responsible and autonomous in learning. The peer examiners experience motivation to learn about learning, they acquire tacit knowledge about assessment and they learn through being involved and trusted. The student‐centred educational context, which requires responsibility throughout the programme, is recognized as very important.     

Assessment of a conceptually based instructor evaluation form

An instrument to be used by students in evaluating faculty was developed with the major focus being five conceptualized interpretations of scales (factors) culled from other research on faculty evaluation. The five conceptualized factors were: Evaluation, Presentation, Preparation, Personality, and Intellect. Thirty-five professors from the Division of Curriculum and Instruction teaching 65 classes (1,122 students) at graduate and undergraduate levels participated in May; and 75 professors teaching 2,804 students participated in a December study. Each of the five factors were found to be: independent; stable across student groups; of high internal consistency and reliability; of a high degree of concurrent validity (faculty evaluating themselves); discriminatory among faculty; and applicable under sundry instructional conditions. The instrument can provide information to instructors for the improvement of teaching, as well as providing information for students concerning individual instructors. As part of a larger evaluation system, the instrument can provide information for career decisions.     

Teaching and learning in higher education: Recent trends

In the last decade, the questioning of traditional teaching and learning methods and instructional systems and the search for more effective alternatives have gained momentum in higher education. Recent trends can be grouped as follows:

Medical educators’ metaphoric talk about their assessment relationships with students: ‘you don’t want to sort of be the one who sticks the knife in them’

Current perspectives in cognitive linguistics highlight the conceptual nature of cognition and how the conceptual metaphors we hold affect ways we think, talk and act. This study examines medical educators’ metaphoric talk to understand ways in which assessment relationships with students are conceptualised in order to understand why educators might ‘fail to fail’ underperformance in medical students. We conducted 10 focus groups with 70 educators at two UK medical schools. The audiotapes were transcribed and analysed using systematic metaphor analysis. The analysis revealed six over‐arching metaphors associated with the target domain of assessment relationships, i.e. assessment relationships as journey, war, sport, parentalism, machine and medicine. While medical educators conceptualised students’ academic failure in relatively innocuous ways (e.g. students’ failure to arrive at their destination; journey), they also conceptualised failure in harmful ways (e.g. students’ death; war). If medical educators hold the concept of assessment relationships as war, it is perhaps unsurprising that they are reluctant to fail students. We discuss these findings in light of existing literature and provide recommendations for designing training sessions through which assessors can address reluctance to fail underperformance. Such faculty development may aid assessors in identifying and using alternative metaphors to conceptualise the assessment relationship.     

The effects of faculty behaviors on the academic achievement of first-year Cambodian urban university students

Research on the faculty impact on students’ academic achievement has been disproportionately confined to the context of countries with developed higher education systems. Few studies have been undertaken in the developing world like Cambodia. This study employed hierarchical linear modeling to examine the relationships between faculty behaviors and the academic achievement of university students in Cambodia, using the data of 923 first-year students from nine universities in Phnom Penh City. Results indicated that faculty behavior, namely their support and feedback to students, was a unique factor that had a strong and positive influence on students’ academic achievement. Its effect was the same for all students regardless of their pre-university academic experience and geographical origin and partially moderated by student engagement in time spent on course-related tasks outside the classroom, assigned homework/tasks, class participation, and class preparation. Contrary to existing findings from faculty impact studies, no relation was found for faculty’s instructional organization and clarity or classroom practices that challenge students on academic achievement. Practical implications for assessment policies and instructional practices are discussed.     

Advising time inventory: Consequences of the general college individualized baccalaureate degree program on faculty advising,activities, and academic load

Faculty of the General College are assigned advising responsibilities in addition to their teaching load. Full-time members, for example, advise 35–40 students per year. When the College initiated an individualized baccalaureate degree program in 1971, little provision was made for consequent changes on the advising function and the academic load. As a result, advisers, faced with the complex and time-consuming matters that adhere to individualized learning programs, felt mounting pressure. Large numbers of evening school students were attracted to the program and sought advice and, upon admission, expected to be assigned a faculty adviser. The College made no formal acknowledgment of the addition of the non-day school advisees to advisers' loads. Strained by the overburden, faculty participated in this study to identify how much real time they spend in advising, with whom, and on what kinds of activities. Evidence from this study was used to support their request for recognition by the College of the changed nature and full scope of their advising work.The author wishes to acknowledge Drs. Paul J. Feltovich and Thomas Brothen for their help with the researching of this paper.     

Evaluation of a High-Engagement Teaching Program for STEM Graduate Students: Outcomes of the Future Academic Scholars in Teaching (FAST) Fellowship Program

Higher education institutions prepare future faculty members for multiple roles, including teaching. However, teaching professional development programs for graduate students vary widely. We present evaluation data from a high engagement program for STEM doctoral students. We analyzed the impact on three cohorts of participants over three academic years and identified the components most influential upon their teaching professional development. Participants found the year-long teaching assessment project and the disciplinary and reflective focus instrumental for improving their knowledge of teaching and learning. We recommend these components for the design of other such high-engagement programs.