THE IMPACT OF FORMATIVE AND SUMMATIVE EVALUATION OF TEACHING IN NORTH AMERICAN UNIVERSITIES

This paper reviews recent research evidence on formative (developmental) and summative (judgemental) evaluation of teaching in North American universities. The major conclusions of the review are as follows: (1) student ratings are the most common method of evaluating teaching in North American universities, and their popularity appears to be increasing; (2) student ratings have been found to provide reliable and valid information on limited aspects of teaching competence; (3) teaching evaluations make a difference in decisions on faculty salary, tenure and promotion; (4) evaluation of teaching has been shown to produce modest improvements in teaching effectiveness; (5) faculty satisfaction with teaching evaluation practices ranges from very high to very low, depending on institutional context; and (6) there is consensus that several specific improvements are needed in the current teaching evaluation system.     

Understanding and using feedback – A review of empirical studies concerning feedback from external evaluations to teachers

To improve the quality of teaching, educational accountability needs to include periodic external evaluations of students’ performance. This requires evaluation formats which support the development of the educational process and provide information which is understandable for teachers. The aims of this study were to review: (i) how teachers understand the feedback they receive from external evaluations; (ii) how they use the feedback; and (iii) how teachers’ understanding and use of such feedback affects the achievement of their students. None of the papers included contained simultaneous a study with all three of these aspects of external evaluations; the review shows that teachers have many problems understanding feedback and mainly focus their use of it on developing strategic teaching tactics. Research that focuses on teachers’ understanding of external evaluation, their use of it, and how the use of the feedback can foster student achievement is needed.     

大学评估何处去?国际评估在中国一流大学的兴起、扩散与制度化

我国的大学评估在较长时期内伴有浓厚的自上而下的行政主导色彩,政府资源在大学间的分配与评估结果存在紧密联系,大学更倾向于凸显自身优势而非探究自身问题。在推动世界一流大学建设和教育评价改革持续深化过程中,中国大学的评估体系逐渐发生变化,国际评估由此兴起并发展成为一流大学开展自我评估的重要机制。本研究通过对早期3个大学的评估案例分析,指出早期开展国际评估的大学进行评估时带有鲜明的问题导向,在其自身蕴含的强烈变革意愿的牵引下,评估促成了大学的人事管理体制机制、校院两级管理体制、跨院系学科建设等多层次的自我变革。此后,国际评估机制在中国一流大学群体中不断扩散。高等教育评估制度环境的变迁是国际评估机制扩散的重要基础,对先行者的学习效仿则是扩散的直接动因,在合法性机制作用下国际评估机制不断扩散且持续制度化并内嵌于大学治理体系中。     

Repositioning Colleges Using Changing Student Quality Perceptions: An Exploratory Analysis

ABSTRACT

Colleges and universities must respond to competitive pressure in the marketplace by embracing the concept of quality as a marketing tool. Many colleges and universities have recently looked to the business arena and have adopted techniques, strategies, and the language of quality management.

This exploratory study focuses on the identification and evaluation of student perceptions of salient attributes. The findings suggest that standards of quality are difficult to establish, since evaluations of university performance are not consistent, even among an undergraduate student population.     

The perceived impact of external evaluation: the system,organisation and individual levels—Estonian case

The purpose of this article is to provide an overview of how the employees of higher education institutions perceive the impact of external evaluations. The study was conducted using the concurrent mixed method and involved 361 employees from Estonian universities and professional higher education institutions. The results indicated that evaluation is mostly deemed necessary at the levels of the organisation and the system; from the personal viewpoint of an employee, positive effects of external evaluations are perceived to a lesser degree. The negative influences mentioned were increased bureaucracy, extra work and stress. Top managers of institutions most often saw the positive influences from external evaluations since they help to implement reorganisations and development activities within an organisation. Managers perceived the positive effects of external evaluations on their everyday work as well, more than other employees did. Researchers felt the positive effects of external evaluations least. Nearly a quarter of all employees of higher education institutions do not perceive the effects of external evaluations on their everyday work at all. Employees of professional higher education institutions perceive the positive effects of external evaluations to a greater extent than employees of universities do, indicating that external evaluation has contributed to the development of their organisations. The differences in opinions of the employees of higher education institutions regarding perceived positive and negative impacts of external evaluations—whether on individual, organisation or system points of view—indicate that employees do not identify closely enough with the organisation and its strategic goals.     

Universities and Mid-Career Professionals: The Policy Potential

To date, few universities have become substantially involved in providing for the learning needs of professionals in mid-career. This paper argues the case for their doing so in the future. It begins with a review of the current provision of formal continuing professional development (CPD) – in which universities can be seen to perform only a limited role – and goes on to examine the attitudes of practitioners towards various forms of learning experience and their resulting interactions with the academic world. The reasons why such interactions are predominantly ad hoc and individualistic, rather than systematic and collective, are briefly considered before the opportunities for greater, more coherent mutual involvement are explored. The concluding section briefly reviews the policy decisions and organisational changes which appear to be needed before the full potentialities are realised.     

The University Presidency: Do Evaluations Make a Difference?

High turnover and short tenures are the norm for state university presidents in the last part of the twentieth century. Depending on the type of institution, the average state university president stays in office about four years, some three years shorter than the national average of seven years for all higher education institutions. The article discusses this trend and relates length of tenure to the formal and informal evaluations of presidents. Beginning with a presentation of presidential tenure (time in office) data and a discussion of the benefits of longer terms, the article continues with a dialogue on the role of the president in state institutions of higher education. It is argued that the position of president necessitates constituent leadership with presidents often expected to please all constituencies. Included in this discussion are the nature, role, and effectiveness of evaluations of university presidents. The authors conclude that evaluations of presidents make little difference in decisions to retain or remove presidents. Even a good or great evaluation does not offset the impact of the unforseen event or changes in political leadership within a state or on a board. The authors agree that annual reviews or reports are valuable in improving a presidents performance and leadership. Evaluations make a difference when they are supervised by individuals with direct knowledge of the job, are fair in terms of expectations from often divergent constituent groups, and focus on maximizing the ability of the president to improve the institution.     

National Report on Universal Education in China (November 2005)

A study of the science research activities in China's institutions of higher learning in recent years indicates that there is a major connection between the current instances of corruption in scientific research at colleges and universities and the evaluations system for scientific research implemented at many of the colleges and universities. This paper analyzes the imperfections in the science research evaluations system. It explores ways for devising scientific systems and criteria for the evaluation of academic research as well as ways to conduct scientific and impartial evaluations of academic research.     

The impact of electronic media on faculty evaluation

With the proliferation of computer networks and the increased use of Internet‐based applications, many forms of social interactions now take place in an on‐line context through Computer‐Mediated Communication (CMC). Many universities are now reaping the benefits of using CMC applications to collect data on student evaluations of faculty, rather than using paper‐based surveys in Face‐To‐Face (FTF) classroom settings. While the relative merits of CMC versus FTF student evaluations have been researched extensively, there is limited research published about the ways students respond to the questions from either mode of data collection. This paper reports on a research study to analyse the communication differences between student scores from FTF student evaluations and CMC evaluation questions from end of semester evaluations from a university in the Middle East region. In addition to the questions about communication mode differences between two evaluation questions, several demographic variables were measured to determine any interaction effects. The results of our study suggest that the type of communication channel mitigates the responses that students make on CMC evaluations vis‐à‐vis FTF evaluations of faculty. In particular, even though there were significant differences found at the aggregate level between CMC and FTF evaluations, when the course and instructor are controlled for, there were no significant differences reported. In addition, several differences were noted depending on the type and level of the course being studied. Also, we found that students are more likely to express more extreme responses to scale questions in CMC than FTF evaluations. Administrators should consider these potential differences when implementing on‐line evaluation systems.     

Final year undergraduates’ perceptions of the integration of soft skills in the formal curriculum: a survey of Malaysian public universities

A recent initiative or skill bridging measure taken by the Malaysian public universities is to build into the formal curriculum soft skills to produce graduates with a right balance of diverse abilities. However, to date, there is no comprehensive attempt to review the integration of soft skills in the formal curriculum (both coursework and training) of university programmes. The paper therefore reviews the adequacy of the infusion and acquisition of the entire range of skills embedded in taught courses and practical/industrial training from the students’ perspective. The specific focus of the study is to identify what worked well and what had not in acquiring the skills designated in the teaching–learning process. The key findings of the study are as follows. First, the perceived infusion–acquisition of skill types differs unequivocally between coursework and training, suggesting the complementary nature of both components of the formal curriculum for the integration of soft skills. Second, the infusion–acquisition of soft skills remains highly concentrated on specific items/skills for both coursework and training. For the coursework component, communication skills explain most of the total variance, whilst moral and professional ethics ranks first in explaining the total variance for the training component. In total, the perceived low and selective appreciation of skills by students signals the need to readdress the existing strategies within the teaching–learning process to ensure a better integration of soft skills.     

对美国高校计划评审活动的研究——以康奈尔大学为例

美国大多高校都有计划评审活动。计划评审活动以改善自我为主要目的,立足院校实际,通过自评和外部评审提高质量的同时也为高校决策提供依据。     

韩国大学的社会评价探析

随着高等教育的发展,韩国的大学评价也逐渐多样化,其中社会评价受到人们越来越多的关注。韩国中央日报社的大学评价与企业视角中的大学评价在评价目标、评价内容、实施方法等方面各具特点,一方面为学生、家长及企业界等教育消费者提供了大学的相关信息,努力促进大学的发展;另一方面也受到大学的批评与质疑,其发展状况值得人们关注。     

Giving Back to the Web: Social Filtering of World Wide Web Resources in High School Science

Can high school students become contributors as well as users on the World Wide Web? This research explores a new Web-based curriculum idea, that of having students write and publish critical “reviews” of scientific resources. Writing reviews can be a means of both practicing critical evaluation of Web resources, and of making an authentic value-added contribution to the Web. This paper presents content analyses of 41 source documents and 63 critical reviews published by 11th grade students in a project-based science class. The source documents are described as to their publishing source, use of organizational elements, and use of graphics. Two aspects of student-written critical evaluation are analyzed: evaluation of organization and evaluation of graphics. While evaluations of graphics were somewhat thin in these reviews, this was due mostly to the lack of good content representations in the source documents. The on-line review form did successfully prompt students to make conceptual connections between organizational structure and their own sense-making process. Reviews also sometimes showed students engaged in perspective-taking related to potential readers. The literature review of this paper examines critical evaluation models and current social filtering models for large, distributed databases. A section on future directions for Web reviews describes a review-publishing system developed for the University of Michigan Digital Library system.     

STUDENTS’ EVALUATIONS OF INSTRUCTION IN HIGHER EDUCATION: A REVIEW

This paper reviews the research work and reports on the subject of students’ evaluations of Instruction in Higher Education. The views of proponents and opponents on student ratings are reviewed. The several variables influencing students’ evaluation of instruction are identified. Recommendations and conclusions are based on the review.     

Analysis of Differences in Nonteaching Factors Influencing Student Evaluation of Teaching between Engineering and Business Classrooms

In recent years, there have been increasing calls from the government and other organizations to provide easy public access to student evaluations of teaching. Indeed, the increasing ease of displaying and viewing large quantities of information, and competition among universities and majors for students, makes it likely that an era of greater transparency of this type of information is at hand. While students’ evaluation of teaching (SET) is one quantitative metric that rates the instructor, it may be influenced by factors that are often beyond the instructor's control. In this study, we analyze a longitudinal data set from both engineering and business schools of a large public university, and identify factors that influence SET. We show which factors have the highest influence on overall SET scores, and contrast these between engineering and business colleges. Colleges within the same university may have differences in the factors affecting SET, and recognition of this is important in effectively and fairly evaluating SET scores. We also provide recommendations regarding information that should be displayed along with the SET, particularly when SET scores are made public, so that instructors are not unduly penalized when their evaluations can be influenced by factors over which they have no control.     

Sustainable development education in Scottish schools: the Sleeping Beauty syndrome

This paper reviews and discusses the development of Sustainable Development Education (SDE) policy within the context of the Scottish formal school system. The focus is on the progress, and lack thereof, of implementation of SDE in schools in the light of some of the key curriculum documents and associated political decisions and advisory reports. The period of the review dates from 1993, which saw the publication of a report that was regarded as the seminal document for the development of environmental education in Scotland, to 2007 and the Scottish Executive's proposals for SDE in the light of curriculum reform for schools for the 21st century. The paper employs, loosely, the metaphor of the Sleeping Beauty to tell the story of SDE in Scotland in three parts: the story's three phases of emergence, obscurity and re‐emergence might serve as a useful metaphor, here.     

New academics negotiating communities of practice: learning to swim with the big fish

Abstract

Student evaluations of teaching constitute a recurring theme for debate within the tertiary education literature particularly in relation to validity and reliability. My aim in this paper is to highlight the dependence of student evaluative capabilities on their own beliefs and assumptions about knowledge constituting ‘ways of knowing’, which will differ from one student to the next, and to advocate that this is taken into account when evaluations of teaching are analysed and interpreted. I begin by addressing the nature of the student evaluation process, its current perceived status in higher education and in relation to student expectations in the present consumerist culture from which universities are not exempt. I then go on to focus on cognitive development implicit in higher education, and consider its impact on student expectations and consequent evaluations. Finally, I suggest that promoting an understanding of changing and developing epistemic assumptions of students should be an essential component of the explicit curriculum to the benefit of both students and lecturing staff, and recommend the exploration of more qualitative approaches to student evaluation.     

A general framework for cultivating innovations in higher education curriculum

ABSTRACT

Systematic changes to higher education curriculum typically occur within the extended timeframes of formal curriculum review processes. Programmes need to be reviewed periodically for internal and external accountability or to determine whether the curriculum has lost its coherence due to the accumulative effect of continual small-scale changes. These programme reviews often lend themselves to the introduction of innovations in teaching and learning however experiences suggest that these innovations are often short-lived. Even with well-thought-through project plans, adequate funding and staffing, and robust project evaluations, a curriculum innovation may fail to take hold and continue beyond the short-term. In this article we work towards developing a general framework that identifies the various factors and drivers that are essential to sustain important curriculum innovation beyond the short-term. The framework is developed from an analysis of several curriculum innovations related to the embedding of graduate attributes to highlight the important factors necessary to ensure longevity in important developments in teaching and learning.     

Young people and alternative provision: Perspectives from participatory–collaborative evaluations in three UK local authorities

This article reports the findings of four separately commissioned evaluations of alternative provision (AP) undertaken in three local authorities in the UK. The evaluations were specifically predicated on the principles of children's rights and used a combination of qualitative research methods and documentary analysis to elicit the experiences of young people in conjunction with the viewpoints of key stakeholders. Data from each evaluation was gathered over a total period of 6 years. The sites and time scales for each evaluation varied from 6-month authority-wide strategic reviews to a 3-year evaluation of an AP free school and an evaluation of pupil referrals in a large school partnership. The evaluations involved 200 participant children and young people, 30 managers and stakeholders, 8 parents of non-attending pupils and local authority officers and school governors. The evaluations report the complexity of needs amongst children and young people; the continuing problem of unsuccessful transitions between key phases/stages of education and the profound consequences of this for young people; assumptions around mainstream reintegration and managed moves; and the curriculum challenges of vocationalism and academic emphasis. While the research data confirms the positive value of multi-agency approaches in AP, it also shows a more recent troubling increase in the number of young people now being referred to AP as a consequence of their exposure to performative school cultures.     

How do instructional designers evaluate? A qualitative study of evaluation in practice

This study employed a qualitative research design to investigate how instructional designers use evaluation in everyday design practice. While past research has examined how designers spend their time, how they generally make decisions, and expert-novice differences, little attention has been paid to use of context, input, process, or product evaluation, from the perspective of practicing designers. Based on interviews of practitioners, our findings included ten themes regarding how designers use evaluation to improve their products. While these results substantiate to some degree the claim that practitioners believe clients will not pay for formal evaluations, they also suggest that practitioners use evaluation in important but less formal ways. Other conclusions regarding the role of evaluation in design are provided and future directions for training and research are discussed.