Improving course evaluations to improve instruction and complex learning in higher education

Recent research has touted the benefits of learner-centered instruction, problem-based learning, and a focus on complex learning. Instructors often struggle to put these goals into practice as well as to measure the effectiveness of these new teaching strategies in terms of mastery of course objectives. Enter the course evaluation, often a standardized tool that yields little practical information for an instructor, but is nonetheless utilized in making high-level career decisions, such as tenure and monetary awards to faculty. The present researchers have developed a new instrument to measure teaching and learning quality (TALQ). In the current study of 464 students in 12 courses, if students agreed that their instructors used First Principles of Instruction and also agreed that they experienced academic learning time (ALT), then students were about 5 times more likely to achieve high levels of mastery of course objectives and 26 times less likely to achieve low levels of mastery, according to independent instructor assessments. TALQ can measure improvements in use of First Principles in teaching and course design. The feedback from this instrument can assist teachers who wish to implement the recommendation made by Kuh et al. (2007) that universities and colleges should focus their assessment efforts on factors that influence student success.     

Distance education for dentists: Improving the quality of online instruction

Abstract

This study describes the development and the evaluation of quality assurance criteria for online dental education. These criteria were developed by the authors and evaluated by oral health professionals through an Internet survey. The survey indicated that oral health professionals ranked the creation of scientifically based, frequently updated courses that clearly define their educational goals and objectives as highly important.     

Improving metacognition in the classroom through instruction,training, and feedback

Accurately judging one’s performance in the classroom can be challenging considering most students tend to be overconfident and overestimate their actual performance. The current work draws upon the metacognition and decision making literatures to examine improving metacognition in the classroom. Using historical data from several semesters of an upper-level undergraduate course (N?=?127), we analyzed students’ judgments of their performance and their actual performance for two exams. Students were instructed on the concepts of overconfidence, received feedback on exams, and were given incentives for accurate calibration. We found results consistent with the “unskilled and unaware” effect Kruger & Dunning (Journal of Personality and Social Psychology, 77(6), 1121–1134, 1999) where lower performing students initially displayed overconfidence and the highest performing students initially displayed underconfidence. Importantly, students were able to change both judgments and performance such that metacognitive accuracy improved significantly from the first to the second exam. In a second study, two additional semesters for the same course used in Study 1 were examined (N?=?90). For one of the semesters feedback was not provided, allowing us to determine whether feedback can improve both metacognitive judgments and performance. Our findings revealed significant improvements in performance paired with decreases in overconfidence on Exam 2, but only for students who received feedback about their performance and judgments. We postulate that feedback may be an important component in improvement metacognitive judgments.     

Comparison of online and classroom-based student evaluations of instruction

In recent years, colleges have been moving from traditional, classroom‐based student evaluations of instruction to online evaluations. Because of the importance of these evaluations in decisions regarding retention, promotion and tenure, instructors are justifiably concerned about how this trend might affect their ratings. We recruited faculty members who were teaching two or more sections of the same course in a single semester and assigned at least one section to receive online evaluations and the other section(s) to receive classroom evaluations. We hypothesised that the online evaluations would yield a lower response rate than the classroom administration. We also predicted that there would be no significant differences in the overall ratings, the number of written comments, and the valence (positive/neutral/negative) of students’ comments. A total of 32 instructors participated in the study over two semesters, providing evaluation data from 2057 students. As expected, online evaluations had a significantly lower response rate than classroom evaluations. Additionally, there were no differences in the mean ratings, the percentage of students who provided written comments or the proportion of comments in the three valence categories. Thus, even with the lower response rate for online evaluations, the two administration formats seemed to produce comparable data.     

Academic discipline and generalizability of student evaluations of instruction

Previous research on the generalizability of student ratings of instruction has raised questions about the effects of academic discipline and item types on the generalizability of these data for making relative decisions about instructors and about courses. In particular, although student evaluation data appear to provide a reasonable basis for making decisions about instructors when generalizing across courses and students, when course is the object of measurement, the data appear to be less generalizable. It was suggested in the literature that this may be due to the type of evaluation items used or it may be due to academic discipline differences in the type of courses selected for study. This study used Biglan's (1973a) model for classifying disciplines along the dimensions of paradigmatic/preparadigmatic (hard/soft) and pure/applied. A nested sampling procedure yielded two sample types: courses within teachers, in which individual instructors taught more than one course; and teachers within courses, in which individual courses were taught by more than one instructor. For each sample type, evaluation forms for twenty courses within each discipline classification were sought. The evaluation items for this study were classified as measuring six dimensions of instruction: organization, breadth of coverage, group interaction, enthusiasm, grading, and individual rapport. Generalizability and decision studies were conducted in which, for one sample, teacher was the object of measurement, and for the second sample, course was the object of measurement. Results indicated that reliable decisions about instructors could reasonably be made from all six of the evaluation dimensions; however, reliability for course decisions varied greatly with the evaluation dimension, being highest for breadth of coverage and lowest for grading. The same general pattern was noted for the paradigmatic disciplines and the preparadigmatic-applied disciplines but not for the preparadigmatic-pure disciplines. It is suggested that a single evaluation instrument may not be uniformly applicable to all discipline areas.     

Improving the quality of student ratings of instruction: A look at two strategies

The effects of rating scale format (behaviorally anchored vs. Likert) and rater training on leniency and halo in student ratings of instruction were investigated. The subjects (N=269) were students enrolled in required courses at a graduate theological seminary in the Southwest United States. A repeated measures design controlling for teacher and course was used. Findings indicated: (a) training was effective in reducing leniency and halo in ratings from both instruments; (b) trained raters exhibited less leniency on two rating dimensions when using behaviorally anchored rating scales (BARS's) than when using the Likert scale; and (c) trained raters exhibited less halo when using the Likert than when using the BARS. The findings demonstrate the importance of focusing efforts to improve quality of ratings on the students rather than on the format of the instrument.Presented at the Twenty-Eighth Annual Forum of the Association for Institutional Research, Phoenix, Ariz., May 1988.     

Do grades contaminate student evaluations of instruction?

The correlation between grades and student satisfaction has been interpreted as providing support for belief in a grading leniency bias hypothesis. That is, easy graders are assumed to receive better evaluations than hard gradersbecause they are easy graders. Howard and Maxwell have demonstrated that the relationship between grades and satisfaction might be viewed as an expected result of important causal relationships of other variables (student motivation and progress in the course) with satisfaction and grades, rather than simply evidence of contamination due to grading leniency. Eighty-three students in a research methodology course provided data at two points in a semester. Cross-lagged panel correlation analysis was employed to ascertain the direction of casuality in the relationship between student satisfaction and grades. The findings replicate the Howard and Maxwell path analytic results in finding no evidence that a grades-influencing-satisfaction interpretation is more likely than its opposite, namely, a satisfaction-causing-grades one. The weak relationship between grades and student satisfaction, and the study's inability to find evidence to impugn a satisfaction-causing-grades interpretation of that relationship, renders the grade contamination objection to student evaluations even less poignant.     

创建精品课程,提高教学质量

根据《饭店管理概论》精品课程建设的实践,提出从加强师资队伍建设,打造精品团队;整合教学资源,优化教学内容;改革教学方法与手段,提高教学效果;强化教学管理,提升教学质量等方面入手,打造精品课程,全方位提高教学质量。     

Improving mathematics instruction through lesson study: a theoretical model and North American case

This article presents a theoretical model of lesson study, an approach to instructional improvement that originated in Japan. The theoretical model includes four lesson study features (investigation, planning, research lesson, and reflection) and three pathways through which lesson study improves instruction: changes in teachers’ knowledge and beliefs; changes in professional community; and changes in teaching–learning resources. The model thus suggests that development of teachers’ knowledge and professional community (not just improved lesson plans) are instructional improvement mechanisms within lesson study. The theoretical model is used to examine the “auditable trail” of data from a North American lesson study case, yielding evidence that the lesson study work affected each of the three pathways. We argue that the case provides an “existence proof” of the potential effectiveness of lesson study outside Japan. Limitations of the case are discussed, including (1) the nature of data available from the “auditable trail” and (2) generalizability to other lesson study efforts.

Relationship with instructor and other variables influencing student evaluations of instruction

This study examined some of the factors which influence college students’ evaluations of their instructors. Data were collected from 557 students enrolled in a basic speech communication course. Overall, a student's relationship with the instructor was the best predictor of the student's evaluation of the instructor. This study lends partial support to the idea that students evaluate male and female instructors differently. The variables examined in this study were better predictors of student evaluation of female than male instructors.     

Measuring the mathematical quality of instruction

In this article, we describe a framework and instrument for measuring the mathematical quality of mathematics instruction. In describing this framework, we argue for the separation of the mathematical quality of instruction (MQI), such as the absence of mathematical errors and the presence of sound mathematical reasoning, from pedagogical method. We argue that conceptualizing this key aspect of mathematics classrooms will enable more clarity in mathematics educators’ research questions and will facilitate study of the mechanisms by which teacher knowledge shapes instruction and subsequent student learning. The instrument we have developed offers an important first step in demonstrating the viability of the construct.     

Improving teaching quality through training: Evidence from the Caucasus

Teacher training constitutes a promising policy area for improving education quality in developing countries. While there is mixed empirical evidence on whether teacher training improves student achievement, some interventions that follow a comprehensive approach on teacher training show positive impacts. In this study we analyze the effect of the Georgian Primary Education Project, an initiative that provided teacher training and other teacher support activities in 122 schools in the Republic of Georgia between 2013 and 2015. We use a quasi-experimental design, specifically a Value-Added model, to estimate the effect of the program on math and Georgian test scores of students that were in grades 1–4 in 2013. We find that the program increased math test scores by 0.27 standard deviations; these gains were observed mostly in students that were in grades 1–3 in 2013, while no effects are observed for students that were in grade 4 in 2013. For Georgian we analyze the results for students that speak Georgian as their native language and for students in minority schools that speak Georgian as a second language. For native speakers we find an average effect of 0.15 standard deviations, in this case, the observed gains are also concentrated in students that were in grades 1–3 in 2013, and no impact is observed on students that were in grade 4 in 2013. We do not find any significant effects on reading Georgian as a second language. This study contributes to the growing literature on comprehensive teacher training as a strategy to improve students’ achievement in developing countries.     

Improving the quality of primary science teaching through a pre-service course

This paper describes a study involving second and third year primary Diploma of Teaching students. It considers their discipline competence, their confidence in teaching particular topic areas and their views of the nature of science teaching in the classes in which they were placed during teaching rounds. It then describes attempts made within the science curriculum course to overcome particular problems. Specializations: Science teacher education, the practicum.     

Improving the quality of education through self-evaluation in Dutch secondary schools

In countries with a governance structure in which responsibility for the quality of education is shared between government and school boards, the past decades school self-evaluation has been stimulated as a way to encourage continuous quality improvement. However, working on the goals of quality assurance and school improvement at the same time is a challenge in general. To make a valuable contribution to both goals, the self-evaluation effort has to be of sufficient quality itself. In this article, we present a research-based framework for school self-evaluation (SSE) composed of both content and process factors that allows to evaluate the quality of self-evaluation in schools. We then used this model to evaluate the experiences in a comprehensive self-evaluation project that has been designed and used to help Dutch secondary schools promote the quality of student care. Our sample encompassed 79 Dutch secondary schools involved in this project. The findings show that the quality of SSE depends on the quality of the instruments (content) and process factors. However, to make a valuable contribution to school improvement and thereby the quality of education in The Netherlands more attention is needed for a balance between internal and external supervision and the role of school managers in the process of SSE.For future research more insight is needed in the challenges of meeting the content and process conditions of school self-evaluations, the governance and supervision issue at the level of schoolboards, the competence of change management in schools and the effects of SSE on the quality of education.