ASSESSING TEACHING EFFECTIVENESS: AN INDIAN PERSPECTIVE

>An investigation is reported which tests the applicability of two American instruments designed to assess tertiary students’ evaluations of teaching effectiveness with 111 Indian graduate students. The scales were found to have generally high internal consistency reliability coefficients, most of the items were seen to be appropriate, and every item was considered of importance by at least some of the students. In addition, all but the Workload/Difficulty items clearly differentiated between ‘good’ and ‘poor’ lecturers. Further analysis generally supported the convergent and discriminant validity of those scales hypothesised to measure similar or dissimilar components of effective teaching. However, this analysis did indicate more overlap between aspects of teaching skill and enthusiasm than evident in Western studies. Factor analysis confirmed this finding as a strong main factor of teaching effectiveness plus minor factors referring to specific aspects of teaching were obtained.     

Students' Evaluations of University Teaching: A China Perspective

This study investigates the applicability of two American instruments, designed to assess students' evaluations of teaching effectiveness, to university students in China. Based on the responses of 367 Chinese undergraduates, the results indicated that the scales had high internal consistency reliability coefficients and the items were generally considered appropriate and clearly distinguished between good and poor lecturers. Moreover, every item was deemed important by at least some of the students. Further analysis lent support to the convergent and discriminant validities of the scales hypothesised to measure similar or dissimilar components of teaching effectiveness. However, this analysis and factor analytic findings did indicate more overlap among aspects of Learning/Value, Organisation/Clarity, Enthusiasm, and Breadth of Coverage, than that found in Western studies. Hence, this study only provided partial support for the cross‐cultural validity of the two instruments and the multidimensional model of teaching effectiveness on which they were based.     

Student evaluations of teaching effectiveness: a Nigerian investigation

An investigation is reported which tests the applicability of two American instruments designed to assess tertiary students' evaluations of teaching effectiveness with 158 Nigerian undergraduates. The scales were found to have generally high internal consistency reliability coefficients, most of the items were seen to be appropriate, and every item was considered of importance by at least some of the students. In addition, all but the Workload/Difficulty items clearly differentiated between good and poor lecturers. Factor analysis found a strong main factor of teaching effectiveness plus a minor factor referring to course workload and difficulty. Further analysis generally supported the convergent and discriminant validity of those scales hypothesized to measure similar or dissimilar components of effective teaching. However, this analysis supported the factor analytic results as more overlap between aspects of teaching skill and enthusiasm was found than has been evident in Western studies. Thus there must be doubt about the cross-cultural validity of a multidimensional model of teaching effectiveness.     

Evaluating tertiary teaching: A New Zealand perspective

The purpose of this research was to test the applicability of two American instruments designed to assess tertiary students' evaluations of teaching effectiveness with New Zealand students. The scales were found to have high internal consistency reliability coefficients, most of the items were seen to be appropriate, and every item was considered of importance by at least some of the students. In addition, all but the Workload/Difficulty items clearly differentiated between “good,” “average,” and “poor” lectures. Further analyses generally supported both the factor structure identified in earlier research and the convergent and discriminate validity of the scales from both instruments. This research has provided strong support for the applicability of these American instruments for evaluating effective teaching at a New Zealand university.     

Measuring twenty-first century skills: development and validation of a scale for in-service and pre-service teachers

A self-report scale that measures teachers’ confidence in teaching students about twenty-first century skills was developed and validated with pre-service and in-service teachers. First, 16 items were created to measure teaching confidence in six areas: information literacy, collaboration, communication, innovation and creativity, problem solving, and responsible citizenship. Then, the items were tested in two groups of pre-service teachers and two groups of in-service teachers respectively. Exploratory factor analyses and confirmatory factor analyses were conducted to examine the scale’s validity and reliability. The analyses identified a three-construct scale including innovation and problem solving, collaboration, and utility of technology for pre-service teachers and a one-dimension scale, cross-functional skills, for in-service teachers. This study suggests that different scales should be used to measure in-service teachers’ and pre-service teachers’ confidence in teaching twenty-first century skills.     

ESTABLISHING THE VALIDITY OF A SET OF SUMMATIVE TEACHING PERFORMANCE SCALES

A policy at a New Zealand University to assess lecturers’ teaching performance as an administrative requirement provided for information to be obtained from the chairperson, colleagues, students and the individual being evaluated.

In designing the summative scales to obtain student feedback, four conditions were seen as important for achieving a system acceptable to both lecturers and students. These conditions—length of scale, time to complete the scale, face validity of the scale and the acceptability of the procedures used to distribute the scales, together with the context within which teaching takes place at the Institution, provided constraints to the design of the summative scales.

Following 98 lecturing and 43 course evaluations, the scales were subjected to a principal factor analysis. This analysis showed that the items most strongly related to overall teaching effectiveness, as assessed by students, included the lecturer's ability to communicate ideas and information and to stimulate interest in the subject.

By pooling student ratings from different teaching situations for any one lecture, a Teaching Performance Profile was prepared. This profile increased the reliability and validity of the information obtained. At the same time it provided a summary suitable for combining with the evaluative . information from the other three sources in making a judgement as to the individual's level of teaching competency.     

High school students’ feelings: Discoveries from a large national survey and an experience sampling study

We investigated students' feelings at high school in a nation-wide survey of 21,678 US students (study 1), and in a four-week study using experience sampling methodology (ESM) with 472 students across 5 high schools (study 2). Both studies combined mixed methods, including open-ended questions and rating scales (e.g., PANAS). In study 1, seventy-five percent of the feelings students reported in their responses to open-ended questions were negative. The three most frequently mentioned feelings were tired, stressed, and bored. Similar findings emerged with rated items, The prevalence of negative feelings was largely similar across demographic groups. Study 2 largely corroborated the findings from study 1. Although the retrospective measures showed similar results to study 1, the in-the-moment measures also showed frequent positive feelings. We discuss the findings in light of the ‘sleep deprivation epidemic’, the achievement motivation literature, and implications for the validity of state- and trait measures of academic emotions.     

Students' evaluations of university instructors: The applicability of American instruments in a Spanish setting

Items from two American instruments designed to measure students' evaluations of teaching effectiveness were translated into Spanish and administered to a sample of Spanish university students. Most of the items were judged by the students to be appropriate, every item was chosen by at least a few as being a most important item, and all but the Workload/Difficulty items clearly differentiated between lecturers whom students indicated to be “good,” “average,” and “poor.” A series of factor analyses clearly identified the factors that the instruments were designed to measure and that have been identified in previous research. Finally, a multitrait-multimethod analysis demonstrated that there was good agreement between factors from the two instruments which were hypothesized to measure the same components of effective teaching, and provided support for both the convergent and discriminant validity of the ratings. The findings illustrate the feasibility of evaluating effective teaching in a Spanish university and the appropriateness of the two American instruments.     

Measuring ocean literacy of high school students: psychometric properties of a Chinese version of the ocean literacy scale

This study established a Chinese scale for measuring high school students’ ocean literacy. This included testing its reliability, validity, and differential item functioning (DIF) with the aim of compensating for the lack of DIF tests focusing on current scales. The construct validity and reliability were verified and tested by analyzing the established scale’s items using the Rasch model, and a gender DIF test was conducted to ensure the test results’ fairness when distinct groups were compared simultaneously. The results indicated that the scale established in this study is unidimensional and possesses favorable internal consistency and construct validity. The gender DIF test results indicated that several items were difficult for either female or male students to correctly answer; however, the experts and scholars discussed these items individually and suggested retaining them. The final Chinese version of the ocean literacy scale developed here comprises 48 items that can reflect high school students’ understanding of ocean literacy—which helps students understand the topics of marine science encountered in real life.     

Validating the Course Experience Questionnaire in West Bengal higher secondary education

The Course Experience Questionnaire (CEQ) is a 36-item instrument that is intended to measure six different aspects of students’ perceptions of the academic quality of their programmes. It has been widely used in Western countries, and it has also been used in non-Western countries, including China, Hong Kong, Japan and Pakistan. Nevertheless, in the latter countries, it has sometimes not been possible to identify the full range of constructs that were supposed to be measured by the original CEQ. We translated the CEQ into Bengali and administered this to 552 science students at 15 higher secondary schools in West Bengal, India. A confirmatory factor analysis found that their responses provided a poor fit to the original six-factor model of the CEQ. An exploratory factor analysis identified just four constructs, which reflected good teaching, generic skills, student support and appropriate workload. The items with salient loadings on the four factors were used to construct four scales. The students’ scores on three of the four scales showed satisfactory levels of internal consistency. A factor analysis of their scores on all four scales yielded one overarching factor that could be interpreted as a measure of perceived academic quality. A reduced version of the CEQ consisting of the 30 items that constitute these four scales can be recommended as a measure of students’ perceptions of the academic quality of programmes in West Bengal.     

浅谈基于网络资源提高学生自主学习能力

目的:文章对目前的教学过程和学习效果进行了调查分析,考查网络资源与学生自主学习之间的联系。方法:通过调查法、德尔菲法选择36条目进行调查,揭示网络资源在学生自主学过程中的作用。结果:网络资源能有效地提高教学质量,但需要与教师传统教学有机结合。结论:通过考查学生利用网络资源自主学习情况,为更好提高教学水平、增强学生解决实际问题能力、培养信息素养和增强学习的有效性提供依据。     

What makes a good educator? The relevance of meta programmes

This paper reports the results of a qualitative study which explores the relevance of meta programmes to students' perceptions of teaching quality. Meta programmes are a model of personality preferences from the discipline of Neuro Linguistic Programming (NLP). Research into teaching effectiveness indicates that students rate as important ‘hygiene factors’ such as the teacher's ‘knowledge of the subject’ as well as less tangible qualities including ‘ability to communicate’ and ‘student–teacher rapport’. This study reveals the need for similar qualities but, in some cases, presence of these ‘hygiene factors’ was insufficient; students cited other factors related to personality. Some students were found to like the approach of certain teachers whilst other students rated the same teachers less favourably. When questioned as to why, factors emerged that relate to the distinct teaching style adopted by these teachers that appeared to suit particular students and not others. This study suggests that a teacher's meta programmes influence the approaches adopted in their teaching and these styles suit students with matching meta programme preferences. Where students have different meta programme preferences from the teacher, then, even where the ‘hygiene factors’ are met, this leaves the student dissatisfied. There is potential for teachers to adopt teaching approaches more appropriate for the meta programme profiles of their students. Also, on the part of the students, an increased awareness of their meta programme preferences offers the potential to improve their learning experience.     

Measuring the causes for the growth in special needs education. A validation of a questionnaire with possible factors explaining the growing demand for special provisions in education

In recent decades, the basic principle in Norway’s education policy has been that all students should receive adapted education within the framework of the mainstream schools. This policy notwithstanding, the rate of students for whom special support was initiated, rose from 6.2% in 2006 to 8.0% in 2014. The study reported here developed measures for the possible causes for this unwelcome development. A questionnaire with 23 statements suggesting possible causes of the increase in the use of special support was presented to 136 teachers in primary and lower secondary school. A factor analysis revealed that 12 items formed three reliable scales for possible causes behind the high demand for special provisions: ‘Focus on students’ problems’, ‘External pressure on schools’ and ‘Early support’. The teachers stated that these causes had moderate impact. The study has resulted in three scales useful in measuring three causes.     

In search of the feeling of ‘belonging’ in higher education: undergraduate students transition into higher education

ABSTRACT

Previous research that evaluated first year students’ transition into university found that the values of ‘being, belonging and becoming’ were important in particular within the first few months and within the first year of university. From our previous work, we reported that three things matter to students: the academic staff they work with, the nature of their academic study and the feeling of belonging. This paper provides a further illumination to our work by reporting on the qualitative data collected in the same study. The study included 530 students from five cohorts over a five-year period. As part of the Student Experience Evaluation instrument, open-ended questions probed students about their early experiences of belonging and transition into university. This original research uses rich data to illuminate the scales and items from previous quantitative data analysis to explore ‘belonging’, triangulated with research from the field. This paper is timely due to increased emphasis placed on learning and teaching with the introduction of the Teaching Excellence Framework. Student satisfaction is not a simplistic measure and this study articulates the complexity of student belonging in Higher Education.     

Development and validation of a questionnaire for teacher effective communication in Qur’an learning

ABSTRACT

A student’s perception of teacher effective communication influences the learning atmosphere. The measurement of perception indicates how students view the quality of learning. As few studies have explored the development of an appropriate measurement tool of students’ perception of learning to read the Qur’an, this study aims to develop such a tool. Conceptual analysis and a survey using open-ended questions resulted in dimensions and items. Six experts and three instructors evaluated the content validity. The questionnaire was then administered to 421 participants – 201 for Exploratory Factor Analysis (EFA) and 220 for Confirmatory Factor Analysis (CFA). The results successfully identified four dimensions that explained 64.6% of the variation. The instrument consisted of 13 items with satisfactory reliability and validity. The dimension with the highest coefficient path was ‘understanding and friendliness’ (UF), while the dimension of ‘learning media’ (LM) was the lowest. There was a significant influence of gender on the dimension of UF, as well as verbal (V) and non-verbal (NV) communication, while there were no differences in the dimension of LM. The questionnaire can be effectively used as a measurement device of dimensions related to students’ perception of teacher effective communication in Qur’an learning.     

Dutch beginning teachers’ intrinsic orientation for the profession: Measurement and consistency during the first year

Compared to the disseminated studies on teacher motivation and positive emotion as separate dimensions, the knowledge base about teachers' intrinsic orientation to the profession (TIOP) as a compound trait is relatively underrepresented in the literature. Particularly, little is known regarding TIOP at the start of teaching trajectories. The present study sought to develop a reliable and valid TIOP scale based on self-determination theory and teaching effectiveness perspectives. By combining three existing subscales, the compound scale of TIOP was designed to measure the hypothesized value-related cognitive (i.e., the value and meaningfulness added to the profession) and the feeling-related affective (i.e., the positive affection experienced from the teaching activity and the subject) dimensions of TIOP. Furthermore, this study aimed to provide empirical evidence concerning the predictive validity and trait consistency of TIOP. The study included 271 Dutch beginning teachers in secondary education at the beginning and the end of their first career year, by means of self-report scales and classroom observation. The analysis results confirm that TIOP can be reliably and validly measured in terms of the three subdimensions of autonomous motivation, enthusiasm for teaching and for the subject. Additionally, TIOP was found to be a consistent teacher quality over the first year of teaching profession and can validly predict teachers’ self-efficacy. However, the results also reveal a possible discrepancy between ‘an enthusiastic and motivated teacher’ and ‘a teacher with effective teaching skills’. In sum, TIOP can provide a relatively explicit and quantified picture of teachers’ global psychological processes and functioning on a long-term basis.     

高校学生评教的有效性问题及优化策略

在高等教育大众化的背景下,如何促进教学质量的整体提高,已成为社会各界普遍关注的重要问题。学生评教是高校教学质量监控的重要环节,对教学质量的提高起到积极的促进作用。但是,学生评教指标体系尚存不合理,学生对评教意义认识不足,受情感因素影响较大,学生评教结果的反馈不及时等问题影响着学生评教的有效性。因此,完善学生评教指标体系,提高学生评教参与率,合理统计和客观分析学生评教结果,及时将评教结果反馈给师生,是提高学生评教有效性的重要措施。     

The role of engagement and academic behavioral skills on young students’ academic performance—A validation across four countries

The aim of this study is to validate an instrument measuring students’ academic behavioral skills and engagement—skills identified as vital for student achievement. We inspect the reliability and validity of the survey with respect to item fit, factorial structure, relations with academic performance, and the fairness of the items across student groups. The fairness analyses are critical to making valid comparisons between groups and across countries. Data comprising 8520 grade 10 students from four countries were analysed using item response theory. We found that both scales were multidimensional, acted fairly across students’ gender, country, immigrant-, and socio-economic background (after removing four items), and were positively and significantly correlated with self-reported and performance-based academic performance.     

The Development of an Instrument to Measure the Project Competences of College Students in Online Project-Based Learning

This study sought to develop a self-report instrument to be used in the assessment of the project competences of college students engaged in online project-based learning. Three scales of the KIPSSE instrument developed for this study, namely, the knowledge integration, project skills, and self-efficacy scales, were based on related theories and the analysis results of three project advisor interviews. Those items of knowledge integration and project skill scales focused on the integration of different disciplines and technological skills separately. Two samples of data were collected from information technology-related courses taught with an online project-based learning strategy over different semesters at a college in southern Taiwan. The validity and reliability of the KIPSSE instrument were confirmed through item analysis and confirmatory factor analysis using structural equation modeling of two samples of students’ online response sets separately. The Cronbach’s alpha reliability coefficient for the entire instrument was 0.931; for each scale, the alpha ranged from 0.832 to 0.907. There was also a significant correlation (r = 0.55, p < 0.01) between the KIPSSE instrument results and the students’ product evaluation scores. The findings of this study confirmed the validity and reliability of the KIPSSE instrument. The confirmation process and related implications are also discussed.     

Students' perceptions of clinical attachments across rural and metropolitan settings

For many years, medical students at the University of Sydney undertook their clinical clerkships in traditional metropolitan teaching hospitals, which were regarded as the ‘gold standard’ for clinical training. In 2001 the university established a rural clinical school at which increasing numbers of students now complete a significant proportion of their medical education. The aim of the study reported here is to examine students’ perceptions of what facilitates their learning in clinical settings and to compare their perceptions across rural and metropolitan settings. Focus groups were conducted to collect students’ views on their experiences of learning in clinical settings. The findings were used to generate a questionnaire with items designed directly from focus‐group data ensuring content validity. The questionnaire was sent to all students in the 2004 cohort. Exploratory factor analysis was used to provide evidence of construct validity. The internal consistency reliability of the questionnaire was assessed using Cronbach’s alpha. Factor scores were computed to compare students’ perceptions across the two settings. Four factors were extracted: (1) clinical teachers’ orientation to teaching; (2) opportunities to develop clinical skills; (3) supportiveness of the clinical setting; and (4) student confidence and sense of self‐efficacy. Students rated the rural experience more highly and positively than the metropolitan hospital experience with regard to all four factors. This study highlights the positive role that rural attachments can play in providing an educationally sound learning experience. The findings are important in the context of both the drive, among medical programs worldwide, to seek out additional and alternative settings for clinical education and the national agenda to foster student interest in rural careers through positive rural training experiences.