Student evaluation of instruction in higher education: exploring issues of validity and reliability

Many personnel committees at colleges and universities in the USA use student evaluation of faculty instruction to make decisions regarding tenure, promotion, merit pay or faculty professional development. This study examines the construct validity and internal consistency reliability of the student evaluation of instruction (SEI) used at a large mid‐western university in the USA for both administrative and instructional purposes. The sample consisted of 73,500 completed SEIs for undergraduate students who self‐reported as freshman, sophomore, junior or senior. Confirmatory factor analysis via structural equation modelling was used to explore the construct validity of the SEI instrument. The internal consistency of students' ratings was reported to provide reliability evidence. The results of this study showed that the model fits the data for the sample. The significance of this study as well as areas for further research are discussed.     

Contextualizing instruction: Leveraging students' prior knowledge and experiences to foster understanding of middle school science

Contextualizing science instruction involves utilizing students' prior knowledge and everyday experiences as a catalyst for understanding challenging science concepts. This study of two middle school science classrooms examined how students utilized the contextualizing aspects of project‐based instruction and its relationship to their science learning. Observations of focus students' participation during instruction were described in terms of a contextualizing score for their use of the project features to support their learning. Pre/posttests were administered and students' final artifacts were collected and evaluated. The results of these assessments were compared with students' contextualizing scores, demonstrating a strong positive correlation between them. These findings provide evidence to support claims of contextualizing instruction as a means to facilitate student learning, and point toward future consideration of this instructional method in broader research studies and the design of science learning environments. © 2007 Wiley Periodicals, Inc. J Res Sci Teach 45: 79–100, 2008     

Students' conceptions of natural selection and its role in evolution: Cases of replication and comparison

The work of Bishop and Anderson (1990) plays a major role in educators' understanding of evolution education. Their findings remind us that the majority of university students do not understand the process of evolution but that conceptual change instruction can be moderately effective in promoting the construction of a scientific understanding. The present article details two studies that represent an effort to focus on and define the limits of the Bishop and Anderson (1990) study. Study A describes a close replication of the work of Bishop and Anderson (1990) using the same conceptual-change teaching module to teach a unit on evolution to students enrolled in a biology course for nonmajors. Study B, a case of comparison, used the same evaluation instrument used in Bishop and Anderson (1990) and Study A, but high school students were the participants and the instruction was based on the inquiry approach to science. Like Bishop and Anderson (1990), Study A showed that the amount of prior instruction and students' beliefs in evolution were not found to be large factors in students' use of scientific conceptions. Unlike the original study, the students in Study A showed only a meager increase in their use of scientific conceptions for evolution. In Study B, students in the experimental group showed significant increases in their use of scientific conceptions. These findings suggest a need to investigate more closely the teachers' theories of learning, their reliance on instructional conversations, and the amount of time devoted to the topic of evolution as we study conceptual change in this area.     

Scaffolding Middle School Students’ Construction of Scientific Explanations: Comparing a cognitive versus a metacognitive evaluation approach

This study investigated the effects of scaffolds as cognitive prompts and as metacognitive evaluation on seventh-grade students' growth of content knowledge and construction of scientific explanations in five inquiry-based biology activities. Students' scores on multiple-choice pretest and posttest and worksheets for five inquiry-based activities were analyzed. The results show that the students' content knowledge in all conditions significantly increased from the pretest to posttest. Incorporating cognitive prompts with the explanation scaffolds better facilitated knowledge integration and resulted in greater learning gains of content knowledge and better quality evidence and reasoning. The metacognitive evaluation instruction improved all explanation components, especially claims and reasoning. This metacognitive approach also significantly reduced students' over- or underestimation during peer-evaluation by refining their internal standards for the quality of scientific explanations. The ability to accurately evaluate the quality of explanations was strongly associated with better performance on explanation construction. The cognitive prompts and metacognitive evaluation instruction address different aspects of the challenges faced by the students, and show different effects on the enhancement of content knowledge and the quality of scientific explanations. Future directions and suggestions are provided for improving the design of the scaffolds to facilitate the construction of scientific explanations.     

Assessment and Modification of Flexibility of Cognitive Structures Created in University Courses

Students' organization of the knowledge that they acquire is an important factor in determining the degree to which it is retained and used. In the past we have used the “fill-in-the-structure” (FITS) task as a direct method of inferring students' cognitive structures of course content (Naveh-Benjamin, Lin & McKeachie, 1995). This study goes one step further by using the FITS task to assess the flexibility of students' cognitive structures of the material learned; that is, whether students are able to relate the same concepts in different ways when the concepts are embedded in two different conceptual frameworks. We assessed the flexibility of students' cognitive structures in three studies by asking students in an ecology course to complete two different structures, each based on a different major dimension in the course. Results of the first study showed that the FITS technique could be used to assess students' ability to use concepts learned in the course appropriately in two different frameworks. The flexibility measures obtained were positively related to academic performance. The second study demonstrated the usefulness of the technique in measuring the development of conceptual flexibility during the course. Finally, the third study employed the technique to show that students' flexible use of concepts can be enhanced by appropriate instruction.     

Development of an Evaluation Model for an Introductory Module on Social Medicine

Evaluation of educational programmes is an essential component of the teaching-learning process. Therefore, efficient and effective ways of evaluating programmes or their components need to be identified. In this paper a focus group approach to evaluation is compared with a written format in relation to the time, quality and nature of feedback obtained from participants. The focus group required significantly less student and faculty time than the written evaluation approach. The focus group also yielded specific and therefore useful information in terms of module development compared with the written evaluations even though the overall impression of the students' experiences of the module as reflected in the written evaluations were more positive.     

Role of reading engagement in mediating effects of reading comprehension instruction on reading outcomes

The engagement model of reading development suggests that instruction improves students' reading comprehension to the extent that it increases students' engagement processes in reading. We compared how Concept‐Oriented Reading Instruction (CORI) (support for cognitive and motivational processes in reading), strategy instruction (support for cognitive strategies in reading), and traditional instruction in fourth‐grade classrooms differentially influenced students' reading comprehension, strategy use, and engagement in reading. Students experiencing CORI were significantly higher than both comparison groups on reading comprehension, reading strategies, and reading engagement. When students' level of reading engagement was statistically controlled, the differences between the treatment groups were not significant. We infer that the level of students' reading engagement during classroom work mediated the instructional effects on reading outcomes. © 2008 Wiley Periodicals, Inc.     

Classical Chinese reading instruction: Current practices and their relationship with students' strategy use and reading motivation

The study examined current practices in Classical Chinese (CC) reading instruction in Hong Kong and the relationship between different instructional practices and students' strategy use and motivation in CC reading. A total of 519 secondary students voluntarily responded to a questionnaire that measured their perception of CC reading instruction, strategy use, and motivation. The findings indicate while teachers frequently teach both the language and content aspects of CC reading, the teacher-centered approach they are now adopting is ineffective in facilitating students' CC reading development. Relations between different instructional approaches and students’ strategy use and motivation in CC reading are discussed.     

Reading motivation,perceptions of reading instruction and reading amount: a comparison of junior and senior secondary students in Hong Kong

This study examined the relations between students' reading motivation, perceptions of reading instruction and reading amount, together with grade differences, in a Chinese educational context. A total of 1,146 students from 19 secondary schools in Hong Kong voluntarily responded to a questionnaire that measured these three sets of variables. The study's findings indicated that students' intrinsic motivation was most strongly related to their reading amount. Students' perceptions of the reading instruction they received in their Chinese language class were significantly related to their reading motivation, but were only indirectly related to their reading amount, being mediated through reading motivation. Consistent with previous studies, significant grade differences were found in all types of reading motivation, students' perceptions of reading instruction and students' reading amount. The findings indicated that junior secondary students had better self‐efficacy, intrinsic motivation, extrinsic motivation and social motivation than senior secondary students. The largest grade difference was in students' self‐efficacy. Junior secondary students also perceived the reading instruction in their Chinese language class as more mastery‐oriented and read more frequently than senior secondary students. The implications of these findings for understanding Chinese students' reading motivation and for planning effective reading instruction to enhance their motivation are discussed.     

Assessing student exposure to and use of computer technologies through an examination of course syllabi

A syllabus analysis instrument was developed to assist program evaluators, administrators and faculty in the identification of skills that students use as they complete their college coursework. While this instrument can be tailored for use with a variety of learning domains, we used it to assess students' use of and exposure to computer technology skills. The reliability and validity of the instrument was examined through an analysis of 88 syllabi from courses within the teacher education program and the core curriculum at a private Midwest US university. Results indicate that the instrument has good inter‐rater reliability and ratings by and interviews with faculty and students provide evidence of construct validity. The use and limitations of the instrument in educational program evaluation are discussed.     

Lower track science students' argumentation and open inquiry instruction

The purpose of this study was to examine the effects of open inquiry instruction with low achieving, marginalized high school students. Students with long histories of scholastic failure were asked to participate in question generation, experimental design, and argument construction as a part of their General Science course instruction. Videotapes were collected from daily science instruction, and entrance and exit instruction interviews were conducted using identical open‐ended problems. From this dataset, comparisons were made between students' entrance and exit interview responses representing change over time. Shifts in student responses coincided with renegotiated classroom norms for scientific discourse. Results are reported for five students in the form of assertions. Students' arguments were observed to shift toward those more consistent with the nature of the scientific arguments including: (1) students' tentativeness of knowledge claims, (2) students' use of evidence, and (3) students' views regarding the source of scientific authority. Implications are discussed for research and practice in light of the national standards' call for universal scientific literacy. © 2000 John Wiley & Sons, Inc. J Res Sci Teach 37: 807–838, 2000     

Student evaluation of teaching quality in higher education: development of an instrument based on 10 Likert‐scales

Students' evaluation of teaching skills has been an important yet controversial tool in the improvement of teaching quality during the last few decades. When searching for an apt student questionnaire to measure instructional skills, it appeared that most existing questionnaires the authors were able to collect are based on a single‐item type of evaluation. Additionally, most of these instruments lack a theoretical foundation and hardly any instrument was tested with modern tests for reliability and validity. The authors managed to create a 31‐item instrument which comprises 10 Likert scales and is based on both the educational theory and empirical data. In this article, they present the different steps in constructing the instrument and discuss its reliability and validity. The results of this study underline the value of the use of a scaling technique in students' evaluation of teacher performance.     

University Students’ Understanding of Electromagnetic Induction

This study examined engineering and physical science students' understanding of the electromagnetic induction (EMI) phenomena. It is assumed that significant knowledge of the EMI theory is a basic prerequisite when students have to think about electromagnetic phenomena. To analyse students' conceptions, we have taken into account the fact that individuals build mental representations to help them understand how a physical system works. Individuals use these representations to explain reality, depending on the context and the contents involved. Therefore, we have designed a questionnaire with an emphasis on explanations and an interview, so as to analyse students' reasoning. We found that most of the students failed to distinguish between macroscopic levels described in terms of fields and microscopic levels described in terms of the actions of fields. It is concluded that although the questionnaire and interviews involved a limited range of phenomena, the identified explanations fall into three main categories that can provide information for curriculum development by identifying the strengths and weaknesses of students' conceptions.     

The Effect of Functional Flow Diagrams on Apprentice Aircraft Mechanics' Technical System Understanding

Cognitive theory suggests that a key to expert performance lies in the internal organization of the expert's knowledge. The authors contend that the type of technical illustration used during instruction influences knowledge organization and greatly impacts students' understanding of the content. This paper describes an experimental study that tested the impact of one type of conceptual illustration on students' understanding of the structure, function, and behavior of complex technical systems. The results show that supplementing traditional technical instruction with functional flow diagrams can improve overall system understanding. The functional flow diagrams were also found to be an effective instructional aid for enhancing students' conceptual understanding of the causal behavior of systems. In addition, the use of the functional flow diagram was found to significantly improve the subjects' ability to construct conceptual models that were similar to those of an expert. The implications of using conceptual diagrams for technical instruction are discussed and recommendations for future research in this area are provided.     

An investigation of Zimbabwe high school chemistry students' laboratory work–based images of the nature of science

This study investigates the proximal and distal images of the nature of science (NOS) that A‐level students develop from their participation in chemistry laboratory work. We also explored the nature of the interactions among the students' proximal and distal images of the NOS and students' participation in laboratory work. Students' views of the NOS and the nature of their chemistry laboratory work were elicited through students' responses to an open‐ended questionnaire and semistructured interviews. The results suggest that students build some understandings of the NOS from their participation in laboratory work. Students' proximal NOS understandings appear to build into and interact with their understandings of the nature and practice of professional science. This interaction appears to be mediated by the nature of instruction. It is posited that each student's conceptual ecological system is replete with interactions, which govern attenuation of proximal understandings into distal images. Methodologically, the study illustrates how students' laboratory work–based proximal and distal images of the NOS can be identified and extracted through analyzing and interpreting their responses to protocols. Implications for A‐level Chemistry instruction and curriculum development are raised. © 2006 Wiley Periodicals, Inc. J Res Sci Teach 43: 127–149, 2006     

The organization of conceptual materials: A methodology for measuring ideal and actual cognitive structures

The teaching-learning process is described in terms of a simple communication model with four components: the sender, the encoding structure, the decoding structure, and the destination. Within this context a method is proposed and illustrated for evaluating a student's understanding of the system of concepts underlying the topic areas in an elementary statistics and measurement course. The method, multidimensional scaling, involves determining the students' and instructor's cognitive maps for the various topics. A cognitive map represents a hypothetical cognitive structure of a student or instructor which characterizes his perceived organization of the concepts in a topic area. These individual cognitive maps were compared to the optimal organization or formal structure, as a basis for assessing the students' under-standing of the material. The present research was exploratory but demonstrated that:

1. With the help of experts it is possible to define the formal structure for the concepts in a topic area.
2. It is possible to measure, in a classroom setting, the cognitive maps that both the students and instructors have for a topic area.
3. By comparing individual students' cognitive maps to the instructor's cognitive map or to the formal structure, the student's understanding of the system of concepts defining the topic area can be evaluated.

     

Exploring high school students' use of theory and evidence in an everyday context: the role of scientific thinking in environmental science decision‐making

This study examined 10th‐grade students' use of theory and evidence in evaluating a socio‐scientific issue: the use of underground water, after students had received a Science, Technology and Society‐oriented instruction. Forty‐five male and 45 female students from two intact, single‐sex, classes participated in this study. A flow‐map method was used to assess the participants' conceptual knowledge. The reasoning mode was assessed using a questionnaire with open‐ended questions. Results showed that, although some weak to moderate associations were found between conceptual organization in memory and reasoning modes, the students' ability to incorporate theory and evidence was in general inadequate. It was also found that students' reasoning modes were consistent with their epistemological perspectives. Moreover, male and female students appear to have different reasoning approaches.     

The nature of advanced reasoning and science instruction

Although the development of reasoning is recognized as an important goal of science instruction, its nature remains somewhat of a mystery. This article discusses two key questions: Does formal thought constitute a structured whole? And what role does propositional logic play in advanced reasoning? Aspects of a model of advanced reasoning are presented in which hypothesis generation and testing are viewed as central processes in intellectual development. It is argued that a number of important advanced reasoning schemata are linked by these processes and should be made a part of science instruction designed to improve students' reasoning abilities. Concerning students' development and use of formal reasoning, Linn (1982) calls for research into practical issues such as the roles of task-specific knowledge and individual differences in performance, roles not emphasized by Piaget in his theory and research. From a science teacher's point of view, this is good advice. Accordingly, this article will expand upon some of the issues raised by Linn in a discussion of the nature of advanced reasoning which attempts to reconcile the apparent contradiction between students' differential use of advanced reasoning schemata in varying contexts with the notion of a general stage of formal thought. Two key questions will be discussed: Does formal thought constitute a structured whole? And what role does propositional logic play in advanced reasoning? The underlying assumption of the present discussion is that, among other things, science instruction should concern itself with the improvement of students' reasoning abilities (cf. Arons, 1976; Arons & Karplus, 1976; Bady, 1979; Bauman, 1976; Educational Policies Commission, 1966; Herron, 1978; Karplus, 1979; Kohlberg & Mayer, 1972; Moshman & Thompson, 1981; Lawson, 1979; Levine & linn, 1977; Pallrand, 1977; Renner & Lawson, 1973; Sayre & Ball, 1975; Schneider & Renner, 1980; Wollman, 1978). The questions are of interest because to date they lack clear answers, yet clear answers are necessary if we hope to design effective instruction in reasoning.     

Teachers' Professional Noticing

Teachers' instructional practice is pedagogically vulnerable when not anchored within detailed and comprehensive observation and understanding of students' immediate responses during instruction. Professional noticing ability, then, is central to, and integrated within, teachers' ability to provide instruction appropriately adapted to students' immediate needs. This article explicates a model of expert noticing for literacy instruction and relates this model to the characteristics of microadaptive teaching decisions. Characteristics, for example, that distinguish expert from less expert noticing during literacy instruction include detailed hypothesizing, breadth and depth of elaboration, observation of learners' metacognitive behavior, and identification of pivotal events. Teacher noticing ability is generally not related to years of experience, but can be learned through participation in sustained, multilevel professional development with an explicit focus on noticing ability. This complex learning integrates teachers' pedagogical and content knowledge with observation of student behaviors, and supports teachers' learning from practice.     

高校思政理论课课堂教学质量学生评价问卷编制

加强和改进高校思政理论课建设,需要对包括课堂教学在内的教学质量进行科学评价。学生是课堂教学质量评价的主体之一。在借鉴国外成熟的课堂教学学生评价量表SEEQ问卷基础上,本研究遵循大胆借鉴,为我所用、合理改造,凸显特征、充分验证,确保科学的基本原则,针对高校思政理论课的特点,对项目内容进行了调整,重新对问卷的因素进行分析并对因素顺序进行了适当调整,编制高校思政理论课教师课堂教学质量学生评价问卷(ISEEQ问卷)。选取安徽某高校13个专业1079名一年级学生作为被测,使用SPSS18.0软件对数据进行分析,对ISEEQ问卷进行可靠性评价,结果显示,问卷具有良好的的维度结构和信度效度。