From program evaluation to intstructional information system

This article describes the emergence of a new and exciting possibility for upgrading teaching and learning in American schools. We call this new phenomenon, instructional information systems (IIS). Instructional information systems are computerized data banks that allow a variety of users to ask important evaluative questions about student learning, classroom function, school management, and district policies. They built on current practice in testing and program evaluation but go way beyond it.     

Small-group problem-based learning as a complex adaptive system

Small-group problem-based learning (PBL) is widely embraced as a method of study in health professions schools and at many different levels of education. Complexity science provides a different lens with which to view and understand the application of this method. It presents new concepts and vocabulary that may be unfamiliar to practitioners of small-group PBL and other educational methods. This article looks at small-group PBL from the perspective of complex adaptive systems (CAS). It begins with a brief review of the current understanding and practice of PBL. Next some of the characteristics of CAS are reviewed using examples from small-group PBL to illustrate how these characteristics are expressed in that context. The principles and the educational theory in which small-group PBL are embedded are related to CAS. Implications for health professions education are discussed.     

Striking a balance: A mixed-methods approach to impact-oriented evaluation

It is commonsensical to assume that a program needs to be implemented in order to have an impact. However, most impact evaluation studies tend to pay little attention to the issue of program implementation. This paper argues for the logical priority of program implementation through a mixed-methods evaluation of a language intervention program initiated in one of the largest urban school districts in the United States. By selecting a mixed-methods evaluation approach, the evaluation yielded results that have implications for addressing implementation issues in an impact-oriented evaluation. At the local district level, the evaluation results of what works, what does not, and how the implementation varies across teachers and classrooms have a direct implication for program staff. They can use this information to monitor the program implementation and make necessary adjustments to enhance the effectiveness of the program so that the ultimate program goal can be accomplished.     

GLOBE in the Czech Republic: a program evaluation

The article presents results of the evaluation of the GLOBE program (Global Learning and Observations to Benefit the Environment) in the Czech Republic. The evaluation explores the implementation of the program in schools and its impact on research skills. Four hundred and sixty six pupils, aged 13, from 28 different schools participated in the evaluation. The evaluation revealed problems with the implementation of the program in schools. The majority of pupils usually collect data and work with worksheets in the program. The other activities, such as the analysis and comparison of collected data or planning GLOBE activities, are done only occasionally or never. Although one of the main goals of the program is to develop pupils' research skills, the program had no measurable effect on this outcome. According to these results, changes in the way the program is implemented are needed. Recommendations for further program development and evaluation are discussed.     

Teachers' voices in program evaluation and improvement: A case study of a TESOL program

In spite of the many ways that schools have changed to accommodate students not yet proficient in English, obstacles still exist that limit students' opportunities to receive adequate instruction. This paper describes the role of evaluation in developing a program to prepare Teachers of English to Speakers of Other Languages (TESOL). The teachers' voices presented in this paper suggest the importance of considering their contributions in refining and improving teacher preparation programs. Implications for on-going program evaluation and improvement are discussed.     

Evaluating the olympic education program: A qualitative approach

This study was conducted to provide further information regarding factors associated with educational evaluation, using a qualitative approach. Fifty-five interviews were carried out with school principals. The results of the present study supported the notion that the qualitative methods could provide information that quantitative procedures could not. Additionally, the findings also provided useful information for solving problems in OEP implementation and enhanced decision-making by indicating points of the program that should be improved.     

A visual-based approach to the mapping of generic skills: its application to a Marketing degree

With increasing complexity in the world, universities continue to face pressure to demonstrate that their graduates have acquired skills beyond discipline-based knowledge. These are generic skills like critical thinking, intellectual curiosity, problem-solving and so forth. In order to demonstrate this, universities have to show how their teaching contributes to the fostering of these skills. This can be a challenge for many reasons. Our mapping approach overcomes most of the obstacles by developing multiple indicators for each generic skill, making it easy to involve the lecturers-in-charge, summarising all their evaluations into important dimensions and, finally, representing the data in a three-dimensional visual map. This allows all the marketing subjects to be simultaneously evaluated in their ability to foster different generic skills, generating useful insights for effective curriculum development in a Marketing program. We illustrate this methodology using eight generic skills and 18 marketing subjects from a major Australian university.     

SCIS-II and the elementary teacher: A program evaluation

A two-part opinionaire was used to identify teachers' perceptions of the SCIS-II elementary science program. Of particular interest were perceived barriers to effective implementation of the program. The first part of the opinionaire was administered to 685 individuals, or 96.8% of the elementary teachers in a metropolitan school district. The second part was completed by a group of teachers at each grade level in each of the 31 elementary schools, with completion rates ranging from 77%–97% for the 13 SCIS-II units. Results indicate that grades K-2 teachers rated the program more positively than grades 3–6 teachers; teachers who rated the program highly perceived fewer barriers to effectively utilizing the program; and teachers rated the program higher for developing social skills and positive attitudes than for teaching science content and processes. Data on problems with individual SCIS-II units indicated that the viability and performance of the living organisms are problematic in all life science units. Further, the small number of insightful evaluative comments received for the last one-third to one-half of some units suggests that a significant portion of teachers are quitting the units before reaching the end.     

Responding to student expectations: a partnership approach to course evaluation

This paper presents a study that used newly developed instruments to investigate student expectations at two points in a semester‐based course. Course convenors were able to make some changes in response to the feedback, explain to students how the feedback had informed the course processes, and work with tutors on delivery enhancements. The administration of these instruments, a partnership approach involving students, tutors and lecturers, contributed to the creation of a responsive teaching and learning environment. In addition to enabling real‐time changes to be made, evaluation of the total teaching and learning experience ensured that improvements were made to subsequent offerings of the same course. Also, insights from one course were evaluated in terms of applicability to other courses offered within the same academic discipline. The study finds that establishing student expectations provides a framework for developing a responsive and iterative approach to achieving excellence in teaching and learning.     

Solving complex problems: A convergent approach to cognitive load measurement

The study challenged the current practices in cognitive load measurement involving complex problem solving by manipulating the presence of pictures in multiple rule‐based problem‐solving situations and examining the cognitive load resulting from both off‐line and online measures associated with complex problem solving. Forty‐eight participants were recruited from a subject pool in the Educational Psychology department in a large research university in western USA. Results showed that the presence of pictures had no effect on learners' complex problem solving as measured by the response time and accuracy. However, the online measure (eg, pupillometric measures) revealed a change in cognitive load associated with the presence of pictures in complex problem solving. The authors concluded that different measures of cognitive load may actually be measuring separate aspects of cognitive load. Discussions were made on how research on multimedia learning and cognitive load could be advanced by carefully considering multiple aspects of cognitive load and by including the use of convergent measurement techniques to capture the variations of cognitive load involved in learning.     

A psychometric approach to the validation of a student evaluation of teaching instrument

The purpose of this study was to conduct a validation analysis of an SET and provide a validation framework of SETs that can be included when designing complete evaluations of teaching within higher education institutions. A series of Rasch analyses was conducted on the results of the SET, examining the responses of students within a college and three departments. Results show the majority of items were moderately difficult to endorse in the college and departments, there were issues with DIF, and two items did not consistently fit the model. The study provides an analysis framework that may aid policymakers and institutional administrators in developing higher quality SETs, and demonstrates the need for validating SETs being implemented in higher education settings.     

Introducing the Microcomputer into English. II. An evaluation of WILT as a program using a problem-solving approach to learning spelling

Use of the spelling program WILT with a class of mixed-ability third-year secondary-level pupils as part of an evaluation study of programs for language development is reported. The behaviour and language activities characteristic of pupils using WILT are described and 14 categories of language activity identified. The results of an analysis of the language activities of four groups are presented, and the types of learning that are encouraged by WILT are identified. Differences between the pupils' and the class teacher's evaluations of WILT are examined, and the educational value of the program assessed.