Historical overview of formative evaluation of instructional media products

Though the term “formative evaluation” was coined only in 1967, formative evaluation activities, by different labels, have been going on for a long time. This article traces those activities—from study of World War I training films in the 1920s to detailed models and microcomputer technology currently employed.     

The principal’s role in the post-evaluation process.—How does the principal engage in the work carried out after the schools self-evaluation?

This article refers to a study on how the school principal engaged in the process after a school self-evaluation. The study examined how two primary schools followed up the evaluation. Although they both used the same evaluation tool, the schools’ understanding and application of results differed greatly. This paper describes and discusses the post evaluation process based on Erik Johnsen’s ideal leadership model (2002). It argues that formal leadership makes a difference in the use of the school evaluation for development by providing a proper context for knowledge sharing and reflection. This involves the prioritization and facilitation of individual and organisational reflection for learning, as well as transformation of knowledge through interaction across the whole school.     

Instructional product verification and revision: 20 questions and 200 speculations

Conclusion For some time now we have suspected that the significant differences due to formative evaluation in many doctoral dissertations is the result of beginning with an intentionally sloppy prototype product. The field of IPVR is full of mystique and myth — even from the initial assumption that IPVR does improve instructional products. I have contributed to the folklore of formative evaluation in this article and have probably confirmed some unjustifiable and unvalidated procedures. However, as Baker and Alkin (1973) point out, research on evaluation is not easy to undertake. We cannot afford to wait for the empirical testing of every aspect of our technology. Fortunately, the philosophy of IPVR suggests that we begin with our best guesses and verify and revise them on the basis of feedback from the field.     

Collective curriculum design as a tool for rethinking scientific literacy

This is a review essay from an educator’s point of view, on the article written by Maria Inês Mafra Goulart and Wolff-Michael Roth Engaging young children in collective curriculum design. The article as well as the commentary essay contributes to the current agenda about the feasibility of social practices in (science) education even from the early childhood. A child centered collective curriculum design can be a tool for rethinking scientific literacy towards a multi-science perspective, participative thinking and dialectical teaching strategies that take account of structure or/and agency relationships. The review is structured on five levels: (a) scientific literacy, (b) multi-science perspective, (c) participative thinking—structure or∣and agency, (d) comments on methodology—the emergency of a new order, (e) conclusions.     

ADAPT<Superscript>IT</Superscript>: Tools for training design and evaluation

This article describes a set of computerized tools that support the design and evaluation of competency-based training programs. The training of complex skills such as air traffic control and process control requires a competency-based approach that focuses on the integration and coordination of constituent skills and transfer of learning. At the heart of the training are authentic whole-task practice situations. The instructional design tools are based on van Merriënboer's 4C/ID^* methodology (1997). The article describes a training design tool (Core) that supports the analysis and design for competency-based training programs and an evaluation tool (Eval) that supports the subsequent revision of this training design.     

Formative Evaluation in Performance Technology

This article describes and discusses how systematic formative evaluation can be applied to different types of human performance technology (HPT) interventions. Formative evaluation is the process of determining the worth of a package in order to improve its cost–effectiveness. Inputs to formative evaluation include interventions, products, and people. Outputs include preference data (for example, user–friendliness) and performance data (immediate outcomes, transfer outcomes, and payoff outcomes). The general process for forma tive evaluation consists of the steps of goals specification, preparation, data collection, data analysis, revision, and recycling. Formative evaluation is usually undertaken in six stages to correspond to different phase of the systematic HP model. The most important aspect of formative evaluation is not the quantity of data but the quality of revisions.     

Design and Development of a Bilingual Multimedia Educational Tool for Teaching Chemistry Concepts to Deaf Students in Greek Sign Language

In this paper the design and development of a multimedia system which serves as a bilingual chemistry educational tool for deaf students is presented. Details of the architecture of the tool, its functionality and formative evaluation results are also presented. Furthermore the process to arrive at the chemistry terminology in Greek Sign Language used throughout the tool is briefly described.     

The Online Academy formative evaluation approach to evaluating online instruction

The Online Academy (HO29K73002) was funded by the Office of Special Education Programs (OSEP) to develop online instructional modules in the content areas of reading, positive behavior support, and technology across the curriculum. Targeted to preservice teacher education programs in institutions of higher education (IHE), to date, the modules have been adopted for implementation by 162 institutions. A requirement of the funding agency was that the content of the modules be research based. A total of 75 lessons in 22 online modules and an authoring software tool were developed. Each module is approximately equivalent to a one-semester credit course. This paper describes the formative evaluation processes that were employed in creating the instructional design, design and production processes, content development, usability and navigational features of the modules, and the national implementation process.     

Engineering design in the classroom: Is it good science education or is it revolting?

The changing emphasis of science education today includes a shift from a narrowly disciplinary view of science to a more socially and technologically situated perspective. This article reports on the implementation of engineering design projects in about two dozen U. S. secondary science classrooms, following an inservice professional development course conducted at a university engineering college. Using data from an evaluation of the project and three sociological themes—secrecy and ownership, social persuasion and the status of facts, and the relationship between money and science—I argue that technological design projects provide a sociologically fruitful approach for teaching new themes in science education. However, teachers must rethink the nature of their subject matter, something that may be difficult if their training was disciplinary in emphasis.     

Formative assessment systems: evaluating the fit between school districts’ needs and assessment systems’ characteristics

Recent legislative and local school accountability efforts have placed a premium on the collection, analysis, and use of student assessment data for educational improvement. As a result, schools have sought assessments that will provide additional information about student performance. In response, a burgeoning boon industry formed—formative educational assessment systems. In this study we describe how districts search for and acquire formative assessment systems to meet their needs. We focus on three school districts that adopted three different formative assessment systems. Our findings suggest the fit between a system’s characteristics and a school district’s intended use is the most important consideration in instituting a successful formative assessment system that will have a positive impact on teacher education and student learning.     

Professionalism in early care and education

Summary Professionalism refers to the utilization of specialized knowledge that its members need to accomplish specific outcomes. It involves a shared set of skills that are used to improve the quality of caregiving practices and interactions between professional caregivers and the children and families that they work with in their respective programs. Professionalism is not an end in itself—a state of being—but an ongoing effort—a process of becoming. New knowledge of children's development needs to be incorporated into professional caregivers' existing repertoire. Their words and actions do matter greatly. As Neil Postman wrote in his book,The Disappearance of Childhood, “Children are the living messages we send to a time we will not see”.     

The role of the content and structure of curriculum material in cognition

Summary and Conclusion What has been reported here is an objective, pre-trial formative evaluation of the core of an ASEP unit designed to show its potential in terms of themes and structure, a revision on the basis cognitive-field learning theory of those sections of the core shown by the analysis to be possibly deficient, and an ongoing field-trial of the original and revised cores. The field-trial results, though only tentative, have demonstrated the use of the Carss (1973) content analysis technique as it may be applied to formative curriculum evaluation since it supplies “... the diagnostic analysis which is at the heart of formative evaluation” (Baumgart; 1972:9). The technique produces, prior to field-trials, usable information with respect to the potential inherant in the text, showing for example where structure is poor or where there is irrelevant or too much information. The technique also allows feedback from field-trials to be used more constructively since general statements or questions, hitherto of little use in giving writers directions for revision, can be linked to themes, the content and distribution of which are available. The use of this technique for the analysis of the manifest content of communication-written or verbal—seems unlimited. The analysis of the teacher and pupils verbal communications in parallel with the textual analysis, and the analysis of “expert” ideas on “what is science?” to obtain the “essence” of science are but two. The technique itself needs further refinement and this refinement and possible uses offer an interesting challenge for future research.     

Debunking the Fear of Peer Review: Combining Supervision and Evaluation and Living to Tell About It

Scholars have argued that combining supervision and evaluation is a conflict of interest that will damage the coaching process. Peer assistance and review (PAR) is an approach to the supervision and evaluation of teachers that runs counter to this argument, as the coach charged with providing professional support to a new or struggling veteran teacher also plays a formal role in that teacher’s summative personnel evaluation. This article presents data from a study of one urban school district in California as it implemented a PAR program, asking: (1) To what degree do PAR mentees trust their coaches, who conduct both formative and summative assessments? (2) How do PAR mentees who do not trust their coaches differ from those who do? Mentees’ trust in their coaches was a point and a half higher than their lack of trust in their coaches, as self-reported on a 4-point Likert scale. Those mentees who did not report a high degree of trust in their coaches were low-performing mentees.     

Integrating formative evaluation into a learner-centred revision course

This article describes a learner-centred course which ran in summer 1988. It was ostensibly a revision exercise, using computer assisted learning (CAL), but an element of formative evaluation was deliberately built into the process, making it contribute to our systematic review and development of our teaching. Our aim was to take snapshots of learning at different stages, through a short but intensive revision class in mathematics for engineering and science undergraduates at a Scottish university
The CALM Project—Computer Aided Learning in Mathematics—has produced 2 5 software units to enhance the teaching of calculus to large groups of undergraduates. The CALM software was piloted by one group of students and a formative evaluation of the packages has been reported elsewhere (Beevers et al 1987, Beevers et al 1988a). This revision course enabled us to confirm many of the previous evaluation findings on the CALM software; we also discovered that a round-up session on problem-solving is an important and integral part of the learning process. The evaluation proved useful to us as teachers of mathematics, and has already influenced the future development of our CAL work.     

Science Teachers’ Use of a Concept Map Marking Guide as a Formative Assessment Tool for the Concept of Energy

In this study, we investigated the value of a concept map marking guide as an alternative formative assessment tool for science teachers to adopt for the topic of energy. Eight high school science teachers marked students’ concept maps using an itemized holistic marking guide. Their marking was compared with the researchers’ marking and the scores of a multiple-choice diagnostic test. Statistical analysis revealed that teachers evaluated students’ concept maps consistently, but the concept map scores were only mildly correlated with the multiple-choice test scores. We explored what each assessment method revealed in terms of students’ understanding with one student’s performance as a showcase. Teachers’ views on using the marking guide of concept maps as an assessment tool were also included. We discuss the value of the concept map marking guide as a formative assessment tool for science teachers to create and modify in order to encourage students’ conceptual learning.     

Formative Assessment: Assessment Is for Self-regulated Learning

The article draws from 199 sources on assessment, learning, and motivation to present a detailed decomposition of the values, theories, and goals of formative assessment. This article will discuss the extent to which formative feedback actualizes and reinforces self-regulated learning (SRL) strategies among students. Theoreticians agree that SRL is predictive of improved academic outcomes and motivation because students acquire the adaptive and autonomous learning characteristics required for an enhanced engagement with the learning process and subsequent successful performance. The theory of formative assessment is found to be a unifying theory of instruction, which guides practice and improves the learning process by developing SRL strategies among learners. In a postmodern era characterized by rapid technical and scientific advance and obsolescence, there is a growing emphasis on the acquisition of learning strategies which people may rely on across the entire span of their life. Research consistently finds that the self-regulation of cognitive and affective states supports the drive for lifelong learning by: enhancing the motivational disposition to learn, enriching reasoning, refining meta-cognitive skills, and improving performance outcomes. The specific purposes of the article are to provide practitioners, administrators and policy-makers with: (a) an account of the very extensive conceptual territory that is the ‘theory of formative assessment’ and (b) how the goals of formative feedback operate to reveal recondite learning processes, thereby reinforcing SRL strategies which support learning, improve outcomes and actualize the drive for lifelong learning.     

Exploring the potential of a computer tool for instructional developers

Information and communication technology tools currently permeate almost every professional domain. Those geared toward the field of instructional development have emerged in recent years. This article explores the potential for linking the domains of computer support and instructional development.This article reports on the design and evaluation of CASCADE (Computer Assisted Curriculum Analysis, Design and Evaluation), a computer system that supports instructional developers during formative evaluation efforts. Five prototypes of CASCADE were created and evaluated on the basis of their validity (reflection of state-of-the-art knowledge and internal consistency); practicality (ability to meet the needs, wishes and contextual constraints of the target group); and effectiveness (improved user task performance).The results of this study suggest that the use of CASCADE could: (a) improve the consistency of formative evaluation plans and activities; (b) motivate developers by elevating their confidence in using formative evaluation activities; (c) save time; and (d) help to provide justifications for decisions made.     

网络环境下基于学习活动的形成性评价——中小学教师教育技术能力培训个案研究

该文从中小学教师教育技术能力培训的实际过程出发,在分析形成性评价概念和学习过程分析的基础上,详细分析了混合式培训模式下学习活动的分类,以及学习活动的观测指标,并描述了基于网络的形成性评价中的关键技术,为形成性评价过程中资料数据的收集、分析、反馈提供了理论和技术基础,真正把形成性评价融于具体的教学活动过程之中,以此来推动中小学教师教育技术能力培训的有效开展。     

Formative Evaluation Alternatives

Traditional formative evaluation methods and tools have a documented history of successful use. Expert, one-to-one, small group, and field-test methods have been used for decades. However, changes in design practice, technology, and lately, in theory, have resulted in alternative evaluation methods that are not as well-known in current design literature. These methods can be used to complement or replace traditional evaluation methods in a formative evaluation project. Some of these alternative methods use different groupings of experts or learners, while others use different evaluation questions or technologies. This article explains each of these formative evaluation alternatives and outlines their advantages, disadvantages, and applicable contexts.