The influence of participants in formative evaluation on the improvement of learning from written instructional materials

Formative evaluation in instructional design is widely advocated as a means to improve instructional materials through tryouts with learners and experts and revision based on this feedback. Research on formative evaluation has sought to prove its effectiveness and to provide guidance as to which methods of collecting feedback and which sources of feedback are most effective in improving learning. It is difficult to determine from the research whether aspects of tryout or of revision are responsible for improved learning. This paper describes a systematic investigation of both tryout and revision to more clearly determine the influence of different formative evaluation participants (experts, learners and revisers) on the improvement of learning from written instructional materials.It was found that revised versions incorporating learner feedback had the most impact on improving learning from the materials. It was also found that revisers have a far more powerful impact on formative evaluation outcomes than was previously supposed, in terms of how they mediate and incorporate the feedback they are given. The results refute contentions that any revision is better than none since versions revised without learner feedback did not improve learning. Implications are discussed.     

The use of evaluative feedback for instructional improvement: A longitudinal perspective

Both the available experimental evidence and recent reviews are unclear regarding the effectiveness of student ratings feedback for instructional improvement. Provision of feedback has not consistently produced improvement. Feedback accompanied by consultation has resulted in more consistent instructor improvement. However, the literature on interventions for instructional improvements has failed to address the longitudinal nature of improvement and development. The present study examined follow-up data over a ten-year period on instructors who had participated in a quasi-experimental study of the effectiveness of consultation and feedback conducted in 1971 and 1972. The results obtained indicate that instructors who had originally received feedback with consultation maintained higher student ratings and used instructional resources more during follow-up than instructors who originally received feedback only. This result must be tempered by several interpretative considerations regarding sample size, power, and experimental control. A number of concerns were also expressed regarding the adequacy of the current literature for describing a developmental event, instructional improvement, through the use of short-term studies. Recommendations are made for increased durations of study and alternative tactics of experimental design and control.     

Feedback about Teaching in Higher Ed: Neglected Opportunities to Promote Change

Despite ongoing dissemination of evidence-based teaching strategies, science teaching at the university level is less than reformed. Most college biology instructors could benefit from more sustained support in implementing these strategies. One-time workshops raise awareness of evidence-based practices, but faculty members are more likely to make significant changes in their teaching practices when supported by coaching and feedback. Currently, most instructional feedback occurs via student evaluations, which typically lack specific feedback for improvement and focus on teacher-centered practices, or via drop-in classroom observations and peer evaluation by other instructors, which raise issues for promotion, tenure, and evaluation. The goals of this essay are to summarize the best practices for providing instructional feedback, recommend specific strategies for providing feedback, and suggest areas for further research. Missed opportunities for feedback in teaching are highlighted, and the sharing of instructional expertise is encouraged.     

The priorities established among data sources when instructional designers revise written materials

The objective of the study was to investigate the priorities instructional designers establish among data sources when they revise written materials and the relationship of their practice to standard models of formative evaluation. Two modules of printed instructional material were revised by each of eight experienced instructional designers under three treatment conditions, using a counterbalanced design. Data were collected using a think-aloud procedure. The think-aloud protocol was segmented, coded and analysed. Results show that instructional designers were significantly more likely to incorporate their own knowledge into revisions than to use feedback data, and when they did use feedback data, they preferred learner comments. They did not accurately assess their use of their own knowledge or of feedback data; they thought they used feedback data more than the results showed. These instructional designers reflected some aspects of the standard instructional design model when they revised instructional materials, but their experience has caused many of the procedures to become internalized, so they are invisible in the protocols. This research was funded in part by a grant from the Social Sciences and Humanities Research Council of Canada.     

An Expanded Control Model for the Use of Instructional Feedback

The strengths of the now classic Kulhavy/Stock model of instructional feedback are summarized and various researchers′ suggestions for improvement are incorporated into an expanded model. Data from two computer drill experiments demonstrate that the use of feedback varies systematically between subjects: the highest achieving subjects use feedback as the Kulhavy/Stock model predicts, the intermediate achievers use of feedback is less predictable, and the lowest achievers use feedback in a nearly random manner. These response patterns can be addressed in a feedback processing model which adds higher level goals related to certitude and correct responding. The expanded model has the following characteristics: feedback processing is not automatic, but varies in mindfulness; learner′s individual goals are incorporated; analysis and modeling are conducted on a subject by subject basis; and instructional feedback is understood more in terms of each learner′s control systems and less in terms of a stimulus-response orientation.     

The Case for Expanding Standards for Teacher Evaluation to Include an Instructional Supervision Perspective

Despite the concerns of scholars in the field of instructional supervision, teacher evaluations continue to emphasize bureaucratic accountability and standardization. This article presents an argument for extending the Joint Committee on Standards' Personnel Evaluation Standards to include standards related to the practice of supervision. The proposed standards call for differentiated procedures, collaborative identification of teachers' professional development goals, multiple sources of data, emphasis on formative evaluation processes, consideration of both teachers' personal development goals and school/program improvement goals, and the formalization of formative evaluation processes to achieve clear and shared understanding of their purpose and goals. These standards would align teacher evaluation with aspects of instructional supervision namely,clarification and shared understanding of the process and purpose of evaluations, interpretation of teaching performance in the context of teachers' classrooms and professional values, and deliberation with teachers about how evaluation evidence depicts and informs their work.Patricia Holland is Associate Professor in Educational Leadership and Cultural Studies at the University of Houston. Her research in the areas of instuctional supervision and teachers’ professional development emphasizes the interpretive nature of practice in these areas.     

Pointing teachers in the wrong direction: understanding Louisiana elementary teachers’ use of <Emphasis Type="Italic">Compass</Emphasis> high-stakes teacher evaluation data

Spurred by Race to the Top, efforts to improve teacher evaluation systems have provided states with an opportunity to get teacher evaluation right. Despite the fact that a core reform area of Race to the Top was the use of teacher evaluation to provide on-going and meaningful feedback for instructional decision making, we still know relatively little about how states’ responses in this area have led to changes in teachers’ use of these sources of data for instructional improvement. Self-determination theory (SDT) and the concept of functional significance was utilized as a lens for understanding and explaining patterns of use (or non-use) of Compass-generated evaluation data by teachers over a period of 3 years in a diverse sample of Louisiana elementary schools. The analysis revealed that the majority of teachers exhibited either controlled or amotivated functional orientations to Compass-generated information, and this resulted in low or superficial use for improvement. Perceptions of the validity/utility of teacher evaluation data were critical determinants of use and were multifaceted: In some cases, teachers had concerns about how state and district assessments would harm vulnerable students, while some questioned the credibility and/or fairness of the feedback. These perceptions were compounded by (a) the lack of experience of evaluators in evaluating teachers with more specialized roles in the school, such as special education teachers; (b) a lack of support in terms of training on Compass and its processes; and (c) lack of teacher autonomy in selecting appropriate assessments and targets for Student Learning Target growth.     

A Model for Peer Review in Instructional Design

Although many professions practice some type of formal peer review, similar support for increasing the quality and efficiency of instructional design products is largely informal. Most designers develop solutions in professional eommunilies-of-practice, but formal design methodologies ordained by many corporations and instructional design schools do not account for the strong influence these communities have on design decisions. This paper describes an instructional design review process that is modeled after a widely accepted practice among software developers in which peers offer feedback through “structured walkthroughs.” The instructional design review helps the practitioner develop stronger designs more quickly. provides a means to scaffold the novice designer in the vagaries of the workplace, and improves organizational memory. An artifact of a design review as well as guidelines and success factors are presented. The paper provides a summary of a formal peer review structure that has been developed and tested over the past three years at a major corporation.     

试论诉讼监督工作评价模式的重构

当前我国的诉讼监督工作评价模式存在着一些问题,某种程度上阻碍了检察机关诉讼监督职能的发挥,应构建独立的、动态的、结果评价与过程评价、个案评价与综合评价、内部评价与外部评价相结合的诉讼监督工作评价模式,以推动诉讼监督工作发展,维护司法公正和公民合法权益。     

Seniority and experience of college teachers as related to evaluations they receive from students

A review of relevant research shows that many studies have found the academic rank, age, and extent of instructional experience of college teachers to be unrelated to their students' global evaluation of them. Yet many other studies have found these indicators of the teacher's seniority and instructional experience to be related to the overall evaluation of teachers, and with a consistent pattern. In this latter group of studies, the teacher's academic rank is positively associated with the overall evaluation of the teacher, whereas both the teacher's age and extent of instructional experience are inversely associated with overall evaluation. Although these associations are generally weak in strength, they are robust enough to hold under a variety of controls (including the size of course enrollment and the gender of the teacher). Also, at least for extent of instructional experience, the number and strength of the associations may be underestimated by considering the relationship in question as linear when in fact it may be curvilinear. Just as the teachers' age and extent of instructional experience in general have been either not related or inversely related to the global evaluation of teachers, so they have been for more specific evaluations of teachers. When academic rank has been found to be related to specific evaluations, the relationships tend to be positive for only certain specific rating dimensions while being inverse for others. This being so, it is puzzling that the associations that have been found between academic rank and global evaluations have generally been positive only. A possible explanation for this is offered and supportive evidence marshaled. Finally, several reasons why associations between any of the indicators of seniority or instructional experience and either specific or overall evaluations may be inverse, positive, or curvilinear in the first place are advanced. These hinge on such factors as the teacher's motivation and career circumstances, the pedagogical training and abilities of different cohorts of teachers, as well as the criteria and practices of colleges in their recruitment of teachers, the extent of an age (or generation) gap between students and teachers, and students' expectations and perceptions of teachers and pedagogical characteristics attributed to them. Those reasons having the greater support empirically as well as those that seem the most plausible are noted.     

How to activate teachers through teacher evaluation?

There is a general doubt on whether teacher evaluation can contribute to teachers’ professional development. Recently, standards-based teacher evaluation has been introduced in many countries to improve teaching practice. This study wants to investigate which teacher evaluation procedural, leadership, and teacher characteristics can stimulate teachers’ feedback utility perceptions and professional learning activities in the context of teacher evaluation. Data were collected in 32 secondary schools from 298 teachers and 32 school leaders, both quantitatively and qualitatively. We put forward a research model which is investigated through path analysis. Results nurture optimism about teacher evaluation indicating that teacher evaluation can stimulate teachers to undertake professional learning activities when they perceive feedback as useful. The important role of the school leader (through both instructional and transformational leadership) is stressed. Teacher participation and teacher collaboration also have a positive influence on teacher evaluation and its outcomes.     

Generic questioning strategies for linking teaching and testing

Modern instructional theory and research suggest that the content of instruction should be closely linked with testing. The content of an instructional program should not focus solely on memorization of facts but should also include higher level thinking. Three uses of tests within any instructional program are: (1) practice on objectives, (2) feedback about mastery of those objectives, and (3) summative evaluation. The context-dependent item set is proposed as a useful tool for measuring many higher level objectives. A generic method for developing context-dependent test item sets is proposed, and several examples are provided. The procedure is useful for developing a larger number of test items that can be used for any of the three uses of tests. The procedure also seems to apply to a wide variety of subject matter.     

Teacher supervision and evaluation challenges: Canadian perspectives on overall instructional leadership

This inquiry focuses on the overall instructional leadership approaches used by exemplary principals in three high performing Canadian provinces to overcome three persistent obstacles to effective teacher supervision and evaluation: (a) the management challenge, (b) the complexity challenge, and (c) the learning challenge. Analysis of data collected from interviews, focus groups, observations, field notes, documents, artifacts, and reflective research journals yielded the following four assertions: (a) Shared, distributed, and collective approaches to overall instructional leadership deepen and widen impact. (b) Effective supervision and evaluation are part of a career-long continuum of practice that fosters teacher growth while ensuring quality teaching. (c) There are multiple learning pathways to effective overall instructional leadership. (d) Policy contexts that place teacher supervision and evaluation practice within a broader conception of overall instructional leadership are beneficial.     

高职院校人才培养质量“三维度”评价诊断体系研究

在高等教育质量保障理论和多年实践成果的指导下,以学校发展状态评价为引导,以专业人才培养质量评价为核心,以教学过程监控为抓手,构建全员、全方位、全过程的人才培养工作质量"三维度"评价监控体系,从多元评价主体视角对学校环境支持、专业发展状态和教学效果达成度开展评价、监控与诊断,不仅为人才培养质量工作的内控和持续改进提供依据,而且理顺了人才培养宏观、中观和微观质量管理工作三个维度之间的逻辑关系,体现了人才培养质量评价诊断工作的科学性、系统性、全面性。     

Metacognitive, cognitive and affective aspects of learning styles and strategies: A phenomenographic analysis

This paper addresses the following questions: how do students perform metacognitive, cognitive and affective learning functions; how is the execution of learning functions regulated by internal and external sources; what learning styles can be discerned from the viewpoint of learning functions and regulation? Subjects were students from an open distance university and a regular university. They were interviewed extensively about their learning strategies, mental models of learning, learning orientations and interpretations and appraisals of instructional measures. The interviews were analyzed in a phenomenographic way. The results indicate that there are large differences among students in the manner in which they carry out learning functions, that these differences are associated with internal and external sources, and that four qualitatively different learning styles can be discerned: an undirected, a reproduction directed, a meaning directed and an application directed learning style. Mental models of learning and learning orientations turn out to be related to the way in which students interpret, appraise and use instructional measures to regulate their learning activities. It is concluded that in many instances instructional measures do not have the intended effects. Suggestions are given regarding the implications of these results for the improvement of teaching practices in higher education.     

Using screen capture feedback to improve academic performance

Feedback is an effective instructional practice for improving achievement. The importance of feedback has been understood by teachers for many years, yet recent research has elevated its status. One characteristic of effective feedback is personalized comments, which assist students in understanding their performance. Most research analyzes the effects of written feedback. Some current studies have examined the effects of alternative delivery methods, such as audio. One new approach for creating and delivering feedback is through screen capture technology. A case study was conducted to examine the effects of this method on the academic performance of 36 undergraduate students in a teacher education course. Results showed that screen capture feedback is effective for guiding corrections on a writing assignment. Students also reported that feedback was useful for knowing what they were doing right or wrong and that they would prefer more feedback of this type.     

Instructional product verification and revision: 20 questions and 200 speculations

Conclusion For some time now we have suspected that the significant differences due to formative evaluation in many doctoral dissertations is the result of beginning with an intentionally sloppy prototype product. The field of IPVR is full of mystique and myth — even from the initial assumption that IPVR does improve instructional products. I have contributed to the folklore of formative evaluation in this article and have probably confirmed some unjustifiable and unvalidated procedures. However, as Baker and Alkin (1973) point out, research on evaluation is not easy to undertake. We cannot afford to wait for the empirical testing of every aspect of our technology. Fortunately, the philosophy of IPVR suggests that we begin with our best guesses and verify and revise them on the basis of feedback from the field.     

From knowledge sharing to knowledge creation: A blended knowledge-management model for improving university students’ creativity

Creativity and knowledge management are both important competences that university students need to strive to develop. This study therefore developed and evaluated an instructional program for improving university students’ creativity based on a blended knowledge-management (KM) model that integrates e-learning and three core processes of KM: knowledge sharing, knowledge internalization, and knowledge creation. Moreover, this study investigated the underlying mechanisms that achieve the effectiveness of this model. A 17-week instructional program was conducted. The findings from both quantitative and qualitative analyses suggest the following. The blended KM model is effective in improving knowledge, dispositions, and abilities of creativity. The online sharing and evaluation of creative products, learning communities and discussions, and the practice of creativity strategies have substantial effects on all three aspects of creativity. The observation and peer evaluation of group assignments and creativity-related feedback enhance the learning of knowledge and dispositions. Finally, the creation of products and scaffolding of a teacher are critical to skill improvement.