Transfer Evaluation of Knowledge and Skill Development: A Case Study of an Instructional Systems Development Training Program

Program evaluation involves a systematic process to collect and convert data into information which is used to improve programs, measure their effects, track quality, act on results, and improve decision making. Beginning in 1992, the LG Group of Korea was involved in an instructional systems development (ISD) training program for its human resource development professionals in cooperation with Indiana University. The LG ISD training program stressed that learning instructional design theories and practices would make LG more effective in global competition. The purposes of the research reported in this article were to evaluate the effectiveness of the ISD training program in terms of transfer of training and to gain information on how to improve future training programs. The findings of the study support the goals that were established for the ISD training program. The program participants are more confident and competent in using the ISD process for analyzing needs for performance improvement and designing and developing quality training products. The article suggests ways of solving barriers to transfer training and evaluation and improving future programs.     

Teachers' Planning Processes: Does ISD Make a Difference?

Do teachers who have a formal educational background in instructional systems design (ISD) differ from those who do not in how they plan and implement instruction? This question guided this pilot study of ten volunteer teachers, five who had formal education in ISD and five who did not. An open and closed item questionnaire was sent to each subject and follow-up interviews were conducted. Teachers were asked about their general planning practices, written planning procedures, mental planning, and how they implement instruction based on their plans. Results indicated that nine of the ten teachers use general ISD skills and processes in planning. Teachers reported that they (a) analyzed learners' needs and abilities, and (b) used objectives to guide the instructional process, specifically for selecting learning activities and evaluation. While there were no major differences between the two groups in their submitted daily plans, the teachers with instructional design backgrounds were more specific in their open-ended responses about the use of instructional design for some aspects of planning (e.g., using hierarchies and taxonomies to sequence instruction), and the importance of consistency among objectives, learning activities, and evaluation. In addition, four of the five ISD teachers believed that knowledge of ISD has improved both their planning processes and their teaching. Because major differences between the two groups were not observed, much of the article focuses on teacher planning processes in general.     

Instructional systems design and public schools

A great deal has been written in the last several years about the use or lack of use of instructional systems design (ISD) in public schools. The general discussion has revolved around why ISD has not been accepted by public schools; why instructional design experts have not been hired by public school systems; and primarily what an educational technologist can do to change the situation. The focus of most discussions by educational technologists is: how can schools and teachers alter their practice to benefit from ISD? In this article, we maintain that one reason why ISD has not been adopted in schools is that there may not be a fit between the philosophies, needs, resources, and constraints of schools and the benefits of adopting ISD. We also argue that for ISD to have a significant influence in public schools, it must be accepted by teachers. The general question we ask in this paper is this: are we attempting to redesign schools and classrooms to accommodate our technologies with insufficient attention to the perceived needs of clients and to the consequences of adopting ISD? Our approach to answering this question was to search the literature to uncover perceived inconsistencies between ISD theory and practice on the one hand and teacher needs, wants, and practices on the other.     

If we can't contextualize it,should we teach it?

Constructivists often present their ideas as the solution to most ills in education. Sometimes explicitly, but often implicitly, instructional systems design (ISD) is blamed for these ills. There is some irony to this as ISD advocates often lament the dearth of ISD in education. Be that as it may, ISD does have limitations and there is much to be learned from constructivism. However, is a paradigm switch the answer? This paper will take a different approach as it will indicate how ISD can help those who want to develop constructivist instruction. The ISD procedures involved in task/learning analyses and formative evaluation are presented as particularly beneficial.     

Constructivist values for instructional systems design: Five principles toward a new mindset

In this article, the implications of constructivism for instructional systems design (ISD) are summarized as five principles that integrate the affective and cognitive domains of learning. In contrast to current views, it is suggested that constructivist philosophy offers instructional designers an alternative set of values that may significantly influence the emphasis of ISD methods without undermining the coherence and consistency of the ISD model. Distinguishing characteristics of the two approaches are described, based on a review of recent literature. The article concludes with the assertion that the influence of constructivist philosophy on ISD should be to focus attention on critical enabling objectives traditionally overlooked by instructional designers.     

Theoretical Frameworks of Performance‐Based Models

The purpose of this article is to highlight theories that support the functions of performance‐based design models and to discuss the implications of integrating divergent models into the system‐oriented human performance technology (HPT) and performance improvement (PI) disciplines. HPT, PI, and instructional systems design (ISD) share a systems framework, along with the influence of common theories such as performance theory, learning theory, adult learning, cognitive psychology, and behavioral psychology (Foshay, Villachica, & Stepich, 2014). This article focuses on the role of theory as a tool in the practitioner's toolbox and as a connection point when working with teams and organizations that have different theoretical orientations. Performance‐based ISD models are discussed, including Robinson and Robinson's (1989) Training for Impact, Brethower and Smalley's (1998) Performance‐Based Instruction, and Bradford and Boler's (2015) Horizon Model. Allen and Sites's (2012) successive approximation model (SAM) retains elements of ADDIE as a process, but the model is iterative rather than systematic in design.     

An Agency Theory Perspective on Student Performance Evaluation

The emphasis in recent research on the responsibility of college and university business instructors to prepare students for future employment underscores a need to refine the evaluation of student performance. In this article, an agency theory framework is used to understand the trade‐offs that may be involved in the selection of various approaches to student evaluation. Understanding these trade‐offs may be particularly important as faculty members seek to balance competing obligations, such as research and service requirements, while ensuring instructional effectiveness. This article presents propositions for examining how various institutional, instructor, and student characteristics influence the selection and use of student performance evaluation techniques (i.e., exams, papers, and group assignments). In conclusion, we suggest that agency theory may serve as a foundation for understanding current evaluation practices and guiding instructors in their selection of appropriate evaluation mechanisms.     

Cognitive task analysis for instructional design: Applications in distance education

The increasing complexity of the subjectmatters taught through distance education calls for a cognitive approach to instructional design. Cognitive task analysis‐based instructional design (CTA‐BID) is especially appropriate for distance education, which requires high‐fidelity instructional materials that teach cognitive content with little or no classroom instruction. CTA‐BID organises instructional materials around the results of a cognitive task analysis that identifies the optimal knowledge structures, mental models, strategies, and skills underlying expertise in the subject matter. CTA‐BID has produced innovative instructional programmes aimed at teaching cognitive skills, accelerating the development of expertise, and improving student performance and training efficiency. This article provides an overview of CTA‐BID and its applications in the design of instructional and testing materials for distance education. I also review recent developments in education, psychology, and instructional design that complement CTA‐BID.     

The Education of Instructional Designers: Reflections on the Tripp Paper

In a recent issue of this journal, Steven D. Tripp proposed some innovations in the education of instructional designers based on his analysis of education practices in other design professions. His principal proposals were that students should analyze great instructional designs of the past and that instructional design studios should form part of the educational preparation of students entering the profession. This paper proposes some extensions and modifications to Tripp's original ideas. These include extending his concept of the analysis of great lessons of the past to the analysis of current examples of expert work, the use of an expanded jury system to incorporate peer and practitioner evaluation of students' work, and the criticality of students developing solutions to actual performance problems during the instructional design studio experience. Finally, it is proposed that opportunities for critical reflection on the process of instructional design be built into the studio experience.     

Enhancing instructional design efficiency: Methodologies employed by instructional designers

Instructional systems design (ISD) has been frequently criticised as taking too long to implement, calling for a reduction in cycle time—the time that elapses between project initiation and delivery. While instructional design research has historically focused on increasing learner efficiencies, the study of what instructional designers do to increase efficiency during the design of instruction, including web‐based training, has not yet been fully examined. The purpose of this qualitative developmental study was to identify and understand the methodologies used by experienced instructional designers to increase the efficiency of the instructional design process. Data were gathered from 11 instructional designers working within two business and industry consulting firms that provide learning solutions to global clients. Results revealed 47 efficiency methodologies found within the four categories of design model, instructional design team member roles, instructional design processes and instructional design tools. Additionally, 14 supporting instructional designer competencies were found within the category of instructional designer knowledge, skills and attitudes.     

网络课程开发的参与式教学设计策略

目前网络课程开发更多涉及微观层面的教学设计流程。而参与式设计作为宏观教学系统设计动态特征的一大表现,是从交互式设计的观点出发,运用工程和产品设计的过程观点来认识教学设计的一种方法, 是一种用户为中心的教学设计方法。该方法在今后的教学资源及网络课程的系统开发中将会得到广泛应用。     

Exploring teachers' instructional design practices from a systems design perspective

Curricular changes in higher vocationaleducation have rendered teachers' instructionaldesign activities increasingly important. Usinga repertory grid technique, this paper sets outto analyse current design activities of tenteacher trainers. Their actual approach iscompared with an instructional systems design(ISD) approach and related to innovativeteacher roles. Teachers' activities show animbalance in two ID phases, that is problemanalysis and evaluation. The results suggestthat they attempt to translate curricular goalsdirectly into concrete lessons and they payrelatively little attention to evaluation. Inline with this finding, they underrate the twoinnovative teacher roles of the `diagnostician'and the `evaluator'. It is argued thatimbalanced or incomplete design approaches andperceived roles may hinder innovation ineducation. Implications for the support ofteachers' design activities are discussed.     

What can we learn from chaos theory? An alternative approach to instructional systems design

In this article, the “goodness of fit” between ISD and chaos theory is explored by applying the key concepts of chaos theory to the process of developing an alternative ISD model. After a brief introduction to chaos theory and an exploration of the limitations of and/or the problems with conventional ISD models, the theoretical implications for developing an alternative ISD model are explored. The assumptions of a conventional ISD model are compared to those of chaos theory and dynamic nonlinear systems in order to derive theoretical implications and recommendations for future research and practice in instructional systems design and development.     

In‐Text Questions: A Note on an Experiment in Instructional Design

The use of instructional design is increasingly being discussed in the distance education and higher education literature. This paper reports an experiment in which self study questions are inserted in the text of lecture note material for external students. An evaluation revealed that students who received notes with in‐text questions were no different from those who received the standard lecture notes as judged by both performance and attitudinal criteria. Possible reasons for the perverse findings are given and it is concluded that professional instructional design assistance would be of value to subject matter experts.     

A Layered Approach to Understanding Your Audience

One of the first areas to be limited in performance improvement or an educational project is the learner‐analysis phase. If done in a precise and purposeful manner, an effective learner analysis will begin the learner‐centered approach and then drive an impactful instructional design. This study has taken Tessmer and Wedman's (1990) layers‐of‐necessity model for instructional design and applied it to enhancing the needs‐analysis phase of instructional design.     

Beliefs about learning and enacted instructional practices: An investigation in postsecondary chemistry education

While research on and development of evidence‐based instructional practices (EBIPs) in STEM education has flourished, implementation of these practices classrooms has not been as prolific. Using the teacher‐centered systemic reform model as a framework, we explore the connection between chemistry instructors’ beliefs about teaching and learning and self‐efficacy beliefs, and their enacted classroom practices. Postsecondary chemistry faculty present a unique population for the study because of their role in teaching prerequisite courses, such as general and organic chemistry, which are key to many science major fields. A measure of teacher beliefs and self‐efficacy was administered to a national survey of postsecondary chemistry faculty members. Instructional practices used in a chemistry course were also collected via self‐report. While instructional practices were not directly observed, a cluster analysis of our data mirrors patterns of instructional practices found in observation‐based studies of chemistry faculty. Significant differences are found on teacher thinking and self‐efficacy measures based on enacted instructional practices. Results support the hypothesized connection between beliefs and instructional practice on a larger scale than in previous studies of this relationship, bolstering the evidence for the importance of this relationship over previously criticized results. These results present a call for reform efforts on fostering change from its core, that is, the beliefs of those who ultimately adopt EBIPs. Dissemination and design should incorporate training and materials that highlight the process by which faculty members interpret reformed practices within their belief system, and explore belief change in the complex context of education reform.     

The backyard human performance technologist: Applying the development research methodology to develop and validate a new instructional design framework

Development research methodology (DRM) has been recommended as a viable research approach to expand the practice‐to‐theory/theory‐to‐practice literature that human performance technology (HPT) practitioners can integrate into the day‐to‐day work flow they already use to develop instructional products. However, little has been written about how it can be applied in a workplace setting to allow HPT practitioners to consider this research approach for adoption into their own activities. This article provides a real‐world application of the DRM to help close this literature gap. After providing background information to establish the case context, the article presents an overview of how this research approach was applied to an effort to develop and validate a new instructional design framework for potentially training National Aeronautics and Space Administration (NASA) astronauts for deep space exploration missions. The result of this case indicates that this research methodology provides a viable approach that HPT practitioners can integrate into their current practices to provide a practice‐based research baseline to contribute to the practice‐to‐theory/theory‐to‐practice literature.     

How Instructional Designers Solve Workplace Problems

This naturalistic inquiry investigated how instructional designers engage in complex and ambiguous problem solving across organizational boundaries in two corporations. Participants represented a range of instructional design experience, from novices to experts. Research methods included a participant background survey, observations of problem‐solving activities, in‐depth interviews, and analysis and evaluation of project documents and other tools. The findings revealed differences between experts and novices with regards to tolerance of ambiguity, expectations about their own roles in finding solutions for their clients, adaptability, attention to appropriate details, and management of workplace stress. The contrast between instructional design processes taught in universities and actual workplace practice was noted by both expert and novice participants. Experienced participants demonstrated adaptability in processes and communications to efficiently arrive at viable solutions for their clients. Expectation setting and relationship building emerged as techniques for creating environments supportive of instructional designers' problem‐solving activities.