Evaluation of Corporate Training Programs: Perspectives and Issues for Further Research

Performance improvement interventions, including training, are investments that can yield identifiable payoffs for an organization in the form of better job performance. Evaluation is vital to continuous improvement of human performance in the workplace. Without measures of effectiveness, organizations do not know whether dollars are being spent wisely and, consequently, whether to continue, modify, or improve performance interventions. There are several approaches for the evaluation of training programs. Few adequately cover the broader perspective of performance improvement. Various schemes and terms are used to describe facets for evaluating training programs. However, sometimes different terms describe the same event. At other times, quite different training evaluation activities are discussed by different authors using the same terms. The present article reviews six overall evaluation perspectives of corporate training programs: Kirkpatrick's four‐level approach; the CIRO approach; Hamblin's five‐level approach; Florida State University approach; Indiana University approach; and Phillips' five‐level approach. And four research areas for further study are recommended: overall evaluation models, causal relationships between evaluation categories, systematic research on how to evaluate the various categories, and appropriate uses of the results of evaluations.     

The ROI fieldbook: Strategies for implementing ROI in HR and training by Patti Pulliam Phillips,CPT, Jack J. Phillips,Ron Drew Stone,and Holly Burkett,CPT

Patti Pulliam Phillips, CPT, PhD, CEO of the ROI Institute, Inc., consults with U.S. and international organizations on implementing the ROI Methodology. Jack J. Phillips, PhD, a world‐renowned expert on measurement and evaluation, developed the ROI Methodology; his expertise in measurement and evaluation comes from nearly 30 years of experience. Ron Drew Stone is an author, international consultant, and authority on improving and measuring performance improvement interventions. Holly Burkett, CPT, MA, SPHR, principal of Evaluation Works, has more than 20 years' experience assisting in the design and measurement of performance improvement initiatives. The ROI Fieldbook: Strategies for Implementing ROI in HR and Training (ISBN: 978‐0‐7506‐7622‐9) is published by Elsevier Butterworth‐Heinemann (price: $44.95; paperback with a CD). This publisher may be reached online at http://www.elsevier.com.     

The application of artificial intelligence principles to teaching and training*

This paper compares and contrasts the use of AI principles in industrial training with more normal computer‐based training (CBT) approaches. A number of applications of CBT are illustrated (for example simulations, tutorial presentations, fault diagnosis, management games, industrial relations exercises) and compared with an alternative approach using AI. An evaluation of the relative merits of the two approaches will be given. Existing CBT packages are used to illustrate the points raised and the emphasis of the arguments will be on the effectiveness of AI and CBT in terms of both cost and learning. The position of AI applications within CBT is discussed, as is the task of getting started in applying these techniques.     

Career Development: The Linkage Between Training And Organizational Development

Improved organizational efficiency can only be accomplished if the employees within an organization have developed the skills, competencies, knowledge, and attitudes to perform at the highest possible levels. To a limited degree this can be accomplished via training, but employees develop the fundamental skills, competencies, knowledge, and attitudes related only to their current job. As a result, the organization is only indirectly impacted. This refers to the micro perspective of improved organizational efficiency. Organizational efficiency can also be accomplished through organizational development, but the individual employee is often overlooked during this process. Without employee commitment to improve efficiency, it will be short-lived. This approach refers to the macro perspective of improved organizational efficiency. Both the micro and macro perspectives of improved organizational efficiency have their weaknesses. Thus, a combination of the two processes is necessary. Career development is that combination. Career development is a process which enables employees to develop beyond the fundamental skills, competencies, knowledge, and attitudes required for their present job assignments. It is a process which is focused on improving organizational efficiency while relying on the development of the individual employee. Performance improvement practitioners can provide learning activities which improve employees' skills, competencies, knowledge, and attitudes by viewing improved organizational efficiency from an individual (micro) perspective while remaining dedicated to the overall enhancement of the organization (macro). However, many performance improvement practitioners fail to see the connection between training and overall organizational development. These terms are often used interchangeably which leads to the increased confusion and misapplication of each process. It is necessary, therefore, to arrive at an acceptable definition in order to communicate effectively the intent as well as the similarities and differences of the processes. This will enable performance improvement practitioners to better understand the commonalities of each process. It is this understanding that serves as the basis of this article and the forthcoming recommendations.     

Funding Government Training Schemes: Mechanisms and Consequences

Abstract

This paper follows the path of public funds allocated to private institutions‐‐Training and Enterprise Councils (TECs)‐‐which, in turn, fund the delivery of Youth Training (soon to be replaced by Youth Credits) and Training for Work. The paper argues that the terms on which funds are allocated to TECs offers little financial incentive for TECs to support high cost, high quality training since their success and cost effectiveness is measured by indicators which take no account of the type of training provided, to whom it is provided and by whom. A TECs performance rating can only be improved if it devotes more of its resources to support for the cheapest, easiest and quickest routes to the government's set of output measures. However, this will do little to cure Britain's well‐known deficiencies in intermediate skills in occupations in which it is costly and lengthy to train.     

Computer: The use of computer‐mediated communication in training students in group problem‐solving and decision‐making techniques

Abstract

A cost‐effectiveness analysis of training Army National Guard soldiers by audio teletraining technology was conducted. The trainees were n = 225 soldiers nationwide. About half of the trainees received training in a three‐week Unit Clerk Course through traditional residence training, and the remainder received the same instruction through audio teletraining. Objective performance data were collected from written tests on sixteen of the forty‐seven tasks taught. The percentage of trainees in the audio teletraining group that reached criterion (93%) on the first test was significantly higher (by statistical test) than the residence group (85%). In comparing costs, the audio teletraining group had lower training costs per trainee. Projected on an annual basis, the Army National Guard can save $292,404 per year through the use of audio teletraining for the Unit Clerk Course.     

Object‐Oriented Performance Improvement

In this paper, a framework to support an object‐oriented approach to performance analysis is described. The framework includes the use of collaboration, automation, visual modeling, and reusable repositories of analysis knowledge. The need for a new framework is related to the increasing concern with the cost effectiveness of student and employee development. Efforts to improve the return‐on‐investment in such development have been hindered by a craft orientation to the design and construction of learning and performance support materials. One solution to this problem has been to enhance the reuse of such materials. Rather than build every new system from scratch in a craft‐oriented manner, it is envisioned that systems will be constructed largely of standardized, reusable objects shared through Web‐based repositories. Currently, the main focus is on the technological framework necessary for an object‐based approach to learning system development. There appears to be little consideration of the changes in analysis and design thinking required for the move towards object‐based systems. Such systems should still be required to be directly linked to performance problems and opportunities at both the organizational and individual system levels.     

基于虚拟现实的动漫制作

从建立与现代企业生产需求相适应的培训及现场模拟体系出发,进行基于虚拟现实的安全培训动漫影片制作关键技术研究,开发出了安全培训动漫影片制作平台并进行了影片制作输出.实用制作及成片培训效果表明,该技术达到了预期研发目的,输出的培训影片提升了培训效率和效果.     

The distance learning mode of training teachers in Kenya: challenges,prospects, and suggested policy framework

Globally, distance learning has gained legitimacy as an effective mode for learning and training. This legitimacy has occurred as a result of, inter alia, its flexibility with respect to time, pace and entry requirements, affordability, cost‐effectiveness, and reputation for high quality. In Kenya, distance learning – although relatively new – is being promoted to attain the Millennium Development Goals, Universal Primary Education, and the Kenya Vision 2030 targets. The majority of those who enrol for distance learning are teachers. This paper examines the utility of a distance learning approach for training teachers in Kenya: its challenges, prospects, and the need for a policy framework. It critically interrogates the readiness of the providers, the learning and policy environments. The paper concludes that current dual‐mode providers do not meet the requirements of the defining features of distance learning, and offers specific quality assuring policy directions.     

Career Development; The Linkage Between Training and Organizational Development Originally published in 1989, PIQ 2.1

Improved organizational efficiency can only be accomplished if the employees within an organization have developed the skills, competencies, knowledge, and attitudes to perform at the highest possible levels. To a limited degree this can be accomplished via training, but employees develop the fundamental skills, competencies, knowledge, and attitudes related only to their current job. As a result, the organization is only indirectly impacted. This refers to the micro perspective of improved organizational efficiency. Organizational efficiency can also be accomplished through organizational development, but the individual employee is often overlooked during this process. Without employee commitment to improve efficiency, it will be short-lived. This approach refers to the macro perspective of improved organizational efficiency. Both the micro and macro perspectives of improved organizational efficiency have their weaknesses. Thus, a combination of the two processes is necessary. Career development is that combination. Career development is a process which enables employees to develop beyond the fundamental skills, competencies, knowledge, and attitudes required for their present job assignments. It is a process which is focused on improving organizational efficiency while relying on the development of the individual employee. Performance improvement practitioners can provide learning activities which improve employees' skills, competencies, knowledge, and attitudes by viewing improved organizational efficiency from, an individual (micro) perspective while remaining dedicated to the overall enhancement of the organization (macro). However, many performance improvement practitioners fail to see the connection between training and overall organizational development. These terms are often used interchangeably which leads to the increased confusion and misapplication of each process. It is necessary, therefore, to arrive at an acceptable definition in order to communicate effectively the intent as well as the similarities and differences of the processes. This will enable performance improvement practitioners to better understand the commonalities of each process. It is this understanding that serves as the basis of this article and the forthcoming recommendations.     

DIRECT BEHAVIORAL CONSULTATION: EFFECTS ON TEACHERS’ PRAISE AND STUDENT DISRUPTIVE BEHAVIOR

Direct behavioral consultation is an extension of traditional behavioral consultation and focuses on assessment and training in the classroom during ongoing classroom activities. This study evaluated direct behavioral consultation services in two elementary alternative classrooms referred following a program evaluation in which data suggested behavior‐specific praise was not being delivered at a desired level. A multiple baseline design across classrooms was used to evaluate indirect and direct training procedures for increasing teachers’ implementation of behavior‐specific praise. Results indicated that indirect training did not result in substantial improvements in teachers’ use of behavior‐specific praise. Direct training procedures resulted in immediate increases in behavior‐specific praise for both teachers, but only one teacher maintained those increases immediately following training. For the other teacher, when performance feedback was added to direct training, increases in behavior‐specific praise were maintained. Data are also presented regarding student disruptive behavior. Results are discussed in terms of direct behavioral consultation theory and practice for educational settings.     

The Effects of Trainee Characteristics on Training Effectiveness in Improving Organizational Performance

Despite the existence of many comprehensive and user‐friendly guides to evaluate training programs, most practitioners have had difficulty assessing training effects on corporate outcomes. Research revealed that trainees, organizational, and training‐related factors might influence the effectiveness of training in terms of organizational performance. The current study examines the effects of trainee characteristics among those factors, specifically what kind of trainee characteristics could affect training effectiveness in terms of job performance in a company. In addition, telephone surveys were conducted to aid in understanding of the reasons for leaving and high turnover of company personnel. The current and terminated employees who went through training programs in the company differed when considering previous sales experience in years and type of sales experience as trainee characteristics factors, with those still employed having a statistically higher average number of years and a higher rated type of sales experience. The current employees demonstrated higher job performance. The implications of these results on the attainment of training effectiveness as well as the selection decisions in the organization are discussed.     

An Evaluation Tool for Reflective Activities in Trades Training: Gilbert's Worthy Performance Theorem Reformulated

Despite pervasive advocacy, there is little empirical support for reflective activities' effect on performance. This has made performance‐focused professionals reluctant to incorporate reflective activities into training interventions, particularly in industries with performance standards for ensuring stakeholder welfare, such as in the building and construction trades. In this article, I present an evaluation tool based on Gilbert's worthy performance theorem for identifying how reflective activities affect performance in trades training. I first outline the requirements for constructing an effective evaluation tool for trades training and next detail the tool's structure. I conclude by discussing its potential applications and implications for research and practice.     

Implementing an ROI Measurement Process at Dell Computer

This return-on-investment (ROI) evaluation study determined the business impact of the sales negotiation training course to Dell Computer Corporation. The specific metrics used include profit margin and margin attainment percentage, units sold and attainment percentage, total revenue and attainment percentage. The study applied a five-step ROI measurement process used at Dell Computer. The process steps include Plan-Develop-Analyze-Communicate-Leverage. Using this process, this particular case study involved 57 participants who attended the Dell Situational Negotiation course and an equivalent control group. To collect data, the corporate sales information database was used to compare pre- and post-training metrics for both training and control groups. A cost benefit analysis was also conducted to determine ROI of the training. Among the key findings of the study are:

• •These results have to be communicated to executive and management staffs for them to recommend the training to those who have not yet attended.
• •Coaching and reinforcement efforts for the course have to be stepped up so that these improvements can be sustained.
• •Due to the significant results of this study, the course should be included in development plans of sales representatives, account executives, sales managers, and regional sales managers.
• •It is suggested that a follow-up study be done to identify specific negotiation skills and tools that are being used on the job.

     

Cost‐effectiveness of online teacher training

This study aims to compare the cost‐effectiveness of an online teacher training method with a face‐to‐face training method in teaching ‘ICT integration in the school curriculum’. In addition, the study explores the possibilities of a school‐based voluntary training method in supporting other approaches to ICT teacher training. The analyses of various quantitative and qualitative data showed that online teacher training was more cost‐effective than face‐to‐face teacher training, mainly due to the lower opportunity cost of the participants. The voluntary teacher training revealed the possibility of providing cost‐effective training, especially in the actual application of ICT in classrooms. Based on the results of the study, factors affecting cost‐effectiveness of the teacher training approaches are discussed and further research areas suggested.     

Dimensions of Secondary School Effectiveness: Comparative Analyses Across Regions

Recent UK government policy implementing new systems of evaluation and accountability have highlighted the use of performance data to inform judgements about secondary schools and stimulate school improvement. However, these developments have been informed by a relatively small number of research studies addressing the methodology of measuring school effectiveness, and often employing limited or incomplete datasets. This paper reports the findings of an Economic Social Research Council (ESRC) funded study that employs 6 extensive and detailed regional datasets (drawn from Lancashire, London, Jersey, Scotland, the Netherlands and England as a whole). The study aims to provide new evidence to assist school staff, policy-makers and academics in understanding the multi-faceted nature of school effectiveness and the need to evaluate school performance in detail. The objectives were to investigate 1) the optimal models for measuring secondary school effectiveness across a range of outcomes in the UK and abroad; 2) the extent of regional differences in the results; 3) the definition of the underlying dimension(s) of school effectiveness across different regional and policy contexts. The findings show that at least 4 dimensions of secondary school effectiveness can be defined, specifically in terms of different outcomes, pupil groups, pupil cohorts and curriculum stages. In addition regional differences appear to exist in the size and impact of school effects, and these are mirrored by differences in regional context in terms of pupil selection. In conclusion it is argued that effectiveness at different levels of the education system (e.g., individual pupils; departments; whole school; region and nationally), as well as interactions between levels, needs to be continually monitored in order to map out the boundaries of school effectiveness and how these change over time. The findings are discussed in relation to developing a value added framework for school evaluation in the UK.     

Formative Evaluation of Computer‐Based Training For a University Financial System

This article describes a formative evaluation of a one‐day introductory computer‐based training (CBT) course for a new on‐line financial and purchasing system at a large public university. The purposes of the evaluation were to evaluate the effectiveness of the training and to identify appropriate revisions and incorporate them into the training program. Participants were 78 university employees who were likely future users of the financial and purchasing system. The mean score on an on‐line performance posttest that simulated real‐work tasks was 94%, and the mean on a 40‐item knowledge posttest covering the CBT content was 74%. Learner attitudes toward the course were positive, averaging 4.4 on a 5‐point scale. Formative evaluation of the program resulted in revisions that had the potential to improve its effectiveness and provided evidence of the value of ongoing formative evaluation of workplace training.