What is Program Evaluation? Generating Knowledge for Improvement

In this introductory article, we discuss the nature of Program Evaluation, describing the concepts that underlie our formal and informal evaluative efforts. Program Evaluation, like any deliberate inquiry process, is about learning. The process explicates program purposes, activities, and outcomes and generates knowledge about their merit and worth. This knowledge can inform planning and lead to program improvement. We present and discuss various definitions of Program Evaluation, focussing on its purposes and uses. We also provide an overview of the inquiry process, grounding the search for merit and worth in the American Evaluation Association's Guiding Principles for Evaluators. Because program evaluations are typically conducted to inform decision makers, we discuss aspects of professional practise that contribute to the use of an evaluation. This chapter draws heavily on previous work by Sharon Rallis and Gretchen Rossman; see the following references. Rallis, S.F. and Rossman, G.B., “Mixed Methods in Evaluation Contexts: A Pragmatic framework”, in A. Tashakkori and C. Teddlie (eds.),Handbook of Mixed Methods in the Social and Behavioral Sciences (Thousand Oaks: Sage, 2002). Rallis, S.F. and Rossman, G.B., “Communicating Quality and Qualities: The Role of the Evaluator as Critical Friend”, in A.P. Benson, D.M. Hinn and C. Lloyd (eds.),Visions of Quality: How Evaluators Define, Understand, and Represent Program Quality (Oxford: JAI Press, 2001), pp. 107–120. Rallis, S.F. and Rossman, G.B. “Dialogue for Learning: Evaluator as Critical Friend”, in R. Hopson and M.Q. Patton (eds.),How and Why Language Matters in Evaluation, New Directions for Evaluation, 86 (San Francisco: Jossey-Bass, 2000). Rossman, G.B. and Rallis, S.F.,Learning in the Field: An Introduction to Qualitive Research, 2nd edition (Thousand Oaks: Sage, 2003). Rossman, G.B. and Rallis, S.F., “Evaluation as Learning: Critical Inquiry and Use as Action”, in V.J. Caracelli and H. Preskill (eds.),The Expanding Scope of Evaluation Use, New Directions in Evaluation 88 (San Francisco: Jossey Bass, 2000).     

Creativity and the Design Process

This research examined the role of creativity in the process of designing human resource development (HRD) interventions. Both qualitative and quantitative methods were used to study the preferences and practices of ten experienced HRD practitioners. Results include a comparison of the subjects' creativity styles/preferences using the Myers‐Briggs Type Indicator and the Kirton Adaption Innovation Inventory as well as a model derived from the data, which describes the creative process used by the subjects. The phases include: 1) diagnosis, 2) immersion, 3) percolation, 4) “aha,” and 5) checking. This model focuses on how creativity is used to make design decisions. The design process used by the subjects in this research is similar to a generic creativity model. However, it includes an extensive preliminary diagnostic phase in which the HRD practitioner explores the business need, searches for the root cause of the problem, and establishes a collaborative relationship with stakeholders in the organization.     

Transfer of Learning From An Action Reflection Learning™ Program*

Learning transfer from training and development programs to the workplace is a central issue of concern for human resource development specialists. Action Reflection Learning™ (ARL™) is a form of action learning that places emphasis on the development of targeted competencies while teams of participants work on real problems which have been given to them by a sponsor. This article examines the kinds of learning and learning transfer that took place in an international food company that was striving to become a global organization. Data were collected through both field observation and formal interviews and analyzed by a team of researchers. Conclusions about the degree of transfer and the mechanisms and process supporting it are presented.     

Planning HRD Interventions; What,Why, and How

There is a large body of literature on the instructional design process but little information about the planning practices of Human Resource Development (HRD) practitioners, those who may be devising training or organization development interventions. This article describes qualitative research conducted on the planning processes of five experienced HRD practitioners. Results indicate that subjects had similar mental models of the planning process, portraying the general framework as: a) discerning the desired outcomes, b) understanding the current situation, and c) designing a process to meet the needs. They characterized the planning process as holistic and iterative, involving the use of scenarios, and paradoxical in nature. This paper will identify how the subjects' values, beliefs, and principles of practice influenced their planning. It will also describe techniques used by the subjects to enhance their planning, which involved activating internal resources (prior experience), seeking external resources (reading, talking to others), and tapping into their subconscious for insights or answers. Implications for education, practice, and further research on the planning practices of HRD practitioners will be proposed.     

A preliminary evaluation of the associations among functional performance tasks and quality of life in obese and healthy weight youth

The associations among lower extremity functional performance and quality of life in obese youth are unknown. The aims of this study were to compare lower extremity strength, lower extremity functional performance, and health related quality of life between obese and healthy-weight youth and evaluate the relationships between lower extremity performance and health related quality of life in obese youth. Twenty obese and 20 age and sex matched healthy-weight youth were recruited. Peak torque of the major lower extremity muscles were measured. Functional performance was measured with single leg hop and single leg balance tests. The Paediatric Quality of Life questionnaire's physical and psychosocial health subscales were used. Paired t-tests and multiple regression analyses were performed. Obese youth demonstrated decreased peak torque in all muscles measured (P < 0.05), poorer functional performance (P < 0.05), and worse physical health related quality of life (P < 0.05) compared to healthy-weight youth. Lower extremity functional performance was associated with aspects of quality of life in the obese group (P = 0.002), but not in the healthy-weight group (P < 0.05). These results may assist in encouraging best practices in the promotion of exercise, physical activity, and quality of life in obese youth.     

If It Doesn't Directly Affect You,You Don't Think About It’: a qualitative study of young people's environmental attitudes in two Australian cities

This article presents a range of findbigs from a qualitative study of the environmental attitudes of young people across their final 2 years of secondary school in the two Australian cities of Melbourne and Brisbane. Focus groups comprising the same 16‐ to 17‐year‐old students in 12 schools were interviewed twice, 12 months apart. Several minor differences were found in the attitudes of students between the two cities, but these pale alongside the common, indeed, overwhelming feelings of environmental concern mixed with frustration, cynicism and action paralysis that were reported. The ambivalence towards the environment that results, together with the individualistic frameworks for explaining environmental issues that were displayed, point to areas for renewed curriculum attention in secondary schools and directions for future research.     

Can a Multiple-Choice Exam Discriminate Between Masters and Nonniasters of Instructional Design?

This study was conducted to determine if a norm-referenced test designed to assess instructional design competency could be statistically validated (i.e., confirmed statistically to discriminate between known masters and known nonmasters of instructional design). The test was composed of items written to assess verified competencies required of instructional design professionals. A total of 257 respondents participated in the study over the course of three stages: initial item bank construction, item analysis to determine those items with discrimination power, and the concurrent validity calculation, including determination of the mastery cut-off score. Mean scores of five groups of respondents were analyzed in the final stage. Statistically significant differences were found among the Professional Masters, Education Graduate Students and Undergraduates, Noneducation Graduate Students, and Noneducation Undergraduates. The article concludes with a discussion of the role of such an instrument in conducting research in the field.     

Stability reliability for elementary-age students on the Woodcock-Johnson psychoeducational battery-Revised (achievement section) and the Kaufman Test of Educational Achievement

These two studies examined the stability reliability for the Woodcock-Johnson-Revised (WJ-R; Woodcock & Johnson, 1989) and the Kaufman Test of Educational Achievment (KTEA; Kaufman & Kaufman, 1985) with approximately a 2-week retest interval for elementary-age students. Results indicated that across grade levels, the Broad Reading Cluster for the WJ-R remained stable. Most correlations for the clusters for mathematics and written language as well as the subtests for reading, mathematics, and written language were less than .90. Correlations for all composites and subtests for the KTEA exceeded .90. These data illustrate the need for more specific information in test manuals on test-retest reliability in order to enable examiners to select the most reliable measures.