Systematic Self‐Evaluation in Graduate Medical Education

This study seeks to examine the utility and efficacy of the application of a systematic self‐evaluation process in a single graduate medical education residency program and to compare the results of this process to previous program self‐evaluation efforts. This case study examined the impact of using a systematic evaluation process, the “impact evaluation process,” as a means for annual program evaluation in an ophthalmology residency program sponsored by a large health care institution. Three years of outcome data (2011–2013) were analyzed and compared to prior evaluation efforts. Surveys with residency program stakeholders were conducted in 2011 and 2013. The findings suggest that over this time, the number of recommended actions for program improvement increased, were more qualitatively robust, and more action items were completed. Recommendations generated using the systematic evaluation process were actionable (specific and measurable) and aligned to program deficits.     

Using SAS® PROC CALIS for path analysis: An introduction

This article presents a straightforward approach for using PROC CALIS programs that will perform path analyses using observed variables. The step‐by‐step approach should help the beginner understand concepts in path analysis and how to use the SAS^® System's PROC CALIS procedure. The approach begins with the development of a program figure, a figure that graphically illustrates the researcher's theoretical model and identifies the parameters to be estimated in the model. A set of 14 rules guide this task. The article illustrates how to convert the program figure into a PROC CALIS program that will estimate parameters in the diagrammed model and how to review the output of PROC CALIS (e.g., residuals, chi‐square test, fit indices) to assess the fit of the model to the data.     

Impaired Students: Applying the Therapeutic Process Model to Graduate Training Programs

This article updates and adds to the literature on graduate trainee impairment provided by L. Forrest, N. Elman, S. Gizara, and T. Vacha‐Haase (1999) in their comprehensive literature review. Based on recommendations by Forrest et al., a program model for dealing with impaired trainees is proposed. The 5‐step therapeutic process model includes (a) informed consent, (b) intake and assessment, (c) evaluation, (d) treatment planning and follow‐up, and (e) termination. Each step is explained within the context of graduate training programs, and key points to consider are discussed.     

Using Class Climate Scales to Evaluate Gifted Inservice Programs

This study represents a novel approach to evaluating the effectiveness of an inservice program designed to help teachers enrich classroom instruction to make it more appropriate for gifted students. Results indicate that the number of inservice courses taken by teachers over a two and one‐half year period was associated with increased emphasis in their classes on higher‐level thinking skills and illustrates the use of class climate scales as a tool for program evaluation.     

Dilemmas of Participatory Enquiry: a case study of method‐in‐action

The purpose of this paper is to problematize the notion of participatory enquiry as a legitimate form of educational evaluation. Using a case study of a new teacher education program the authors describe how they negotiated and enacted an alternative evaluation methodology within a traditional context. The evaluation process was intended to actively engage faculty members and students, as well as representatives of the educational system and the broader community, in self‐reflection and self‐evaluation of their own program. The authors then attempt to turn the methodology back on itself by critically analyzing their own participatory process in terms of a number of dilemmas that arose during the three‐year course of the evaluation. These dilemmas were addressed directly as political issues of method as a means of eliciting discussion and debate about the appropriateness of various forms of enquiry in the assessment and evaluation of higher education.     

IDENTIFICATION AND LONGITUDINAL STUDY OF DIFFERENT FORMS OF GIFTEDNESS IN OLDER ADOLESCENTS

This study involved an adaptation and evaluation of the Munich differential diagnostic instrument for the identification of gifted and highly gifted schoolchildren (KFT). An analysis was carried out of the relationship between various forms of giftedness and actual performance. A two step procedure was applied to select gifted students using a Teachers' Checklist, the KFT and tests of creativity. About 70 gifted youngsters were selected from 600 students. The label “gifted” was used for the top 6%‐10%, “highly gifted” for the top 3%–5% and “extremely gifted” for the highest scoring members of the group. The control group (70 unselected students) was investigated with the same methods. There were clear differences between the gifted, highly gifted and extremely gifted students in every domain of giftedness. Different forms of giftedness developed in the students during the three year period.     

USE OF CRITICAL INCIDENTS FOR PLANNING,IMPLEMENTING, AND EVALUATING A TRAINING PROGRAM FOR FOSTER GRANDPARENTS

The authors report the planning, implementation, and evaluation of a training program for foster grandparents employed at a residential diagnostic and treatment center for disturbed children and youth. The training needs were assessed by task and performance analysis procedures, including the use of critical incident technique. A scoring procedure derived values for various responses to a set of problem situations presented to each foster grandparent. The training objectives focused on the role of the foster grandparents in helping their charges to develop socially acceptable behavior. A cognitive problem‐solving strategy was taught by use primarily of role‐modeling and role‐playing methods. The effectiveness of training was evaluated by comparing pretraining scores on a problem situation test with posttraining scores. Foster grandparents who had been highly rated by supervisors on an independent criterion improved with training; the others did not.     

Utilization of an Accelerated Queso Fresco Recipe to Teach Concepts of Food Science in a Didactic Program in Dietetics

Students in the Didactic Program in Dietetics (DPD) at the University of Maine were exposed to the cheese‐making process, within a lab setting of two hours, utilizing an accelerated recipe for a Queso Fresco‐style cheese. The purpose of this project was to provide students with a novel, hands‐on learning experience, which covered concepts of coagulating milk with enzymes and the treatment of curds during the cheese‐making process. The cheese recipe was tested three times by faculty and a teaching assistant to ensure standard results. The recipe, directions and procedures were adjusted and recorded. A standardized recipe and procedure were created for use in the student laboratory. Comparison of knowledge pre‐ and post‐experiment were analyzed using the Wilcoxon Signed‐Rank Test using Statistical Package for Social Sciences (SPSS) version 23.0. Thirty‐four students participated in the laboratory and when surveyed, expressed significant improvement in their understanding of the cheese‐making process (p <0.001). Students also rated themselves on a six‐step confidence scale from “not confident at all” to “very confident” in their understanding and ability in cheese‐making with 70.5 % of the 34 students participating reporting a three‐step increase in their confidence level. Students also completed a sensory evaluation of the recipe, utilizing a 9‐point Hedonic scale (1‐Dislike Extremely to 9‐Like Extremely), with over 88 % of students giving the product a positive overall acceptability rating. Through experience gained during the cheese‐making laboratory, students were able to apply concepts in food preparation, food science and sensory evaluation, all part of the curriculum content requirements of a DPD.     

Program evaluation of a special education day school for conduct problem adolescents

A procedure for program evaluation of a special education day school is described, and empirical evidence of its application in a school for conduct problem adolescents is reported. Use of the procedure enables a program evaluator to: (a) identify the priority evaluation information needs of a school principal and school staff, (b) involve those persons in evaluation design and implementation, and (c) determine the utility of the evaluation for program decision-making purposes.     

Psychoeducational Support for Parents of Children with Intellectual Disability: An Outcome Study

The pressures frequently encountered by the parents of children with chronic conditions and the concomitant impact on family life have been well‐documented. Family‐focussed interventions have been advocated to address parents’ difficulties. The present evaluation study was undertaken to ascertain long‐term outcomes of the psychoeducational support program, Caring for Parent Caregivers. The aim of the program is to empower the individual and thereby strengthen family resources. Quantitative and qualitative analytical procedures, which included the written evaluations of treatment subjects, were employed to examine program effectiveness. For the fathers and mothers of children with disabilities, psychological health and well‐being was assessed on the General Health Questionnaire, using a pretest and follow‐up control group design. Results showed a statistically significant difference between groups, 12 months after treatment, with program participants displaying less emotional distress than control group subjects. The positive direction of behavioural and attitudinal outcomes, which emerged from evaluation of the small group intervention, were similar for both mothers and fathers. Overall satisfaction with the program was very high, with useful recommendations for program development and for future research and practice being identified.     

The concept and practice of second-level screening

The purpose of the study was to substantiate the premise that second-level screening of preschool children failing the Denver Developmental Screening Test (DDST) is an effective, personnel- and cost-efficient procedure for identifying preschool handicapped children. The sample included 280 subjects, age three to five, enrolled in a Head Start program in Southern Illinois. The DDST was administered to these subjects by staff members of the Head Start Center. Children who failed the DDST received a second-level screening to determine whether a full-scale psychological evaluation was in order. Results demonstrate that the second-level screening procedure is an effective method for ferreting out children in need of full-scale psychological evaluation to determine area(s) of handicapping condition and generation of appropriate services.     

Skill Acquisition and Reflection‐based Decision Making in a Teaching Laboratory: an evaluative study

This paper continues the tradition of research, development and evaluation of teaching laboratories, and, at the same time, extends earlier research in two directions: by placing more emphasis on the improvement of a reflection‐based decision‐making ability in becoming a teacher (beyond effective skill acquisition), and by concentrating on the improvement of presentation skills, an area neglected in teacher training. Based on research on teaching and teacher training, a training program was developed for enhancing the trainees’ nonverbal expressiveness in presentations that, by fostering reflection‐based decision making through experimenting with presentation skills in a laboratory with feedback and discussions, also improves the clarity, interest and social climate of presentations. The evaluation of the most recent Version of this program reveals that significant gains could be achieved not only in nonverbal expressiveness as an indicator for successful skill acquisition, but also in the global aspects of successful lecturing indicating that the program was also effective for the improvement of the trainees’ reflection‐based decision making.     

A pilot study of the Australasian Readwell program

Abstract

The “Readwell” program is an Australian‐designed approach to remediation of children's reading difficulties. Although used in most Australian states, the program has not been evaluated. This quasi‐experimental pilot study reports on a first phase of an evaluation of the program. It involves a preliminary examination of the program with a group of 18 grade six and seven students with significant reading delays. It was conducted over a ten‐week period during which students received 45‐minute group instruction with the Readwell program each day. In addition, interviews with the students and their teachers were conducted to examine perceived outcomes of the program. Surveys with selected parents of students who had completed the program 12 months previously were also conducted to gauge the endurance of students’ gains in reading competence. Some comparisons with the literature reporting on Reading Recovery programs are also included and some initial conclusions are draw in relation to the place of the Readwell program in remediation practices for students with a reading difficulty.     

Mandatory voice: Implementation of a district‐led student‐voice program in urban high schools

This qualitative case study explores how urban high schools implement a district‐initiated student‐voice program, referred to as Student Advisory Committees, intended to improve school climate. The study uses an implementation science framework to analyze interview data from principals and advisors in 22 urban high schools. An iterative, top–down thematic analytic procedure was used for analysis and interpretation of the data. The results suggest wide variation in how the program was implemented in terms of fidelity, dosage, quality, program reach, and adaptation, with contextual school factors—namely, principal and school characteristics—that help to explain this variation. In addition, the findings indicate that these contextual factors had a significant influence on how the school district implemented the student advisory committees. The authors discuss the implications for program implementation in the context of urban school districts and for school psychologists.     

Control Group Methods for HPT Program Evaluation and Measurement

This research contributes to the methodologies in HPT program evaluation and measurement that are fairly lacking to date. First, a theoretical foundation for a control group is established based on a brief review of control group applications in various fields. Then, four types of control groups applicable to HPT program evaluation and measurement are defined and classified, and threats to internal and external validity in control group applications are explored. Lastly, four evaluation and measurement scenarios are presented for an E‐learning program to demonstrate the applicability of the control group methods for HPT program evaluation and ROI measurement.     

ASSESSMENT OF ENTERING STUDENT WRITING SKILL IN THE COMMUNITY COLLEGE

This article describes an assessment of writing skills (writing sample, objective test, and self‐assessment) made by one community college. Information was gathered to design an entry‐level assessment procedure for placement in English composition and developmental writing courses. Comparing the three approaches, each of which measured different aspects of writing ability, showed that the best predictor of grade in an English composition course was the objective test. However, the statistical results of the study form only one component of an entry‐level assessment program. It may be that a writing sample is needed as part of entry‐level assessment because it points out the value of writing skill in attempting college‐credit courses.     

Testing the Engagement Theory of Program Quality in CACREP‐Accredited Counselor Education Programs

This study examined the engagement theory of program quality ( Haworth & Conrad, 1997 ), which highlights positive student learning outcomes that result from stakeholder involvement in program evaluation within master's‐level graduate programs. A total of 481 master's‐level counseling students and 63 faculty members, representing 68 Council for Accreditation of Counseling and Related Educational Programs (CACREP)–accredited counselor education programs, participated in the study. Findings reveal that engagement theory is a potentially useful quality assessment resource for CACREP‐accredited programs in their efforts at enhancing and sustaining program quality.