Evaluation of the effectiveness of education: Some problems and prospects

Two case histories of efforts at educational evaluation—the Pennsylvania Evaluation of Quality Education and the Educational Testing Service Longitudinal Study of Head Start Programs—are considered here to illustrate a number of the major problems facing the field of evaluation. These include the perils of intuitionist holism and the matching dangers of psychometric trivialization; the strain between the hyper-enthusiasm of the politics of educational reform and the inherent restraint of scientific evidence regarding alternatives. A third case study of an attempt to institue Advanced Placement Tests in Art studies, designed to legitimate art as a hard subject, suggests a range of possible negative consequences that arise from attempting to introduce structured evaluation too early and too uniformly in an unstructured and variegated field. The results of a two-year effort at devising a model for measuring and comparing the effectivencess of educational systems are reported, indicating significant but instructive failure. A number of the major lessons learned during that effort are recited. indicating, overall, the importance of treating evaluation as a transactive process that must be weaned through varius evolutionary stages into full productivity.

---

Résumé L'auteur examine deux cas où on a fait des tentatives à l'évaluation éducationnelle; il traite de ces deux exemples—Pennsylvania Evaluation of Quality Education et Educational Testing Service Longitudinal Study of Head Start Programs—afin d'illustrer les problèmes principaux de l'évaluation. Ces problèmes incluent les dangers du holism et les dangers correspondants de la banalité psychométrique, et la tension entre l'enthusiasme de la politique de la réforme éducationnelle et la contrainte inhérente à l'évidence scientifique concernant les alternatives. Un troisième exemple d'une tentative à instituer des études qui s'appellentAdvanced Placement Tests in Art, et destinées à légitimer l'art comme un sujet concret, suggère qu'il y a la possibilité des conséquences négatives si on essaie d'introduire l'évaluation structurée trop tôt dans un sujet nonstructuré et varié. L'auteur fait un rapport sur les résultants d'un projet de deux ans le but duquel était d'arriver à un modèle pour mesurer et mettre en conmparaison l'efficacité des systèmes éducationnels; ces résultats montrent un échec significatif mais instructif. On relate les leçons principales de cet effort-là qui indiquent qu'il faut considérer l'évaluation comme un procès transactif; l'évaluation doit subir plusieurs phases d'évolution avant d'atteindre sa pleine productivité.

---

---

Visiting Research Sociologist, Educational Testing Service

---

This paper was deliverd as a seminar to the General Staff of Educational Testing Service.     

The Impact of Two- and Three-Generational Black Family Structure on Perceived Family Climate

This study examined the effect of black family structure on the perceived social climate as measured by the Family Environment Scale. The 64 black families were selected to fill the four cells of a 2 × 2 factorial design of 1- versus 2-parent black families by grandmother residing in the home of the family versus living 10 or more miles away. The family climate was significantly different among the four family structures and between this sample and the normative sample of Moos and his associates. No significant differences were found among the family structures on their Family Incongruence scores. The subscale scores of this black sample were similar to the scores of other black and partly black samples. These results suggest that black families are not a homogeneous group, but rather different black family structures are associated with different family climates. Further research is needed to understand the impact family climate has on child-rearing practices and child outcomes within these diverse family structures.     

Longitudinal comparisons of wechsler scales in educable mentally handicapped children and adults

The change in IQ scores over an extended period of time (approximately 17 years) of subjects who had been placed in classes for the Educable Mentally Handicapped was investigated. Analysis indicated that a significant difference existed between WISC scores administered at the time of initial placement (mean age, 11–9) and WAIS scores administered at a later date (mean age, 17–6). In addition, although a difference was revealed with respect to WAIS scores when the students were divided into two groups based upon initial WISC scores (⩽70 vs. >70), it was determined that the change was in a uniform fashion over time; i.e., neither regression nor interaction effects were detected. A final analysis revealed that those students who were located for a follow-up testing approximately 10 years later (mean age 28–8) continued to show a significant increase in IQ scores. The importance placed upon IQ scores for initial placement of students in EMH programs appears to remain suspect.     

Some Conceptions are Difficult to Change: One Middle School Mathematics Teacher's Struggle

A framework for investigating approaches to mathematics teaching in light of recent reforms is described. This framework is used to explore the struggles of a veteran middle school mathematics teacher as he attempted to modify his practices. The report was developed during a two-year period during which the teacher was interviewed and observed regularly. The teacher was successful in broadening some aspects of his pedagogical thinking and practice. For example, he reformed his teaching to emphasize important mathematical concepts. However, at least in part because of his views of pedagogical authority, he was unable to let his students explore ideas, solve open-ended problems, and debate intended meanings of mathematical ideas and solutions on a regular basis. This revised version was published online in July 2006 with corrections to the Cover Date.