Measurement of teacher performance: A study in instrument development

The procedures used to produce a research-based teaching evaluation system, containing low-inference indicators of effective and ineffective teacher behavior, included instrument development, content validation, and field testing. An extensive reliability study produced estimates for three types of consistency: intercoder agreement (r = 0.85), stability across teaching situations r = 0.86, and discriminant reliability among teachers (r = 0.79). The norming component of the study conducted in 45 schools, generated over 1200 observations of current teacher practice at all grade levels and in various subject areas. The results indicated the substantially generic nature of these behaviors; only grade level and instructional method (interactive, lecture, and independent seatwork) produced meaningful differences among groups of teachers. In addition, the average teacher used most of the effective behaviors (70%), and a few of the ineffective behaviors (8%) during observation. These results support the instrument's value in teacher training, remediation, and evaluation.     

Contextualizing the Interpretation of Reliability Data

What is the relationship between the context and substantive meaning of a measure and the assessment of the reliability of the measure? Does the interpretation of a reliability coefficient depend on the nature of the measure? How does this approach to reliability relate to alternative assessments?     

Establishing reliability of biomechanical data using univariate and multivariate approaches

The purpose of this study was to use generalizability theory with both univariate and multivariate approaches to examine reliability of total body center of mass (CM) values calculated from cinematographical data. Twenty-eight college-aged male volunteers were filmed by a LOCAM camera at 100 fps while performing the basic locomotion skill of walking. Film analysis was conducted on each subject using six frames of film depicting a one-stride walking cycle consisting of right heel strike, right foot flat, left toe-off, left heel strike, left foot flat, and right toe-off. Nineteen segmental endpoints and a reference point were digitized by three experienced plotters. The digitizing sequence was replicated three times by each plotter. A FORTRAN program calculated nine CM values (three plotters by three repetitions) for each subject filmed in each of the six positions of the stride. The x- and y-coordinates of the CM values were the dependent variables analyzed by fully crossed 3-way univariate and multivariate ANOVAs (subjects by plotters by repetitions). All measurement facets were considered to be random. Results indicate that there was very little repetition error but considerable interplotter error for most frames. Phi-coefficients for x- and y-coordinates, separately, fluctuated across frames. The univariate values for the x-coordinates were similar but slightly less than the multivariate values. The Phi-coefficients for Y, however, were considerably lower than the multivariate values. The multivariate Phi-coefficients for generalizing over three plotters and three repetitions ranged from .82 to .90.(ABSTRACT TRUNCATED AT 250 WORDS)