Teacher behavior under performance pay incentives

Over the last decade many districts implemented performance pay incentives to reward teachers for improving student achievement. Economic theory suggests that these programs could alter teacher work effort, cooperation, and retention. Because teachers can choose to work in a performance pay district that has characteristics correlated with teacher behavior, I use the distance between a teacher's undergraduate institution and the nearest performance pay district as an instrumental variable. Using data from the 2003 and 2007 waves of the Schools and Staffing Survey, I find that teachers respond to performance pay incentives by working fewer hours per week. Performance pay also decreases participation in unpaid cooperative school activities, while there is suggestive evidence that teacher turnover decreases. The treatment effects are heterogeneous; male teachers respond more positively than female teachers. In Florida, which restricts state performance pay funding to individual teachers, I find that work effort and teacher turnover increase.     

表现性评价在美国教师资格认定实践中的应用及其启示

自20世纪80年代起,美国佛罗里达州、乔治亚州、得克萨斯州等开始在专业教师资格认定中使用表现性评价。目前,少数州又将表现性评价应用于初级教师资格认定。表现性评价的应用增进了教师资格认定的真实性和预测效度。我国可以尝试在教师资格认定实践中应用表现性评价,也可以将表现性评价拓展应用到师范生教育实习以及在职教师专业发展领域,但同时要充分预见并积极应对可能面临的挑战。     

Threats to the validity of science performance assessments for English language learners

In recent years, at the same time that performance assessments in science have become more popular, the number of English language learners (ELLs) (i.e., students whose native language is other than English) served by the U.S. educational system has also increased rapidly. While the research base is growing in each of these areas independently, little attention has been paid to their intersection. This case study of the use of a science performance assessment with 96 ELLs in five high school science classes investigated the face, construct, and consequential validity of this intersection. Qualitative and quantitative data analyses showed that both teachers and students had an overall favorable response to the assessment, although students' English comprehension and expression skills were determining factors for certain items. While most responses were reliably scored, ELL spelling and syntax on certain responses were significant sources of error. The degree of specificity of teachers' guidance also significantly affected students' scores. Recommendations from this study include increasing the clarity of an assessment's design, allowing ELLs more time to complete assessments, and scoring by raters who are knowledgeable about typical patterns in written English for this student population. Furthermore, it is recommended that the use of performance assessments with ELLs be exploratory until such time as their validity and reliability with this population can be more adequately established. J Res Sci Teach 34: 721–743, 1997.     

The work sampling system: Reliability and validity of a performance assessment for young children

Performance assessment, an alternative approach to assessing students' achievements in school, refers to assessment methods that allow students to demonstrate their skills, knowledge, behavior, and accomplishments across a wide variety of classroom domains on multiple occasions. This article presents data concerning the reliability and validity of the Work Sampling System with 100 kindergarten-age children. A psychometric design was implemented in which children were enrolled in classrooms where the Work Sampling System was used and were also given individually-administered norm-referenced assessments in the fall and spring; in addition, their teachers completed a behavior rating scale in the spring. Results show that the Work Sampling checklist and summary report have very high internal and moderately high interrater reliability. The Work Sampling System accurately predicts performance on the norm-referenced achievement battery, even when the potential effects of gender, maturation (age), and initial ability are controlled. These data provide empirical support for the reliability and criterion validity of this performance assessment system as a measure of children's overall school achievement in kindergarten. The discussion covers issues raised by the study's design and by the use of performance assessment in general.     

The relationship of estimated learning potential to performance on a learning task and achievement

The Estimated Learning Potential (ELP), a score obtained through the System of Multicultural Pluralistic Assessment (SOMPA), is purported to be a more appropriate index of potential for socioculturally different children. Prior studies have suggested a lower correlation with academic achievement than that of the traditional IQ score. The validity of the ELP has, therefore, been questioned. This study attempted to determine whether the ELP or IQ score would be the better predictor of performance on a learning task for 31 black and 32 white EMR children. The learning task was presented in a test-teach-test paradigm. Results suggest that the ELP did not correlate more highly with learning task performance than did the IQ score. However, the ELP may be more differentially sensitive to declassifying black children as retarded. The validity and the social consequences of the use of the IQ, ELP, and performance, after teaching, on the learning task for classification decisions also are discussed.     

Effects of response options on the mathematics performance of secondary students with emotional or behavioral disorders

ABSTRACT

This study examined the effects of three response options (traditional responding, response cards, and response systems) on the mathematics performance, participation, and time on-task of secondary students with emotional or behavioral disorders (EBD). A three-way crossover design was utilized to evaluate the efficacy of response options in secondary mathematics classrooms. Thirty-three students with EBD attending an urban high school and their teachers served as participants. Results indicated that the use of response cards (white boards) or response systems (ActivResponders) significantly increased students’ mathematics performance, participation, and time on-task when compared to traditional responding. Further, the use of response cards resulted in significant increases in performance and response accuracy over the increases found when using response systems. Social validity data indicated that students and teachers felt they benefited from the use of response cards and systems. Limitations, discussions, and implications for practice and future research are presented.     

Using student performance data for improving individualized instructional units

Summary The improvement of the educational process is an ongoing activity. The examples presented have shown the data upon which revisions of PLAN modules were made and have shown student performance results on the revised materials. Presented as part of a symposium, PLAN in Operation—A Summary of Four Years’ Experience in the Evolution of an Educational System, for Divisions 5 and 15, American Psychological Association, Miami Beach, Florida, September 5, 1970. The work discussed in this paper was conducted while Mr. Rahmlow was associated with the American Institutes for Research.     

Criterion-related validity of the WRAT-R and K-TEA with teacher estimates of actual classroom academic performance

The validity of most psychological and educational tests is established using correlational procedures examining the linear relationship between performance on the two instruments. Concurrent validity developed in this manner is commonly viewed as verification of the acceptability of a test. Few studies exist examining the degree to which test performance covaries with real-life performance appraisals. This study examined the concurrent validity of the WRAT-R and the K-TEA with teacher estimates of actual classroom levels of performance in reading and mathematics. Participants were 134 third and fourth graders enrolled in a regular education setting. In addition, this study compared the test performance of average students on two widely used standardized educational achievement tests in order to determine whether the tests yielded significantly different performance estimates relative to grade level functioning.     

Validity inquiry process: Practical guidance for examining performance assessments and building a validity argument

Given the increased use of performance assessments (PAs) in higher education to evaluate achievement of learning outcomes, it is important to address the barriers related to ensuring quality for this type of assessment. This article presents a design-based research (DBR) study that resulted in the development of a Validity Inquiry Process (VIP). The study’s aim was to support faculty in examining the validity and reliability of the interpretation and use of results from locally developed PAs. DBR was determined to be an appropriate method because it is used to study interventions such as an instructional innovation, type of assessment, technology integration, or administrative activity (Anderson & Shattuck, 2012). The VIP provides a collection of instruments and utilizes a reflective practice approach integrating concepts of quality criteria and development of a validity argument as outlined in the literature (M.T. Kane, 2013; Linn, Baker, & Dunbar, 1991; Messick, 1994).     

Adult undergraduate students: Evaluating the appropriateness of a traditional model of academic performance

Adult learners (age 25 or older) now comprise approximately 40 percent of under-graduate enrollment. However, predictive models of undergraduate academic success are usually based on traditional young undergraduate students, presenting a problematic picture for the adult undergraduate population. Past research indicates that many older adult learners enter higher education from family backgrounds that tend to place them at an academic disadvantage, bring with them deficiencies in academic skills, and are less involved in academic and social aspects of campus life. Therefore, conventional wisdom would suggest that older students should have lower cumulative grade-point averages than younger students. However, past research on academic performance of adult undergraduates does not substantiate this prediction. This study focused on the validity of generalizing a traditional model of academic performance to older adult students. Results from this study indicate that a traditional model of academic performance prediction is inappropriate for use with older adult undergraduates. This research was supported in part by a grant from the Association for Continuing Higher Education.     

Reliability and validity of a mathematics performance assessment

This study evaluated the reliability and validity of a performance assessment designed to measure students' thinking and reasoning skills in mathematics. The QUASAR Cognitive Assessment Instrument (QCA1) was administered to over 1.700 sixth and seventh grade students of various ethnic backgrounds in six schools that are participating in the QUASAR project. The consistency of students' responses across tasks and the validity for inferences drawn from the scores on the assessment to the more broadly-defined construct domain were examined. The intertask consistency and the dimensionality of the assessment was assessed through the use of polychoric correlations and confirmatory factor analysis, and the generalizability of the derived scores was examined through the use of generalizability theory. The results from the confirmatory factor analysis indicate that a one-factor model fits the data for each of the four QCAI forms. The major findings from the generalizability studies (person x task and person x rater x task) indicate that, for each of the four forms, the person x task variance component accounts for the largest percentage of the total variability and the percentage of variance accounted for by the variance components that include the rater effect is negligible. The variance components that-include the rater effect were negligible. The generalizability and dependability coefficients for the person x task decision studies (n_t, = 9) range from .71-.84. These results indicate that the use of nine tasks may not be adequate for generalizing to the larger domain of mathematics for individual student level scores. The QUASAR project, however, is interested in assessing mathematics achievement at the program level not the student level; therefore, these coefficients are not alarmingly low.     

Structural and convergent validity of the homework performance questionnaire

Homework is a requirement for most school-age children, but research on the benefits and drawbacks of homework is limited by lack of psychometrically sound measurement of homework performance. This study examined the structural and convergent validity of scores from the newly developed Homework Performance Questionnaire – Teacher Scale (HPQ-T). Participants were 112 teachers of 224 students in six Illinois school districts. Common factor analysis with principal axis extraction and promax rotation was used for data analysis. Results revealed three salient factors: Parent support, student competence and homework completion. Subsequently, convergent validity of HPQ-T subscale scores with subscale scores from the Learning Behaviours Scale was examined. Findings suggest that the HPQ-T may potentially be a useful tool for improving research on homework and identifying strengths and weaknesses in student homework performance. However, modifications are recommended to optimise the utility of the scores.     

The impact of a rubric and friendship on peer assessment: Effects on construct validity,performance, and perceptions of fairness and comfort

Construct validity of peer assessment (PA) is important for PA application, yet difficult to achieve. The present study investigated the impact of an assessment rubric and friendship between the assessor and assessee on construct validity of PA. Two-hundred nine bachelor students participated: half of them assessed a peer's concept map with a rubric whereas the other half did not use a rubric. The results revealed a substantial reliability and construct validity for PA. All students over-score their peers’ performance, but students using a rubric were more valid. Moreover, when using a rubric a high level of friendship between assessor and assessee resulted in more over-scoring. Use of a rubric resulted in higher quality concept maps for peer and expert ratings.     

Self-confidence and performance goal orientation interactively predict performance in a reasoning test with accuracy feedback

This study takes an individual differences' perspective on performance feedback effects in psychometric testing. A total of 105 students in a mainstream secondary school in North East England undertook a cognitive ability test on two occasions. In one condition, students received item-specific accuracy feedback while in the other (standard condition) no feedback was provided. While accuracy feedback had, on average, no effect on test performance, differential effects were observed. Performance goal orientation and self-confidence moderated feedback effects on performance. The provision of accuracy feedback seemed to have performance optimising effects upon test takers with high performance goal orientation in combination with low self-confidence. The implications of these findings in relation to the validity of findings from psychological testing are discussed.     

Using digital technologies and contemporary psychometrics in the assessment of performance on complex practical tasks

There is a critical need for research into the use of digital technologies to support the assessment of performance on complex tasks in schools. This paper reports on a component of a pilot study aimed at investigating the use of digital forms of performance assessment, manageable within schools, with high levels of reliability and capable of being scaled up for statewide implementation. Student practical performances were represented in digital files, placed in an online repository and marked using both a standards‐reference and comparative pairs method. It was found that the short performance tasks exam was readily implemented and the work easily marked. The resulting scores were shown to be highly reliable and well correlated when compared with the other case studies and suitable for ranking students. However, the limitations of the tasks, time and scaffolding of the tasks reduced the study’s validity in assessing student capability in the course. Overall the results were encouraging and provide a basis for a larger and longer study that has commenced.     

Promoting academic satisfaction and performance: Building academic resilience through coping strategies

The aim of this article is twofold: (a) to validate the psychometric properties of the Academic Resilience Scale (AR‐S) in a Spanish university context; and (b) to test a model where different coping strategies are antecedents of AR, and where academic satisfaction and performance are its consequences. The studies were conducted with 185 (study 1) and 780 (study 2) students. In study 1, the construct validity, reliability, and convergent and divergent validity of the AR‐S were analyzed. The hypothesized model (study 2) was tested using structural equation modeling. The results confirmed the good psychometric properties of the AR‐S in the Spanish context, providing evidence of its validity. Furthermore, the distinctive role of different coping strategies in resilience was confirmed, as well as the relationship between resilience and academic satisfaction. No direct relationship between resilience and performance was found, highlighting the full mediation of satisfaction.     

Raven's test performance of sub-Saharan Africans: Average performance,psychometric properties,and the Flynn Effect

This paper presents a systematic review of published data on the performance of sub-Saharan Africans on Raven's Progressive Matrices. The specific goals were to estimate the average level of performance, to study the Flynn Effect in African samples, and to examine the psychometric meaning of Raven's test scores as measures of general intelligence. Convergent validity of the Raven's tests is found to be relatively poor, although reliability and predictive validity are comparable to western samples. Factor analyses indicate that the Raven's tests are relatively weak indicators of general intelligence among Africans, and often measure additional factors, besides general intelligence. The degree to which Raven's scores of Africans reflect levels of general intelligence is unknown. Average IQ of Africans is approximately 80 when compared to US norms. Raven's scores among African adults have shown secular increases over the years. It is concluded that the Flynn Effect has yet to take hold in sub-Saharan Africa.     

自始履行不能对合同效力影响的比较研究

在正确理解大陆法系和英美法系自始履行不能概念的基础上,对大陆法系各国、英美法系以及国际立法上自始履行不能对合同效力的不同影响作了探讨,并对其各自可能存在的弊端及立法意图做了剖析,指出在我国法律现代化进程中应如何处理好法的移植、法的发展的内部动力及法的改革问题。