| Abstract: | This study was undertaken to evaluate two procedures for establishing a minimum performance standard for the essay subtest of the National Teacher Examinations (NTE) Communication Skills test. Twenty public school teachers and 20 teacher educators were randomly assigned to either a "blind" or "informed" review panel. Both panels were directed to evaluate the same set of 12 sample essays. Those assigned to the informed panel were apprised of the scores previously awarded to each essay and were asked to decide upon the lowest score they would judge to be acceptable for a beginning teacher. Those assigned to the blind panel were not informed of the scores and were asked to judge whether each essay was at least minimally acceptable for a beginning teacher. Following their blind reviews, the scores were revealed to this group, and they were also asked to indicate the lowest score they would judge to be acceptable. No significant mean differences were observed for the standards associated with blind and informed reviews, but a significant mean difference was observed between the standards set by teachers and teacher educators; the former being more stringent. Based upon criteria other than mean differences, it was concluded that the preferred procedure for setting standards on essays should involve a blind review followed by an informed review. This strategy was subsequently implemented in a state-wide validation study in Virginia, and the results of this full-scale study are presented along with results reported for a study using informed reviews in another state. |
