The application of pairwise comparisons to form scaled exemplars as a basis for setting and exemplifying standards in teacher education

ABSTRACT

This paper details a two-stage process for standard setting using an extended pairwise comparison method. It describes the application of a new process of establishing a set of scaled and ordered performances as a basis for exemplifying standards. In stage one, the method of pairwise comparison was used to generate data from pairwise judgements made at the criterion level and overall grade. In stage two, judges identified where the pass/fail boundary lay in relation to a selected subset of ordered performances generated from stage one. The combined evidence from the two-stage process demonstrates the method’s application to establish exemplified standards using on-balance judgements across multiple criteria. The approach combines expert judgement and statistical data to provide a basis for comparable cross-institution assessments against standards in the case of a summative assessment of professional competence in teacher education. It may be relevant to any discipline with professional standards and complex performance assessments.