Extending participation in standard setting: an online judging proposal

In order for standard setting to retain public confidence, it will be argued there are two important requirements. One is that the judges’ allocation of students to performance bands would yield results broadly consistent with the expectation of the wider educational community. Secondly, in the absence of any change in educational performance, that the percentages in the corresponding bands should be stable over time. It is argued that the use of a small team of judges makes it more difficult to satisfy these conditions. However the cost and logistics of organizing a larger number of judges in the time-pressured atmosphere of public examining can lead to sub-optimal standard setting. Two parallel systems of awarding performance bands are empirically compared, one based on teams of six judges, the other based on a population of teachers. It is shown that the latter system gives more stable results over time for the same large student population. A proposal is outlined for extending participation in standard setting through the web-based presentation of materials and the capturing of cutscores from a population of teachers.