Aspects of ‘relevance’ in the alignment of curriculum with educational standards

With over 60,000 US K-12 science and mathematics education standards and a rapid proliferation of Web-enabled curriculum, retrieving curriculum that aligns with the standards to which teachers must teach is a key objective for educational digital libraries. However, previous studies of such alignment use single-dimensional and binary measures of the alignment concept. As a consequence, they suffer from low inter-rater reliability (IRR), with experts agreeing about alignments only some 20–40% of the time. We present the results of an experiment in which the alignment variable was operationalized using the Saracevic model of relevance ‘clues’ taken from the everyday practice of K-12 teaching. Results show high IRR across all clues with IRR on several specific alignment dimensions significantly higher than on overall alignment. In addition, a model of overall alignment is derived and estimated. The structure and explanatory power of the model as well as the relationships between alignment clues differ significantly between alignments of curriculum found by users themselves and curriculum found by others. These results illustrate the usefulness of clue-based relevance measures for information retrieval and have important consequences for both the formulation of automated retrieval mechanisms and the construction of a gold standard or benchmark set of standard-curriculum alignments.