A Written Knowledge Test for Postgraduate Medical Students: Reliability in Relation to Different Educational Goals

ABSTRACT

The use of written knowledge tests in (medical) education is widespread. Only few of them are thoroughly validated. Usually, validity studies are restricted to establishing ‘face‐validity’, the apparent similarity between test‐material and real life problems. Reliability studies are usually restricted to estimation of the coefficient alpha, representing the reproducibility of rank‐ordering of students at repeated test administration. This study addresses reliability from a broader perspective, using generalizability theory. The approach enables faculty to gain insight into the suitability of the test to serve different educational goals.

A written knowledge test was examined, applied in postgraduate training for general practice in the Netherlands. Test‐reliability was approached from different perspectives: the norm‐oriented perspective, aiming at rank‐ordering (groups of) students, the domain‐oriented perspective, aiming at determining the absolute score level of (groups of) students and the decision‐oriented perspective, aiming at taking pass‐fail decisions.

Reliability estimates differed for the different perspectives. The implication of the results and feasible options to increase reliability are discussed.