Differences in expert practice: a case from formative evaluation

A clear case has been made in the literature regarding the role of the learner in formative evaluation of instructional materials. In contrast, research on the role of the expert has been limited. This paper describes formative evaluation by instructional designers and subject matter experts, highlighting their task interpretation, their focus on text features, and their strategies. Content analysis of protocols based on think-aloud data suggest that instructional designers take on the role of a generalist, use a comparative method of review, and appear to be directed by the heuristics of the Instructional Systems Design Model. In contrast, subject matter experts approach the task as a specialist, use a sequential method of review, and seem to be directed by the domain knowledge. These differences lead to the identification of different problems and the generation of different revision recommendations. The implications of findings are discussed.