An Experimental Test of Student Verbal Reports and Teacher Evaluations as a Source of Validity Evidence for Test Development

The Standards for Educational and Psychological Testing indicate that test instructions, and by extension item objectives, presented to examinees should be sufficiently clear and detailed to help ensure that they respond as developers intend them to respond (Standard 3.20; AERA, APA, & NCME, 1999 American Educational Research Association (AERA), American Psychological Association (APA), National Council on Measurement in Education (NCME). 1999. Standards for educational and psychological testing, Washington, DC: Author.  [Google Scholar]). The present study investigates the use of verbal reports, one of many sources of evidence for validity arguments, as a way to evaluate the content clarity of 30 items from a large-scale science assessment. Student reports were used to edit items and create a student-modified test form. Evaluations from expert preservice teachers were used to edit the items and create an expert-modified test form. Both experimental forms, along with the original set of 30 items, were then randomly assigned to a sample of 264 examinees. Hierarchical regression analyses indicated that examinee performance on the student-modified and expert-modified forms was similar relative to performance on the original test items. Item statistics indicated that student-modified test items were equally difficult and discriminating as expert-modified test items. The implications of using student and teacher evaluations are discussed for informing test development.     

The Effect of Hints and Model Answers in a Student-Controlled Problem-Solving Program for Secondary Physics Education

Many students experience difficulties in solving applied physics problems. Most programs that want students to improve problem-solving skills are concerned with the development of content knowledge. Physhint is an example of a student-controlled computer program that supports students in developing their strategic knowledge in combination with support at the level of content knowledge. The program allows students to ask for hints related to the episodes involved in solving a problem. The main question to be answered in this article is whether the program succeeds in improving strategic knowledge by allowing for more effective practice time for the student (practice effect) and/or by focusing on the systematic use of the available help (systematic hint-use effect). Analysis of qualitative data from an experimental study conducted previously show that both the expected effectiveness of practice and the systematic use of episode-related hints account for the enhanced problem-solving skills of students.