An Alternative Multiple‐Choice Question Format to Guide Feedback Using Student Self‐Assessment of Knowledge

Management science professors who teach large classes often assess students with multiple‐choice questions (MCQs) because it is efficient. However, traditional MCQ formats are ill‐fitted for constructive feedback. We propose the reward for omission with confidence in knowledge (ROCK) format as an original formative assessment technique to help guide feedback associated with MCQs in an introductory undergraduate management science course. Our study contributes to theory by empirically showing that students can self‐assess their state of knowledge, signal it to the professor, and use proper answering options. In practice, ROCK is an easily implementable MCQ format that allows professors to gain information on student learning based on answers selected. ROCK identifies lack of knowledge or misinformation at both individual and collective levels thus providing opportunities for better feedback in class and during office hours. Limitations of the application of ROCK are also discussed.     

Students’ perceptions about evaluation and assessment in higher education: a review1

In educational contexts, understanding the student’s learning must take account of the student’s construction of reality. Reality as experienced by the student has an important additional value. This assumption also applies to a student’s perception of evaluation and assessment. Students’ study behaviour is not only determined by the examination or assessment modes that are used. Students’ perceptions about evaluation methods also play a significant role. This review aims to examine evaluation and assessment from the student’s point of view. Research findings reveal that students’ perceptions about assessment significantly influence their approaches to learning and studying. Conversely, students’ approaches to study influence the ways in which they perceive evaluation and assessment. Findings suggest that students hold strong views about different assessment and evaluation formats. In this respect students favour multiple‐choice format exams to essay type questions. However, when compared with more innovative assessment methods, students call the ‘fairness’ of these well‐known evaluation modes into question.     

Stability of Country Rankings Across Item Formats in the Third International Mathematics and Science Study

Multiple‐choice, short‐answer, and extended‐response item formats were used in the Third International Mathematics and Science Study to assess student achievement in mathematics and science at Grades 7 and 8 in more than 40 countries around the world. Data pertaining to science indicate that the standings of some countries relative to others change when performance is measured via the different item formats. The question addressed in the present article is the following: Can the instability of ranks in this case be attributed principally to item format, or are other important factors at work? It is argued that the findings provide further evidence that comparing student achievement across countries is a very complex undertaking indeed.     

The effect of distributed questioning with varied examples on exam performance on inference questions

The purpose of this study was to investigate whether the distributed presentation of different versions of a question would produce better performance on a new version of the question than distributed presentation of the same version of the question. A total of 48 four question sets of five alternative multiple‐choice questions were presented during a college lecture course. The answers to all the four questions in each set required an inference from the same fact statement. One question in each set appeared on both a block exam and the final. Each of the other three questions was presented either on an online quiz before class, during class or on an online quiz a week after class. Answering different versions of a question before class, during class and after class resulted in better performance on a fourth version of the question on the exam and final than answering the same question before class, during class and after class. On the final exam, 44 questions were answered twice, first under a time constraint of 90 seconds and then when response time was unlimited. The two responses were virtually identical.     

The relationship of deep and surface study approaches on factual and applied test‐bank multiple‐choice question performance

With the advent of online test banks and large introductory classes, instructors have often turned to textbook publisher‐generated multiple‐choice question (MCQ) exams in their courses. Multiple‐choice questions are often divided into categories of factual or applied, thereby implicating levels of cognitive processing. This investigation examined the influence student study approaches have on performance of shallow compared to deep, cognitive process outcomes measured by scores on factual versus application MCQ test bank questions respectively. Fifty‐six introduction to psychology students, aged 18–52 years, completed a modified version of the Revised Two‐Factor Study Process Questionnaire (R‐SPQ‐2F) to access deep or surface study approaches. Mid‐term and final exam MCQ were equally divided between factual and application questions from the textbook publisher test bank. Overall, students performed significantly better on factual MCQ, with older students achieving higher factual and applied scores. Results suggest younger students tend to use a surface approach to study, with reproduction of what was taught to meet minimum requirements. With age statistically controlled, analyses demonstrated that the surface learning approach negatively impacts MCQ performance on questions categorised as factual and applied more than a deep learning approach benefits MCQ performance.     

Using student-generated questions for student-centred assessment

In small groups, medical students were involved in generating questions to contribute to an online item bank. This study sought to support collaborative question‐writing and enhance students’ metacognitive abilities, in particular, their ability to self‐regulate learning and moderate understanding of subject material. The study focused on supporting students to write questions requiring higher order cognitive processes. End‐of‐year formal examinations comprised 25% student‐generated questions (SGQs), while mid‐year examination items were completely unseen. Data were gathered from repeated administration of a questionnaire and from examination results. No statistically significant changes were identified in self‐rated monitoring of understanding and regulation of learning. The activity of generating questions supported students to work collaboratively in developing questions and answers. The bank of questions was appreciated by students as a source of revision material, even though it was not strongly focused on higher order processes. Based on scores, it would appear that many students chose to memorise the question bank as a ‘high‐yield’ strategy for mark inflation, paradoxically favouring surface rather than deep learning. The study has not identified directly improvements in metacognitive capacity and this is an area for further investigation. Continual refinement of the study method will be undertaken, with an emphasis on education of students in developing questions addressing higher order cognitive processes. Although students may have memorised the questions and answers, there is no evidence that they do not understand the information.     

Combined online and in‐class pretesting improves exam performance in general psychology

This study examined the effect of distributed questioning on learning and retention in a college lecture course. A total of 48 question pairs were presented over four exams. The 16 question pairs associated with each of the three blocks of the course appeared on the block exams, and all 48 appeared on the final exam. The two questions in each pair were related to each other, so that knowing the answer to one question usually implied knowing the answer to the other. One question in each pair was included in an experimental condition, in which questions were presented online, in class, or both online and in class, before appearing in exams. These conditions were counter‐balanced across the sample. The control questions appeared only in exams. Providing a question online in advance of class, as well as in class, had a significant long‐term effect on the probability of knowing the answers to both experimental and control questions when they appeared in exams. These results demonstrate that coordinated online and in‐class instruction can significantly improve exam performance. These results are consistent with the hypothesis that distributed instruction creates more robust memory traces, rather than the hypothesis that it creates additional memory traces.     

Effects of student‐generated questions as the source of online drill‐and‐practice activities on learning

This study investigated the effects of online drill‐and‐practice activities using student‐generated questions on academic performance and motivation as compared with online drill‐and‐practice using teacher‐generated questions and no drill‐and‐practice in a student question‐generation (SQG) learning context. A quasi‐experimental research method was adopted for the purposes of this study. Six fifth‐grade classes (n = 145) participated in a weekly online activity for 5 weeks. Analysis of covariance results showed significant differences among the different treatment groups with regard to both academic performance and motivation. Post hoc comparisons using simultaneous confidence intervals, however, did not demonstrate the use of online SQG combined with answering peer‐generated questions to be more conducive to learning with regard to any of the observed variables as compared with the use of online SQG combined with answering teacher‐generated questions. Furthermore, answering student‐generated questions in addition to online SQG did not lead to added gains in learning as compared with the online SQG‐alone group. Some reasons for the unexpected findings are proposed, and the significance of this study, as well as suggestions for instructional implementations and future research, are provided.     

Game assessment: fun as well as effective

Alternative methods of assessing student knowledge are useful since assessment is increasingly being emphasized by administrators, accrediting agencies and legislators. A game (team) format to assess students' knowledge of course material was experimentally compared with the traditional format of testing a student who answers individually. Team‐game scores were more often correct than individual scores in all comparisons. Team decisions, which involved cooperation within one's group while at the same time competing against other teams, were rarely the result of a single individual deciding on the team answer. Students preferred the game‐assessment condition more and also perceived it as a more accurate measure of their course knowledge.     

Learning approaches: examination type,discipline of study,and gender

A sample of 248 Australian university students participated in a study that investigated the influence of assessment type and discipline of study on students' learning approaches. Participants (93 psychology and 155 business students) responded to the Study Process Questionnaire in a context of preparation for a hypothetical examination. The hypothetical examination is in either a multiple‐choice or an essay format. Results indicated that assessment type had no significant influence on how students approach their learning, while discipline did have significant influence on student learning.     

Using computer animation and illustration activities to improve high school students' achievement in molecular genetics

Our main goal in this study was to determine whether the use of computer animation and illustration activities in high school can contribute to student achievement in molecular genetics. Three comparable groups of eleventh‐ and twelfth‐grade students participated: the control group (116 students) was taught in the traditional lecture format, whereas the experimental groups received instructions that integrated a computer animation (61 students) or illustration (71 students) activities. We used three research instruments: a multiple‐choice questionnaire; an open‐ended, written questionnaire; and personal interviews. Five of the multiple‐choice questions were also given to students before they received their genetics instruction (pretest). We found that students who participate in the experimental groups improved their knowledge in molecular genetics compared with the control group. However, the open‐ended questions revealed that the computer animation activity was significantly more effective than the illustration activity. On the basis of these findings, we conclude that it is advisable to use computer animations in molecular genetics, especially when teaching about dynamic processes; however, engaging students in illustration activities can still improve their achievement in comparison to traditional instruction. © 2007 Wiley Periodicals, Inc. J Res Sci Teach 45: 273–292, 2008     

Anatomy practical examinations: How does student performance on computerized evaluation compare with the traditional format?

Practical examinations in anatomy are usually conducted on specimens in the anatomy laboratory (referred to here as the “traditional” method). Recently, we have started to administer similar examinations online using the quiz facility in Moodle?. In this study, we compare student scores between two assessment environments viz. online and traditional environments. We hypothesized that regardless of the examination medium (traditional or online) overall student performance would not be significantly different. For the online medium, radiological images, prosected specimens, and short video clips demonstrating muscle action were first acquired from resources used for teaching during anatomy practical classes. These were optimized for online viewing and then uploaded onto Moodle learning management software. With regards to the traditional format, actual specimens were usually laid out in a circular stream. Identification tags were then attached to specific spots on the specimens and questions asked regarding those identified spots. A cohort of students taking practical examinations in six courses was studied. The courses were divided into three pairs with each pair credit‐weight matched. Each pair consisted of a course where the practical examination was conducted online and the other in the traditional format. There was no significant difference in the mean scores within each course pair. In addition, a significant positive correlation between score in traditional and online formats was found. We conclude that mean grades in anatomy practical examination conducted either online or in the traditional format were comparable. These findings should reassure teachers intending to use either format for their practical examinations. Anat Sci Educ. © 2011 American Association of Anatomists.     

Assessment of an online piloted module targeted toward home‐based food operators in Iowa

Understanding safe food practice is important for home‐based food operators to prevent foodborne illness. Earlier work has found that home‐based food operators lack food safety knowledge and may benefit from training that is specifically tailored to their needs. Unfortunately, home‐based food operators may be deterred from enrolling in traditional educational formats due to their busy schedules. The objective of this study was to pilot and evaluate the effectiveness of an online food safety education module for home‐based food operators in Iowa through three learning assessments. Twenty‐one participants enrolled in a blended workshop in which participants completed one online module before attending a face‐to‐face session where they completed the remaining five modules. The effectiveness of the online module was measured by examining the first‐attempt average scores on learning assessments, the number of assessment attempts required to achieve 100%, and the first‐attempt performance by question type. The three learning assessment tools resulted in first‐attempt averages of approximately 86%, 90%, and 83%, surpassing our standard of effectiveness of 75% and showing good potential for the online format. The learning assessment attempt numbers of 4.65, 1.67, and 3.81 showed difficulty with knowledge transfer for some topics. Comprehension and analysis‐style questions had first‐attempt success rates of approximately 85% and 88%, respectively. Scores on knowledge and application‐style questions were lower with first‐attempt success rates of approximately 80% and 75%, respectively. These findings were used to improve the first online module and guide the transition of the remaining five modules to the online format.     

Online or face-to-face learning? Exploring the personal factors that predict students' choice of instructional format

Notwithstanding the growth of online learning, little is known about the personal factors that predict student decisions to enroll in online courses. This study examined the relations between several personal factors and students' choice of instructional format. After completing an online course, service academy undergraduates (N = 564) completed a survey that assessed their motivational beliefs (task value and self-efficacy), achievement-related emotions (enjoyment, boredom, and frustration), and satisfaction with the course. Results indicated that students who preferred to take future courses in an online format (as opposed to face-to-face) also reported greater confidence in their ability to learn online (self-efficacy) and greater satisfaction with their recent online learning experience. Logistic regression analysis confirmed and extended these findings, indicating that membership in the online preference group was predicted by higher self-efficacy and satisfaction, and lower task value beliefs. Taken together, results from this study provide researchers and practitioners with a better understanding of how several personal factors relate to students' choice of instructional format.     

A NEW APPROACH TO THE PROBLEM OF CHANGING INITIAL RESPONSES TO MULTIPLE CHOICE QUESTIONS

The hypothesis that it is unwise to change answers to multiple choice questions was tested using the technique of multiple regression analysis. The net number of correct answers as a result of changing responses was regressed against final grade in the course, numeric score on the exam, percent of total answers changed for all questions and for analytical questions, sex of the student, and scope of the exam.
The results show that there are gains to be made by changing responses. The variables which proved to be significant indicated that students who did well on the test changed a large percentage of answers, and that those who were taking a final exam tended to gain more. Final grades, sex of the student, and analytical questions had no significant impact on gains from changing responses. On the basis of the results gathered, the authors reject the hypothesis that changing responses is unwise.     

Language Background and Assessment in the Physical Sciences

A project was initiated at the University of Technology, Sydney to look at the role of language and gender in assessment of first‐year university students in Physics. Research on student learning has shown that assessment, to a large extent, determines the curriculum for students and their approaches to learning. Assessment plays a crucial role in students’ interest and motivation in a subject and the quality of their learning outcomes. In this paper, we report on a three‐part study. The first involved an analysis of assessment items. The second involved the use of paired questions to investigate the effect of the format of the question. The third involved investigating the effects of including a range of assessment types. The effects of language background were important and were found to interact with gender. Strategies will be discussed for changing assessments in the physical sciences to make them more equitable.     

To risk or not to risk it: student (non‐)engagement with seen examination questions

This case study explores students’ perceptions of seen examination questions about topics not covered by the formal curriculum of a final‐year economics module and of the associated group support sessions. Eight semi‐structured interviews with a total of 13 students were analysed. Contrary to expectations, learners taking a strategic approach to the module were not attracted by the seen questions. The uncertainty of an unfamiliar assessment format and the prospect of undertaking independent research and group work were perceived as involving more risks than the familiar unseen examination. Take‐up for the seen examination questions was low, and the students who did not answer a seen question tended to make workload considerations as well as concerns about group work for assessment purposes responsible for their decision. Despite not participating in the group support sessions, a few students researched the seen question in conjunction with trusted fellow students or on their own. The students who answered a seen question enjoyed the autonomy which the seen exam questions provided, while other students were critical of the way in which their autonomy might be externally controlled. The study provides insight into the impact of assessment on risk taking and students’ perception of risk associated with this type of assessment.     

Performance and duration differences between online and paper–pencil tests

Digital technologies have been used for measurement purposes and whether the test medium influences the user is an important issue. The aim of this study is to investigate students?? performances and duration differences between online and paper?Cpencil tests. An online testing tool was developed and administered in order to determine the differences between the traditional paper?Cpencil tests and online tests concerning students?? performances and the duration on tests. This tool enables to add questions that utilize an online database and which are in the form of multiple choice (with 5 or 4 options), true?Cfalse, matching, filling in the blanks, with multiple answers, with short answers, with long answers, and it also enables to prepare tests and to turn them into paper?Cpencil test mode. Performance test was applied with both online and paper?Cpencil modes on junior students at one of the universities in Turkey. Besides, the online testing tool developed within the context of the study was evaluated by instructors with respect to usability, relevance to the purpose and design. Instructor and student questionnaires are developed to determine the opinions on the online testing tool and online tests. Results showed that there was no significant differences between the performances on online and paper?Cpencil tests. On the other hand, the time they spent on the online test has been longer than the time they spent on paper?Cpencil test. Students found the online testing tool easy to use and stated that online test medium is more comfortable than paper?Cpencil tests. However, they complained about external noises, tiredness, and focusing problems regarding the online examination mediums. Generally, instructors have also appreciated the online testing tool??s design and they agree on the fact that it serves for its purposes.