The influence of assessment method on students' learning approaches: Multiple choice question examination versus assignment essay

A sample of 206 second-year Education students completed questionnaires on issues relating to their preparation for and perceptions of two methods of assessment of the same course: an assignment essay and an end-of-course multiple choice question (MCQ) examination. The questionnaire required a simultaneous response for each assessment method to statements focusing on their learning approaches, their perceptions of the levels of intellectual abilities being assessed, and their preference for either the assignment essay or MCQ examination as an assessment method of the course and the reasons for their choices. The above variables were analysed in relation to each other and to performance outcome in both assessment tasks. Results suggest distinct patterns according to assessment method. Students were more likely to employ surface learning approaches in the MCQ examination context and to perceive MCQ examinations as assessing knowledge-based (lower levels of) intellectual processing. Poorer performance in the MCQ examination was associated with the employment of deep learning strategies. In contrast, students were more likely to employ deep learning approaches when preparing their assignment essays which they perceived as assessing higher levels of cognitive processing. Poorer performance in the assignment essays was associated with the employment of surface strategies. The implications of these findings are discussed.     

The relationship of deep and surface study approaches on factual and applied test‐bank multiple‐choice question performance

With the advent of online test banks and large introductory classes, instructors have often turned to textbook publisher‐generated multiple‐choice question (MCQ) exams in their courses. Multiple‐choice questions are often divided into categories of factual or applied, thereby implicating levels of cognitive processing. This investigation examined the influence student study approaches have on performance of shallow compared to deep, cognitive process outcomes measured by scores on factual versus application MCQ test bank questions respectively. Fifty‐six introduction to psychology students, aged 18–52 years, completed a modified version of the Revised Two‐Factor Study Process Questionnaire (R‐SPQ‐2F) to access deep or surface study approaches. Mid‐term and final exam MCQ were equally divided between factual and application questions from the textbook publisher test bank. Overall, students performed significantly better on factual MCQ, with older students achieving higher factual and applied scores. Results suggest younger students tend to use a surface approach to study, with reproduction of what was taught to meet minimum requirements. With age statistically controlled, analyses demonstrated that the surface learning approach negatively impacts MCQ performance on questions categorised as factual and applied more than a deep learning approach benefits MCQ performance.     

Emotional factors of causal coherence in text comprehension

The current study tested the effects of positive and negative emotion at the beginning and end of texts, as well as the consistency in valence throughout a text on readers’ response choice to items on a multiple-choice reading comprehension assessment. Multinomial logistic regressions were performed on 1,161 college participants’ assessment responses to test whether emotion in narrative and informational text items significantly predicted which distractor response options readers chose in comparison to the correct response option. Overall, consistent emotion throughout text and positive and negative emotion at the beginning and end of text were significant predictors of readers’ response choices. The results are discussed in terms of emotion being an indicator of readers’ causal processing. Specifically, findings provide a better understanding of how emotional features in narrative and informational texts may influence how readers develop causal coherence and comprehension during reading. This understanding could also help inform the development of instructional tools that encourage readers to focus on aspects of text (i.e., emotion) which could, in turn, help improve comprehension for readers who struggle.     

A discursive question: Supporting student-authored multiple choice questions through peer-learning software in non-STEMM disciplines

Peer-learning that engages students in multiple choice question (MCQ) formulation promotes higher task engagement and deeper learning than simply answering MCQ’s in summative assessment. Yet presently, the literature detailing deployments of student-authored MCQ software is biased towards accounts from Science, Technology, Engineering, Maths and Medicine (STEMM) subjects, rather than discursive subjects or disciplines where content may contain fewer absolute facts and objective metrics and more nuance. We report on qualitative and quantitative findings from a semester-long deployment of a peer-learning software package (PeerWise) in a 140-student course on Interaction Design. PeerWise enables students to author, rate and comment upon their peers’ MCQ questions. The platform was enthusiastically adopted as a revision aid, yet overall question quality was poor and students expressed difficulty in translating the discursive nature of the course content into MCQs with only one correct answer. In addressing these shortcomings, this paper offers specific recommendations to instructors of more discursive subjects using student-led MCQ authoring platforms, and further, how platforms such as PeerWise may be adapted to better suit disciplines characterised by discursive content. We propose alternative approaches to moderation and two suggestions for potential amendments to the software itself.     

Analysis of testing with multiple choice versus open‐ended questions: Outcome‐based observations in an anatomy course

The pedagogical approach for both didactic and laboratory teaching of anatomy has changed in the last 25 years and continues to evolve; however, assessment of student anatomical knowledge has not changed despite the awareness of Bloom's taxonomy. For economic reasons most schools rely on multiple choice questions (MCQ) that test knowledge mastered while competences such as critical thinking and skill development are not typically assessed. In contrast, open‐ended question (OEQ) examinations demand knowledge construction and a higher order of thinking, but more time is required from the faculty to score the constructed responses. This study compares performances on MCQ and OEQ examinations administered to a small group of incoming first year medical students in a preparatory (enrichment) anatomy course that covered the thorax and abdomen. In the thorax module, the OEQ examination score was lower than the MCQ examination score; however, in the abdomen module, the OEQ examination score improved compared to the thorax OEQ score. Many students attributed their improved performance to a change from simple memorization (superficial learning) for cued responses to conceptual understanding (deeper learning) for constructed responses. The results support the view that assessment with OEQs, which requires in depth knowledge, would result in student better performance in the examination. Anat Sci Educ 11: 254–261. © 2017 American Association of Anatomists.     

The Effect of the Most-Attractive-Distractor Location on Multiple-Choice Item Difficulty

Abstract

The arrangement of response options in multiple-choice (MC) items, especially the location of the most attractive distractor, is considered critical in constructing high-quality MC items. In the current study, a sample of 496 undergraduate students taking an educational assessment course was given three test forms consisting of the same items but the positions of the most attractive distractor varied across the forms. Using a multiple-indicators–multiple-causes (MIMIC) approach, the effects of the most attractive distractor's positions on item difficulty were investigated. The results indicated that the relative placement of the most attractive distractor and the distance between the most attractive distractor and the keyed option affected students’ response behaviors. Moreover, low-achieving students were more susceptible to response-position changes than high-achieving students.     

A case for written examinations in undergraduate medical education: experiences with modified essay examinations

Abstract

A modified essay is an open-ended question about clinical presentations, with the goal to test application of basic science principles to novel scenarios. Due to high reliability and validity, multiple-choice question (MCQ) examinations are more commonly used. We have employed a combination of MCQ and modified essay examinations in medical school. The correlation between MCQ and modified essay grades is positive but small, suggesting they may be testing different cognitive levels/skills. In this paper, we evaluate the impact of a non-graded mock modified essay on student’s performance and satisfaction, and outline our best practices. We have found that it is important to involve faculty from multiple disciplines to construct robust cases, and to review examinations and rubrics. The mock modified essay resulted in higher performance and student satisfaction, and fewer students’ comments related to subjectivity, frustration and anxiety. Yet, there was not an improvement in confidence. Moreover, there was no correlation between grades in the mock and graded modified essays, suggesting that the mock modified essay may not be useful to predict academic performance. Yet, the incorporation of a mock modified essay was useful to clarify expectations and provide guidance, potentially reducing the anxiety induced by an unfamiliar assessment.     

The validity of multiple choice practical examinations as an alternative to traditional free response examination formats in gross anatomy

Traditionally, an anatomy practical examination is conducted using a free response format (FRF). However, this format is resource‐intensive, as it requires a relatively large time investment from anatomy course faculty in preparation and grading. Thus, several interventions have been reported where the response format was changed to a selected response format (SRF). However, validity evidence from those interventions has not proved entirely adequate for the practical anatomy examination, and thus, further investigation was required. In this study, the validity evidence of SRF was examined using multiple choice questions (MCQs) constructed according to different levels of Bloom's taxonomy in comparison with the traditional free response format. A group of 100 medical students registered in a gross anatomy course volunteered to be enrolled in this study. The experimental MCQ examinations were part of graded midterm and final steeplechase practical examination. Volunteer students were instructed to complete the practical examinations twice, once in each of two separate examination rooms. The two separate examinations consisted of a traditional free response format and MCQ format. Scores from the two examinations (FRF and MCQ) displayed a strong correlation, even with higher level Bloom's taxonomy questions. In conclusion, the results of this study provide empirical evidence that the SRF (MCQ) response format is a valid method and can be used as an alternative to the traditional FRF steeplechase examination. Anat Sci Educ. © 2013 American Association of Anatomists.     

An Australian study comparing the use of multiple-choice questionnaires with assignments as interim,summative law school assessment

To the author’s knowledge, this is the first Australian study to empirically compare the use of a multiple-choice questionnaire (MCQ) with the use of a written assignment for interim, summative law school assessment. This study also surveyed the same student sample as to what types of assessments are preferred and why. In total, 182 undergraduate property law students participated in this study. Results showed that scores for the MCQ (assessing five topics) and assignment (assessing one topic) followed a similar distribution. This indicates that an MCQ does not necessarily skew students towards higher grades than an assignment. Results also showed significant but low correlations of test scores across instruments. When asked which instrument best assessed their knowledge of property law, students expressed a strong preference for an assignment over an MCQ or examination. Comments revealed a strong belief that, because lawyers write, law schools must assess legal writing – a skill not captured by MCQs. This study is important as many Australian law schools face increasing marking loads due to higher student numbers and compulsory mid-term assessments. This article endorses the use of MCQs but only as part of a diverse suite of law school assessment.     

Differentials of a State Reading Assessment: Item Functioning, Distractor Functioning, and Omission Frequency for Disability Categories

Large data sets from a state reading assessment for third and fifth graders were analyzed to examine differential item functioning (DIF), differential distractor functioning (DDF), and differential omission frequency (DOF) between students with particular categories of disabilities (speech/language impairments, learning disabilities, and emotional behavior disorders) and students without disabilities. Multinomial logistic regression was employed to compare response characteristic curves (RCCs) of individual test items. Although no evidence for serious test bias was found for the state assessment examined in this study, the results indicated that students in different disability categories showed different patterns of DIF, DDF, and DOF, and that the use of RCCs helps clarify the implications of DIF and DDF.     

Enhancing the effectiveness of concept inventories using textual analysis: investigations in an electrical engineering subject

ABSTRACT

Concept inventories (CIs) are assessment instruments designed to measure students’ conceptual understanding of fundamental concepts in particular fields. CIs utilise multiple-choice questions (MCQs), and specifically designed response selections, to help identify misconceptions. One shortcoming of this assessment instrument is that it fails to provide evidence of the causes of the misconceptions, or the nature of students’ conceptual understanding. In this article, we present the results of conducting textual analysis on students’ written explanations in order to provide better judgements into their conceptual understanding. We compared students’ MCQ scores in Signals and Systems Concept Inventory questions, with the textual analysis utilising vector analysis approaches. Our analysis of the textual data provided the ability to detect answers that students identified as a ‘guessed’ response. However, the analysis was unable to detect if conceptually correct ideas existed within the ‘guessed’ responses. The presented approach can be used as a framework to analyse assessment instruments that utilise textual, short-answer responses. This analysis framework is best suited for the restricted conditions imposed by the short-answer structure.     

Of Small Beauties and Large Beasts: The Quality of Distractors on Multiple-Choice Tests Is More Important Than Their Quantity

In multiple-choice tests, the quality of distractors may be more important than their number. We therefore examined the joint influence of distractor quality and quantity on test functioning by providing a sample of 5,793 participants with five parallel test sets consisting of items that differed in the number and quality of distractors. Surprisingly, we found that items in which only the one best distractor was presented together with the solution provided the strongest criterion-related evidence of the validity of test scores and thus allowed for the most valid conclusions on the general knowledge level of test takers. Items that included the best distractor produced more reliable test scores irrespective of option number. Increasing the number of options increased item difficulty, but did not increase internal consistency when testing time was controlled for.     

Metacognition as a Mediator of the Effect of Test Anxiety on a Surface Approach to Studying

This study investigated the role of metacognition as a mediator of the effect of test anxiety on a surface approach to studying. The following scales were completed by 109 undergraduate social sciences students: Approaches and Study Skills Inventories for Students (ASSIST), Metacognitions Questionnaire (MCQ), and Test Anxiety Scale (TAS). Positive and significant correlations were observed between test anxiety and a surface approach to studying, and between all five dimensions of metacognition and test anxiety. Positive and significant correlations were also found between four of the five dimensions of metacognition and a surface approach to studying. Structural equation modelling was used to test a mediational model in which test anxiety predicted metacognition which in turn predicted a surface approach to studying. The results supported the hypothesis that the effect of test anxiety on a surface approach to studying is entirely mediated by metacognition. The practical implications of these findings are outlined.     

Test anxiety and GCSE performance: the effect of gender and socio‐economic background

Despite a well established body of international literature describing the effect of test anxiety on student performance in a range of assessments, there has been little work conducted on samples of students from the UK. The purpose of this exploratory study is two‐fold. First, to establish the relationship between test anxiety and assessment performance in a group of students in their final year of compulsory secondary schooling, in the politicised educational context of the UK. Second, to establish if this relationship is moderated by gender and socio‐economic background. Data were gathered on trait test anxiety, GSCE examination performance in Mathematics, English Language and Science, gender and socio‐economic background from 557 mixed ability Year 11 students drawn from three secondary schools in the UK. A hierarchical regression analysis was used to establish the moderating influence of gender and socio‐economic background. Results suggest a small, but significant inverse relationship between test anxiety scores and mean examination performance and that the cognitive component of test anxiety accounts for 7% of variance in examination performance. A differential test anxiety–assessment performance relationship was reported for socio‐economic background but not gender. Although the data reported for the test anxiety–assessment performance relationship are similar to those reported in numerous other studies, it is hypothesised that contextualised features associated with secondary education in the UK, particularly efforts to raise attainment, may have influenced these results.     

高校学生评教结果有效性分析方法研究——Z高校背景因素方差分析案例

学生评教作为一种课堂教学质量评价模式,能够实现课堂教学全过程覆盖、学生全员参与。支持派和质疑派对学生评教结果有效性始终存在较大争议,焦点就在于学生评教结果是否会受到背景因素的影响。为应对各方主体对学生评教结果有效性的质疑,以Z高校为例,将教师、课程、学生三方面18类背景因素作为自变量,从方法论角度入手,通过分析显著性概率、效应量、方差解释率和统计检验力,尝试建立一套较为系统的单因素方差分析方法体系,为学生评教结果有效性论证提供路径。     

Students’ conceptions of assessment purposes in a low stakes secondary-school context: A mixed methodology approach

Though assessment in its early history was conceived as a measurement device, alternative approaches to assessment of student achievement have appeared the past few decades. In this context, studies have been conducted to investigate students’ conceptions of assessment approaches and purposes, as there is evidence that these will affect their learning and ultimately their achievement. This study uses a sequential mixed methods design to investigate students’ conceptions about the purposes of assessment and their definitions of assessment. A questionnaire was administered to 599 lower secondary school students in Cyprus, followed by group interviews with 15 students to explain and build upon initial quantitative findings. Results show that students’ perceptions are in alignment with the current shift in assessment to be used primarily for enhancing teaching and learning. Students seem to agree with the legitimate purposes of assessment and realize that it is an ongoing procedure linked with teaching and learning.     

An Odds Ratio Approach for Assessing Differential Distractor Functioning Effects under the Nominal Response Model

Investigations of differential distractor functioning (DDF) can provide valuable information concerning the location and possible causes of measurement invariance within a multiple‐choice item. In this article, I propose an odds ratio estimator of the DDF effect as modeled under the nominal response model. In addition, I propose a simultaneous distractor‐level (SDL) test of invariance based on the results of the distractor‐level tests of DDF. The results of a simulation study indicated that the DDF effect estimator maintained good statistical properties under a variety of conditions, and the SDL test displayed substantially higher power than the traditional Mantel‐Haenszel test of no DIF when the DDF effect varied in magnitude and/or size across the distractors.     

Music during lectures: Will students learn better?

We investigated the influence of music during learning on the academic performance of undergraduate students, and more particularly the influence of affects induced by music. Altogether 249 students were involved in the study, divided into a control group and an experimental group. Both groups attended the same videotaped lecture, with the addition of classical music for the experimental group. Just after the lecture, both groups had to fill out a multiple choice questionnaire (MCQ) aimed to assess their learning during the lecture. The MCQ score of the experimental group was significantly higher than that of the control group. Affect just before the exam predicted significantly the MCQ-score, however music accounted for an additional part of variance. The effect of music on academic performance is not only about affects, and further research should investigate the underlying mechanisms resulting from a change in the perception of the learning environment.     

Listening to children’s views of spiritual and metaphysical concepts: A new dimension to educational psychology practice?

Educational psychologists (EPs) have for many years been developing techniques for listening to children. The aim of the present research was to investigate ways of listening to the “spiritual” views of children in order to develop questions that educational psychologists might use as part of the assessment repertoire. The study explored children’s spiritual concepts such as their purpose for “being on the planet” and the “meaning of their life”. The findings suggest that primary and secondary school children from mixed religious backgrounds can respond to spiritual questions in a clear way. In addition, most children believed that spiritual views influence how children behave. Further research is required to explore whether children’s views of spiritual concepts are linked to learning and progress in school and in life.