An Alternative Multiple‐Choice Question Format to Guide Feedback Using Student Self‐Assessment of Knowledge

Management science professors who teach large classes often assess students with multiple‐choice questions (MCQs) because it is efficient. However, traditional MCQ formats are ill‐fitted for constructive feedback. We propose the reward for omission with confidence in knowledge (ROCK) format as an original formative assessment technique to help guide feedback associated with MCQs in an introductory undergraduate management science course. Our study contributes to theory by empirically showing that students can self‐assess their state of knowledge, signal it to the professor, and use proper answering options. In practice, ROCK is an easily implementable MCQ format that allows professors to gain information on student learning based on answers selected. ROCK identifies lack of knowledge or misinformation at both individual and collective levels thus providing opportunities for better feedback in class and during office hours. Limitations of the application of ROCK are also discussed.     

Multiple-Choice Questions and Written Questions matched according to levels of cognitive ability in an applied course: Evidence and practical implications

Whether or not the scores for multiple-choice questions (MCQs) and written questions (WQs) in formative assessments are the same, has been a subject of intense scrutiny. However, the evidence for their similarity at different levels of cognitive ability in applied courses has not been sufficiently documented. This study analysed the comparability of scores for equivalent MCQs and WQs at each level of cognitive ability, namely, ‘application’, ‘analysis’, ‘synthesis’ and ‘evaluation’, in an applied course. It was found that MCQ scores were higher than WQ scores at the levels of ‘application’ and ‘analysis’, while they were the same as WQ scores at the levels of ‘synthesis’ and ‘evaluation’. Furthermore, MCQ ranking of students’ scores at the level of ‘evaluation’ was inconsistent with its ranking at lower levels of cognitive ability. Thus, it is recommended that MCQs be pitched for sufficiently high levels of cognitive ability, albeit not the highest, to achieve similar scores to WQs.     

Longitudinal assessment of progress in reasoning capacity and relation with self-estimation of knowledge base

The aim of the study was to investigate progress in reasoning capacity and knowledge base appraisal in a longitudinal analysis of data from summative evaluation throughout a medical problem-based learning curriculum. The scores in multidisciplinary discussion of a clinical case and multiple choice questionnaires (MCQs) were studied longitudinally for 213 students from years 2 to 5. The capacity of core knowledge delimitation was calculated as the difference between the levels of average ascertainment degrees given for correct and incorrect answers at MCQ. For both multidisciplinary discussion of a clinical case evaluation and self-estimation of core knowledge, the capacity increases throughout the curriculum. The reasoning capacity assessed through multidisciplinary discussion of a clinical case is positively correlated with MCQ scores and the capacity to discriminate the mastered core knowledge. In conclusion, this study indicates that self-estimation of core knowledge is associated with an increase in reasoning performance through a well-organised knowledge base. Since that ability is related to success or failure, it is suggested that student awareness about delimitation of mastered core knowledge is considered as part of learning.     

The influence of distractor strength and response order on MCQ responding

Multiple-choice questions (MCQs) play a key role in standardised testing and in-class assessment. Research into the influence of within-item response order on MCQ characteristics has been mixed. While some researchers have shown preferential selection of response options presented earlier in the answer list, others have failed to replicate these results. This paper investigates a possible explanation for these mixed findings by assessing the influence of distractor strength on MCQ response order effects. A real-world assessment was administered to 232 undergraduates in which the response order within items was systematically varied. A generalised multilevel model was then used to show a significant interaction between distractor strength and response position with regard to response behaviour. Furthermore, the effect was found to be independent of student ability. These findings have implications for MCQ test construction, minimising the negative consequences of MCQ test exposure and promoting improved test-taking strategies.     

Investigating peer-assessment strategies for mathematics pre-service teacher learning on formative assessment

Formative assessment practices for secondary mathematics have been advocated as valuable for students, but difficult for teachers to learn. There have been calls in the literature to increase the emphasis on formative assessment in mathematics teacher preparation courses. This study explored the use of peer-assessment strategies for helping pre-service secondary mathematics teachers (PSTs) cultivate formative assessment principles and practices for assessing school students. Twenty-seven PSTs participated in a peer-assessment cycle comprised of: sourcing a rich mathematics task; constructing an assessment rubric for it; and collecting and analysing a selection of secondary student responses to the task. Each PST then provided written and verbal feedback to a peer on his/her rubric and student solution assessments. We draw on theoretical conceptions of Teacher Assessment Literacy in Practice to characterize the PSTs’ perceptions of their experience of formative assessment processes for learning to assess school students, in terms of cognitive and affective dimensions of their conceptions of assessment. The cohort evidenced a wide range of levels of confidence with the various aspects of formative assessment practices but on average less confidence in assessing school student task responses themselves than in assessing peer work. In addition to highlighting specific changes to different types of assessment knowledge, the PSTs also evidenced an awareness of shifts in their attitudes, in coming to view student task responses with more appreciation and humility.

     

The influence of assessment method on students' learning approaches: Multiple choice question examination versus assignment essay

A sample of 206 second-year Education students completed questionnaires on issues relating to their preparation for and perceptions of two methods of assessment of the same course: an assignment essay and an end-of-course multiple choice question (MCQ) examination. The questionnaire required a simultaneous response for each assessment method to statements focusing on their learning approaches, their perceptions of the levels of intellectual abilities being assessed, and their preference for either the assignment essay or MCQ examination as an assessment method of the course and the reasons for their choices. The above variables were analysed in relation to each other and to performance outcome in both assessment tasks. Results suggest distinct patterns according to assessment method. Students were more likely to employ surface learning approaches in the MCQ examination context and to perceive MCQ examinations as assessing knowledge-based (lower levels of) intellectual processing. Poorer performance in the MCQ examination was associated with the employment of deep learning strategies. In contrast, students were more likely to employ deep learning approaches when preparing their assignment essays which they perceived as assessing higher levels of cognitive processing. Poorer performance in the assignment essays was associated with the employment of surface strategies. The implications of these findings are discussed.     

Climbing Bloom's taxonomy pyramid: Lessons from a graduate histology course

Bloom's taxonomy was adopted to create a subject‐specific scoring tool for histology multiple‐choice questions (MCQs). This Bloom's Taxonomy Histology Tool (BTHT) was used to analyze teacher‐ and student‐generated quiz and examination questions from a graduate level histology course. Multiple‐choice questions using histological images were generally assigned a higher BTHT level than simple text questions. The type of microscopy technique (light or electron microscopy) used for these image‐based questions did not result in any significant differences in their Bloom's taxonomy scores. The BTHT levels for teacher‐generated MCQs correlated positively with higher discrimination indices and inversely with the percent of students answering these questions correctly (difficulty index), suggesting that higher‐level Bloom's taxonomy questions differentiate well between higher‐ and lower‐performing students. When examining BTHT scores for MCQs that were written by students in a Multiple‐Choice Item Development Assignment (MCIDA) there was no significant correlation between these scores and the students' ability to answer teacher‐generated MCQs. This suggests that the ability to answer histology MCQs relies on a different skill set than the aptitude to construct higher‐level Bloom's taxonomy questions. However, students significantly improved their average BTHT scores from the midterm to the final MCIDA task, which indicates that practice, experience and feedback increased their MCQ writing proficiency. Anat Sci Educ 10: 456–464. © 2017 American Association of Anatomists.     

Multi‐level Assessment of Scientific Content Knowledge Gains Associated with Socioscientific Issues‐based Instruction

This study explored the impact of using a socioscientific issue (SSI) based curriculum on developing science content knowledge. Using a multi‐level assessment design, student content knowledge gains were measured before and after implementation of a three‐week unit on global warming (a prominent SSI) that explored both the relevant science content and the controversy surrounding global warming. Measures of student content knowledge were made using a standards‐aligned content knowledge exam (distal assessment) and a curriculum‐aligned exam (proximal assessment). Data were collected from 108 students enrolled from two schools. Quantitative analysis of the distal assessment indicated that student post‐test scores were statistically significantly different than their pre‐test scores (F = 15.31, p<0.001). Qualitative analyses of student responses from the proximal assessment indicated that students, on average, expressed more accurate, more detailed, and more sophisticated understandings of global warming, the greenhouse effect, and the controversy and challenges associated with these issues following the three‐week unit. Combined results from the proximal and distal assessments explored in this study offer important evidence in supporting the efficacy of using SSI as contexts for science education. In addition to a discussion of the components of an SSI‐based curriculum, this study provides support for the use of SSI as a context for learning science content.     

The Aptitude–Achievement Function: An Aid for Allocating Educational Resources, with an Advanced Placement Example

We fit a functional relationship between aptitude and achievement test scores and show how to use it to allocate educational resources. As an example we use the PSAT–Mathematics test to predict performance on the College Board's Advanced Placement Test in calculus, as a guide to student and school participation, for school or system assessment, and to project future nationwide expansion. In addition to the PSAT-AP test score relations, we consider the distribution of student ability, school policies of student selection and recruitment, and teacher skill in presenting the material and in motivating students. This overall result provides an indication of just how remarkable was Jaime Escalante's accomplishment in Los Angeles's Garfield High School. We find little evidence for differences in educational quality between such diverse schools as in the inner city of Detroit and the affluent suburb of La Cañada, California. We comment briefly on the role of the AP in international assessments.     

How Can Released State Test Items Support Interim Assessment Purposes in an Educational Crisis?

State testing programs regularly release previously administered test items to the public. We provide an open-source recipe for state, district, and school assessment coordinators to combine these items flexibly to produce scores linked to established state score scales. These would enable estimation of student score distributions and achievement levels. We discuss how educators can use resulting scores to estimate achievement distributions at the classroom and school level. We emphasize that any use of such tests should be tertiary, with no stakes for students, educators, and schools, particularly in the context of a crisis like the COVID-19 pandemic. These tests and their results should also be lower in priority than assessments of physical, mental, and social–emotional health, and lower in priority than classroom and district assessments that may already be in place. We encourage state testing programs to release all the ingredients for this recipe to support low-stakes, aggregate-level assessments. This is particularly urgent during a crisis where scores may be declining and gaps increasing at unknown rates.     

School-Level IRT Scaling of Writing Assessment Data

School-level assessment of student writing ability using a group-level, polytomous item response theory (IRT) model was illustrated in this study. The study supported the viability of an IRT-based school assessment as an alternative to the conventional approach based on aggregation of individual scores. The precision provided by the assumed assessment design varied dramatically depending on school size and school average ability. For small schools and students with low average abilities, differences in average school performance had to be quite large to be trustworthy. In contrast, the design provided greater precision in detecting differences for large schools and students with high average abilities. An operational use of this design would require great care in the reporting of results to ensure that unreliable school comparisons are clearly identified.     

Analysis of testing with multiple choice versus open‐ended questions: Outcome‐based observations in an anatomy course

The pedagogical approach for both didactic and laboratory teaching of anatomy has changed in the last 25 years and continues to evolve; however, assessment of student anatomical knowledge has not changed despite the awareness of Bloom's taxonomy. For economic reasons most schools rely on multiple choice questions (MCQ) that test knowledge mastered while competences such as critical thinking and skill development are not typically assessed. In contrast, open‐ended question (OEQ) examinations demand knowledge construction and a higher order of thinking, but more time is required from the faculty to score the constructed responses. This study compares performances on MCQ and OEQ examinations administered to a small group of incoming first year medical students in a preparatory (enrichment) anatomy course that covered the thorax and abdomen. In the thorax module, the OEQ examination score was lower than the MCQ examination score; however, in the abdomen module, the OEQ examination score improved compared to the thorax OEQ score. Many students attributed their improved performance to a change from simple memorization (superficial learning) for cued responses to conceptual understanding (deeper learning) for constructed responses. The results support the view that assessment with OEQs, which requires in depth knowledge, would result in student better performance in the examination. Anat Sci Educ 11: 254–261. © 2017 American Association of Anatomists.     

Differential Improvement in Student Understanding of Mathematical Principles Following Formative Assessment Intervention

ABSTRACT

The authors describe results from a study of a middle school mathematics formative assessment strategy. They employed a randomized, controlled design to address the following question: Does using our strategy improve student performance on assessments of key mathematical ideas relative to a comparison group? Eighty-five teachers and 4,091 students were included. Students took a pretest and a transfer measure at the end of the year. Treatment students completed formative assessments. Treatment teachers had exposure to professional development and instructional resources. Results indicated students with higher pretest scores benefited more from the treatment compared to students with lower pretest scores. In addition treatment students significantly outperformed control students on distributive property items. This effect was larger as pretest scores increased. Results, limitations, and future directions are discussed.     

Item Type and Cognitive Ability Measured: The Validity Evidence for Multiple True-False Items in Medical Specialty Certification

This study compared the criterion-related validity evidence and other psycho- metric characteristics of multiple-choice (MCQ) and multiple true-false (MTF) items in medical specialty certifying examinations in internal medicine and its subspecialties. Results showed that MTF items were more reliable than MCQs and that the format scores were highly correlated. However, MCQs were more highly correlated with an independent performance measure than were MTF items. MTF items were classified primarily as measuring knowledge rather than synthesis or judgment. These results may have implications for examination construction, especially if criterion-related validity evidence is important.     

Thinking beyond the score: Multidimensional analysis of student performance to inform the next generation of science assessments

Conventional assessment analysis of student results, referred to as rubric-based assessments (RBA), has emphasized numeric scores as the primary way of communicating information to teachers about their students’ learning. In this light, rethinking and reflecting on not only how scores are generated but also what analyses are done with them to inform classroom practices is of utmost importance. Informed by Systemic Functional Linguistics and Latent Dirichlet Allocation analyses, this study utilizes an innovative bilingual (Spanish–English) constructed response assessment of science and language practices for middle and high school students to perform a multilayered analysis of student responses. We explore multiple ways of looking at students’ performance through their written assessments and discuss features of student responses that are made visible through these analyses. Findings from this study suggest that science educators would benefit from a multidimensional model which deploys complementary ways in which we can interpret student performance. This understanding leads us to think that researchers and developers in the field of assessment need to promote approaches that analyze student science performance as a multilayered phenomenon.     

National survey of accommodations and alternate assessments for students who are deaf or hard of hearing in the United States

This paper reports the results of the National Survey of Accommodations and Alternate Assessments for Students who are Deaf or Hard of Hearing in the United States (National Survey). This study focused on the use of accommodations and alternate assessments in statewide assessments used with students who are deaf or hard of hearing. A total of 258 participants responded to the survey, including 32 representing schools for the deaf, 168 from districtwide/school programs, and 58 from mainstreamed settings. These schools and programs served a total of nearly 12,000 students who are deaf or hard of hearing nationwide. The most prevalent accommodations used in 2003-2004 statewide standardized assessments in mathematics and reading were extended time, an interpreter for directions, and a separate room for test administration. Read aloud and signed question-response accommodations were often prevalent, used more often for mathematics than in reading assessments. Participants from mainstreamed settings reported a more frequent use of accommodations than those in schools for the deaf or districtwide/school programs. In contrast, schools for the deaf were most likely to have students participate in alternate assessments. The top three alternate assessment formats used across all settings were out-of-level testing, work samples, and portfolios. Using the National Survey results as a starting point, future research will need to investigate the validity of accommodations used with students who are deaf or hard of hearing. In the context of the No Child Left Behind Act of 2001 accountability policies, the accommodations and alternate assessment formats used with students who are deaf or hard of hearing may result in restrictions in how scores are integrated into state accountability frameworks.     

Energy and Energy Waste: a Topic for Science Education

The Science Foundation Programme (SFP) was launched in 1991 at the University of Natal, Pietermaritzburg, South Africa in an attempt to equip a selected number of matriculants from historically disadvantaged schools with the skills, resources and self-confidence needed to embark on their tertiary studies. Previous research within the SFP biology component suggests that a major contributor to poor achievement and low retention rates among English second language (ESL) students in the Life Sciences is the inadequate background knowledge in natural history. In this study, SFP student background knowledge was assessed along a continuum of language dependency using a set of three probes. Improved student performance in each of the respective assessments examined the extent to which a sound natural history background facilitated meaningful learning relative to ESL proficiency. Student profiles and attitudes to biology were also examined. Results indicated that students did not perceive language to be a problem in biology. However, analysis of the student performance in the assessment probes indicated that, although the marine course provided the students with the background knowledge that they were initially lacking, they continued to perform better in the drawing and MCQ tools in the post-tests, suggesting that it is their inability to express themselves in the written form that hampers their development. These results have implications for curriculum development within the constructivist framework of the SFP.     

Impact of interactive online units on learning science among students with learning disabilities and English learners

The purpose of this study was to document the design, classroom implementation, and effectiveness of interactive online units to enhance science learning over 3 years among students with learning disabilities, English learners, and general education students. Results of a randomised controlled trial with 2,303 middle school students and 71 teachers across 13 schools in two states indicated that online units effectively deepened science knowledge across all three student groups. Comparing all treatment and control students on pretest-to-posttest improvement on standards-based content-specific assessments, there were statistically significant mean differences (17% improvement treatment vs. 6% control; p?相似文献   

Learning to Learn From Benchmark Assessment Data: How Teachers Analyze Results

Although interim assessments are currently promoted as a mechanism for improving teaching and student learning, we know little about how teachers use this data to modify instruction. This article presents findings from a larger study on teachers’ use of interim assessment information in elementary mathematics. We address the following questions: (a) How do the Philadelphia teachers in our sample analyze benchmark assessment results, (b) how do they plan instruction based on these results, and (c) what are their reported instructional responses to such results? To answer these questions, we interviewed all 3rd- and 5th-grade teachers in five average- and above-average-performing elementary schools three times during the 2006–07 school year. We found that although the teachers in our study used interim assessment results to gain information about students’ learning in mathematics, teachers did not use interim assessments to make sense of students’ conceptual understanding. Furthermore, teachers’ tendency to interpret student errors as procedural missteps was paralleled by a trend toward procedural instructional responses.     

Student experiences of NAPLAN: sharing insights from two school sites

This paper provides insight into middle school students’ perceptions and reactions to their participation in the Australian National Assessment Program—Literacy and Numeracy (NAPLAN). A case study was conducted over 10 months at two Queensland schools with different approaches to NAPLAN implementation. Student voice was elicited via focus groups and 35 students provided drawings and words describing their experience in four stages: preparing, sitting, completing and receiving their results. Thematic content analysis of the textual data and trait and holistic coding of the visual data revealed five themes and suggests that the approach adopted by the school may impact on students’ NAPLAN experiences. This study privileges student voice and enables access to student experiences as they participate in a testing regime which is now a feature of the Australian school assessment landscape.