Enhancing the effectiveness of concept inventories using textual analysis: investigations in an electrical engineering subject

ABSTRACT

Concept inventories (CIs) are assessment instruments designed to measure students’ conceptual understanding of fundamental concepts in particular fields. CIs utilise multiple-choice questions (MCQs), and specifically designed response selections, to help identify misconceptions. One shortcoming of this assessment instrument is that it fails to provide evidence of the causes of the misconceptions, or the nature of students’ conceptual understanding. In this article, we present the results of conducting textual analysis on students’ written explanations in order to provide better judgements into their conceptual understanding. We compared students’ MCQ scores in Signals and Systems Concept Inventory questions, with the textual analysis utilising vector analysis approaches. Our analysis of the textual data provided the ability to detect answers that students identified as a ‘guessed’ response. However, the analysis was unable to detect if conceptually correct ideas existed within the ‘guessed’ responses. The presented approach can be used as a framework to analyse assessment instruments that utilise textual, short-answer responses. This analysis framework is best suited for the restricted conditions imposed by the short-answer structure.     

基于OBE理念的思想政治理论课实践教学体系探析 ——以卫生类高职院校为例

基于OBE理念,结合卫生类高职院校人才培养目标,以成果为导向,通过评估学生能力为基础制定思想政治理论课实践教学的目标;以学生为中心,按照学生的需求为标准设计思想政治理论课实践教学的过程;以能力为根本,遵循学生个性发展规律改革思想政治理论课实践教学的考核方式;以发展为重点,采用持续改进的方式构建思想政治理论课实践教学的体...     

USING ITEM RESPONSE THEORY TO CONDUCT A DISTRACTER ANALYSIS ON CONCEPTUAL INVENTORY OF NATURAL SELECTION

Concept inventories are often used to assess current student understanding although conceptual change models are problematic. Due to controversies with conceptual change models and the realities of student assessment, it is important that concept inventories are evaluated using a variety of theoretical models to improve quality. This study used a modified item response theory model to determine university nonmajor biology students’ levels of understanding of natural selection (n = 1,192). Using Conceptual Inventory of Natural Selection, we have reported how we applied Bock’s modified nominal item response theory model and the distracter test item analysis. We found that the use of this model can define student levels of understanding and identify problematic distracters.     

Analysing Concept Maps as an Assessment Tool in Teaching Physics and Comparison with the Achievement Tests

Concept mapping is a technique that paves the way to represent knowledge schematically. In this research, concept mapping was used as an assessment method on the impulse–momentum topic. The purpose of this study was to determine teacher candidates’ knowledge about understanding of the concepts of impulse and momentum by comparing and contrasting two different methods; namely, students’ concept maps and an achievement test. The mean of teacher candidates’ concept map scores are extremely low when compared with the scores of the achievement test. In addition, it was seen that although a great number of concepts were written down, not many relationships were established between these concepts. There is a weak correlation between the achievement test and the concept map scores since concept maps assess the students’ knowledge from a conceptual perspective while the achievement tests measure the level of students’ knowledge on the topic and his/her ability to apply this knowledge on different occasions.     

多元智力理论评价观及其对教育评价改革的启示

加德纳提出的多元智力理论评价观强调在情境观、动态观、多元观和独特观等方面对学生进行评价,为教育评价改革提供了理论基础。本文首先介绍了多元智力理论的评价观,然后阐述了多元智力理论评价观对教育评价改革的启示:注重在真实情境中评价学生的实践能力;关注学生的成长历程;提倡多元化评价学生。     

“物流系统建模与仿真”课程混合式实验教学探索——基于OBE教育理念

物流仿真类课程教学理论和实践兼顾,强调学生综合实践能力的培养,而混合式学习能够整合线上和线下教学的优势,为提高教学效果提供了有力支撑。基于OBE教育理念,以培养学生综合能力为导向,借用线上教学平台和虚拟仿真工具,对“物流系统建模与仿真”课程的实验教学进行层次化、模块化和迭代化设计;合理分配教与学学时,兼顾线上学习的灵活性和课堂教学的互动优势,采用混合模式培养学生的沟通、协作和思辨能力;对课程考核进行持续改进,考核环节贯穿全过程,构建多维评价视角,保证课程目标的实现。     

关于概念图在数学教学评价中的应用研究问题

概念图是组织和表征知识的工具,它的节点、连线、层级、分支和交叉连接都可以用来检测学生的学习.数学概念之间的关系可以拓展为对象与对象、对象与过程,或是过程与过程等之间的复杂关系,也牵涉到与运算之间的关系.概念图的评价形式有读图改错题、填图题和构图题.不同形式的概念图检验试题可以设计不同的评分标准,有节点连线评分法、编码评分法和等级评分法.     

Identifying threshold concepts: case study of an open catchment hydraulics course

The Threshold Concept Framework is used to initiate a dialogue on an empirically supported pedagogy that focuses on students’ conceptual understanding required for solving application-based problems. The present paper uses a triangulation approach to identify the threshold concept in a third-year undergraduate civil engineering course on open channel hydraulics. Evidence from teachers, students, and assessment data point to ‘critical flow’ as the threshold concept – a concept that is transformative, integrative, and troublesome. Identifying the threshold concept by engaging various course stakeholders in a dialogue about conceptual understanding and capabilities makes learning visible for all participants in the process. Implementing this approach can result in an empirically driven rationale for adjusting pedagogies and assessments to foster enhanced student learning outcomes.     

概念图在信息技术课堂中的应用

概念图作为一种新型的学与教的工具,可作为信息技术课学生自主学习、教师教学及评价的一种方法。学生自己动手制作概念图,不仅可以促进学生对信息技术有关概念的掌握,而且还能提高学生信息技术的操作能力,有助于培养学生的创造性思维。     

多元智能理论下普职融通试点班学生评价研究

普职融通试点班的学生如果达到一定的标准,可以分流到与其结对的普通高中进行学习和生活。目前,分流的标准是依据学生的考试成绩,但单一的纸笔测验并不能真正反映出学生的综合素质和能力。为了促进学生的全面发展以及个性化发展,运用访谈法与文献分析相结合的方法,从多元智能理论的视角出发,得出普职融通试点班应采用可以量化的纸笔测验和基于学生表现的质性评价即档案袋评价相结合的方法进行学生评价的结论。     

Using Concept Cartoons in Formative Assessment: Scaffolding students’ argumentation

The purpose of this study was to investigate how concept cartoons, together with other diagnostic and scaffolding tools, could be used in formative assessment, to stimulate talk and argumentation among students in small groups, as part of peer‐assessment and self‐assessment; and to provide diagnostic feedback about students’ misconceptions to the teacher for teaching towards conceptual change. Two classes of Primary 5 and 6 students worked in small groups to discuss the opposing viewpoints posed by the cartoon characters, using scaffolding tools to guide their discussions and to evaluate, challenge, and document each others’ ideas. Students also used drawings to depict their ideas. The conversation from one group was audio‐taped. These tools provided a record of students’ thinking in a form that was accessible to the teacher for monitoring and feedback purposes. Findings showed dialogic talk and interactive argumentation among students where they made their reasoning visible. Students’ assertions and questions had formative potential as they encouraged exploratory and reflective discourse by drawing upon each others’ ideas. The teacher’s discursive practices, as well as her role in designing scaffolding structures for supporting ‘assessment conversations’ when using concept cartoons and in devising strategies that take into account students’ conceptual and epistemic thinking, are emphasised.     

英语教学法课学业成就发展性评价研究

本文针对高师英语教学法课测试性评价偏重考察学生对知识的认知情况,缺乏对学生在学习过程中的情感态度、应用能力形成性的价值判断,导致学生不能及时地反思和调整自己的学习状态,非智力因素和后继学习的潜能得不到有效地开发等问题展开研究。重点探讨如何运用发展性评价的原则、方法和工具对学生在英语教学法课学习过程中的认知能力、情感态度和动作技能进行科学、全面地评价,旨在证实发展性评价对促进英语师范生综合能力全面提高的良好效应。     

案例教学法在高校“业绩管理”(F5)课程教学中的应用研究

"业绩管理"(F5)课程涵括成本核算与管理、预算编制与控制、管理决策制定以及业绩评价等内容,要求学生具备一定的综合应用能力,需要学生根据不同商业情景选择适当的分析工具,做出正确的评估。但是在校本科学生缺乏实际工作经验,对外部商业环境知之甚少,这就迫切需要教师在该课程教学中引入案例教学,因为案例教学可深化学生对外部商业环境的了解,而且在常规化观点碰撞和交流中,学生的主观能动性和综合应用能力潜力都会得到充分的刺激和发挥。"业绩管理"(F5)课程的特点要求案例选择和设计时要国际化和本土化并举,要具有针对性和层次性,同时还要与课后习题挂钩。好的案例教学模式既有利于学生发散思维,累积能力,也有助于教师在案例教学准备、实施和总结过程中因材施教,教学相长。     

Psychological Assessment of High Ability: worldwide‐used Wechsler's intelligence scales and their psychometric shortcomings

Because the psychological assessment of high ability usually concentrates on intelligence testing, it is pertinent to discuss the validity of intelligence test batteries. The well‐known Wechsler's scales are analyzed and evaluated. Based on psychometric models, especially the Rasch model, analyses are made of some German editions, which show that hardly a single subtest scores fairly. That is, the true extent of testees’ abilities will not be correctly represented by the scores obtained under current scoring rules. Since many of the items of the analyzed editions correspond to items of the American edition (WISC‐R), the same shortcomings must also be suspect for that test battery. In this light, the administration of these tests is no longer acceptable. However, it is shown that Wechsler's basic concept is worthwhile when accompanied by (modern) psychometric tools: a new (German) test battery, AID, is introduced which, in particular, conforms to economic requirements if high ability is to be assessed.     

Harnessing technology to improve formative assessment of student conceptions in STEM: forging a national network

Concept inventories, consisting of multiple-choice questions designed around common student misconceptions, are designed to reveal student thinking. However, students often have complex, heterogeneous ideas about scientific concepts. Constructed-response assessments, in which students must create their own answer, may better reveal students' thinking, but are time- and resource-intensive to evaluate. This report describes the initial meeting of a National Science Foundation-funded cross-institutional collaboration of interdisciplinary science, technology, engineering, and mathematics (STEM) education researchers interested in exploring the use of automated text analysis to evaluate constructed-response assessments. Participants at the meeting shared existing work on lexical analysis and concept inventories, participated in technology demonstrations and workshops, and discussed research goals. We are seeking interested collaborators to join our research community.     

Enhancing Force Concept Inventory diagnostics to identify dominant misconceptions in first-year engineering physics

This work presents the results of a study whose aim is to detect systematic errors about the concept of force among freshmen students. The researchers analysed the results of the Force Concept Inventory test, which was administered to two different groups of students. The results show that, although there were significant performance variations between the two groups, they, nonetheless, shared common incorrect answers that were consistently triggered by the same misconceptions. The analysis proposed in this paper could also be applied in other universities to reveal the students’ a priori mindset in Newtonian mechanics and serve as a guideline for developing effective computer simulations or other tools.     

New Looks in the Assessment of Cognitive Ability

The past 30 years have produced major changes in the measurement of cognitive ability and the interpretation of assessment results. Theory describing the factorial structure of cognitive ability has blossomed, and the results are visible in several recently published batteries of intellectual ability. The application of better theory to new assessment instruments has been facilitated by advances in the psychometric and statistical tools available to test developers. Attention is drawn to a concern about the capability of many clinicians to appreciate the importance of these changes and to apply them in practice without adequate continuing education.     

高职学生思维风格的实证研究

研究根据田运思维科学的相关理论,从信息学的角度编制了一份二维思维风格问卷。问卷包含主观—客观和能动—惰性两个维度,具有较好的心理测量学指标,可用于相关的研究和实践。调查结果表明:高职学生有4种不同的思维风格类型,总体倾向主观—惰性型,学生的思维风格在不同的地区、年级、专业、性别上均存在显著差异,可用于教师整体地理解学生的个体差异,进而提高对他们行为的预测效度。     

“工业微生物育种”项目化教学实施建设探索

基于“构思—设计—实现—运作”的教育模式和“成果导向”的教育理念,对“工业微生物育种”进行了项目化教学改革,在拆解重组原有课程目标的基础上,以行业内最新且应用广泛的案例项目为教学载体;强调理论与实践有机结合,让学生成为学习过程的主导,激发学习兴趣;最终达到帮助学生重构专业知识能力、形成实践能力及提升人文素养的人才培养要求。同时,采用多元化、过程化的考核手段,从学生的知识、能力、素养等方面做出多维度的评价,建立过程化考核评价体系。课程实施改革后取得了良好的教学效果,得到了学生的普遍认可,提高了学生学习专业知识的兴趣和主动性。     

Building, using, and maximizing the impact of concept inventories in the biological sciences: report on a National Science Foundation sponsored conference on the construction of concept inventories in the biological sciences

The meeting “Conceptual Assessment in the Biological Sciences” was held March 3–4, 2007, in Boulder, Colorado. Sponsored by the National Science Foundation and hosted by University of Colorado, Boulder's Biology Concept Inventory Team, the meeting drew together 21 participants from 13 institutions, all of whom had received National Science Foundation funding for biology education. Topics of interest included Introductory Biology, Genetics, Evolution, Ecology, and the Nature of Science. The goal of the meeting was to organize and leverage current efforts to develop concept inventories for each of these topics. These diagnostic tools are inspired by the success of the Force Concept Inventory, developed by the community of physics educators to identify student misconceptions about Newtonian mechanics. By working together, participants hope to lessen the risk that groups might develop competing rather than complementary inventories.