A strategy for detection of inconsistency in evaluation of essay type answers

The quality of evaluation of essay type answer books involving multiple evaluators for courses with large number of enrollments is likely to be affected due to heterogeneity in experience, expertise and maturity of evaluators. In this paper, we present a strategy to detect anomalies in evaluation of essay type answers by multiple evaluators based on the relationship between marks/grades awarded and symbolic markers, opinionated words recorded in answer books during evaluation. Our strategy is based on the results of our survey with evaluators, analysis of large number of essay type evaluated answer books and our own experiences regarding grievances of students regarding marks/grades. Results of both survey and analysis of evaluated answer books identified underline, tick and cross as frequently used markers compared to circle and question mark. Further, both opinionated words and symbolic markers identified through the survey are used by evaluators to express either positive or negative sentiments. They have differential usage pattern of these symbols as single evaluator and as one amongst multiple evaluators. Tick and cross have well define purposes and have strong correlation with marks awarded. However, the underline marker is being used for dual purpose of expressing both correctness and incorrectness of answers. Our strategy of inconsistency detection first identifies outliers based on the relationship between marks/grades awarded and number of symbols and/or opinionated words used in evaluation. Subsequently, marks and number of symbolic markers of outliers are compared with peer non-outlier answer books having same marks but different number of markers used. Such outlier answer books are termed as anomalous. We discovered 36 anomalies out of total 425 evaluated answer books. We have developed a prototype tool to facilitate online evaluation of answer book and to proactively alert evaluators of possible anomalies.     

医患交流中患者扩展回答的会话分析研究

面对医生的确认类提问,患者进行扩展回答是医患交流中一种较为常见的现象.尽管扩展回答在交流层面违背提问对回答的结构限定,而且还占用更多的交流时间,但患者的扩展回答并不是一种消极的交流现象.本文采用会话分析的研究方法,以发生在泌尿外科门诊的医患交流为语料,探究患者扩展回答的具体原因及序列语境,分析扩展回答的价值和意义.通过...     

Dialogue in the elementary school mathematics classroom: A comparative study between expert and novice teacher

Using classroom observation and video recording methods, we performed a comparative study on the forms and content of dialogues in the classrooms between expert and novice teachers. Of the 55 lessons surveyed, it was found that expert teachers tend to use analytical and comparative questions more frequently to detect students’ mathematical reasoning. Students and teachers work together to determine the answer to a question and the dialogue in the classroom takes place in a way that students present an answer, the teacher and the other students question the answer, and then the students explain the answer. On the other hand, a novice teacher often tends to give students hints, or utilize simple questions to jog the memory. The novice teacher recognizes students’ logic but does not incorporate them into his/her teaching. In this case, the teacher becomes the sole judge for the appropriateness of the answers and the typical dialogue in the classroom occurs in a way that the teacher asks a question, students answer, and the teacher comments.     

小学数学课堂对话的特点:对专家教师与非专家教师的比较

Using classroom observation and video recording methods, we performed a comparative study on the forms and content of dialogues in the classrooms between expert and novice teachers. Of the 55 lessons surveyed, it was found that expert teachers tend to use analytical and comparative questions more frequently to detect students’ mathematical reasoning. Students and teachers work together to determine the answer to a question and the dialogue in the classroom takes place in a way that students present an answer, the teacher and the other students question the answer, and then the students explain the answer. On the other hand, a novice teacher often tends to give students hints, or utilize simple questions to jog the memory. The novice teacher recognizes students’ logic but does not incorporate them into his/her teaching. In this case, the teacher becomes the sole judge for the appropriateness of the answers and the typical dialogue in the classroom occurs in a way that the teacher asks a question, students answer, and the teacher comments.     

Written justifications to multiple-choice concept questions during active learning in class

Increasingly, instructors of large, introductory STEM courses are having students actively engage during class by answering multiple-choice concept questions individually and in groups. This study investigates the use of a technology-based tool that allows students to answer such questions during class. The tool also allows the instructor to prompt students to provide written responses to justify the selection of the multiple-choice answer that they have chosen. We hypothesize that prompting students to explain and elaborate on their answer choices leads to greater focus and use of normative scientific reasoning processes, and will allow them to answer questions correctly more often. The study contains two parts. First, a crossover quasi-experimental design is employed to determine the influence of asking students to individually provide written explanations (treatment condition) of their answer choices to 39 concept questions as compared to students who do not. Second, we analyze a subset of the questions to see whether students identify the salient concepts and use appropriate reasoning in their explanations. Results show that soliciting written explanations can have a significant influence on answer choice and, when it does, that influence is usually positive. However, students are not always able to articulate the correct reason for their answer.     

High School Teachers’ Understanding of Blackbody Radiation

This study is a detailed look at the level of understanding of fundamental ideas about blackbody radiation (BBR) among physics teachers. The aim is to explore associations and ideas that teachers have regarding blackbody radiation: a concept used routinely in physics and chemistry, which is necessary to understand fundamentals of quantum physics. In this explorative study, the responses of 106 high school physics teachers from 18 different cities in Turkey were examined using an online questionnaire during the 2014/2015 academic year. Male (88 %) and female (12 %) teachers with an average teaching experience of 17 years participated in this study. Teachers were asked about the concept of a “blackbody” and its radiation as well as about the closely related topics such as Wien’s law and the historic effect of understanding BBR on the development of modern physics. In this study, teachers’ spontaneous ideas about the BBR are depicted by “answer sculptures.” The answer sculpture is a new way of representing and categorizing responses provided by the participants. The analysis of answer sculptures revealed that the physics teachers’ factual knowledge and their conceptual understanding of the BBR are incomplete and even misleading. Here is what teachers we surveyed make of BBR as presented in our answer sculptures. This study is left at investigating teachers’ understandings of (1) blackbody radiation and (2) the significance of this scientific breakthrough.     

企业技术技能人才向职教教师转换——德国的经验与启示

《国家职业教育改革实施方案》规定2019年起职业院校将从企业招聘职教教师.为此,我国职教教师教育与研究亟需解决"如何选拔企业人才进入职教师资队伍"及"如何促使其最终向'双师型'教师转换"问题.德国的经验显示:构建在线测试、专业咨询和实习的完整过程,采用"做中学"培养方式,结合多方、多阶段评价,有助于实现企业技术技能人才...     

The development of speech act comprehension in mentally retarded individuals and nonretarded children

Understanding the speech act(s) a sentence conveys requires that a listener follow conversational rules and use clues provided by the sentence and the context in which it is spoken. To trace the development of speech act comprehension in mentally retarded and nonretarded school age children, we examined their responses to sentences such as Would you open the telephone book? and Could you roll the shoebox? These sentences can be intended as questions or as directives. Nonretarded adults follow the "answer obviousness" rule and interpret a sentence of this type as a question if the answer to the question is nonobvious to the speaker, but as a directive if the question has an obvious answer. We manipulated answer obviousness by varying both the context and linguistic properties of the interrogative. Retarded and nonretarded individuals at the nonverbal MAs of 5, 7, and 9 years were studied. Retarded and nonretarded individuals at all MAs followed the answer obviousness rule and used the contextual and linguistic clues available. There were only minor differences between MA-matched retarded and nonretarded subjects despite the fact that the retarded subjects had serious deficits in receptive linguistic competence. The implications for understanding the development of comprehension in retarded persons and for understanding the relations among language, cognition, and communication in general are considered.     

Introduction and training of students to use separate answer sheets: Effects on standardized test scores

Previous research has found conflicting evidence regarding how early children can effectively use separate answer sheets with achievement tests. This study looked at the effects of separate answer sheets on the California Achievement Test (CAT) scores of third, fourth, and fifth graders. The Mathematics Computation and the Reading Comprehension subtests of the CAT were used. Seventy-one classrooms were randomly assigned to have students record their answers on either: (a) their test booklets, (b) separate answer sheets, or (c) separate answer sheets after being given training in the use of separate answer sheets. The results were consistent across both subtests and grades; no response mode treatment effect was found. Further, no evidence of a treatment by ability interaction was found, which was contrary to previously reported research. The results of this study suggest that students can, as early as grade three, effectively use separate answer sheets without prior training.     

A New Statistic for Detection of Aberrant Answer Changes

As noted by Fremer and Olson, analysis of answer changes is often used to investigate testing irregularities because the analysis is readily performed and has proven its value in practice. Researchers such as Belov, Sinharay and Johnson, van der Linden and Jeon, van der Linden and Lewis, and Wollack, Cohen, and Eckerly have suggested several statistics for detection of aberrant answer changes. This article suggests a new statistic that is based on the likelihood ratio test. An advantage of the new statistic is that it follows the standard normal distribution under the null hypothesis of no aberrant answer changes. It is demonstrated in a detailed simulation study that the Type I error rate of the new statistic is very close to the nominal level and the power of the new statistic is satisfactory in comparison to those of several existing statistics for detecting aberrant answer changes. The new statistic and several existing statistics were shown to provide useful information for a real data set. Given the increasing interest in analysis of answer changes, the new statistic promises to be useful to measurement practitioners.     

Score Gains, Attitudes, and Behavior Changes Due to Answer-Changing Instruction

Six undergraduate and three graduate classes were given multiple-choice tests with subsequent evaluation of answer changes. The 300 students were tested twice, once before and once after instruction on answer changing. After each test, students were asked to complete two forms. The forms evaluated attitude toward answer changing, reasons for changing, and confidence in final answers. Students showed a significant increase in favorability toward answer changing after instruction. No significant change was found in number of answers changed. Psychology students were found to change significantly more items than were business students. Mean gain score did not change significantly after instruction. It was concluded that although instruction does lead to a change in attitude in answer changing, the number of changes and overall gain due to answer changing do not change. It was also determined that students continue to make significant gains even when their confidence in the final answer is less than 50 on a 100-point scale.     

Answer Changing on Multiple-Choice Test Items Among Eighth-Grade Readers

This study was done to examine the effect of answer changing on multiple-choice test performance among good and poor readers in the eighth grade. Although the gains of poor readers were higher than those of good readers, all subjects profited significantly from changing their answers on items. For all subjects, when a single response was changed, there was a two-to-one chance that the new response would raise rather than lower the final score. Gains from answer changing on test items were slightly higher for poor readers as a group than were those for good readers. However, the result was determined not to be significant. More important, this hypothesis is strengthened by the fact that all subjects profited from answer changing. Therefore, the results were interpreted as lending support to the notion that answer-changing response among young examinees should be encouraged if there is a reasonable doubt about their “first impression.”     

Answer changing in testing situations: the role of metacognition in deciding which answers to review

The purpose of our study was to examine the issue of answer changing in relation to students’ abilities to monitor their behaviour accurately while responding to multiple-choice tests. The data for this study were obtained from the final examination administered to students in an educational psychology course. The results of the study indicate that decision making regarding answer changing can be informed by metacognitive monitoring when a student's attention is drawn to cues solicited during testing. However, answer changing can be a “tricky” test-taking strategy, and how it relates to test performance is based on an interplay of variables both at a person level as well as at a task level. This is probably why this strategy is frequently misunderstood by both test-takers and educators.     

TEST RELIABILITY AS A FUNCTION OF ANSWER SHEET MODE

The purpose of this study was to examine the effect of using separate, machine scorable answer sheets on the number of marking errors made by third-, fourth-, and sixth-grade students. Half of the Ss at each grade level answered in the test booklet (AIB), and half answered on separate answer sheets (SAS). The number of marking errors for the SAS group was approximately three times as great as for the AIB group at each grade level. This indicates that test standardization information must specify the answer sheet mode used to collect normative data.     

Partial Identification of Answer Reviewing Effects in Multiple-Choice Exams

Does reviewing previous answers during multiple-choice exams help examinees increase their final score? This article formalizes the question using a rigorous causal framework, the potential outcomes framework. Viewing examinees’ reviewing status as a treatment and their final score as an outcome, the article first explains the challenges of identifying the causal effect of answer reviewing in regular exam-taking settings. In addition to the incapability of randomizing the treatment selection (reviewing status) and the lack of other information to make this selection process ignorable, the treatment variable itself is not fully known to researchers. Looking at examinees’ answer sheet data, it is unclear whether an examinee who did not change his or her answer on a specific item reviewed it but retained the initial answer (treatment condition) or chose not to review it (control condition). Despite such challenges, however, the article develops partial identification strategies and shows that the sign of the answer reviewing effect can be reasonably inferred. By analyzing a statewide math assessment data set, the article finds that reviewing initial answers is generally beneficial for examinees.     

How does rocket propulsion work?

The most common answer to the above question is — hot jet of gas comes out of the nozzle of the rocket engine at high speeds and as a reaction the rocket moves (is propelled) in the opposite direction [1]. But is this answer right? Let us explore what goes on inside a rocket engine and arrive at the right answer.     

文学研究与教育是否该有"标准答案"

文章有感于中国文学研究及教育的传统与现状,提出当前文学研究与教育中的一个问题,即究竟(应该)有无“标准答案”的问题,并给出初步的理论上的回答。问题提出的价值在于它的确是个问题。文章重在“解构”,而旨在引起关注并对由此问题必然而至的文学研究与教育中一系列的具体相关问题进行深入的讨论。     

Clickers in college classrooms: Fostering learning with questioning methods in large lecture classes

What can be done to promote student–instructor interaction in a large lecture class? One approach is to use a personal response system (or “clickers”) in which students press a button on a hand-held remote control device corresponding to their answer to a multiple choice question projected on a screen, then see the class distribution of answers on a screen, and discuss the thinking that leads to the correct answer. Students scored significantly higher on the course exams in a college-level educational psychology class when they used clickers to answer 2 to 4 questions per lecture (clicker group), as compared to an identical class with in-class questions presented without clickers (no-clicker group, d = 0.38) or with no in-class questions (control group, d = 0.40). The clicker treatment produced a gain of approximately 1/3 of a grade point over the no-clicker and control groups, which did not differ significantly from each other. Results are consistent with the generative theory of learning, which predicts students in the clicker group are more cognitively engaged during learning.     

光电读卡机可行性设计

光电读卡机由上下位机两部分组成。下位机是反射式光电传感器作为答题卡信息采集工具,辨别出答题卡上相应的答案是否被涂黑。传感器输出的信号经电压比较器调理变为比较规则的信号送至微控制器,进行简单处理,把获取的信息变成选择题答案的形式暂存起来。每读完一张卡,把单片机暂存的数据通过串口通讯RS -22发送到上位机。然后,由基于虚拟仪器开发平台Labview的评分系统对读取的答案数据与标准答案比对,得出考生最终得分,并对所有考生的成绩进行相关处理和存储。     

THE USE OF SEPARATE ANSWER SHEETS BY PRIMARY AGE CHILDREN1

The purpose of this study was to determine the effects on test scores of employing a separate answer sheet using the California Test of Mental Maturity with 117 pupils in the primary grades of one through three. The CTMM was administered twice to all students; once employing the usual answer marking format contained in the test booklet and once employing a separate answer sheet. A counterbalanced design was used to control for order effects. The mean raw differences between marking format conditions decreased from 23.67 to 10.79 to 3.32 for students in grades one through three respectively, indicating that the need for the test booklet marking format decreases sharply as the grade level of the Ss increases.