To Score or Not to Score: Factors Influencing Performance and Feasibility of Automatic Content Scoring of Text Responses

In this article, we systematize the factors influencing performance and feasibility of automatic content scoring methods for short text responses. We argue that performance (i.e., how well an automatic system agrees with human judgments) mainly depends on the linguistic variance seen in the responses and that this variance is indirectly influenced by other factors such as target population or input modality. Extending previous work, we distinguish conceptual, realization, and nonconformity variance, which are differentially impacted by the various factors. While conceptual variance relates to different concepts embedded in the text responses, realization variance refers to their diverse manifestation through natural language. Nonconformity variance is added by aberrant response behavior. Furthermore, besides its performance, the feasibility of using an automatic scoring system depends on external factors, such as ethical or computational constraints, which influence whether a system with a given performance is accepted by stakeholders. Our work provides (i) a framework for assessment practitioners to decide a priori whether automatic content scoring can be successfully applied in a given setup as well as (ii) new empirical findings and the integration of empirical findings from the literature on factors that influence automatic systems' performance.     

To Block or Not To Block: That's Not the Question

Abstract

The block schedule emerged in the late 1980s as one structuring practice that allowed educators to address several problems associated with the traditional high school schedule. Among its posited advantages is the opportunity for teachers to engage students in active teaching strategies that require more time than the traditional schedule allows. Research on the use of teaching strategies in traditional and block schedule high schools is limited. Similarities and differences in survey responses of more than 2,000 teachers working in traditional- and block-schedule schools were described and analyzed in this study. In general, few differences were evident in opinions about level of use, appropriateness, and training for a variety of instruction approaches.     

To Be or Not to Be:A Comparison between Two Novels

Both the two short novels The Short Happy Life of Francis Macombev by Hemingway and Barn Burning by Faulkner are well-known.Macomber and Sartoris are protagonists in these two novels respectively.Both of them were betrayers to some one and to themselves;both of them had the important transformation in their life and grew mature.This article is making a comparison between these two novels from the perspective of rite of passage to explore their common theme:To Be is much more important than Not To Be,a life with self-determination and self-respect is much more significant than a life without self-judgment and dominated.     

To Boxplot or Not to Boxplot?

The lower quartile is the median of all of the values to the left of the median of the whole set of data; the upper quartile is the median of all of the values to the right of the median of the whole set of data.     

To Challenge or Not To Challenge: The Mentor's Dilemma

This paper focuses upon open and distance learning initial teacher education in both primary and secondary sectors and explores the ways in which mentors challenge PGCE students in their professional thinking and classroom practice. It draws upon ethnographic research on mentors and student teachers during one presentation of the course. The data are based on observations of mentors working with students on final student placement, conversations with specialist subject mentor trainers as well as questionnaire and other source material. An analysis of how mentors articulate their knowledge base; the dialogue between mentor and student and the challenge of moving from novice to expert for the student teacher are all considered. Our conclusions have implications for all ITT and will also be a source of valuable practical help for student teachers and mentors.     

To Feedback or Not to Feedback in Student Self-assessment

Extensive research (Black & Wiliam, 1998) notes the importance of feedback for learning. It is posited that even final-year undergraduate students will benefit from feedback and would thus value tutor feedback as an integral part of the student self-assessment process. At a British university, students were trained to carry out self-assessment of two types: firstly, self-assessment prior to peer and tutor feedback and secondly, self-assessment which integrated feedback as part of the process. Students overwhelmingly favoured the latter. In addition, analysis of student errors showed that, contrary to expectations, the majority of students' errors in translation were principally caused by their native language. Furthermore, discussion and questions on criteria revealed that students' expectations of their grades were closely related to the amount of time and effort which they had invested in their work. This was external to the actual standard of their work and their awareness of this.     

Validity and Automad Scoring: It's Not Only the Scoring

What are the validity issues involved in automated scoring of tests? What is the nature of the interplay among construct definition, task design, examinee interface, tutorial, test development tools, and automated scoring and reporting?     

To Apply or Not to Apply: FAFSA Completion and Financial Aid Gaps

In the United States, college students must complete the Free Application for Student Federal Aid (FAFSA) to access federal aid. However, many eligible students do not apply and consequently forgo significant amounts of financial aid. If students have perfect information about aid eligibility, we would expect that all eligible students complete FAFSA and no aid would go unclaimed. Using data from the National Postsecondary Student Aid Survey, I estimate a multinomial logit model which controls for all variables that contribute to aid eligibility and other student characteristics that may deter FAFSA completion. I find that students who are lower middle income, white, male and independent from parents are less likely to complete FAFSA even when they are eligible for aid. Using propensity score matching, I find that each year applicants forgo $9,741.05 in total aid (including grant and loan aid) which includes $1,281.00 of Pell Grants, $2,439.50 of the balance subsidized student loans, $1,986.65 of the balance of unsubsidized student loans, and $1,016.04 of institutional grants. These aid totals aggregate to $24 billion annually.     

Subscores Based on Classical Test Theory: To Report or Not to Report

There is an increasing interest in reporting subscores, both at examinee level and at aggregate levels. However, it is important to ensure reasonable subscore performance in terms of high reliability and validity to minimize incorrect instructional and remediation decisions. This article employs a statistical measure based on classical test theory that is conceptually similar to the test reliability measure and can be used to determine when subscores have any added value over total scores. The usefulness of subscores is examined both at the level of the examinees and at the level of the institutions that the examinees belong to. The suggested approach is applied to two data sets from a basic skills test. The results provide little support in favor of reporting subscores for either examinees or institutions for the tests studied here.     

To See or Not To See: Preservice Teacher Attitudes Toward Color Blindness

This study examined the perspectives of 85 undergraduate preservice teacher candidates engaged in a required survey course on diversity at a midsized Midwestern university. Candidates completed the Common Beliefs Survey at both the beginning and end of the 16-week semester. Quantitative analysis of the pre- and postcourse Likert ratings of candidates indicated a significant negative shift relative to the statement, “I don't think of my students in terms of their race or ethnicity; I am color-blind when it comes to my teaching.” Further qualitative analysis of candidates who provided written rationale for their numeric response indicated prevailing themes of conscious bias, intentional color-blindness, and movement toward appreciation of racial difference as a critical cognizance. Implications for teacher education include experiences beyond the scope of a survey course if significant schematic shift leading to the potential for educational equity is to be realized.     

To Be Polite or Not to Be Polite, That Is A Big Problen

Ｉ.ＴＨＥＴＷＯＰＲＩＮＣＩＰＬＥＳＩｎ 1 96 7ＧｒｉｃｅｐｒｏｐｏｓｅｄｔｈｅｌａｔｅｒｆａｍｏｕｓＣＯＯＰＥＲＡＴＩＶＥＰＲＩＮＣＩＰＬＥｗｈｉｃｈｈａｓｆｏｕｒｍａｘｉｍｓａｓｆｏｌｌｏｗｓ:(1 )ＴｈｅｍａｘｉｍｏｆｑｕａｌｉｔｙＴｒｙｔｏｍａｋｅｙｏｕｒｃｏｎｔｒｉｂｕｔｉｏｎｏｎｅｔｈａｔｉｓｔｒｕｅ,ｓｐｅｃｉｆｉｃａｌｌｙ :(ｉ)ｄｏｎｏｔｓａｙｗｈａｔｙｏｕｂｅｌｉｅｖｅｔｏｂｅｆａｌｓｅ(ｉｉ)ｄｏｎｏｔｓａｙｔｈａｔｆｏｒｗｈｉｃｈｙｏｕｌａｃｋａｄｅｑｕａｔｅｅｖｉｄｅｎｃｅ(2 )ｔ…     

捐,还是不捐,这是一个问题

2004年岁末,突如其来的海啸震惊了全世界, 也包括我所在的学校。灾难发生后几天,高二有个班 发起了募捐活动,很快,学校德育处接手了此事,一 场为灾民献爱心的活动轰轰烈烈地开展起来了。学 校规定,学生捐款以班为单位,班主任随本班捐款, 并准备在周一的升旗仪式上举行一个捐赠仪式。     

To Parcel or Not to Parcel: Exploring the Question,Weighing the Merits

We examine the controversial practice of using parcels of items as manifest variables in structural equation modeling (SEM) procedures. After detailing arguments pro and con, we conclude that the unconsidered use of parcels is never warranted, while, at the same time, the considered use of parcels cannot be dismissed out of hand. In large part, the decision to parcel or not depends on one's philosophical stance regarding scientific inquiry (e.g., empiricist vs. pragmatist) and the substantive goal of a study (e.g., to understand the structure of a set of items or to examine the nature of a set of constructs). Prior to creating parcels, however, we recommend strongly that investigators acquire a thorough understanding of the nature and dimensionality of the items to be parceled. With this knowledge in hand, various techniques for creating parcels can be utilized to minimize potential pitfalls and to optimize the measurement structure of constructs in SEM procedures. A number of parceling techniques are described, noting their strengths and weaknesses.