Automated Scoring of Constructed‐Response Science Items: Prospects and Obstacles

Content‐based automated scoring has been applied in a variety of science domains. However, many prior applications involved simplified scoring rubrics without considering rubrics representing multiple levels of understanding. This study tested a concept‐based scoring tool for content‐based scoring, c‐rater?, for four science items with rubrics aiming to differentiate among multiple levels of understanding. The items showed moderate to good agreement with human scores. The findings suggest that automated scoring has the potential to score constructed‐response items with complex scoring rubrics, but in its current design cannot replace human raters. This article discusses sources of disagreement and factors that could potentially improve the accuracy of concept‐based automated scoring.     

Generalization of the Lord‐Wingersky Algorithm to Computing the Distribution of Summed Test Scores Based on Real‐Number Item Scores

With known item response theory (IRT) item parameters, Lord and Wingersky provided a recursive algorithm for computing the conditional frequency distribution of number‐correct test scores, given proficiency. This article presents a generalized algorithm for computing the conditional distribution of summed test scores involving real‐number item scores. The generalized algorithm is distinct from the Lord‐Wingersky algorithm in that it explicitly incorporates the task of figuring out all possible unique real‐number test scores in each recursion. Some applications of the generalized recursive algorithm, such as IRT test score reliability estimation and IRT proficiency estimation based on summed test scores, are illustrated with a short test by varying scoring schemes for its items.     

无锚题测验等值设计方法研究进展

由于测验安全性、试卷组卷不当等问题,有些测验的题本相互之间不能或者没有设置锚题。对作答不同题本的被试进行分数比较时,需要用到测验等值技术。不同于有锚题测验能通过题本之间的锚题进行等值,无锚题情境下的测验需要借助于一些特殊方法进行等值。目前,对无锚题测验进行等值主要有三种方式,一种是通过测验中具体的题目,也就是构建相同的"锚题"来进行等值,如构造随机等组测验法和利用题目先验信息进行等值的方法;一种是通过构建相同被试组来进行等值,即构造随机等组样本法;还有一种是借助于测验题目所考查的认知属性来进行等值,一般是基于一种认知诊断模型——规则空间模型来进行操作。     

A Common Pitfall in the Use of the Kolmogorov-Smirnov One Sample Test

This article concerns the Kolmogorov-Smirnov one sample test which was introduced by the Russian mathematician A. N. Kolmogorov in 1933.     

试后试题全公开背景下分数分布的跨年度比较——日本全国性测验与地方性测验的链接

本研究的目的有三：（1）提出试后试题全公开背景下分数分布的跨年度比较方案,即通过组合日本的全国性测验与地区性测验的设计,应用测验理论中的链接原理提出跨年度比较分数分布的方法;（2）讨论实现该方案的可行性,具体讨论了使用测验数据的可能性、地区性协作的方式以及对于被试群体的要求;（3）进行实际数据的证实,即呈现2006年度与2009年度初中三年级学生国语测验分数的跨年度比较结果,发现无论哪个测验的分数分布都基本上没有变化。     

Performance of a Proposed Method for the Linking of Mixed Format Tests With Constructed Response and Multiple Choice Items

The error associated with a proposed linking method for tests consisting of both constructed response and multiple choice items was investigated in a simulation study. Study factors that were varied included the relative proportion of constructed response items in the test, the size of the year-to-year change in the ability metric, the number of anchor items, the number of linking papers to be reassessed, and the presence of guessing. The results supported the use of the proposed linking method, In addition, simulations were used to illustrate possible linking bias resulting from (a) the use of the traditional linking method and (b) the use of only multiple choice anchor items in the presence of test multidimensionality.     

A Validity Argument in Support of the Use of College Admissions Test Scores for Federal Accountability

In 2018, 26 states administered a college admissions test to all public school juniors. Nearly half of those states proposed to use those scores as their academic achievement indicators for federal accountability under the Every Student Succeeds Act (ESSA); many others are planning to use those scores for other accountability purposes. Accountability encompasses a number of different uses and subsumes a variety of claims. For states proposing to use summative tests for accountability, a validity argument needs to be developed, which entails delineating each specific use of test scores associated with accountability, identifying appropriate evidence, and offering a rebuttal to counterclaims. The aim of this article is to support states in developing a validity argument for use of college admission test scores for accountability by identifying claims that are applicable across states, along with summarizing existing evidence as it relates to each of these claims. As outlined by The Standards for Educational and Psychological Testing, multiple sources of evidence are used to address each claim. A series of threats to the validity argument, including weaker alignment with content standards and potential influences in narrowing teaching, are reviewed. Finally, the article contrasts validity evidence, primarily from research on the ACT, with regulatory requirements from ESSA. The Standards and guidance addressing the use of a “nationally recognized high school academic assessment” (Elementary and Secondary Education Act (ESEA), Negotiated Rulemaking Committee; Department of Education) are the primary sources for the organization of validity evidence.     

用LONGFORD方法对主观评分中的异常分数进行分数调整

分数不确切代表被试的真实语言能力的问题是语言测量学界一个最本质、最棘手的问题——效度问题。以往我们采取的一些诸如增加评分员数量、重评等办法虽然在一定程度上改善了效度,但是却都无法从真正意义上得到一个与真分数尽可能近似的客观的分数。Longford针对主观评分中的信度问题提出了四种分数调整模型来解决这一问题。本文运用严厉度调整模型对HSK高等作文评分中的异常评分者所评的分数进行了调整,调整后分数得到很大改善。因此在以后的考试当中基本上可以用这种数学的调整方法代替以往组织评分员重评的方法。     

Preschoolers’ Use of Talker Information in On‐Line Comprehension

A crucial part of language development is learning how various social and contextual language‐external factors constrain an utterance’s meaning. This learning process is poorly understood. Five experiments addressed one hundred thirty‐one 3‐ to 5‐year‐old children’s use of one such socially relevant information source: talker characteristics. Participants learned 2 characters’ favorite colors; then, those characters asked participants to select colored shapes, as eye movements were tracked. Results suggest that by preschool, children use voice characteristics predictively to constrain a talker’s domain of reference, visually fixating the talker’s preferred color shapes. Indicating flexibility, children used talker information when the talker made a request for herself but not when she made a request for the other character. Children’s ease at using voice characteristics and possible developmental changes are discussed.     

Controlling Bias in Both Constructed Response and Multiple‐Choice Items When Analyzed With the Dichotomous Rasch Model

Even though guessing biases difficulty estimates as a function of item difficulty in the dichotomous Rasch model, assessment programs with tests which include multiple‐choice items often construct scales using this model. Research has shown that when all items are multiple‐choice, this bias can largely be eliminated. However, many assessments have a combination of multiple‐choice and constructed response items. Using vertically scaled numeracy assessments from a large‐scale assessment program, this article shows that eliminating the bias on estimates of the multiple‐choice items also impacts on the difficulty estimates of the constructed response items. This implies that the original estimates of the constructed response items were biased by the guessing on the multiple‐choice items. This bias has implications for both defining difficulties in item banks for use in adaptive testing composed of both multiple‐choice and constructed response items, and for the construction of proficiency scales.     

Comparison of Methods for Combining the Minimum Passing Levels for Individual Items into a Passing Score for a Test

The purpose of this study was to compare several methods for determining a passing score on an examination from the individual raters' estimates of minimal pass levels for the items. The methods investigated differ in the weighting that the estimates for each item receive in the aggregation process. An IRT-based simulation method was used to model a variety of error components of minimum pass levels. The results indicate little difference in estimated passing scores across the three methods. Less error was present when the ability level of the minimally competent candidates matched the expected difficulty level of the test. No meaningful improvement in passing score estimation was achieved for a 50-item test as opposed to a 25-item test; however, the RMSE values for estimates with 10 raters were smaller than those for 5 raters. The results suggest that the simplest method for aggregating minimum pass levels across the items in a test–adding them up–is the preferred method.     

The Impact of Examinee Performance Information on Judges’ Cut Scores in Modified Angoff Standard‐Setting Exercises

This research evaluated the impact of a common modification to Angoff standard‐setting exercises: the provision of examinee performance data. Data from 18 independent standard‐setting panels across three different medical licensing examinations were examined to investigate whether and how the provision of performance information impacted judgments and the resulting cut scores. Results varied by panel but in general indicated that both the variability among the panelists and the resulting cut scores were affected by the data. After the review of performance data, panelist variability generally decreased. In addition, for all panels and examinations pre‐ and post‐data cut scores were significantly different. Investigation of the practical significance of the findings indicated that nontrivial fail rate changes were associated with the cut score changes for a majority of standard‐setting exercises. This study is the first to provide a large‐scale, systematic evaluation of the impact of a common standard setting practice, and the results can provide practitioners with insight into how the practice influences panelist variability and resulting cut scores.     

Assessing Fit of Item Response Models Using the Information Matrix Test

The information matrix can equivalently be determined via the expectation of the Hessian matrix or the expectation of the outer product of the score vector. The identity of these two matrices, however, is only valid in case of a correctly specified model. Therefore, differences between the two versions of the observed information matrix indicate model misfit. The equality of both matrices can be tested with the so‐called information matrix test as a general test of misspecification. This test can be adapted to item response models in order to evaluate the fit of single items and the fit of the whole scale. The performance of different versions of the test is compared in a simulation study with existing tests of model fit, among them the test of Orlando and Thissen, the score test of local independence due to Glas and Suarez‐Falcon, and the limited information approach of Maydeu‐Olivares and Joe. In general, the different versions of the information matrix test adhere to the nominal Type I error rate and have high power for detecting misspecified item characteristic curves. Additionally, some versions of the test can be used in order to detect violations of the local independence assumption.     

SPSS软件在高等数学试卷成绩分析中的应用

利用SPSS统计软件对高等数学试卷成绩进行频数分析、难度分析、区分度分析、信度分析,给出了较合理的评价.分析过程表明,应用SPSS统计软件对学生的高等数学试卷成绩进行分析是非常方便适用的.     

Predicting Freshman Grade‐Point Average from Test Scores: Effects of Variation Within and Between High Schools

Most studies predicting college performance from high‐school grade point average (HSGPA) and college admissions test scores use single‐level regression models that conflate relationships within and between high schools. Because grading standards vary among high schools, these relationships are likely to differ within and between schools. We used two‐level regression models to predict freshman grade point average from HSGPA and scores on both college admissions and state tests. When HSGPA and scores are considered together, HSGPA predicts more strongly within high schools than between, as expected in the light of variations in grading standards. In contrast, test scores, particularly mathematics scores, predict more strongly between schools than within. Within‐school variation in mathematics scores has no net predictive value, but between‐school variation is substantially predictive. Whereas other studies have shown that adding test scores to HSGPA yields only a minor improvement in aggregate prediction, our findings suggest that a potentially more important effect of admissions tests is statistical moderation, that is, partially offsetting differences in grading standards across high schools.     

Simulating the Use of Disclosed Items in Computerized Adaptive Testing

Regular use of questions previously made available to the public (i.e., disclosed items) may provide one way to meet the requirement for large numbers of questions in a continuous testing environment, that is, an environment in which testing is offered at test taker convenience throughout the year rather than on a few prespecified test dates. First it must be shown that such use has effects on test scores small enough to be acceptable. In this study simulations are used to explore the use of disclosed items under a worst-case scenario which assumes that disclosed items are always answered correctly. Some item pool and test designs were identified in which the use of disclosed items produces effects on test scores that may be viewed as negligible.     

求多元函数最值的常用方法

求最值问题是中等数学永恒的话题,其中,多元函数求最值是难点.求多元函数最值的常用方法有:消元法、均值不等式法、换元法、数形结合法、柯西不等式法、向量法等,结合例题将这些方法加以总结.     

求多元函数最值的常用方法

求最值问题是中等数学永恒的话题,其中,多元函数求最值是难点。求多元函数最值的常用方法有：消元法、均值不等式法、换元法、数形结合法、柯西不等式法、向量法等,结合例题将这些方法加以总结。