Estimating High School GPA Weighting Parameters With a Graded Response Model

The high school grade point average (GPA) is often adjusted to account for nominal indicators of course rigor, such as “honors” or “advanced placement.” Adjusted GPAs—also known as weighted GPAs—are frequently used for computing students’ rank in class and in the college admission process. Despite the high stakes attached to GPA, weighting policies vary considerably across states and high schools. Previous methods of estimating weighting parameters have used regression models with college course performance as the dependent variable. We discuss and demonstrate the suitability of the graded response model for estimating GPA weighting parameters and evaluating traditional weighting schemes. In our sample, which was limited to self‐reported performance in high school mathematics courses, we found that commonly used policies award more than twice the bonus points necessary to create parity for standard and advanced courses.     

A Comparison of Limited-Information and Full-Information Methods in Mplus for Estimating Item Response Theory Parameters for Nonnormal Populations

In structural equation modeling software, either limited-information (bivariate proportions) or full-information item parameter estimation routines could be used for the 2-parameter item response theory (IRT) model. Limited-information methods assume the continuous variable underlying an item response is normally distributed. For skewed and platykurtic latent variable distributions, 3 methods were compared in Mplus: limited information, full information integrating over a normal distribution, and full information integrating over the known underlying distribution. Interfactor correlation estimates were similar for all 3 estimation methods. For the platykurtic distribution, estimation method made little difference for the item parameter estimates. When the latent variable was negatively skewed, for the most discriminating easy or difficult items, limited-information estimates of both parameters were considerably biased. Full-information estimates obtained by marginalizing over a normal distribution were somewhat biased. Full-information estimates obtained by integrating over the true latent distribution were essentially unbiased. For the a parameters, standard errors were larger for the limited-information estimates when the bias was positive but smaller when the bias was negative. For the d parameters, standard errors were larger for the limited-information estimates of the easiest, most discriminating items. Otherwise, they were generally similar for the limited- and full-information estimates. Sample size did not substantially impact the differences between the estimation methods; limited information did not gain an advantage for smaller samples.     

结构方程模型和IRT等级反应模型在人格量表项目筛选中的对比研究

为比较结构方程模型和 IRT等级反应模型在人格量表项目筛选上的作用,以《中国大学生人格量表》的7229个实际测量数据为基础,针对因素二“爽直”分别以Lisrel8.70和Multilog7.03进行结构方程模型和等级反应模型的参数估计与拟合,比较两种方法的项目筛选结果.二者统计结果均认为项目5、6、7、8拟合度不佳,在结构方程模型上表现为因子负荷较低,整体拟合指数不理想;在等级反应模型上表现为区分度参数和位置参数不理想,相关项目的特征曲线和信息曲线形态较差.但结构方程模型倾向于项目6、8更差,而等级反应模型则倾向于项目5、6更差.结构方程模型和 IRT等级反应模型对人格量表项目的统计推断结果从总体上讲是一致的,但在个别项目上略有差异.二者各有优势,可以结合使用.     

Parameter Recovery in the Graded Response Model Using MULTILOG

The graded response model can be used to describe test-taking behavior when item responses are classified into ordered categories. In this study, parameter recovery in the graded response model was investigated using the MULTILOG computer program under default conditions. Based on items having five response categories, 36 simulated data sets were generated that varied on true θ distribution, true item discrimination distribution, and calibration sample size. The findings suggest, first, the correlations between the true and estimated parameters were consistently greater than 0.85 with sample sizes of at least 500. Second, the root mean square error differences between true and estimated parameters were comparable with results from binary data parameter recovery studies. Of special note was the finding that the calibration sample size had little influence on the recovery of the true ability parameter but did influence item-parameter recovery. Therefore, it appeared that item-parameter estimation error, due to small calibration samples, did not result in poor person-parameter estimation. It was concluded that at least 500 examinees are needed to achieve an adequate calibration under the graded model.     

HSK[高等]口语考试分数处理

本研究根据前人的研究成果,选用单维等级反应模型（GRM）,对高等汉语水平考试（简称HSK[高等]）口试进行了实验分析。实验假设,等级反应模型下的评分能够更加精细地区分被试的能力。最终实验结果证实了该假设。     

A Comparison of Exposure Control Procedures in CATs Using the GPC Model

The current study compares the progressive-restricted standard error (PR-SE) exposure control method with the Sympson-Hetter, randomesque, and no exposure control (maximum information) procedures using the generalized partial credit model with fixed- and variable-length CATs and two item pools. The PR-SE method administered the entire item pool for all conditions; whereas the Sympson-Hetter and randomesque procedures did not administer 27%–28% and 14%, respectively, of item pool 1 and about 45%–50% and 27%–29% of item pool 2, respectively, of the items that were not administered. PR-SE also resulted in the smallest amount of mean item overlap averaged across replications. These results were obtained with similar measurement precision compared to the other methods while improving on the utilization of the item pools, except for very low theta levels (less than ?2) for item pool 2, where a mismatch with the trait distribution occurs.     

Optimal Linking Design for Response Model Parameters

Linking functions adjust for differences between identifiability restrictions used in different instances of the estimation of item response model parameters. These adjustments are necessary when results from those instances are to be compared. As linking functions are derived from estimated item response model parameters, parameter estimation error automatically propagates into linking error. This article explores an optimal linking design approach in which mixed‐integer programming is used to select linking items to minimize linking error. Results indicate that the method holds promise for selection of linking items.     

Estimating Standardized SEM Parameters Given Nonnormal Data and Incorrect Model: Methods and Comparison

When both model misspecifications and nonnormal data are present, it is unknown how trustworthy various point estimates, standard errors (SEs), and confidence intervals (CIs) are for standardized structural equation modeling parameters. We conducted simulations to evaluate maximum likelihood (ML), conventional robust SE estimator (MLM), Huber–White robust SE estimator (MLR), and the bootstrap (BS). We found (a) ML point estimates can sometimes be quite biased at finite sample sizes if misfit and nonnormality are serious; (b) ML and MLM generally give egregiously biased SEs and CIs regardless of the degree of misfit and nonnormality; (c) MLR and BS provide trustworthy SEs and CIs given medium misfit and nonnormality, but BS is better; and (d) given severe misfit and nonnormality, MLR tends to break down and BS begins to struggle.     

Performance of the S[... formula ...]2 Statistic for the Multidimensional Graded Response Model

S−χ2 is a popular item fit index that is available in commercial software packages such as flexMIRT. However, no research has systematically examined the performance of S−χ2 for detecting item misfit within the context of the multidimensional graded response model (MGRM). The primary goal of this study was to evaluate the performance of S−χ2 under two practical misfit scenarios: first, all items are misfitting due to model misspecification, and second, a small subset of items violate the underlying assumptions of the MGRM. Simulation studies showed that caution should be exercised when reporting item fit results of polytomous items using S−χ2 within the context of the MGRM, because of its inflated false positive rates (FPRs), especially with a small sample size and a long test. S−χ2 performed well when detecting overall model misfit as well as item misfit for a small subset of items when the ordinality assumption was violated. However, under a number of conditions of model misspecification or items violating the homogeneous discrimination assumption, even though true positive rates (TPRs) of S−χ2 were high when a small sample size was coupled with a long test, the inflated FPRs were generally directly related to increasing TPRs. There was also a suggestion that performance of S−χ2 was affected by the magnitude of misfit within an item. There was no evidence that FPRs for fitting items were exacerbated by the presence of a small percentage of misfitting items among them.     

A Method of Estimating Rater Reliability

A method for assessing rater reliability by means of a design of overlapping rater teams is presented. The products to be rated are split randomly into m disjoint subsamples, m equaling the number of raters. Each rater rates at least two subsamples according to a prefixed design. The covariances or correlations of the ratings can be analyzed with LISREL models, resulting in estimates of the rater reliabilities. Models in which the rater reliabilities are congeneric, tauequivalent, or parallel can be tested. We address problems concerning the identification and the degrees of freedom of the models and present two examples based on essay ratings.     

A Multidimensional Item Response Theory Model for Continuous and Graded Responses With Error in Persons and Items

Item response theory “dual” models (DMs) in which both items and individuals are viewed as sources of differential measurement error so far have been proposed only for unidimensional measures. This article proposes two multidimensional extensions of existing DMs: the M-DTCRM (dual Thurstonian continuous response model), intended for (approximately) continuous responses, and the M-DTGRM (dual Thurstonian graded response model), intended for ordered-categorical responses (including binary). A rationale for the extension to the multiple-content-dimensions case, which is based on the concept of the multidimensional location index, is first proposed and discussed. Then, the models are described using both the factor-analytic and the item response theory parameterizations. Procedures for (a) calibrating the items, (b) scoring individuals, (c) assessing model appropriateness, and (d) assessing measurement precision are finally discussed. The simulation results suggest that the proposal is quite feasible, and an illustrative example based on personality data is also provided. The proposals are submitted to be of particular interest for the case of multidimensional questionnaires in which the number of items per scale would not be enough for arriving at stable estimates if the existing unidimensional DMs were fitted on a separate-scale basis.     

认知语用参数下的人称指示语研究

人称指示语虽然是言语事件参与角色关系的客观体现,但在很大程度上体现了语境参与下参与的主观认知态度。认知语用参数下的人称指示系统主要包括移情指示系统和礼貌指示系统两个子系统。在生命度层级和可及性层级两个参数的作用下,人称指示系统呈现出多层次、超符号的立体网络系统,使人称指示这一封闭的语言系统具有了开放的性质。     

A Simple Illustration for the Need of Multiple Comparison Procedures

Statistical adjustments to accommodate multiple comparisons are routinely covered in introductory statistical courses. The fundamental rationale for such adjustments, however, may not be readily understood. This article presents a simple illustration to help remedy this.     

Examination of Indices of High School Performance Based on the Graded Response Model

We examined summary indices of high school performance (coursework, grades, and test scores) based on the graded response model (GRM). The indices varied by inclusion of ACT test scores and whether high school courses were constrained to have the same difficulty and discrimination across groups of schools. The indices were examined with respect to skewness, incremental prediction of college degree attainment, and differences across racial/ethnic and socioeconomic subgroups. The most difficult high school courses to earn an “A” grade included calculus, chemistry, trigonometry, other advanced math, physics, algebra 2, and geometry. The GRM‐based indices were less skewed than simple high school grade point average (HSGPA) and had higher correlations with ACT Composite score. The index that included ACT test scores and allowed item parameters to vary by school group was most predictive of college degree attainment, but had larger subgroup differences. Implications for implementing multiple measure models for college readiness are discussed.     

A Comparison of IRT Linking Procedures

Various applications of item response theory often require linking to achieve a common scale for item parameter estimates obtained from different groups. This article used a simulation to examine the relative performance of four different item response theory (IRT) linking procedures in a random groups equating design: concurrent calibration with multiple groups, separate calibration with the Stocking-Lord method, separate calibration with the Haebara method, and proficiency transformation. The simulation conditions used in this article included three sampling designs, two levels of sample size, and two levels of the number of items. In general, the separate calibration procedures performed better than the concurrent calibration and proficiency transformation procedures, even though some inconsistent results were observed across different simulation conditions. Some advantages and disadvantages of the linking procedures are discussed.     

How Reliable Are Our Assessment Data?: A Comparison of the Reliability of Data Produced in Graded and Un-Graded Conditions

Motivating students to perform well on assessment tests is difficult when students know the results have no academic consequence. The present study evaluates the influence of assessment context (graded vs. non-graded) on the reliability of an assessment measure. Results indicate the graded condition produces higher reliability (r= .71) than the non-graded condition (r = .29), which leads to unacceptably low reliability. Moreover, the graded condition produces significantly higher scores (M = 64%), than the non-graded condition (M = 43%). Only students in the graded condition (41%) obtained passing scores of 70% or above.     

A Comparison of Two Procedures for Computing IRT Equating Coefficients

In order to equate tests under Item Response Theory (IRT), one must obtain the slope and intercept coefficients of the appropriate linear transformation. This article compares two methods for computing such equating coefficients–Loyd and Hoover (1980) and Stocking and Lord (1983). The former is based upon summary statistics of the test calibrations; the latter is based upon matching test characteristic curves by minimizing a quadratic loss function. Three types of equating situations: horizontal, vertical, and that inherent in IRT parameter recovery studies–were investigated. The results showed that the two computing procedures generally yielded similar equating coefficients in all three situations. In addition, two sets of SAT data were equated via the two procedures, and little difference in the obtained results was observed. Overall, the results suggest that the Loyd and Hoover procedure usually yields acceptable equating coefficients. The Stocking and Lord procedure improves upon the Loyd and Hoover values and appears to be less sensitive to atypical test characteristics. When the user has reason to suspect that the test calibrations may be associated with data sets that are typically troublesome to calibrate, the Stocking and Lord procedure is to be preferred.     

非专业计算机网络基础课程的分层教学模式探讨

文章结合笔者“计算机网络基础”课程教学实践,针对全校选修课存在的问题,提出了“自适应分层”教学法,阐述了笔者对选修课的一些新建议。     

A Comparison of Procedures for Content-Sensitive Item Selection in Computerized Adaptive Tests

This simulation study compared two procedures to enable an adaptive test to select items in correspondence with a content blueprint. Trait level estimates obtained from testlet-based and constrained adaptive tests administered to 10,000 simulated examinees under two trait distributions and three item pool sizes were compared to the trait level estimates obtained from traditional adaptive tests in terms of mean absolute error, bias, and information. Results indicate that using constrained adaptive testing requires an increase of 5% to 11% in test length over the traditional adaptive test to reach the same error level and, using testlets requires an increase of 43% to 104% in test length over the traditional adaptive test. Given these results, the use of constrained computerized adaptive testing is recommended for situations in which an adaptive test must adhere to particular content specifications.     

Assessing Goodness of Fit of Item Response Theory Models: A Comparison of Traditional and Alternative Procedures

Testing the goodness of fit of item response theory (IRT) models is relevant to validating IRT models, and new procedures have been proposed. These alternatives compare observed and expected response frequencies conditional on observed total scores, and use posterior probabilities for responses across θ levels rather than cross-classifying examinees using point estimates of θ and score responses. This research compared these alternatives with regard to their methods, properties (Type 1 error rates and empirical power), available research, and practical issues (computational demands, treatment of missing data, effects of sample size and sparse data, and available computer programs). Different advantages and disadvantages related to these characteristics are discussed. A simulation study provided additional information about empirical power and Type 1 error rates.