Investigating the Effectiveness of Equating Designs for Constructed-Response Tests in Large-Scale Assessments

Using data from a large-scale exam, in this study we compared various designs for equating constructed-response (CR) tests to determine which design was most effective in producing equivalent scores across the two tests to be equated. In the context of classical equating methods, four linking designs were examined: (a) an anchor set containing common CR items, (b) an anchor set incorporating common CR items rescored, (c) an external multiple-choice (MC) anchor test, and (d) an equivalent groups design incorporating rescored CR items (no anchor test). The use of CR items without rescoring resulted in much larger bias than the other designs. The use of an external MC anchor resulted in the next largest bias. The use of a rescored CR anchor and the equivalent groups design led to similar levels of equating error.     

Comparisons among Designs for Equating Mixed-Format Tests in Large-Scale Assessments

In this study we examined variations of the nonequivalent groups equating design for tests containing both multiple-choice (MC) and constructed-response (CR) items to determine which design was most effective in producing equivalent scores across the two tests to be equated. Using data from a large-scale exam, this study investigated the use of anchor CR item rescoring (known as trend scoring) in the context of classical equating methods. Four linking designs were examined: an anchor with only MC items, a mixed-format anchor test containing both MC and CR items; a mixed-format anchor test incorporating common CR item rescoring; and an equivalent groups (EG) design with CR item rescoring, thereby avoiding the need for an anchor test. Designs using either MC items alone or a mixed anchor without CR item rescoring resulted in much larger bias than the other two designs. The EG design with trend scoring resulted in the smallest bias, leading to the smallest root mean squared error value.     

Local Linear Observed‐Score Equating

Two methods of local linear observed‐score equating for use with anchor‐test and single‐group designs are introduced. In an empirical study, the two methods were compared with the current traditional linear methods for observed‐score equating. As a criterion, the bias in the equated scores relative to true equating based on Lord's (1980) definition of equity was used. The local method for the anchor‐test design yielded minimum bias, even for considerable variation of the relative difficulties of the two test forms and the length of the anchor test. Among the traditional methods, the method of chain equating performed best. The local method for single‐group designs yielded equated scores with bias comparable to the traditional methods. This method, however, appears to be of theoretical interest because it forces us to rethink the relationship between score equating and regression.     

NCME 2008 Presidential Address: The Impact of Anchor Test Configuration on Student Proficiency Rates

Examined in this study were the effects of reducing anchor test length on student proficiency rates for 12 multiple‐choice tests administered in an annual, large‐scale, high‐stakes assessment. The anchor tests contained 15 items, 10 items, or five items. Five content representative samples of items were drawn at each anchor test length from a small universe of items in order to investigate the stability of equating results over anchor test samples. The operational tests were calibrated using the one‐parameter model and equated using the mean b‐value method. The findings indicated that student proficiency rates could display important variability over anchor test samples when 15 anchor items were used. Notable increases in this variability were found for some tests when shorter anchor tests were used. For these tests, some of the anchor items had parameters that changed somewhat in relative difficulty from one year to the next. It is recommended that anchor sets with more than 15 items be used to mitigate the instability in equating results due to anchor item sampling. Also, the optimal allocation method of stratified sampling should be evaluated as one means of improving the stability and precision of equating results.     

The Chain and Post-Stratification Methods for Observed-Score Equating: Their Relationship to Population Invariance

The Non-Equivalent-groups Anchor Test (NEAT) design has been in wide use since at least the early 1940s. It involves two populations of test takers, P and Q, and makes use of an anchor test to link them. Two linking methods used for NEAT designs are those (a) based on chain equating and (b) that use the anchor test to post-stratify the distributions of the two operational test scores to a common population (i.e., Tucker equating and frequency estimation). We show that, under different sets of assumptions, both methods are observed score equating methods and we give conditions under which the methods give identical results. In addition, we develop analogues of the Dorans and Holland (2000) RMSD measures of population invariance of equating methods for the NEAT design for both chain and post-stratification equating methods.     

全测验与锚测验题型分值比对等值误差的影响

为探讨全测验与锚测验不同的客观题与主观题分值比对等值误差造成的影响,本文设计两种全测验与锚测验题型分值比,以等值标准误为因变量,构建2X2的两因素完全随机化设计进行等值误差的方差分析。结果表明,全测验题型分值比与锚测验题型分值比两因素的主效应显著（P〈0．001）,交互作用显著（P〈0．01）,简单效应检验表明两因素在各水平上差异显著（P〈0．01）。全测验题型分值比与锚测验题型分值比对等值误差产生一定的影响,在等值过程中应该考虑这两个影响因素,为了减小等值过程的误差,锚测验题型分值比应该尽量与全测验题型分值比相一致。     

Local Observed‐Score Kernel Equating

Three local observed‐score kernel equating methods that integrate methods from the local equating and kernel equating frameworks are proposed. The new methods were compared with their earlier counterparts with respect to such measures as bias—as defined by Lord's criterion of equity—and percent relative error. The local kernel item response theory observed‐score equating method, which can be used for any of the common equating designs, had a small amount of bias, a low percent relative error, and a relatively low kernel standard error of equating, even when the accuracy of the test was reduced. The local kernel equating methods for the nonequivalent groups with anchor test generally had low bias and were quite stable against changes in the accuracy or length of the anchor test. Although all proposed methods showed small percent relative errors, the local kernel equating methods for the nonequivalent groups with anchor test design had somewhat larger standard error of equating than their kernel method counterparts.     

Applying Research Methods to a Gerontological Population: Matching Data Collection to Characteristics of Older Persons

As Baby Boomers reach 65 years of age and methods of studying older populations are becoming increasingly varied (e.g., including mixed methods designs, on-line surveys, and video-based environments), there is renewed interest in evaluating methodologies used to collect data with older persons. The goal of this article is to examine data-collection methodologies commonly used with older persons (closed-ended, structured surveys with standardized scales, performance-based measures, secondary data sets, open-ended interviewing, grounded theory, and ethnographic work) to illustrate issues arising from the use of each research design with older adults in real-life settings. Experiences from five studies are included to show the potential role of normative age-related sensory and functional changes as well as the role of age and cohort upon method. Specific guidelines are provided about how to improve existing data-collection methods for older persons and ways to improve selection of methodologies for use in future research on aging.     

Small-Sample Equating Using a Single-Group Nearly Equivalent Test (SiGNET) Design

A single-group (SG) equating with nearly equivalent test forms (SiGNET) design was developed by Grant to equate small-volume tests. Under this design, the scored items for the operational form are divided into testlets or mini tests. An additional testlet is created but not scored for the first form. If the scored testlets are testlets 1–6 and the unscored testlet is testlet 7, then the first form is composed of testlets 1–6 and the second form is composed of testlets 2–7. The seven testlets are administered as a single administered form, and when a sufficient number of examinees have taken the administered form, the second form (testlets 2–7) is equated to the first form (testlets 1–6) using an SG equating design. As evident, this design facilitates the use of an SG equating and allows for the accumulation of data, both of which may reduce equating error. This study compared equatings under the SiGNET and common-item equating designs and found lower equating error for the SiGNET design in very small sample size conditions (e.g., N = 10).     

A Comparison of Angoff's Design I and Design II for Vertical Equating Using Traditional and IRT Methodology

Practical considerations in conducting an equating study often require a trade-off between testing time and sample size. A counterbalanced design (Angoff's Design II) is often selected because, as each examinee is administered both test forms and therefore the errors are correlated, sample sizes can be dramatically reduced over those required by a spiraling design (Angoff's Design I), where each examinee is administered only one test form. However, the counterbalanced design may be subject to fatigue, practice, or context effects. This article investigated these two data collection designs (for a given sample size) with equipercentile and IRT equating methodology in the vertical equating of two mathematics achievement tests. Both designs and both methodologies were judged to adequately meet an equivalent expected score criterion; Design II was found to exhibit more stability over different samples.     

The Effects of Incomplete Rating Designs in Combination With Rater Effects

Researchers have explored a variety of topics related to identifying and distinguishing among specific types of rater effects, as well as the implications of different types of incomplete data collection designs for rater‐mediated assessments. In this study, we used simulated data to examine the sensitivity of latent trait model indicators of three rater effects (leniency, central tendency, and severity) in combination with different types of incomplete rating designs (systematic links, anchor performances, and spiral). We used the rating scale model and the partial credit model to calculate rater location estimates, standard errors of rater estimates, model–data fit statistics, and the standard deviation of rating scale category thresholds as indicators of rater effects and we explored the sensitivity of these indicators to rater effects under different conditions. Our results suggest that it is possible to detect rater effects when each of the three types of rating designs is used. However, there are differences in the sensitivity of each indicator related to type of rater effect, type of rating design, and the overall proportion of effect raters. We discuss implications for research and practice related to rater‐mediated assessments.     

传感器实验装置计算机控制系统

设计开发了传感器实验装置计算机控制系统,课题涵盖了检测技术、传感器原理、非电量测量技术、光电变换技术以及Visual Basic6.0语言等方面的知识。设计内容包括:将各种非电量按一定规律转换成便于处理和传输的电量(标准电信号),用不同的测试电路,对各种传感器进行标定和测试,利用微机数据采集技术,由计算机实时地进行实验数据采集和处理,并以图表的形式显示,实现实验数据的保存和查询,报表的打印输出。     

Cognitive diagnostic model of best choice: a study of reading comprehension

Abstract

The present study compared the performance of six cognitive diagnostic models (CDMs) to explore inter skill relationship in a reading comprehension test. To this end, item responses of about 21,642 test-takers to a high-stakes reading comprehension test were analyzed. The models were compared in terms of model fit at both test and item levels, classification consistency and accuracy, and proportion of skill mastery profiles. The results showed that the G-DINA performed the best and the C-RUM, NC-RUM, and ACDM showed the closest affinity to the G-DINA. In terms of some criteria, the DINA showed comparable performance to the G-DINA. The test-level results were corroborated by the item-level model comparison, where DINA, DINO, and ACDM variously fit some of the items. The results of the study suggested that relationships among the subskills of reading comprehension might be a combination of compensatory and non-compensatory. Therefore, it is suggested that the choice of the CDM be carried out at item level rather than test level.     

Test fairness: Examining differential functioning of the reading comprehension section of the GSEEE in China

This study investigated differential item functioning (DIF), differential bundle functioning (DBF), and differential test functioning (DTF) across gender of the reading comprehension section of the Graduate School Entrance English Exam in China. The datasets included 10,000 test-takers’ item-level responses to 6 five-item testlets. Both DIF and DBF were examined by using poly-simultaneous item bias test and item-response-theory-likelihood-ratio test, and DTF was investigated with multi-group confirmatory factor analyses (MG-CFA). The results indicated that although none of the 30 items exhibited statistically and practically significant DIF across gender at the item level, 2 testlets were consistently identified as having significant DBF at the testlet level by the two procedures. Nonetheless, DBF does not manifest itself at the overall test score level to produce DTF based on MG-CFA. This suggests that the relationship between item-level DIF and test-level DTF is a complicated issue with the mediating effect of testlets in testlet-based language assessment.     

On the Choice of Anchor Tests in Equating

The choice of anchor tests is crucial in applications of the nonequivalent groups with anchor test design of equating. Sinharay and Holland (2006, 2007) suggested “miditests,” which are anchor tests that are content‐representative and have the same mean item difficulty as the total test but have a smaller spread of item difficulties. Sinharay and Holland (2006, 2007), Cho, Wall, Lee, and Harris (2010), Fitzpatrick and Skorupski (2016), Liu, Sinharay, Holland, Curley, and Feigenbaum (2011a), Liu, Sinharay, Holland, Feigenbaum, and Curley (2011b), and Yi (2009) found the miditests to lead to better equating than minitests, which are representative of the total test with respect to content and difficulty. However, these findings recently came into question as Trierweiler, Lewis, and Smith (2016) concluded, based on a comparison of correlation coefficients of miditests and minitests with the total test, that making an anchor test a miditest does not generally increase the anchor to total score correlation and recommended the continuation of the practice of using minitests over miditests. Their recommendation raises the question, “Should miditests continue to be considered in practice?” This note defends the miditests by citing literature that favors miditests and then by showing that miditests perform as well as the minitests in most realistic situations considered in Trierweiler et al. (2016), which implies that miditests should continue to be seriously considered by equating practitioners.     

Quality of life of nursing home residents before and after reminiscence therapy

This study was conducted to determine the effects of reminiscence therapy as a nursing intervention on the quality of life (QOL) of nursing home residents. This was a quasi-experimental study with a single-group pretest and posttest design. Sampling inclusion criteria were being 65 years of age or older, ability to communicate, lack of hearing impairment, lack of dementia or any given psychiatric disorder, and volunteering to participate in the study. Participants included 32 nursing home residents, who were divided into groups of 2–5 and received a total of 12 sessions of group reminiscence therapy. After the intervention, a decrease was noticed in the total score and subdomain test scores of a QOL scale. Although the study design did not allow confirmation of the effect of reminiscence, analysis showed some useful indications for further research. Additionally, participants stated that the reminiscence program might be a meaningful activity for nursing home residents.     

脉冲变压器铁氧体磁芯监测与测量

该文论述了仿真条件下测取各磁参数总贡献(而非参数本身)这一测量模式的优越性,指出这将明显地优化脉冲变压器设计(及生产现场监测)条件;并给出了为避免使用昂贵的专用磁测量系统而拟订的脉冲法仿真测量系统有关技术细节描述及试样测量实例。     

Systematic Review of Two Decades (1995 to 2014) of Research on Synchronous Online Learning

Systematic reviews of literature are studies that strategically search for published research on a specific topic in order to synthesize what is known about the topic. This systematic review describes 157 articles on synchronous online learning (SOL) from thirty-four different countries on instructional setting, content areas, participant demographics, research designs, independent and dependent variables, SOL technologies, and data-collection tools.     

Small‐Sample Equating Using a Synthetic Linking Function

This study addressed the sampling error and linking bias that occur with small samples in a nonequivalent groups anchor test design. We proposed a linking method called the synthetic function, which is a weighted average of the identity function and a traditional equating function (in this case, the chained linear equating function). Specifically, we compared the synthetic, identity, and chained linear functions for various‐sized samples from two types of national assessments. One design used a highly reliable test and an external anchor, and the other used a relatively low‐reliability test and an internal anchor. The results from each of these methods were compared to the criterion equating function derived from the total samples with respect to linking bias and error. The study indicated that the synthetic functions might be a better choice than the chained linear equating method when samples are not large and, as a result, unrepresentative.     

Equating With Miditests Using IRT

The equating performance of two internal anchor test structures—miditests and minitests—is studied for four IRT equating methods using simulated data. Originally proposed by Sinharay and Holland, miditests are anchors that have the same mean difficulty as the overall test but less variance in item difficulties. Four popular IRT equating methods were tested, and both the means and SDs of the true ability of the group to be equated were varied. We evaluate equating accuracy marginally and conditional on true ability. Our results suggest miditests perform about as well as traditional minitests for most conditions. Findings are discussed in terms of comparability to the typical minitest design and the trade‐off between accuracy and flexibility in test construction.