Software Use in Psychometric Research

Research on psychometric methods is heavily dependent on software. The quality, availability, and documentation of such software are critical to the advancement of the field. In 2000, an ad hoc committee of NCME recommended that NCME adopt policies that promote greater availability and better documentation of software. This article follows the ad hoc committee's report by examining the use of software in four top-tiered journals in recent years. The results indicated that the most frequently cited programs were those written by the articles' authors. The documentation and availability for these programs are often not clear, particularly for software used for simulations. The use of proprietary software was not widespread in the four journals, but there is still room for concern in the future. This article recommends that NCME form a permanent committee to address software issues.     

Accuracy of Random Groups Equating with Very Small Samples

This study investigated the effectiveness of equating with very small samples using the random groups design. Of particular interest was equating accuracy at specific scores where performance standards might be set. Two sets of simulations were carried out, one in which the two forms were identical and one in which they differed by a tenth of a standard deviation in overall difficulty. These forms were equated using mean equating, linear equating, unsmoothed equipercentile equating, and equipercentile equating using two through six moments of log-linear presmoothing with samples of 25, 50, 75, 100, 150, and 200. The results indicated that identity equating was preferable to any equating method when samples were as small as 25. For samples of 50 and above, the choice of an equating method over identity equating depended on the location of the passing score relative to examinee performance. If passing scores were located below the mean, where data were sparser, mean equating produced the smallest percentage of misclassified examinees. For passing scores near the mean, all methods produced similar results with linear equating being the most accurate. For passing scores above the mean, equipercentile equating with 2- and 3-moment presmoothing were the best equating methods. Higher levels of presmoothing did not improve the results.     

The Consistency of Detecting Item Bias Across Different Test Administrations: Implications of Another Failure

Several item bias detection methods were applied to the analysis of bias among males and females for items from a curriculum-based mathematics test. The focus of this analysis was the consistency of the methods across different test administrations of the same items. The results indicated that, of the methods studied, the Mantel–Haenszel (M–H) and IRT-based sum-of-squares methods were the most consistent. However, the degree of reliability and agreement for these methods was modest at best. As with most prior research, no reasonable explanation could be found for the most consistently flagged items. A likely reason for this lies in the confusion of visible genetic group characteristics with their instructional backgrounds. A multidimensional perspective of item bias is proposed for future research that will take such confounding into account.     

Conceptualizing the Classroom of Target Students: A Qualitative Investigation of Panelists' Experiences during Standard Setting

Increasingly, research has focused on the cognitive processes associated with various standard-setting activities. This qualitative study involved an examination of 16 third-grade reading teachers' experiences with the cognitive task of conceptualizing an entire classroom of hypothetical target students when the single-passage bookmark method or the yes/no method was used during a one-day mock panel meeting. Data were collected using in-depth focus group interviews with eight participants from each of the panel meetings, and a whole-text analysis revealed three categories. Most participants experienced difficulty in attempting to conceive of an entire classroom of target students. Most of them were ultimately unable to do so and made use of alternative cognitive strategies. More fundamental issues also contributed to the difficulties experienced in attempting to complete the required cognitive task. Implications of the findings for standard setting and for further research are also discussed.     

Service-Learning and Experiential Learning in Criminal Justice Education: An Exploratory Examination of Faculty Perspectives

Scholarship on service-learning and experiential learning consistently demonstrates benefits for students, but little evidence has emerged examining the perspectives of faculty. The current study examines the conflicting perspectives from faculty affiliated with a university undergraduate criminology major that utilizes both experiential learning and service-learning. Focus groups of departmental faculty revealed fears over curriculum rigor, concerns over forming partnerships and investing resources into service-learning and experiential learning activities without university incentives, and recognition of student and department benefits derived from service-learning and experiential learning activities.     

My Website Reads at an Eighth Grade Level: Why Plain Language Benefits Your Users (and You)

ABSTRACT

Writing in plain language aims to help users find what they need and ensures that the information is both useful and understandable. This is key for distance students whose primary library interaction may be with the library Website. A mix of user research and readability scores can be used to measure whether content is findable, useful, and understandable. There are several strategies authors can adopt to help them write in plain language, including keeping the users' needs and tasks in mind and structuring the content so that it can be quickly scanned. Converting existing Website content to a plain language format can be a large task; running a content audit can help determine which pages should be prioritized for revision. Once the Website is written in plain language, an ongoing content strategy is necessary to help it stay that way.     

Diagnostic Profiles: A Standard Setting Method for Use With a Cognitive Diagnostic Model

This article introduces the Diagnostic Profiles (DP) standard setting method for setting a performance standard on a test developed from a cognitive diagnostic model (CDM), the outcome of which is a profile of mastered and not‐mastered skills or attributes rather than a single test score. In the DP method, the key judgment task for panelists is a decision on whether or not individual cognitive skill profiles meet the performance standard. A randomized experiment was carried out in which secondary mathematics teachers were randomly assigned to either the DP method or the modified Angoff method. The standard setting methods were applied to a test of student readiness to enter high school algebra (Algebra I). While the DP profile judgments were perceived to be more difficult than the Angoff item judgments, there was a high degree of agreement among the panelists for most of the profiles. In order to compare the methods, cut scores were generated from the DP method. The results of the DP group were comparable to the Angoff group, with less cut score variability in the DP group. The DP method shows promise for testing situations in which diagnostic information is needed about examinees and where that information needs to be linked to a performance standard.     

Setting Passing Scores on Passage-Based Tests: A Comparison of Traditional and Single-Passage Bookmark Methods

In this study, a variation of the bookmark standard setting procedure for passage-based tests is proposed in which separate ordered item booklets are created for the items associated with each passage. This variation is compared to the traditional bookmark procedure for a fifth-grade reading test. The results showed that the single-passage bookmark method produced greater consistency among the participants' cutscores, and most participants' bookmark placements did not change after the first round. In addition, participants reported greater understanding of the bookmarking task and greater confidence in their recommended cutscores. Both procedures required approximately the same amount of time to complete, but it is likely that the single-passage bookmark method could be carried out in two, or possibly even one, round of bookmarking rather than the three rounds used in traditional bookmarking. On the other hand, there are several concerns about the single-passage bookmark method that warrant further research. These include floor and ceiling effects, training issues, optimal booklet length, and multiple standards.