Studying Score Stability with a Harmonic Regression Family: A Comparison of Three Approaches to Adjustment of Examinee‐Specific Demographic Data

For assessments that use different forms in different administrations, equating methods are applied to ensure comparability of scores over time. Ideally, a score scale is well maintained throughout the life of a testing program. In reality, instability of a score scale can result from a variety of causes, some are expected while others may be unforeseen. The situation is more challenging for assessments that assemble many different forms and deliver frequent administrations per year. Harmonic regression, a seasonal‐adjustment method, has been found useful in achieving the goal of differentiating between possible known sources of variability and unknown sources so as to study score stability for such assessments. As an extension, this paper presents a family of three approaches that incorporate examinees' demographic data into harmonic regression in different ways. A generic evaluation method based on jackknifing is developed to compare the approaches within the family. The three approaches are compared using real data from an international language assessment. Results suggest that all approaches perform similarly and are effective in meeting the goal. The paper also discusses the properties and limitations of the three approaches, along with inferences about score (in)stability based on the harmonic regression results.     

The Utility of Augmented Subscores in a Licensure Exam: An Evaluation of Methods Using Empirical Data

Will subscores provide additional information than what is provided by the total score? Is there a method that can estimate more trustworthy subscores than observed subscores? To answer the first question, this study evaluated whether the true subscore was more accurately predicted by the observed subscore or total score. To answer the second question, three subscore estimation methods (i.e., subscore estimated from the observed subscore, total score, or a combination of both the subscore and total score) were compared. Analyses were conducted using data from six licensure tests. Results indicated that reporting subscores at the examinee level may not be necessary as they did not provide much additional information over what is provided by the total score. However, at the institutional level (for institution size ≥ 30), reporting subscores may not be harmful, although they may be redundant because the subscores were predicted equally well by the observed subscores or total scores. Finally, results indicated that estimating the subscore using a combination of observed subscore and total score resulted in the highest reliability.     

Small‐Sample Equating Using a Synthetic Linking Function

This study addressed the sampling error and linking bias that occur with small samples in a nonequivalent groups anchor test design. We proposed a linking method called the synthetic function, which is a weighted average of the identity function and a traditional equating function (in this case, the chained linear equating function). Specifically, we compared the synthetic, identity, and chained linear functions for various‐sized samples from two types of national assessments. One design used a highly reliable test and an external anchor, and the other used a relatively low‐reliability test and an internal anchor. The results from each of these methods were compared to the criterion equating function derived from the total samples with respect to linking bias and error. The study indicated that the synthetic functions might be a better choice than the chained linear equating method when samples are not large and, as a result, unrepresentative.     

The deletion of broadcast licenses by the federal communications commission

Primary purpose of this investigation was to uncover basic trends in the revocation and non‐renewal of broadcasting licenses spanning the history of the Federal Communications Commission through 1969.     

On understanding variability in data: a study of graph interpretation in an advanced experimental biology laboratory

Data analysis is constitutive of the discovery sciences. Few studies in mathematics education, however, investigate how people deal with (statistical) variability and statistical variance in the data to be interpreted. And even fewer, if any, focus on the uncertainties with which scientists wrestle before they are confident in the data they produce. The purpose of this study is to exhibit the work of coping with variability in one advanced research laboratory, as exemplified in a typical data analysis session. The study shows that when the scientists are confronted with novel data, their understanding of the variability does not arise in straightforward fashion, and a lot of normally invisible (interactional) work is required to constitute understanding. Tentative conclusions are provided for the implication to mathematics education.     

Case Study Research: A Valuable Learning Tool for Performance Improvement Professionals

Although it is sometimes recommended that performance improvement (PI) professionals include experimental research designs in their repertoire of PI tools and methods, it has been long understood that experimental designs can be difficult to implement due to impediments resulting from the complex nature of the organizational settings. However, the utilization of case study research has proven to be an effective alternative to aid in the identification of strengths and opportunities for the improvement of organizational procedures, policies, processes, or programs. Case study research helps managers and practitioners make sense of real world problems. This article presents a summary of steps in the design of case study research and provides examples of how these methods have been used within organizational settings. Implications for PI practitioners are provided.     

Preparing Reflective Community College Leaders in a Graduate Leadership Course

This study measured the knowledge construction and reflection that occurred in a doctoral course designed to introduce major leadership theories and encourage reflection as a community college leadership practice. At each session, students completed forms reporting new learning and insights they experienced, as well as responding to reflection catalysts. Three themes emerged from the data: reflection on previous internal perspectives about leadership, critical review of personal and theoretical constructs, and embracing relational leadership. By the end of the course, students acknowledged the value of collaborative learning, and demonstrated changed perspectives regarding leadership practices, together with increased habits of reflection. Implications for graduate-level instruction and further research are proposed, based on the conclusion that the New Learning process is valuable for certain types of classes, is beneficial for professional development, and conforms to principles of transformative learning among adults.     

The Development of Critical Consciousness and its Relation to Academic Achievement in Adolescents of Color

Critical consciousness, the process by which individuals come to understand, analyze, and take action against systems of oppression, is associated with several positive youth outcomes. However, little research has considered how the core components of critical consciousness (critical reflection, political agency, critical action) are associated with academic achievement. The present study explored the extent to which the developing critical consciousness of adolescents of color (N = 364) over 4 years of high school predicted academic achievement, as measured by grade point average (GPA) and the Scholastic Aptitude Test (SAT). Results demonstrated that adolescents’ critical reflection and critical action intercepts predicted SAT scores whereas critical reflection and critical action slopes predicted GPAs. Political agency was not predictive of either academic outcome.     

Repeater Analysis for Combining Information From Different Assessments

Admission decisions frequently rely on multiple assessments. As a consequence, it is important to explore rational approaches to combine the information from different educational tests. For example, U.S. graduate schools usually receive both TOEFL iBT^® scores and GRE^® General scores of foreign applicants for admission; however, little guidance has been given to combine information from these two assessments, even though the relationships between such sections as GRE Verbal and TOEFL iBT Reading are obvious. In this study, principles are provided to explore the extent to which different assessments complement one another and are distinguishable. Augmentation approaches developed for individual tests are applied to provide an accurate evaluation of combined assessments. Because augmentation methods require estimates of measurement error and internal reliability data are unavailable, required estimates of measurement error are obtained from repeaters, examinees who took the same test more than once. Because repeaters are not representative of all examinees in typical assessments, minimum discriminant information adjustment techniques are applied to the available sample of repeaters to treat the effect of selection bias. To illustrate methodology, combining information from TOEFL iBT scores and GRE General scores is examined. Analysis suggests that information from the GRE General and TOEFL iBT assessments is complementary but not redundant, indicating that the two tests measure related but somewhat different constructs. The proposed methodology can be readily applied to other situations where multiple assessments are needed.     

Statistical Theory and Assessment Practice

Examples of the impact of statistical theory on assessment practice are provided from the perspective of a statistician trained in theoretical statistics who began to work on assessments. Goodness of fit of item-response models is examined in terms of restricted likelihood-ratio tests and generalized residuals. Minimum discriminant information adjustment is used for linking with no anchors or problematic anchors and for repeater analysis. Assessment issues are examined in cases in which the number of parameters is large relative to the number of observations.