An NCME Instructional Module on Using Differential Step Functioning to Refine the Analysis of DIF in Polytomous Items

Traditional methods for examining differential item functioning (DIF) in polytomously scored test items yield a single item‐level index of DIF and thus provide no information concerning which score levels are implicated in the DIF effect. To address this limitation of DIF methodology, the framework of differential step functioning (DSF) has recently been proposed, whereby measurement invariance is examined within each step underlying the polytomous response variable. The examination of DSF can provide valuable information concerning the nature of the DIF effect (i.e., is the DIF an item‐level effect or an effect isolated to specific score levels), the location of the DIF effect (i.e., precisely which score levels are manifesting the DIF effect), and the potential causes of a DIF effect (i.e., what properties of the item stem or task are potentially biasing). This article presents a didactic overview of the DSF framework and provides specific guidance and recommendations on how DSF can be used to enhance the examination of DIF in polytomous items. An example with real testing data is presented to illustrate the comprehensive information provided by a DSF analysis.     

Comparison and Comparability: Fitness Tracking Between Youths with Different Physical Activity Levels

This study compared a three-year tracking of health-related physical fitness between two comparable samples of six graders that enrolled either in a PE4life middle school (n?=?154) or another school with a traditional PE program (n?=?93) in the United States. For the cohort, the FITNESSGRAM^TM (Cooper Institute for Aerobics Research, 1992 Cooper Institute for Aerobics Research. 1992. The Prudential FITNESSGRAM^®test administration manual, Dallas, TX: The Cooper Institute.  [Google Scholar]) battery was administered twice with a three-year interval, and moderate and vigorous physical activity levels in PE classes were assessed. Fitness tracking then was assessed with six statistical techniques and compared between the two samples. Students in the PE4life Program demonstrated significantly higher moderate and vigorous physical activity level (p < .001) in PE classes and improved significantly more in most fitness measures at follow-up (ps < .05–.001) than those in the traditional program. Results of tracking assessed with Spearman correlations, Kappa statistics, and Kendall's tau-b did not reflect the significant differences in fitness changes between the two samples, whereas those assessed using stability in extreme quartiles, changes in percentage of students in at-risk groups, and odds ratios reflected the significant differences in the fitness changes between the two samples. The last three tracking methods identify differences in fitness changes in addition to assessing tracking stability and are recommended in fitness tracking research.     

Reliability of a Digital Method to Determine Frontal Area of a Cyclist

Eight cyclists were photographed with a digital camera for three trials while positioned on their own bicycle wearing their helmet. The positions were different from each other and described as hands on the brake hoods, hands below the curve of the brake hoods on the handlebars, and using aerobars. Twenty-four trials were digitized by two different individuals three times to estimate the inter- and intra-rater reliability of the method. The intraclass correlation coefficient (p < .05) value for the intra-rater (test–retest) reliability was .993, and for inter-rater consistency, it was .976. There were significant differences (p < .05) between digitizers and between trials, apparently due to a learning effect that disappeared by the third trial. Due to small differences between digitizers and trials, caution is recommended when considering the use of this method.     

Experimenter Effects in Exercise Tolerance Testing: The Race and Gender of the Tester and the Tester/Subject Relationship

Abstract

Two studies examined the effects of the experimenter on heart rate (HR), rating of perceived exertion (RPE), and time to voluntary exhaustion on a graded treadmill test in college students. The first study determined the effects of the gender and race of the tester on black and white subjects of both sexes. Four testers (black female—BFT, white female—WFT, black male—BMT, and white male—WMT) tested 5 subjects of each gender and race. Differences based on gender and race were determined by a 2 × 2 × 2 × 2 ANOVA. The second study dealt with the tester/subject relationship. One tester tested 20 subjects. Ten were his friends (5 males, 5 females) and 10 (5 males, 5 females) did not know him. Differences were determined by 2 × 2 ANOVA. In both studies, the female subjects had poorer performance than the males. HR during moderate and heavy exercise was not affected by the race and gender of the tester or the tester/subject relationship. For time to exhaustion, white female subjects (WFS) exercised longer when being tested by a female but black female subjects (BFS) had a longer total time with male testers. Significant interactions occurred for HR at rest and at various times during light work for both studies. There were also significant interactions for RPE at various stages of light, moderate and heavy exercise for both studies. For subjects unaccustomed to exercise testing, it appears that the effects of the race and gender of the tester and the tester/subject relationship predominate at rest and during light work. Fewer effects are observed during moderate and heavy work. Many of the results in these studies are consistent with societal roles and expectations for males and females, blacks and whites. A consistent testing environment can minimize the influence of extrinsic testing variables.     

An Approach for Categorizing DIF in Polytomous Items

A widely used approach for categorizing the level of differential item functioning (DIF) in dichotomous items is the scheme proposed by Educational Testing Service (ETS) based on a transformation of the Mantel-Haeszel common odds ratio. In this article two classification schemes for DIF in polytomous items (referred to as the P1 and P2 schemes) are proposed that parallel the criteria set forth in the ETS scheme for dichotomous items. The theoretical equivalence of the P1 and P2 schemes to the ETS scheme is described, and the results of a simulation study conducted to examine the empirical equivalence of the P1 and P2 schemes to the ETS scheme are presented.     

A validation study of the Teacher Collaboration Assessment Survey

Although teacher collaboration is a school improvement imperative, it persists as an under-empiricized construct that has proven difficult to establish and assess with certainty. In this article, the authors present a validation study of the Teacher Collaboration Assessment Survey (TCAS). The TCAS operationalizes and measures 4 key domains of teacher collaboration: dialogue, decision making, action, and evaluation, and has been used to examine the quality of teacher teaming in district-wide comprehensive school reform efforts in the Northeastern and Mid-Atlantic regions of the United States. Five sources of validity evidence recommended by Standards for Educational and Psychological Testing (AERA, APA, & NCME, 1999 American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association. [Google Scholar]) are explicated, which establish a strong argument in support of the instruments' validity. The authors discuss how educational leaders and researchers can use the TCAS for leveraging teacher collaboration for instructional innovation and student achievement, and to systematically examine teacher teaming and its relationship to other educational outcomes.