How invariant and accurate are domain ratings in writing assessment?

The use of evidence to guide policy and practice in education (Cooper, Levin, & Campbell, 2009) has included an increased emphasis on constructed-response items, such as essays and portfolios. Because assessments that go beyond selected-response items and incorporate constructed-response items are rater-mediated (Engelhard, 2002, Engelhard, 2013), it is necessary to develop evidence-based indices of quality for the rating processes used to evaluate student performances. This study proposes a set of criteria for evaluating the quality of ratings based on the concepts of measurement invariance and accuracy within the context of a large-scale writing assessment. Two measurement models are used to explore indices of quality for raters and ratings: the first model provides evidence for the invariance of ratings, and the second model provides evidence for rater accuracy. Rating quality is examined within four writing domains from an analytic rubric. Further, this study explores the alignment between indices of rating quality based on these invariance and accuracy models within each of the four domains of writing. Major findings suggest that rating quality varies across analytic rubric domains, and that there is some correspondence between indices of rating quality based on the invariance and accuracy models. Implications for research and practice are discussed.     

Multilevel Factorial Invariance in n-Level Structural Equation Modeling (nSEM)

We present a multigroup multilevel confirmatory factor analysis (CFA) model and a procedure for testing multilevel factorial invariance in n-level structural equation modeling (nSEM). Multigroup multilevel CFA introduces a complexity when the group membership at the lower level intersects the clustered structure, because the observations in different groups but in the same cluster are not independent of one another. nSEM provides a framework in which the multigroup multilevel data structure is represented with the dependency between groups at the lower level properly taken into account. The procedure for testing multilevel factorial invariance is illustrated with an empirical example using an R package xxm2.     

用光子模型讨论多普勒效应

本文用光子模型来讨论多普勒效应及相位不变性,与波动法有殊途同归的效果.     

Getting Better All the Time? An Illustrative Investigation of Multilevel Longitudinal Measurement Invariance Based on School Climate Surveys

School climate surveys are central to school improvement and principal evaluation policies. The quality of school climate has been linked both to student achievement and to teacher retention. Oftentimes, policymakers and practitioners are concerned with monitoring change in school climate quality in each academic year. Such applications assume longitudinal factorial invariance—it is presupposed that the surveys are measuring the same things in the same metric at each time point. While there is considerable research examining the validity of inferences based on survey‐derived climate indicators, this research is almost exclusively based on cross‐sectional data. There is little literature describing procedures for gathering evidence of factorial invariance of school climate indicators. This study proposes to adapt existing methods for evaluating factorial invariance in longitudinal designs into multilevel frameworks, and in doing so, articulates a novel method for evaluating longitudinal measurement invariance in school climate research. This technique is illustrated on a widely used school climate survey.     

浅谈射影几何的优越性

从三个方面阐述了运用射影几何学中有关原理解决问题具有一定的优越性:可使证明过程简化;揭示内在联系;发现新结论,并举例说明该观点.     

Self-concept in reading: Factor structure,cross-cultural invariance and relationships with reading achievement in an international context (PIRLS 2011)

Using data from 48 countries, this study investigated the factorial structure and tested the cross-cultural invariance of the PIRLS 2011 reading self-concept scale and its relationships with reading achievement. The study showed that a two-factorial structure of the self-concept scale in reading had the best fit with the data. Configural invariance and metric invariance were achieved, scalar invariance was not. The two dimensions ‘Perception of competence’ and ‘Perception of difficulty’ showed robust within-country correlations with reading achievement, especially the second one. At the country level, the ‘Perception of competence’ was negatively related with reading, illustrating the attitudes-achievement paradox: countries in which students on average reported a more positive self-concept performed lower. On the contrary, for the ‘Perception of difficulty’, the sign of the correlation remained the same at the within-country and at the country level.     

Testing Factorial Invariance of the Amotivation Inventory-Physical Education Across Gender for Urban Adolescents

Lack of motivation, or amotiation, is emerging as a critical issue in high school physical education. The Amotivation Inventory-Physical Education (Shen, Winger, Li, Sun, & Rukavina, 2010) was developed to measure the multidimensional nature of amotivation construct in physical education. This study was designed to examine the consistency of the metric properties of Amotivation Inventory-Physical Education scores by evaluating their factorial structure for invariance across gender in a sample of 985 urban inner-city high school adolescents. Latent mean invariance was also tested. Results revealed that constraining factor loadings and intercepts in sequential configural, metric, and scalar invariances had no meaningful impact on model fit. However, gender might play a role in the magnitude of factor scores. Findings suggest that the four-factor structure of the Amotivation Inventory-Physical Education is a satisfactory representation of motivational deficits for urban inner-city adolescents and provide significant validity evidence for the scale scores in urban high school settings.     

Testing Measurement Equivalence of Eudaimonic and Hedonic Entertainment Motivations in a Cross-cultural Comparison

Within Hofstede’s framework of individualistic and collectivistic cultures, this contribution examines measurement equivalence of hedonic and eudaimonic entertainment motivations in two different cultures, namely Germany representing a more individualistic culture (N = 180) and Turkey representing a more collectivistic culture (N = 97). By means of a multi-group confirmatory factor analysis, we could secure configural invariance for both hedonic and eudaimonic entertainment motivations across the German and Turkish sample. Metric invariance, however, could only be obtained for hedonic, but not for eudaimonic motivations. Scalar invariance was obtained for neither of the two entertainment motivations. The study points to the importance of equivalence testing when conducting cross-cultural research.     

Validity and reliability of the Very Short form of the Physical Self-Inventory among Turkish adolescents

The study aimed to test the validity and reliability of the Very Short form of the Physical Self-Inventory (PSI-VS) among a sample of 635 Turkish adolescents. These adolescents have completed the 12 original items of the PSI-VS, plus a positively worded reformulation of the single reverse-keyed item of the physical attractiveness subscale. A series of confirmatory factor analyses was used to examine the psychometric properties of the original and modified versions of the PSI-VS. Findings revealed superior psychometric properties with the modified version than with the original set of items. The modified version was thus subsequently used to examine its factor structure invariance across sexes, age groups and sport practice involvement. Results not only report that this version was fully invariant across, but also that latent means were significantly different across sexes and sport practice involvement. In sum, the Turkish modified version of the PSI-VS presents acceptable psychometric properties and may be used to repeatedly and/or intensively assess participants’ physical self-perceptions in the context of sport and exercise interventions.