The potential of anchoring vignettes to increase intercultural comparability of non-cognitive factors

ABSTRACT

We investigate whether Anchoring Vignettes (AV) improve intercultural comparability of non-cognitive student-directed factors (e.g., procrastination). So far, correlation analyses for anchored and non-anchored scores with a criterion have been used to demonstrate the effectiveness of AV in improving data quality. However, correlation analyses are often used to investigate external validity of a scale. Nonetheless, before testing for validity, the reliability of the measurement of a construct should be examined. In the present study, we tested for measurement invariance across countries and languages and compared anchored and non-anchored student-directed self-reports that are highly relevant for the students’ self and their behaviour and performance. In addition, we apply further criteria for testing reliability. The results indicate that the data quality for some of the constructs can – in fact – be improved slightly by anchoring; whereas, for other self-reports, anchoring is less successful than was hoped. We discuss with regard to possible consequences for research methodology.     

Psychometric properties of the Brunel Mood Scale in Chinese adolescents and adults

Abstract

Building on the work of Terry and colleagues (Terry, P. C., Lane, A. M., Lane, H. J., & Keohane, L. (1999). Development and validation of a mood measure for adolescents. Journal of Sports Sciences, 17, 861–872; Terry, P. C., Lane, A. M., & Fogarty, G. J. (2003). Construct validity of the Profile of Mood States-Adolescents for use with adults. Psychology of Sport & Exercise, 4, 125–139.), the present study examined the validity and internal consistency reliability of the Chinese version of the Brunel Mood Scale (BRUMS-C) among 2,548 participants, comprising adolescent athletes (n = 520), adult athletes (n = 434), adolescent students (n = 673), and adult students (n = 921). Both adolescent and adult athletes completed the BRUMS-C before, during, or after regular training and both adolescent and adult students completed the BRUMS-C in a classroom setting. Confirmatory factor analyses (CFAs) provided support for the factorial validity of a 23-item six-factor model, with one item removed from the hypothesised measurement model. Internal consistency reliabilities were satisfactory for all subscales across each of the four samples. Criterion validity was supported with strong relationships between the BRUMS-C, abbreviated POMS, and Chinese Affect Scale consistent with theoretical predictions. Multi-sample CFAs showed the BRUMS-C to be invariant at the configural, metric, strong, and structural levels for all samples. Furthermore, latent mean difference analyses showed that athletes reported significantly higher levels of fatigue than students while maintaining almost the same levels of vigour, and adolescent students reported significantly higher levels of depressed mood than the other three samples.     

Some Personality Dimensions of Doctors of Education

The comparison of scores from linguistically different tests is a twofold matter: the adaptation of tests and the comparison of scores. These 2 aspects of measurement invariance intersect at the need to guarantee the psychometric equivalence between the original and adapted versions. In this study, the authors examined comparability in 2 stages. First, they conducted a thorough study of progressive factorial variance through which they defined an anchor test. Second, they defined an observed score-equated function to establish equivalences between the original test and the adapted test; they used a design of common item nonequivalent groups for this purpose.     

Investigating the Sensitivity of Goodness-of-Fit Indices to Detect Measurement Invariance in a Bifactor Model

A Monte Carlo simulation study was conducted to evaluate the sensitivities of the likelihood ratio test and five commonly used delta goodness-of-fit (ΔGOF) indices (i.e., ΔGamma, ΔMcDonald’s, ΔCFI, ΔRMSEA, and ΔSRMR) to detect a lack of metric invariance in a bifactor model. Experimental conditions included factor loading differences, location and number of noninvariant items, and sample size. The results indicated all ΔGOF indices held Type I error to a minimum and overall had adequate power for the study. For detecting the violation of metric invariance, only ΔGamma and ΔCFI, in addition to Δχ², are recommended to use in the bifactor model with values of ?.016 to ?.023 and ?.003 to ?.004, respectively. Moreover, in the variance component analysis, the magnitude of the factor loading differences contributed the most variation to all ΔGOF indices, whereas sample size affected Δχ² the most.     

Measurement invariance across information and communication technology development index and gender: The case of the Pearson Test of English Academic reading

The computerization of reading assessments has presented a set of new challenges to test designers. From the vantage point of measurement invariance, test designers must investigate whether the traditionally recognized causes for violating invariance are still a concern in computer-mediated assessments. In addition, it is necessary to understand the technology-related causes of measurement invariance among test-taking populations. In this study, we used the available data (n = 800) from the previous administrations of the Pearson Test of English Academic (PTE Academic) reading, an international test of English comprising 10 test items, to investigate measurement invariance across gender and the Information and Communication Technology Development index (IDI). We conducted a multi-group confirmatory factor analysis (CFA) to assess invariance at four levels: configural, metric, scalar, and structural. Overall, we were able to confirm structural invariance for the PTE Academic, which is a necessary condition for conducting fair assessments. Implications for computer-based education and the assessment of reading are discussed.     

浅议矩阵相关变换的不变性

“不变性”是线性代数中的重要思想方法，它主要反映矩阵在某种条件下的秩数不变、特征值不变及正定性不变等方面。以初等变换为首选的变换方法能够充分认识矩阵或线性方程组相关变换的本质。     

Evaluation of Structural Relationships in Autoregressive Cross-Lagged Models Under Longitudinal Approximate Invariance:A Bayesian Analysis

To infer longitudinal relationships among latent factors, traditional analyses assume that the measurement model is invariant across measurement occasions. Alternative to placing cross-occasion equality constraints on parameters, approximate measurement invariance (MI) can be analyzed by specifying informative priors on parameter differences between occasions. This study evaluated the estimation of structural coefficients in multiple-indicator autoregressive cross-lagged models under various conditions of approximate MI using Bayesian structural equation modeling. Design factors included factor structures, conditions of non-invariance, sizes of structural coefficients, and sample sizes. Models were analyzed using two sets of small-variance priors on select model parameters. Results showed that autoregressive coefficient estimates were more accurate for the mixed pattern than the decreasing pattern of non-invariance. When a model included cross-loadings, an interaction was found between the cross-lagged estimates and the non-invariance conditions. Implications of findings and future research directions are discussed.     

体育院校学生生涯决策自我效能的跨组分析

目的:考察《大学生生涯决策自我效能问卷》在我国体育院校大学生人群的适用性,比较该人群生涯决策自我效能水平在专业类别和年级上的差异。方法:运用上述工具对1,740名体育院校大学生施测,采用验证性因素分析及Cronbachα系数考察该工具使用的效度和信度,在检验测量恒等性基础上,进一步跨组比较分析专业类别和年级的差异,并对体育院校系毕业生、各年级学生与普通高校学生的生涯决策自我效能进行平均数差异性检验。结果:二阶因素模型的拟合指数良好,各条目的因素负载在0.44～0.69间,均达到显著性水平,各分量Cronbachα系数在0.80～0.85间。各单样本生涯决策自我效能结构形态恒等,跨母群因素恒等模型可以接受。体艺类专业与非体艺类专业大学生在生涯决策自我效能上无显著差异,大四毕业生的水平著低于其他年级学生。体育院校学生的生涯决策自我效能与当前普通高校学生无显著差异,但大四学生的水平显著低于九年前普通高校毕业生水平。结论:这一测量工具适用于测量我国体育院校大学生生涯决策自我效能的水平,专业类别并不影响体育院校系大学生的生涯决策自我效能,这进一步证明生涯决策自我效能是一个与专业类别关联不大的概念结构。与同时期其他普通高校大学生一样,体育院校系毕业生临近就业,生涯自我效能水平明显低于低年级学生,这一现状与过去、国外状况不同。     

Two-Step Approach to Partial Factorial Invariance: Selecting a Reference Variable and Identifying the Source of Noninvariance

To date, no effective empirical method has been available to identify a truly invariant reference variable (RV) in testing measurement invariance under a multiple-group confirmatory factor analysis. This study proposes a method that, in selecting an RV, uses the smallest modification index (min-mod). The method’s performance is evaluated using 2 models: (a) a full invariance model, and (b) a partial invariance model. Results indicate that for both models the min-mod successfully identifies a truly invariant RV (Study 1). In Study 2, we use the RV found in Study 1 to further evaluate the performance of item-by-item Wald tests at locating a noninvariant variable. The results indicate that Wald tests overall performed better with an RV selected in a partial invariance model than an RV selected in a full invariance model, although in certain conditions their performances were rather similar. Implications and limitations of the study are also discussed.     

Sensitivity of the Soccer Defending Skill Scale: A comparison between teams

Abstract

The aim of the present study was to examine the sensitivity of the Soccer Defending Skill Scale (SDSS) using multiple-group analysis and a structured mean model. The data were 469 defending performances in the finals of the FIFA World Cup Korea/Japan 2002^TM, which were measured as distances, angles between attackers and defenders, and the number of players using 6-point interval scales. The results of multiple-group analysis indicated that the model, which had factor loading, intercept and factor correlation, and F3/F4/F6 variances invariant across Brazil and Germany, represented the best fit to the data (e.g. Tucker-Lewis index = 1.001, confirmatory fit index = 1.000, root mean square error of approximation = 0.000). Furthermore, the latent mean model comparison between teams revealed significant differences in latent mean with respect to “defence to attackers without the ball” (Brazil < Germany). We conclude that our results provide strong support for the sensitivity of the SDSS.