Data-Generating Mechanisms Versus Constructively Defined Latent Variables in Multitrait–Multimethod Analysis: A Comment on Castro-Schilo,Widaman, and Grimm (2013)

In a recent article, Castro-Schilo, Widaman, and Grimm (2013) compared different approaches for relating multitrait–multimethod (MTMM) data to external variables. Castro-Schilo et al. reported that estimated associations with external variables were in part biased when either the correlated traits–correlated uniqueness (CT-CU) or correlated traits–correlated (methods–1) [CT-C(M–1)] models were fit to data generated from the correlated traits–correlated methods (CT-CM) model, whereas the data-generating CT-CM model accurately reproduced these associations. Castro-Schilo et al. argued that the CT-CM model adequately represents the data-generating mechanism in MTMM studies, whereas the CT-CU and CT-C(M–1) models do not fully represent the MTMM structure. In this comment, we question whether the CT-CM model is more plausible as a data-generating model for MTMM data than the CT-C(M–1) model. We show that the CT-C(M–1) model can be formulated as a reparameterization of a basic MTMM true score model that leads to a meaningful and parsimonious representation of MTMM data. We advocate the use confirmatory factor analysis MTMM models in which latent trait, method, and error variables are explicitly and constructively defined based on psychometric theory.     

Bayesian Versus Maximum Likelihood Estimation of Multitrait–Multimethod Confirmatory Factor Models

This article compares maximum likelihood and Bayesian estimation of the correlated trait–correlated method (CT–CM) confirmatory factor model for multitrait–multimethod (MTMM) data. In particular, Bayesian estimation with minimally informative prior distributions—that is, prior distributions that prescribe equal probability across the known mathematical range of a parameter—are investigated as a source of information to aid convergence. Results from a simulation study indicate that Bayesian estimation with minimally informative priors produces admissible solutions more often maximum likelihood estimation (100.00% for Bayesian estimation, 49.82% for maximum likelihood). Extra convergence does not come at the cost of parameter accuracy; Bayesian parameter estimates showed comparable bias and better efficiency compared to maximum likelihood estimates. The results are echoed via 2 empirical examples. Hence, Bayesian estimation with minimally informative priors outperforms enables admissible solutions of the CT–CM model for MTMM data.     

An Expansion of the Trait-State-Occasion Model: Accounting for Shared Method Variance

This article examines 4 approaches for explaining shared method variance, each applied to a longitudinal trait–state–occasion (TSO) model. Many approaches have been developed to account for shared method variance in multitrait-multimethod (MTMM) data. Some of these MTMM approaches (correlated method, orthogonal method, correlated method minus one, correlated uniqueness) were therefore borrowed in these analyses such that their effectiveness could be evaluated in conjunction with a TSO model. To this end, datasets were generated according to 4 different covariance matrices (each created according to specifications of a model built with 1 of the 4 approaches) and each model was crossed with each type of data. Whereas the correlated method and correlated method minus one approaches encountered many difficulties in convergence, fit, or parameter estimates, the correlated uniqueness and orthogonal method approaches proved to be quite versatile.     

Analyzing Different Types of Moderated Method Effects in Confirmatory Factor Models for Structurally Different Methods

An extension of two confirmatory factor models for multitrait-multimethod measurement designs with structurally different methods to the analysis of latent interaction effects is presented: the nonlinear latent difference (NL-LD) model and the nonlinear correlated trait–correlated method-minus-one (NL-CTC[M – 1]) model. Both models are compared with regard to (a) the psychometric definition of the latent variables, (b) the capabilities of explaining latent method effects, and (c) the analysis of latent interaction effects. Using the latent moderated structural equation approach, we show how moderated method effects can be examined in the NL-CTC(M – 1) model. This fine-grained analysis of method effects is not feasible using the classical NL-LD model. We propose an extended version of the NL-LD model, which recovers the results of the NL-CTC(M – 1) model. The different versions of the nonlinear multimethod models are illustrated using real data from a multirater study. Finally, the advantages and challenges of incorporating latent interaction effects in complex CFA–MTMM models are discussed.     

Augmenting the Correlated Trait–Correlated Method Model for Multitrait–Multimethod Data

We introduce an approach for ensuring empirical identification of the correlated trait–correlated method (CT–CM) model under a variety of conditions. A set of models are referred to as augmented correlated trait–correlated method (ACT–CM) models because they are based on systematically augmenting the multitrait–multimethod matrix put forth by Campbell and Fiske (1959). We show results from a Monte Carlo simulation study in which data characteristics lead to an empirically underidentified standard CT–CM model, but a well-identified fully augmented correlated trait–correlated method (FACT–CM) model. This improved identification occurs even for a model in which equality constraints are imposed on loadings on each trait factor and loadings on each method factor—a specific case shown to lead to an empirically underidentified CT–CM model.     

Additive Models for Multitrait–Multimethod Data With a Multiplicative Trait–Method Relationship: A Simulation Study

Confirmatory factor analysis (CFA) is widely used for analyzing multitrait-multimethod (MTMM) data. But there is no consensus about whether multiplicative or additive trait-method effect of its parameterization, most appropriately represents the underlying structure of MTMM data. Given the popularity of two additive CFA models, the CT-CM model and the CT-CU model, this simulation investigates their performance for multiplicative MTMM data. Results showed that the CT-CM model had much lower convergence and proper solution rates than the CT-CU model. Although both models had adequate fit for the converged solutions, all parameter estimates from the CT-CM model were unacceptably biased. The CT-CU model worked well in most conditions and was quite robust to the multiplicative data when the matrix size was 3T3M.     

Neglect the Structure of Multitrait-Multimethod Data at Your Peril: Implications for Associations With External Variables

In 1959, Campbell and Fiske introduced the use of multitrait–multimethod (MTMM) matrices in psychology, and for the past 4 decades confirmatory factor analysis (CFA) has commonly been used to analyze MTMM data. However, researchers do not always fit CFA models when MTMM data are available; when CFA modeling is used, multiple models are available that have attendant strengths and weaknesses. In this article, we used a Monte Carlo simulation to investigate the drawbacks of either using CFA models that fail to match the data-generating model or completely ignore the MTMM structure of data when the research goal is to uncover associations between trait constructs and external variables. We then used data from the National Institute of Child Health and Human Development Study of Early Child Care and Youth Development to illustrate the substantive implications of fitting models that partially or completely ignore MTMM data structures. Results from analyses of both simulated and empirical data show noticeable biases when the MTMM data structure is partially or completely neglected.     

Multitrait–Multimethod Assessment of Giftedness: An Application of the Correlated Traits–Correlated (Methods – 1) Model

Although the use of multiple criteria and informants is one of the most universally agreed on practices in the identification of gifted children, few studies to date have examined the convergent validity of multiple informants and objective ability tests in gifted identification. In this study, we illustrate the use of the correlated traits–correlated (methods – 1) or CT–C(M – 1) model (Eid, Lischetzke, Nussbeck, & Trierweiler, 2003) to examine the convergent validity of self, parent, and teacher ratings relative to objective cognitive ability tests in a sample of 145 4th to 6th graders. The CT–C(M – 1) analyses revealed that teacher ratings showed the highest convergence with the objective assessments, whereas self-ratings had the lowest reliabilities and insufficient validity. Parent ratings were more reliable and valid than self-reports, but were outperformed by teacher ratings for most abilities. Overall, the CT–C(M – 1) analyses showed that the convergent validity of the ratings relative to the objective test battery was highest for numerical and lowest for creative abilities. Furthermore, whereas part of the shared variance between parent and teacher ratings reflected true convergent validity, agreement between parent and self-reports was entirely due to a shared rater variance. Our analyses demonstrate the usefulness and proper interpretation of the CT–C(M – 1) approach for examining convergent validity and method effects in multitrait–multimethod data.     

Testing Nested Additive,Multiplicative, and General Multitrait-Multimethod Models

The article gives alternatives to Campbell and O'Connell's (1967) definitions of additive and multiplicative method effects in multitrait-multimethod (MTMM) data. The alternative definitions can be formulated by means of constraints in the parameters of the correlated uniqueness (CU) model (Marsh, 1989), which is first reviewed. The definitions have 2 major advantages. First, they allow the researcher to test for additive and multiplicative method effects in a straightforward manner by simply testing the appropriate constraints. An illustration of these tests is given. Second, the alternative definitions are closely linked to other currently used models. The article shows that CU models with additive constraints are equivalent to constrained versions of the confirmatory factor analysis model for MTMM data (Althauser, Heberlein, & Scott, 1971; Werts & Linn, 1970). In addition, Coenders and Saris (1998) showed that, for designs with 3 methods, a CU model with multiplicative constraints is equivalent to the direct product model (Browne, 1984).     

Examining Trait × Method Interactions Using Mixture Distribution Multitrait–Multimethod Models

Multitrait-multimethod (MTMM) analyses are used in psychology to assess convergent and discriminant validity and to study method effects. Most current MTMM approaches assume that measures have equal convergent and discriminant validity across the entire range of trait values and thus do not account for potential trait × method interactions. A novel approach is presented that allows analyzing trait × method interactions using factor mixture modeling. The new MTMM mixture model allows identifying latent classes of individuals who differ with respect to convergent and discriminant validity. The new approach was applied to mother’s and father’s ratings of children’s attention deficit hyperactivity disorder (ADHD) symptoms (N = 618). Results revealed four latent classes: one with no symptom levels, two with low symptom levels, and one with moderate symptom levels. Three classes showed evidence for convergent and discriminant validity, whereas a low symptom class lacked convergent validity for ratings of inattention.

     

Mapping of quantitative trait loci using the skew-normal distribution

In standard interval mapping (IM) of quantitative trait loci (QTL), the QTL effect is described by a normal mixture model. When this assumption of normality is violated, the most commonly adopted strategy is to use the previous model after data transformation. However, an appropriate transformation may not exist or may be difficult to find. Also this approach can raise interpretation issues. An interesting alternative is to consider a skew-normal mixture model in standard IM, and the resulting method is here denoted as skew-normal IM. This flexible model that includes the usual symmetric normal distribution as a special case is important, allowing continuous variation from normality to non-normality. In this paper we briefly introduce the main peculiarities of the skew-normal distribution. The maximum likelihood estimates of parameters of the skew-normal distribution are obtained by the expectation-maximization (EM) algorithm. The proposed model is illustrated with real data from an intercross experiment that shows a significant departure from the normality assumption. The performance of the skew-normal IM is assessed via stochastic simulation. The results indicate that the skew-normal IM has higher power for QTL detection and better precision of QTL location as compared to standard IM and nonparametric IM.     

The effect of a pretest in an interactive,multimodal pretraining system for learning science concepts

Multiple traits of language proficiency as well as test method effects were concurrently analyzed to investigate interrelations of construct validity, convergent validity, and discriminant validity using multitrait-multimethod (MTMM) matrices. A total of 585 test takers' scores were derived from the field test of the Pearson Test of English Academic. An MTMM confirmatory factor analysis model was parameterized using 4 traits and 3 assessment methods. The 4 traits included listening, reading, speaking, and integrated skills, while the 3 methods included prescribed multiple-choice responses, constructed responses, and summarized responses. The trait factor loadings were systematically greater than those of methods, providing evidence that the indicators were strongly related to their latent constructs, after adjusting for the method effects. The results showed robust convergent validity, moderate discriminant validity, and insignificant method effects. Implications are discussed.     

A Monte Carlo Examination of an MTMM Model With Planned Incomplete Data Structures

The classic approach for partitioning and assessing reliability and validity has been through the use of the multitrait-multimethod (MTMM) model. The MTMM approach generally involves 3 different groups (method) evaluating 3 traits. This approach can be reconceptualized for questionnaire evaluation, so that the method becomes 3 different scaling types, which are administered to the same respondents on different occasions to avoid carryover effects. A serious limitation of this MTMM model is that data are required from respondents on at least 3 different occasions, thus placing a heavy burden on the researcher and respondents. Planned incomplete data designs for the purpose of substantially reducing the amount of data required for MTMM models were investigated: 1st, a design that reduces the amount of data collected at the 3rd administration by 22%; and 2nd, a design in which data need only be collected at 2 occasions. The performance of Listwise Deletion, Pairwise Deletion, and the expectation maximization (EM) algorithm at dealing with planned incomplete data are examined through a series of simulations. Results indicate that EM was generally precise and efficient.     

Fit of Different Models for Multitrait-Multimethod Experiments

In the past, several models have been developed for the estimation of the reliability and validity of measurement instruments from multitrait-multimethod (MTMM) experiments. Suggestions have been made for additive, multiplicative and correlated uniqueness models, whereas recently Coenders and Saris (2000) suggested a procedure to test these models against one another. In this article, the different models suggested for the analysis of MTMM matrixes have been compared for their fit to 87 data sets collected in the United States (Andrews, 1984; Rodgers, Andrews, & Herzog, 1992), Austria (Koltringer, 1995), and the Netherlands (Scherpenzeel & Saris, 1997). As most variables are categorical, the analysis has been carried out on the basis of polychoric-polyserial correlation coefficients and of Pearson correlations. The fit of the models based on polychoric correlations is much worse than the fit of models based on product moment correlations, but in both cases a model that assumes additive method effects fits most data sets better than the other models, including the so-called multiplicative models.     

Special education transition issues over the past 60 years: social network analysis on Web of Science exploration

This research identified the potential special education transition development trajectories by tracing the priorities and hierarchical structure of published transition issues in special education. Social network analysis (SNA) of the 1900–2015 data from the Web of Science (WOS) database achieves a transition exploration flow by citation-based main path analysis (MPA). The first main finding was that 517 papers were published relating to special education transition over the past 60 years. Special education transition literature emerged in 1957, was systematically explored after 1987, and grew rapidly in the 1990s. The second main finding was that the trajectories contained various topics that can be broken down into two mainstream fields: curriculum and employment. The third main finding was that there are five sub-branches of the mainstream fields: legislation; team working; accountability; individualised transition programmes (ITP); and self-determination. This research is the first to use MPA techniques in SNA for special education transition. The findings indicate that more foci deserve to be explored to attain the transition working exhaustively and smooth transitions between each career stage for individuals with disabilities.     

基于大数据刍议普通高考引领“学科核心素养”落地的作用——以2019年高考(天津卷)数学(文史类)试题为例

依据2019年高考(天津卷)数学(文史类)试卷的实测数据,参照《天津市高考数学学科考生水平表现标准》和《高中数学课程标准(2017年版)》中对学科核心素养的学业质量水平的划分,对2019年高考(天津卷)数学(文史类)试卷的命题特色、考生数学核心素养的作答表现以及反映出的教学问题进行分析和评价,并提出厘清核心要义、加强单元设计、理解课标教材、关注数学文化等教学建议。     

Thinking critically: a look at students' critiques of a research article

The ability to reason, analyse and evaluate issues critically is a valued skill and ranks highly in the list of attributes expected of graduates. Much has been written about the importance and application of critical thinking in various domains, but studies on the actual manifestation of such skills in students' writing have attracted only modest interest. Even less has been written about critiques in relation to critical thinking. This study sought to investigate the form and nature of issues raised by 119 second-year biology undergraduates in their critiques of the introduction section of a research article. The study revealed that the vast majority of students tended to raise surface issues in their critiques, focusing on visible textual features such as rhetorical structure and language-related issues. The minority who raised depth issues addressed the arguments used in the reading and their significance. In light of the skewed results, a two-stage process – involving (1) summary writing and (2) the use of evaluative criteria and the Toulmin model as an overarching framework – is recommended to enhance the teaching of critical thinking within the curriculum. This study offers a glimpse into the outcomes of critical thinking, as represented by the students' critiques. It provides a bottom-up approach to our understanding of the issues raised by students in a task centred on critical thinking and so focuses our attention on specific areas for further consideration or remediation.     

Examining and Controlling for Wording Effect in a Self-Report Measure: A Monte Carlo Simulation Study

Wording effect refers to the systematic method variance caused by positive and negative item wordings on a self-report measure. This Monte Carlo simulation study investigated the impact of ignoring wording effect on the reliability and validity estimates of a self-report measure. Four factors were considered in the simulation design: (a) the number of positively and negatively worded items, (b) the loadings on the trait and the wording effect factors, (c) sample size, and (d) the magnitude of population validity coefficient. The findings suggest that the unidimensional model that ignores the negative wording effect would underestimate the composite reliability and criterion-related validity, but overestimate the homogeneity coefficient. The magnitude of relative bias of the composite reliability was generally small and acceptable, whereas the relative bias for the homogeneity coefficient and criterion-related validity coefficient was negatively correlated with the strength of the general trait factor.     

A preliminary investigation of the concurrent validity of reading comprehension rate: A direct,dynamic measure of reading comprehension

Reading comprehension rate (RCR) is a direct measure of reading skills that may be useful in formatively evaluating students reading beyond the fourth‐grade level. To investigate the concurrent validity of RCR, we correlated RCR, reading comprehension level (RCL), and words correct per minute (WC/M) with the Broad Reading Cluster Scores of the Woodcock‐Johnson III Tests of Achievement (WJ‐III ACH) across 88 students in 4th, 5th, and 10th grades. Results showed that aloud‐RCR was significantly correlated with the WJ‐III ACH scores for 4th‐grade (r = .90; n = 22), 5th‐grade (r = .87; n = 29), and 10th‐grade (r = .65; n = 37) students. Regression analysis specified a one‐predictor model for 4th‐grade students (aloud‐RCR), a two‐predictor model for 5th‐grade students (WC/M and aloud‐RCR), and a one‐predictor model for 10th‐grade students (WC/M). Discussion focuses on directions for future research and applied issues related to RCR probe passage development. © 2007 Wiley Periodicals, Inc. Psychol Schs 44: 373–388, 2007.     

Ethical issues in learning analytics: a review of the field

Learning analytics (LA) collects, analyses, and reports big data about learners to optimise learning. LA ethics is an interdisciplinary field of study that addresses moral, legal, and social issues; therefore, institutions are responsible for implementing frameworks that integrate these topics. Many of the ethical issues raised apply equally to educational data sets of any size. However, in this study, we focus on big data that increases the scale and granularity of data gathered. The purpose of this study is twofold: (a) to critically review the published (2011–2018) scientific literature on LA ethics issues and (b) to identify current trends and answer research questions in the field. This study’s research questions are as follows: what is essential in LA ethics for key educational stakeholders, and what should a proposed checklist for LA ethics include for specific educational stakeholders? After systematically searching online bibliographic databases, journals, and conferences, a literature review identified 53 articles from a sample of 562. The selected articles, based on critical and qualitative content analysis, were exhaustively analysed. The findings demonstrate the shortage of empirical evidence-based guidelines on LA ethics and highlight the need to establish codes of practices to monitor and evaluate LA ethics policies. Finally, this work proposes a useful checklist as an instructional design model for scholars, policymakers, and instructional designers, so that trusted partners may use LA responsibly to improve teaching and learning.