Assessing the comparability of teacher-related constructs in TIMSS 2015 across 46 education systems: an alignment optimization approach

Research related to the “teacher characteristics” dimension of teacher quality has proven inconclusive and weakly related to student success, and addressing the teaching contexts may be crucial for furthering this line of inquiry. International large-scale assessments are well positioned to undertake such questions due to their systematic sampling of students, schools, and education systems. However, researchers are frequently prohibited from answering such questions due to measurement invariance related issues. This study uses the traditional multiple group confirmatory factor analysis (MGCFA) and an alignment optimization method to examine measurement invariance in several constructs from the teacher questionnaires in the Trends in International Mathematics and Science Study (TIMSS) 2015 across 46 education systems. Constructs included mathematics teacher’s Job satisfaction, School emphasis on academic success, School condition and resources, Safe and orderly school, and teacher’s Self-efficacy. The MGCFA results show that just three constructs achieve invariance at the metric level. However, an alignment optimization method is applied, and results show that all five constructs fall within the threshold of acceptable measurement non-invariance. This study therefore presents an argument that they can be validly compared across education systems, and a subsequent comparison of latent factor means compares differences across the groups. Future research may utilize the estimated factor means from the aligned models in order to further investigate the role of teacher characteristics and contexts in student outcomes.

     

Family and motivation effects on mathematics achievement: Analyses of students in 41 countries

This study examines family and motivation effects on student mathematics achievement across 41 countries. The Rasch estimates of PISA mathematics test scores and questionnaire responses of 107,975 15-year-old students were analyzed via multilevel analyses. Students scored higher in richer or more egalitarian countries; when living with two parents, without grandparents, with fewer siblings (especially fewer older siblings); with higher family SES, more books, cultural possessions, or cultural communication; or when they had greater interest in mathematics, more effort and perseverance, and higher self-efficacy or self-concept. Family structure effects were stronger in individualistic or richer countries. Richer countries showed stronger family cultural communication effects, suggesting stronger, intangible resource effects.     

Using multi-group confirmatory factor analysis to evaluate cross-cultural research: identifying and understanding non-invariance

Multi-group confirmatory factor analysis (MGCFA) allows researchers to determine whether a research inventory elicits similar response patterns across samples. If statistical equivalence in responding is found, then scale score comparisons become possible and samples can be said to be from the same population. This paper illustrates the use of MGCFA by examining survey results relating to practising teachers' conceptions of feedback in two very different jurisdictions (Louisiana, USA, n?=?308; New Zealand, n?=?518), highlighting challenges which can occur when conducting this kind of cross-cultural research. As the two contexts had very different policies and practices around educational assessment, it was considered possible that a common research inventory may elicit non-equivalent responding, leading to non-invariance. Independent models for each group and a joint model for all participants were tested for invariance using MGCFA and all were inadmissible for one of the two groups. Inspection of joint model differences in item loadings, scale reliabilities, and scale inter-correlations established the extent of non-invariance. This paper discusses the implications of non-invariance within this particular study and identifies difficulties in using an inventory in cross-cultural settings. It also provides suggestions about how to increase the likelihood that a common factor structure can be recovered.     

Testing measurement invariance of PISA 2015 mathematics,science, and ICT scales using the alignment method

This study investigates measurement invariance of the mathematics, science, and ICT scales across the 47 countries that participated in the PISA 2015 ICT Familiarity Questionnaire. Knowing whether the same constructs and measurements can be reliably compared across countries constitutes an important goal. The Alignment method is employed to test the measurement invariance of the three scales. The results show that mathematics and science scores are highly invariant and can be used to compare countries, whereas the ICT scale is mostly non-invariant and cannot be used to reliably compare ICT means across all participating countries. Implications and limitations are discussed.     

Evaluating item response theory linking and model fit for data from PISA 2000–2012

ABSTRACT

Based on concerns about the item response theory (IRT) linking approach used in the Programme for International Student Assessment (PISA) until 2012 as well as the desire to include new, more complex, interactive items with the introduction of computer-based assessments, alternative IRT linking methods were implemented in the 2015 PISA round. The new linking method represents a concurrent calibration using all available data, enabling us to find item parameters that maximize fit across all groups and allowing us to investigate measurement invariance across groups. Apart from the Rasch model that historically has been used in PISA operational analyses, we compared our method against more general IRT models that can incorporate item-by-country interactions. The results suggest that our proposed method holds promise not only to provide a strong linkage across countries and cycles but also to serve as a tool for investigating measurement invariance.     

PISA Country Rankings Valid? Results for Canada and Finland

This article examines whether the way that PISA models item outcomes in mathematics affects the validity of its country rankings. As an alternative to PISA methodology a two-parameter model is applied to PISA mathematics item data from Canada and Finland for the year 2012. In the estimation procedure item difficulty and dispersion parameters are allowed to differ across the two countries and samples are restricted to respondents who actually answered items in a mathematics cluster. Different normalizations for identifying the distribution parameters are also considered. The choice of normalization is shown to be crucial in guaranteeing certain invariance properties required by item response models. The ability scores obtained from the methods employed here are significantly higher for Finland, in sharp contrast to PISA results, which gave both countries very similar ranks in mathematics.     

Comparability of educational achievement and learning attitudes across nations

We estimated the invariance of educational achievement (EA) and learning attitudes (LA) measures across nations. A multi-group confirmatory factor analysis was used to estimate the invariance of educational achievement and learning attitudes across 55 nations (Programme for International Student Assessment [PISA] 2006 data, N?=?354,203). The constructs had the same meaning (factor loadings) but different scales (intercepts). Our conclusion is that comparisons of the relationships between educational achievement and learning attitudes across countries need to take into consideration two sources of variability: individual differences of students and group differences of educational systems. The lack of scalar invariance in EA and LA measures means that the relationships between EA and LA may have a different meaning at the level of nations and at the student level within countries. In other words, as PISA measures are not invariant in scalar sense, the comparisons across countries with nationally aggregated scores are not justified.     

A Constructive Program in Silent Reading

ABSTRACT

The authors examined whether self-concept, self-efficacy, and self-esteem show differential predictive utility for academic achievement across age groups and domains. More specifically, the relationships of 3 self-constructs with achievement were examined in mathematics for elementary school students and mathematics and language arts for middle school students in Korea. Task value and test anxiety were hypothesized to mediate these relationships. Consistent with previous reports, domain-specific self-constructs such as self-efficacy and self-concept were better predictors of task value and achievement than was general self-esteem. Task value and test anxiety significantly mediated only the relationships of self-efficacy assessed by the Bandura-type scale to achievement. These domain-specific relationships tended to be stronger for middle school than elementary school students and in mathematics than language arts.     

Students with special educational needs within PISA

Abstract

Since 2003, the Programme for International Student Assessment (PISA) has included students with special educational needs (SEN), identified as those with functional disabilities, those with cognitive/behavioural/emotional disabilities and those with limited test language proficiency. While the number of countries and included students has increased with each test administration, the percentage of students with SEN remains extremely low. The inclusion of these students is not an intentional PISA design parameter but rather a response to the interaction between the need to maintain strict sampling criteria and country-level educational mandates to include SEN students in standardised testing. Based on the analysis of student participation and performance across four cycles of PISA (2003–2012), this paper examines the challenges that exist in current PISA procedures related to: student sampling, eligibility and identification; assessment methodology; and reporting results. PISA practices, their limitations for scientific inferences and recommendations for design improvements are given.     

Benchmarking the performance of Bhutanese students with the performance of the students from the OECD's PISA countries

Setting international benchmarks for education systems of the Organisation for Economic Co-operation and Development (OECD) countries is one of the goals of the OECD's Programme for International Student Assessment (PISA). However, some countries are not able to participate in PISA, despite their desire to set international benchmarks for their education systems. This article presents a method of setting international benchmarks for a country's school education system, without necessarily participating in PISA, by designing a test using the test items released by PISA for public consumption. The method has been implemented in a study that involved 1,500 Grade 10 students across 60 schools in Bhutan. The students were administered a mathematics test constructed from the PISA Mathematical Literacy test items. The study showed that the performance of Bhutanese students was comparable with the performance of the students from the countries that participated in PISA 2003 and that Bhutan could learn from both high- and low-performing school education systems of those countries.     

PISA Country Rankings for Italy and Spain: Revised Results

This article examines whether the way that PISA models item outcomes in mathematics affects the validity of its country rankings. As an alternative to PISA methodology, a two-parameter logistic model is applied to PISA mathematics item data from Italy and Spain for the year 2009. In the estimation procedure, item difficulty and dispersion parameters were allowed to differ across the two countries and samples were restricted to respondents who actually answered items in a mathematics cluster. Different normalizations for identifying the distribution parameters were also considered. The choice of normalization is shown to be crucial in guaranteeing certain invariance properties required by item response models. The ability or proficiency scores obtained from the methods employed here are significantly higher for Spain, in sharp contrast to PISA results, which gave both countries virtually the same rank in mathematics (489 for Italy and 488 for Spain). These results raise serious questions about PISA methodology and the role that PISA results play in the formulation educational policy in member countries.     

Measurement invariance of information,communication and technology (ICT) engagement and its relationship with student academic literacy: Evidence from PISA 2018

The present study aimed to examine the measurement invariance of the information, communication and technology (ICT) engagement questionnaire in the Programme for International Student Assessment (PISA) across countries, and further to explore the effects of ICT engagement on student literacy. The sample included 98,758 15-year-old students from 16 countries in PISA 2018. The multi-group confirmatory factor analyses results indicated that the ICT engagement questionnaire was invariant at the residual level so that meaningful cross-country comparisons can be made at the observed level. Furthermore, the multi-level model results indicated that perceived ICT autonomy was a positive predictor, while use of social media was a negative predictor of student literacy across the countries. However, the relations of interest in ICT and perceived ICT competence to student literacy were inconsistent, with some countries showing positive and linear relationships, while others negative and linear relationships.     

Classroom disciplinary climate of schools and gender – evidence from the Nordic countries

ABSTRACT

Classroom disciplinary climate has emerged as a crucial factor with regard to student achievement. However, most previous studies have not explored potential gender differences in both students’ perceptions of the classroom disciplinary climate and the association between classroom disciplinary climate and student learning. Using data from the Organisation for Economic Co-operation and Development (OECD) Programme for International Student Assessment (PISA) 2012 for the Nordic countries, we found a significant association between the perceived classroom disciplinary climate of schools and students’ mathematics performance across countries. On the basis of an analysis of a pooled sample consisting of all 5 Nordic countries, we found that the correlation between classroom disciplinary climate of schools and maths achievement is significantly stronger for boys than for girls. Further analyses showed that this finding may partly be attributable to gender differences in the perception of the disciplinary climate of schools, whereby boys seemed to perceive the classroom disciplinary climate of schools more positively than girls.     

BUILDING MATHEMATICS ACHIEVEMENT MODELS IN FOUR COUNTRIES USING TIMSS 2003

Using the Trends in International Mathematics and Science Study 2003 data, this study built mathematics achievement models of 8th graders in four countries: the USA, Russia, Singapore and South Africa. These 4 countries represent the full spectrum of mathematics achievement. In addition, they represent 4 continents, and they include 2 countries hugely influential in world events (the USA and Russia). In each country, students’ self-concept of ability in mathematics, mathematics values, perception of school, teachers’ and principals’ perceptions of school and other characteristics related to the classroom and school were incorporated to build an achievement model through hierarchical linear modelling. The final achievement models suggested that among student variables, self-concept of ability in mathematics had the highest relation to 8th graders’ mathematics achievement in all 4 countries. The relation between mathematics achievement and other student characteristics, along with the family, teacher and school variables, differed across the 4 countries. This suggests that self-concept of ability is a key variable for understanding achievement in high and low achieving countries and that other contextual variables vary in the magnitude of relations to mathematics achievement across countries.     

The Measurement Structure,Stability and Mediating Effects of Achievement Goals in Math with Middle-School Student Data

This study examined the measurement structure, cross-year stability of achievement goals, and mediating effects of achievement goals between self-efficacy and math grades in a national sample of Taiwan middle school students. The measurement model with factorial structure showed good fit to the data. In the panel data (N?=?343), four achievement goals showed strong measurement invariance, suggesting factor loadings and intercepts of the items remained invariant across a year. Though mean scores of the four latent achievement goals held quite stable, the rank order of students across two time-points changed more profoundly in the two avoidance goals than in the approached goals. In the cross-sectional data (N?=?748), we found approach-based goals were positive mediators between self-efficacy and math grades while avoidance-based goals were negative mediators. This result could be relevant for middle-school students in learning mathematics. Some instructional implications are provided.     

Science Engagement and Literacy: A retrospective analysis for students in Canada and Australia

Given international concerns about students' pursuit (or more correctly, non-pursuit) of courses and careers in science, technology, engineering and mathematics, this study is about achieving a better understanding of factors related to high school students' engagement in science. The study builds on previous secondary analyses of Programme for International Student Assessment (PISA) datasets for New Zealand and Australia. For the current study, we compared patterns of science engagement and science literacy for male and female students in Canada and Australia. The study's secondary analysis revealed that for all PISA measures included under the conceptual umbrella of engagement in science (i.e. interest, enjoyment, valuing, self-efficacy, self-concept and motivation), 15-year-old students in Australia lagged their Canadian counterparts to varying, albeit modest, degrees. Our retrospective analysis further shows, however, that gender equity in science engagement and science literacy is evident in both Canadian and Australian contexts. Additionally, and consistent with our previous findings for indigenous and non-indigenous students in New Zealand and Australia, we found that for male and female students in both countries, the factor most strongly associated with variations in engagement in science was the extent to which students participate in science activities outside of school. In contrast, and again for both Canadian and Australian students, the factors most strongly associated with science literacy were students' socioeconomic backgrounds, and the amount of formal time spent doing science. The implications of these results for science educators and researchers are discussed.     

The role that mathematics plays in college- and career-readiness: evidence from PISA

Many studies have found a strong relationship between the mathematics students study in school and their performance on an academic or school mathematics assessment but not on an assessment of mathematics literacy (ML). With many countries, like the USA, placing emphasis on finishing secondary education being mathematically literate and prepared for college or career, this raises the question about the relationship between the mathematics studied in school and any ML students may have. The Programme for International Student Assessment (PISA) ML assessment is embedded in real-world contexts that provide an important window on how ready students are to tackle the situations and problems that await them whether they intend to pursue further education beyond high school or intend to go directly into the labour force. In this paper, we draw upon the PISA 2012 data to investigate the extent to which the cumulative exposure to rigorous mathematics content, such as that embedded in college- and career-ready standards, is associated with ML as assessed in PISA. Results reveal that both exposure to rigorous school mathematics and experiencing the instruction of this mathematics through real-world applications are significantly related to all the real-world contextualized PISA ML scores.     

The causal effect of an extra year of schooling on skills and knowledge in Latin America. Evidence from PISA

Abstract

In this paper, we estimate the causal effect of an extra year of schooling on mathematics performance for seven Latin American countries based on PISA 2012. To that end we exploit exogenous variation in students’ birthdates around the school entry cut-off date using both sharp and fuzzy Regression Discontinuity designs. We find strong effects of an extra year of schooling in most countries, which amount to a 30% increase in PISA test scores in Brazil, 18% in Uruguay, 7% in Argentina and 6% in Costa Rica. These effects differ from the typical estimates obtained from simple regressions or multilevel models and are large enough to allow 15-year-old students to reach higher proficiency levels, suggesting significant potential gains of reducing dropout rates in the region. Finally, we stress the importance of taking into account the effects of school entry cut-off dates on PISA samples to avoid making unfair international comparisons.     

Evaluation of Structural Relationships in Autoregressive Cross-Lagged Models Under Longitudinal Approximate Invariance:A Bayesian Analysis

To infer longitudinal relationships among latent factors, traditional analyses assume that the measurement model is invariant across measurement occasions. Alternative to placing cross-occasion equality constraints on parameters, approximate measurement invariance (MI) can be analyzed by specifying informative priors on parameter differences between occasions. This study evaluated the estimation of structural coefficients in multiple-indicator autoregressive cross-lagged models under various conditions of approximate MI using Bayesian structural equation modeling. Design factors included factor structures, conditions of non-invariance, sizes of structural coefficients, and sample sizes. Models were analyzed using two sets of small-variance priors on select model parameters. Results showed that autoregressive coefficient estimates were more accurate for the mixed pattern than the decreasing pattern of non-invariance. When a model included cross-loadings, an interaction was found between the cross-lagged estimates and the non-invariance conditions. Implications of findings and future research directions are discussed.     

Exploring the validity of a teachers’ self-efficacy scale in five countries

The purpose of this article was twofold. The first purpose was to test the validity of the Teachers’ Sense of Self-Efficacy Scale (TSES) in five settings—Canada, Cyprus, Korea, Singapore, and the United States. The second purpose was, by extension, to establish the importance of the teacher self-efficacy construct across diverse teaching conditions. Multi-group confirmatory factor analysis was used to better understand the measurement invariance of the scale across countries, after which the relationship between the TSES, its three factors, and job satisfaction was explored. The TSES showed convincing evidence of reliability and measurement invariance across the five countries, and the relationship between the TSES and job satisfaction was similar across settings. The study provides general evidence that teachers’ self-efficacy is a valid construct across culturally diverse settings and specific evidence that teachers’ self-efficacy showed a similar relationship with teachers’ job satisfaction in five contrasting settings.