Measurement invariance of information,communication and technology (ICT) engagement and its relationship with student academic literacy: Evidence from PISA 2018

The present study aimed to examine the measurement invariance of the information, communication and technology (ICT) engagement questionnaire in the Programme for International Student Assessment (PISA) across countries, and further to explore the effects of ICT engagement on student literacy. The sample included 98,758 15-year-old students from 16 countries in PISA 2018. The multi-group confirmatory factor analyses results indicated that the ICT engagement questionnaire was invariant at the residual level so that meaningful cross-country comparisons can be made at the observed level. Furthermore, the multi-level model results indicated that perceived ICT autonomy was a positive predictor, while use of social media was a negative predictor of student literacy across the countries. However, the relations of interest in ICT and perceived ICT competence to student literacy were inconsistent, with some countries showing positive and linear relationships, while others negative and linear relationships.     

Testing measurement invariance of mathematics self-concept and self-efficacy in PISA using MGCFA and the alignment method

The aim of the study is to investigate the measurement invariance of mathematics self-concept and self-efficacy across 40 countries that participated in the Programme for International Student Assessment (PISA) 2003 and 2012 cycles. The sample of the study consists of 271,760 students in PISA 2003 and 333,804 students in PISA 2012. Firstly, the traditional measurement invariance testing was applied in the multiple-group confirmatory factor analysis (MGCFA). Then, the alignment analyses were performed, allowing non-invariance to a minimum to estimate all of the parameters. Results from MGCFA indicate that mathematics self-concept and self-efficacy hold metric invariance across the 80 groups (cycle by country). The alignment method results suggest that a large proportion of non-invariance exists in both mathematics self-concept and self-efficacy factors, and the factor means cannot be compared across all participating countries. Results of the Monte Carlo simulation show that the alignment results are trustworthy. Implications and limitations are discussed, and some recommendations for future research are proposed.

     

Language effects in international testing: the case of PISA 2006 science items

We investigate the extent to which language versions (English, French and Arabic) of the same science test are comparable in terms of item difficulty and demands. We argue that language is an inextricable part of the scientific literacy construct, be it intended or not by the examiner. This argument has considerable implications on methodologies used to address the equivalence of multiple language versions of the same assessment, including in the context of international assessment where cross-cultural fairness is a concern. We also argue that none of the available statistical or qualitative techniques are capable of teasing out the language variable and neutralising its potential effects on item difficulty and demands. Exploring the use of automated text analysis tools at the quality control stage may be successful in addressing some of these challenges.     

评估科学素养 培育关键能力——OECD／PISA科学素养之构想、设计与评估

科学素养被认为是21世纪公民生活和工作所必须具备的关键能力的核心,而培养学生的科学素养就成了教育的重要使命之一,如何测评学生的科学素养也因此成了研究的重点。本文试图通过介绍欧洲经济合作与发展组织的“国际学生评价项目”对科学素养的界定和评估框架,为评价学生的综合能力提供借鉴。     

Examining factorial validity and measurement invariance of the Student-Teacher Relationship Scale

The purposes of this study were to (a) test the hypothesized factor structure of the Student-Teacher Relationship Scale (STRS; Pianta, 2001) for 308 African American (AA) and European American (EA) children using confirmatory factor analysis (CFA) and (b) examine the measurement invariance of the factor structure across AA and EA children. CFA of the hypothesized three-factor model with correlated latent factors did not yield an optimal model fit. Parameter estimates obtained from CFA identified items with low factor loadings and R² values, suggesting that content revision is required for those items on the STRS. Deletion of two items from the scale yielded a good model fit, suggesting that the remaining 26 items reliably and validly measure the constructs for the whole sample. Tests for configural invariance, however, revealed that the underlying constructs may differ for AA and EA groups. Subsequent exploratory factor analyses (EFAs) for AA and EA children were carried out to investigate the comparability of the measurement model of the STRS across the groups. The results of EFAs provided evidence suggesting differential factor models of the STRS across AA and EA groups. This study provides implications for construct validity research and substantive research using the STRS given that the STRS is extensively used in intervention and research in early childhood education.     

Think aloud: using cognitive interviewing to validate the PISA assessment of student self-efficacy in mathematics

Cognitive interviewing (CI) provides a method of systematically collecting validity evidence of response processes for questionnaire items. CI involves a range of techniques for prompting individuals to verbalise their responses to items. One such technique is concurrent verbalisation, as developed in Think Aloud Protocol (TAP). This article investigates the value of the technique for validating questionnaire items administered to young people in international surveys. To date, the literature on TAP has focused on allaying concerns about reactivity – whether response processes are affected by thinking aloud. This article investigates another concern, namely the completeness of concurrent verbalisations – the extent to which respondents verbalise their response processes. An independent, exploratory validation of the PISA assessment of student self-efficacy in mathematics by a small international team of researchers using CI with concurrent verbalisation in four education systems (England, Estonia, Hong Kong, and the Netherlands) provided the basis for this investigation. The researchers found that students generally thought aloud in response to each of the items, thereby providing validity evidence of responses processes varying within and between the education systems, but that practical steps could be taken to increase the completeness of concurrent verbalisations in future validations.     

Pathways to reading,mathematics, and science: Examining domain-general correlates in young Chinese children

Early competencies in reading, mathematics, and science are associated with later school achievement and adulthood socioeconomic status. This cross-sectional study examined how fundamental domain-general capacities, including language, spatial, and self-regulatory skills, together relate to competencies in reading, mathematics, and science in young Chinese children. A total of 584 Chinese children aged approximately six years were tested individually on their language (receptive vocabulary), spatial (spatial perception, spatial visualization, and mental rotation), and self-regulation (behavioral regulation and working memory) skills, as well as their academic competencies in reading, mathematics, and science. The results showed that vocabulary, spatial, and self-regulatory skills were all associated with Chinese reading, mathematics, and life sciences, whereas only vocabulary was related to earth and physical sciences. The relation between vocabulary and formal mathematics and that between mental rotation and life sciences were found to be stronger in boys than in girls. The findings suggest that foundational domain-general skills may provide the building blocks for children’s academic competencies.     

Science teachers' conceptions of teaching and learning,ICT efficacy,ICT professional development and ICT practices enacted in their classrooms

This study assessed several precursors that might be connected to Information and Communication Technology (ICT) practices enacted in classrooms, namely, science teachers' conception of traditional versus constructivist teaching and learning, their sense of efficacy, ICT efficacy, and ICT professional development. Data were gathered from 303 science teachers. According to the results, constructivist conceptions exerted moderate effects on the teachers' sense of efficacy constructs, which in turn increased their ICT efficacy. Additional results pointed to the positive mediation role of teachers' ICT professional development in linking their ICT efficacy to ICT use for constructivist activities in their science classroom.     

科学教育与ICT角色文献综述:使命、问题与未来的方向

通过追溯英国科学教育的发展,对英国中学科学教育所发生的变革进行描述和境脉介绍,提出了英国科学教育的目标及目前教学存在的问题,重点讨论了ICT应用对科学教学的价值、影响和目前的局限。     

Reflections on and implications of the Programme for International Student Assessment 2015 (PISA 2015) performance of students in Taiwan: The role of epistemic beliefs about science in scientific literacy

The 2015 Programme for International Student Assessment (PISA) has drawn a substantial amount of attention from science educators and educational policymakers because it marked the first time that PISA assessed students' ability to evaluate and design scientific inquiry using computer-based simulations. We undertook a secondary analysis of the PISA 2015 Taiwan dataset of 7,973 students from 214 schools to identify critical issues of student learning and potentially reshape our educational system and policies. Thus, this study sought to identify potential latent clusters of students' scientific literacy performance according to a set of focus variables selected from the PISA student questionnaires. In addition, significant determinants of students' scientific literacy and resiliency were analyzed. Cluster analysis results demonstrated the presence of four clusters of high, medium, low, and inferior scientific literacy/epistemology/affective dispositions. Specifically, students in cluster 1 compared with other clusters showed that the higher the scientific literacy scores are, the more positive epistemic beliefs about science, achievement motivation, enjoyment of science, interests in broad science, science self-efficacy, information and communications technology (ICT) interest, ICT autonomy, more learning time, more teacher supports and teacher-directed instructions are. Regression results indicated that the most robust predictor of students' scientific literacy performance is epistemic beliefs about science, followed by learning time, interest in broad science topics, achievement motivation, inquiry-based science teaching and learning practice, and science self-efficacy. Decision tree model results showed that the descending order of the variables in terms of their importance in differentiating students as high- versus low-performing were epistemic beliefs about science, learning time, self-efficacy, interest in broad science, and scientific inquiry, respectively. A similar decision tree model to determine students as resilient versus non-resilient also was found. Various interpretations of these results are discussed, as are their implications for science education research, science teaching, and science education policy.     

The negative effect of school-average ability on science self-concept in the UK,the UK countries and the world: the Big-Fish-Little-Pond-Effect for PISA 2006

Research on the relation between students’ achievement (ACH) and their academic self-concept (ASC) has consistently shown a Big-Fish-Little-Pond-Effect (BFLPE); ASC is positively affected by individual ACH, but negatively affected by school-average ACH. Surprisingly, however, there are few good UK studies of the BFLPE and few anywhere in the world based on science self-concept (S-ASC). Addressing this substantive limitation in existing research with data from PISA 2006, we extend new multigroup doubly-latent multilevel structural equation models – a substantive-methodological synergy. BFLPE predictions for S-ASC are supported for: the total international sample; the total UK sample; each of the four UK countries considered separately. The BFLPE was marginally larger in the UK than the international sample. However, consistent with the selective nature of school systems in the UK, the BFLPE was larger in Northern Ireland and, to a lesser extent, England than in Scotland or Wales.     

Integrating social justice with mathematics and science: An analysis of student teacher lessons

Student teachers have difficulty planning lessons that fully integrate social justice with mathematics/science content. This study was a content analysis of 26 poster presentations of mathematics or science lessons incorporating social justice issues made by student teachers (20F, 6M) at a mid-sized college in central New York State. The presented lessons applied four pedagogical approaches to integration (data collection followed by graphing analysis; discussion of text/video; modeling; library/internet investigation) and addressed three major social justice themes (diversity, system disparities in human communities, and in stewardship of earth). Deeper content knowledge, faculty lesson modeling/reflection and practice delivering lessons are recommended.     

Exploring plausible causes of differential item functioning in the PISA science assessment: language,curriculum or culture

In recent years, large-scale international assessments have been increasingly used to evaluate and compare the quality of education across regions and countries. However, measurement variance between different versions of these assessments often posts threats to the validity of such cross-cultural comparisons. In this study, we investigated the cross-language, cross-cultural validity of the Programme for International Student Assessment 2006 Science assessment via three differential item functioning (DIF) analyses between the USA and Canada, Chinese Hong Kong and mainland China, and between the USA and mainland China. Furthermore, we explored three plausible causes of DIF via content analysis, namely language, curriculum and cultural differences. Our results revealed that differential curriculum coverage was the most serious cause of DIF among the three factors we investigated in this study, and differential content familiarity also contributed to DIF here. We discussed the implications of the findings for future international assessment development, and for how to best define ‘scientific literacy’ for students around the world.     

Assessing construct of DIBELS: Measurement invariance of DIBELS subscales,from kindergarten to the first grade

A challenge using the Dynamic Indicators of Basic Early Literacy Skills (DIBELS) in studying reading growth is that reading skills children exhibit change by age. In order to study growth using changing subscales, it is necessary to examine measurement invariance and measurement structure underlying the different subscales. The purpose of this paper is to examine the measurement structure of the DIBELS subscales, particular measurement invariance. The results indicate that the DIBELS subscales do not seem to have metric invariance but they do share a common factor over time, suggesting that the same construct of reading skills were measured but they manifested in the different fashion over time.     

Structural validity and invariance of the Feedback Perceptions Questionnaire

Despite a growing interest in instructional feedback, students’ feedback perceptions received limited attention. We examined the structural validity and measurement invariance of the Feedback Perceptions Questionnaire (FPQ). The FPQ measures feedback perceptions in terms of perceived fairness, usefulness, acceptance, willingness to improve, and affect. Secondary school students (N = 1486) received a fictional scenario containing Concise General Feedback or Elaborated Specific Feedback by a fictional peer. Students rated their perceptions as if they had received the feedback themselves. Confirmatory Factor Analysis (CFA) supports the structural validity of the FPQ and its invariance for the two types of peer feedback, gender, four grade-levels and two tracks. Perceived fairness of peer feedback was a strong positive predictor of willingness to improve and affect, whereas perceived usefulness and acceptance of peer feedback showed a more complex pattern in predicting willingness to improve and affect.     

Testing for factorial invariance of the Modified Leadership Scale for Sports: using a Japanese version

The objective of this study was to provide empirical evidence to support psychometric properties of a modified four-dimensional model of the Leadership Scale for Sports (LSS). The study tested invariance of all parameters (i.e., factor loadings, error variances, and factor variances–covariances) in the four-dimensional measurement model between two groups of student-athletes. For testing multi-group invariance of the proposed scale, 335 middle school and 320 high school student-athletes in Japan participated in this study. The modified version of the LSS consists of 35 items representing training instruction, democratic behaviour, positive feedback, and social support. A chi-square difference test was employed for model comparisons. The results supported configural, metric, scalar and factor variance–covariance invariance in the modified LSS across the two student-athlete groups.     

Maximizing the usage of technology-enhanced teaching and learning of science and mathematics in English program in the Malaysian secondary schools system

This research is attempting to examine the effectiveness in the application of ICT （information and communication technology） and standardize courseware in ETeMS （English for Teaching Mathematics and Science） or PPSMI （Pengajaran and Pembelajaran Sains and Matematic dalam Bahasa Inggeris） in English program in the Malaysian secondary school system. Five dimensions of secondary school （type of school, demographic, leadership quality, teaching and learning culture, and teachers＇ personal and working experience） were examined to determine the maximizing usage of TEC （technology-enhanced classroom） in the learning program. In general, there is a significant difference among factors stated above with maximizing usage of TEC in teaching and learning of science and mathematics in English program. The learning culture is also closely related. In addition, the school and teaching staff are another two essential factors found to be the strong fundamental of maximizing the usage of technology in TEC teaching and learning with towering quality. This study represents an addition to the extant literature on maximizing the usage of technology-enhanced teaching and learning towards the excellence of education in the secondary school system. The maximizing usage of TEC teaching and learning environment within the secondary school system is pivotal towards achieving high quality human capital and improving the efficiency and integrity of technology-enhanced learning of science and mathematics in English program. This study provides further groundwork to assist existing education managers to improve work quality and deliver the maximizing usage of TEC teaching and learning towards the excellence in secondary education.     

Measurement invariance across information and communication technology development index and gender: The case of the Pearson Test of English Academic reading

The computerization of reading assessments has presented a set of new challenges to test designers. From the vantage point of measurement invariance, test designers must investigate whether the traditionally recognized causes for violating invariance are still a concern in computer-mediated assessments. In addition, it is necessary to understand the technology-related causes of measurement invariance among test-taking populations. In this study, we used the available data (n = 800) from the previous administrations of the Pearson Test of English Academic (PTE Academic) reading, an international test of English comprising 10 test items, to investigate measurement invariance across gender and the Information and Communication Technology Development index (IDI). We conducted a multi-group confirmatory factor analysis (CFA) to assess invariance at four levels: configural, metric, scalar, and structural. Overall, we were able to confirm structural invariance for the PTE Academic, which is a necessary condition for conducting fair assessments. Implications for computer-based education and the assessment of reading are discussed.     

Visions for ICT, Ethics and the Practice of Teachers

The failure of educational computing is often attributed to deficiencies of practice: resisting teachers and lack of will, competence and sincerity on behalf of the teaching profession. But the complexity of school life and teachers work has been underestimated by the entrepreneurs of educational computing. Theoretically there is a need to approach this area from a different angle. We will suggest there is a need to reframe the issue of ICT in schools with educational policy and discourse with education, rather than with technology in mind. In order to reach that goal there is a need to inculcate technology as an ordinary part of teachers vocabulary about their own work and as a part of their professional ethos. Teachers need to examine the moral values inherent in their practices and use those values as bases for debate. Not taking up these issues and instead adopting defensive stances is dangerous to the profession. The danger is that defensiveness can be seen by others outside the school as rigidity and play into the hands of those who are impatient with teachers. It would serve teachers well if they had a more robust ability to communicate the nature of the difficulties they face and how they deal with them.