When Assessment Validation Neglects Any Strand of Validity Evidence: An Instructive Example from PISA

The Standards for Educational and Psychological Testing identify several strands of validity evidence that may be needed as support for particular interpretations and uses of assessments. Yet assessment validation often does not seem guided by these Standards, with validations lacking a particular strand even when it appears relevant to an assessment. Consequently, the degree to which validity evidence supports the proposed interpretation and use of the assessment may be compromised. Guided by the Standards, this article presents an independent validation of OECD's PISA assessment of mathematical self-efficacy (MSE) as an instructive example of this issue. OECD identifies MSE as one of a number of “factors” explaining student performance in mathematics, thereby serving the “policy orientation” of PISA. However, this independent validation identifies significant shortcomings in the strands of validity evidence available to support this interpretation and use of the assessment. The article therefore demonstrates how the Standards can guide the planning of a validation to ensure it generates the validity evidence relevant to an interpretive argument, particularly for an international large-scale assessment such as PISA. The implication is that assessment validation could yet benefit from the Standards as what Zumbo calls “a global force for testing”.     

PISA与TIMSS中有关数学评价的比较分析

PISA与TIMSS是近年来较为活跃的两个国际评价项目,它们在评价的目的、使用的评价框架以及试题的形式等方面有所不同,但其中又包含一定的相似的成分,对二者异同的分析将有助于我们进一步认识数学课程实施及数学素养评价的要素和关键。     

Student perceptions of assessment and student self-efficacy in competence-based education

The purpose of this study was to provide insight into the interplay between student perceptions of competence-based assessment and student self-efficacy, and how this influences student learning outcomes. Results reveal that student perceptions of the form authenticity aspect and the quality feedback aspect of assessment do predict student self-efficacy, confirming the role of mastery experiences and social persuasions in enhancing student self-efficacy as stated by social cognitive theory. Findings do not confirm mastery experiences as being a stronger source of self-efficacy information than social persuasions. Study results confirm the predictive role of students’ self-efficacy on their competence outcomes. Mediation analysis results indicate that student’s perceptions of assessment have an indirect effect on student’s competence evaluation outcomes through student’s self-efficacy. Study findings highlight which assessment characteristics, positively influencing students’ learning, contribute to the effectiveness of competence-based education. Limitations of the study and directions for future research are indicated.     

The contribution of assessment experiences to student teachers' self-efficacy in competence-based education

Earlier research argues that educational programmes based on social cognitive theory are successful in improving students' self-efficacy. Focussing on some formative assessment characteristics, this qualitative research intends to study in-depth how student teachers' assessment experiences contribute to their self-efficacy. We interviewed 15 s year student teachers enrolled in a competence based teacher educational programme. Thematic content analysis results reveal that the assessment characteristics ‘authenticity’ and ‘feedback’ exert a positive influence on student teachers self-efficacy during all phases of the portfolio competence assessment. The results provide a fine-grained view of several types of self-efficacy information connected with these assessment phases.     

Language effects in international testing: the case of PISA 2006 science items

We investigate the extent to which language versions (English, French and Arabic) of the same science test are comparable in terms of item difficulty and demands. We argue that language is an inextricable part of the scientific literacy construct, be it intended or not by the examiner. This argument has considerable implications on methodologies used to address the equivalence of multiple language versions of the same assessment, including in the context of international assessment where cross-cultural fairness is a concern. We also argue that none of the available statistical or qualitative techniques are capable of teasing out the language variable and neutralising its potential effects on item difficulty and demands. Exploring the use of automated text analysis tools at the quality control stage may be successful in addressing some of these challenges.     

中学生数学学习自我效能感的初步研究

以班杜拉的自我效能感的理论为指导,通过问卷形式,对海口市第一中192名学生的数学学习效能感的现状进行研究,分析影响数学学习效能感的因素。研究发现:中学生的数学学习自我效能感与数学学业成就关系密切,成高度正相关;性别差异对数学学习自我效能感的影响无显著性差异;与父母的文化程度关系密切,家长受过高等教育的孩子的数学学习自我效能感明显高于家长没有受过高等教育的孩子。     

The role of cultural values in teacher and student self-efficacy: Evidence from 16 nations

Data from a large study (PISA, 2015) involving more than 132,000 children and 22,000 of their teachers, in 16 nations, were used to investigate how teachers convey self-efficacy to students when they teach and whether this is culturally grounded. Using a multilevel data analysis framework, we aimed to: (1) test a path linking teacher and student self-efficacy; (2) examine teaching practices as mediators of the links between teachers and student self-efficacy; (3) evaluate the moderating roles of cultural values on those links. Results indicated that teacher and student self-efficacy were linked indirectly through the use of teaching practices, more strongly through inquiry-based practices. We found cross-cultural differences on the associations between student-perceived teaching practices and student self-efficacy that were moderated by two country-level cultural values: individualism and uncertainty avoidance. This study highlights that, although academic self-efficacy is considered universal, we found cultural differences in its sources and manifestations.     

The impact of PISA and the interrelation and development of assessment policy and assessment theory in Japan

PISA presents a new image for academic achievement, which has prompted Japanese education reforms over the past decade to innovate teaching and learning for ‘PISA-style literacy’. Supported by theoretical foundations, particularly with regard to the concept of ‘PISA literacy’ and ‘authentic assessment’, these reforms have accomplished progress in the focus on higher order competencies, such as application and the development of new assessment strategies. However, more recently, various critical discussions of ‘PISA literacy’ are underway in the Japanese academy. They interrogate it more critically and reveal the narrow emphasis on functional application and technical operation. Current assessment practices, which tend to fall into ‘criteria compliance’, are in urgent need of review. There is a need to extend the critical discussions in progress to the new assessment strategies. This paper responds to this, by considering the Japanese acceptance of ‘PISA literacy’ and its assessment, discussing the features and limitations.     

A comparison of US and Korean students' mathematics skills using a cognitive diagnostic testing method: linkage to instruction

The purposes of the present study were (a) to compare US and Korean 8th graders' mastery of knowledge and skills in the mathematics test of the Trends in International Mathematics and Science Study (TIMSS) 2003 using a cognitive diagnostic testing method and (b) to find links between teachers' instruction and students' mastery of mathematics knowledge and skills. The participants included 740 US and 439 Korean 8th graders who took the Booklet 3 mathematics test. The results showed substantial differences between the US and Korean students' performance in problem restructuring and reasoning, measurement, and geometry. The most helpful instructional strategy for both Korean and US students was encouraging students' independent problem solving. Reviewing, re-teaching, and clarifying content were especially effective for the US students. Implications for teaching and learning are discussed.     

Influence of curriculum reform: An analysis of student mathematics achievement in Mainland China

This study investigated curriculum influences on student mathematics achievement by following two groups of students from fifth to sixth grade that were taught either the reformed curriculum or the conventional curriculum. Analyses with three-level modeling were conducted to examine learning outcomes of the students who were assessed three times over a period of 18 months. Achievement was measured with regard to computation, routine problem solving, and complex problem solving. Affective aspects included self-reported interest in learning mathematics, classroom participation, views of the nature of mathematics, and views of learning mathematics. The results showed overall improved performance among all the students over the time on computation, routine problem solving, and complex problem solving but not on the affective measures. There were differentiated patterns of performance between the groups. On the initial assessment, the reform group performed better than the non-reform group on calculation, complex problem solving, and indicated higher interest in learning mathematics. The two groups did not differ on the other achievement and affective measures at the first time of assessment. There was no significant difference in growth rate between the groups on the cognitive and affective measures except that the non-reform group progressed at a faster pace on calculation. Therefore, the non-reform group outperformed the reform group on computation at the third (last) assessment. These results are discussed with respect to the possible influence of the curriculum on student learning.     

Competitive comparison and PISA bragging rights: sub-national uses of the OECD's PISA in Canada and the USA

Participation in the Organization for Economic Co-operation and Development's (OECD) Program for International Student Assessment (PISA) has continuously expanded: from 43 systems in 2000 to 65 systems in the 2012 cycle, with 71 signed up for PISA 2015. There also has been a growth in sub-national participation, expanding PISA's reach beyond the nation-state. This paper explores sub-national PISA participation in Canada and the USA, asking how PISA is being used within sub-national education policy spaces. We draw on analysis of documents and data from interviews with officials at sub-national, national, and international levels. Findings illustrate some of the diverse motivations and uses of PISA, providing insights into the effects of PISA at the sub-national scale. As such, we argue that competitive comparison in education has deepened through the enhanced granularity of international large-scale assessment data to new scales beyond the nation-state.     

指向创造性思维:PISA 2021的启示与借鉴

创造性思维是人类发展所需的必要能力,可以帮助人们适应不断变化的世界和应对充满挑战的未来。经济合作与发展组织确定在PISA 2021中增加对创造性思维能力的评估,其发布的《PISA 2021创造性思维评估框架草案(第三版)》明确阐述了创造性思维的内涵、表现形式和促成因素,以系统的通用框架、科学简易的"三维度四领域"能力模型向公众提供了一个操作性强的评估系统。通过此次评估,各参与国家和地区可获得学生创造性思维能力的可比数据,为未来教育政策的制定和教育实践的改进提供支持。基于PISA的经验,为了更好地评估和培养学生的创造性思维,我国可借鉴创造性思维能力模型,细化学科核心素养的考查;构建创造性课堂,加强学校创新氛围的建设;在课堂教学中以真实情境和实际问题为载体,培育和评价学生的创造性思维。     

Early mathematics assessment: validation of the short form of a prekindergarten and kindergarten mathematics measure

In recent years, there has been increased interest in improving early mathematics curricula and instruction. Subsequently, there has also been a rise in demand for better early mathematics assessments, as most current measures are limited in their content and/or their sensitivity to detect differences in early mathematics development among young children. In this article, using data from two large samples of diverse populations of prekindergarten and kindergarten children, we provide evidence regarding the psychometric validity of a new theory-based early mathematics assessment. The new measure is the short form of a longer, validated measure. Our results suggest the short form assessment is valid for assessing prekindergarten and kindergarten children’s numeracy and geometry skills and is sensitive to differences in early mathematics development among young children.     

评估关键能力 培育生活智慧——OECD／PISA问题解决能力之构想、设计与评估

培养学生的问题解决能力是学校教育的重点内容。“国际学生评价项目”在2003年增加了对学生问题解决能力的测评。该评估项目旨在考察学生综合运用学科领域的知识,识别问题关键特征及其内在关系,能够明确界定问题,合理表征问题和有效解决问题,并能够对问题解决方案进行真实性评估、判断与交流。     

Exploring plausible causes of differential item functioning in the PISA science assessment: language,curriculum or culture

In recent years, large-scale international assessments have been increasingly used to evaluate and compare the quality of education across regions and countries. However, measurement variance between different versions of these assessments often posts threats to the validity of such cross-cultural comparisons. In this study, we investigated the cross-language, cross-cultural validity of the Programme for International Student Assessment 2006 Science assessment via three differential item functioning (DIF) analyses between the USA and Canada, Chinese Hong Kong and mainland China, and between the USA and mainland China. Furthermore, we explored three plausible causes of DIF via content analysis, namely language, curriculum and cultural differences. Our results revealed that differential curriculum coverage was the most serious cause of DIF among the three factors we investigated in this study, and differential content familiarity also contributed to DIF here. We discussed the implications of the findings for future international assessment development, and for how to best define ‘scientific literacy’ for students around the world.     

Testing the effectiveness of classroom formative assessment in Dutch primary mathematics education

Classroom formative assessment (CFA) is considered to be a fundamental part of effective teaching, as it is presumed to enhance student performance. However, there is only limited empirical evidence to support this notion. In this effect study, a quasi-experiment was conducted to compare 2 conditions. In the treatment condition, 17 teachers implemented a CFA model containing both daily and weekly goal-directed instruction, assessment, and immediate instructional feedback for students who needed additional support. In the control condition, 17 teachers implemented a modification to their usual practice. They assessed their students’ mastery of learning goals on the basis of half-yearly mathematics tests, and prepared weekly pre-teaching sessions for groups of low-achieving students. The posttests showed no significant differences in student performance between the 2 conditions after controlling for student and teacher characteristics. The degree of implementation of the CFA model, however, appeared to be positively related to the 5th-grade students’ performance.     

将PISA应用于我国义务教育质量监测不可无视的两个问题

PISA2009上海项目,开启了用PISA监测我国义务教育质量的先河。继而有关学者主张,借鉴PISA建构我国的义务教育质量监测体系,以提供中国同世界上其他国家或地区间可比较的教育质量指标,从而为国家教育政策的制定和调整提供依据。然而,当我们将PISA移植于中国义务教育质量监测,并将其结果在国际上进行比较的时候,不可无视两个问题:一是学生生存质量问题;二是教育效率问题。前者关注:学生在教育过程中幸福感如何、承受的压力有多强、失去了多少原本不该失去的东西;后者关注:某一PISA成绩的取得,消耗了多少劳动,花费了多长时间。     

Quantifying participation in,and the effectiveness of,remediating assessment in a university mathematics module

In Maths for Business, a large first-year mathematics module, the continuous assessment component comprises 10 weekly quizzes which combine to contribute 40% of the final module mark. If students did not receive the full five marks on their weekly quiz, they were provided with the opportunity to resubmit their corrected weekly quiz with an explanation of their error(s) for one additional mark. We refer to this process as ‘remediation’. Of the students who had the opportunity to remediate, ～70% did. Through examining learning management system data, we show that the remediation process encouraged students to access module resources. Furthermore, by using a Bayesian hierarchical model to account for students’ level of participation, achievement and prior knowledge, we show that participation in the remediation process positively impacted the final examination marks of moderate to high-achieving students (based on initial continuous assessment marks). However, participation in the remediation process provided limited benefit to low-achieving students. We conjecture this is because these students had not achieved a level of understanding whereby participation in the remediation process could progress their knowledge.     

Example-based learning: Effects of different types of examples on student performance,cognitive load and self-efficacy in a statistical learning task

Previous research has indicated the disconnect between example-based research focusing on worked examples (WEs) and that focusing on modeling examples. The purpose of this study was to examine and compare the effect of four different types of examples from the two separate lines of research, including standard WEs, erroneous WEs, expert (masterly) modeling examples, and peer (coping) modeling examples, on student performance (knowledge retention, near transfer, and far transfer), cognitive load, and self-efficacy. One hundred and sixteen students participated in the study by undergoing computer-based instruction in one of the four versions differing in how examples were provided. The results showed that, overall, expert modeling examples were most effective in promoting knowledge retention, near transfer, and far transfer, while peer modeling examples were shown to be superior in fostering self-efficacy among the four different types of examples.