NCME Presidential Address 2020: Valuing Educational Measurement

The community of educational measurement researchers and practitioners has made many positive contributions to education, but has also become complacent and lost the public trust. In this article, reasons for the lack of public trust in educational testing are described, and core values for educational measurement are proposed. Reasons for distrust of educational measurement include hypocritical practices that conflict with our professional standards, a biased and selected presentation of the history of testing, and inattention to social problems associated with educational measurement. The five core values proposed to help educational measurement serve education are: (a) everyone is capable of learning; (b) there are no differences in the capacity to learn across groups defined by race, ethnicity, or sex; (c) all educational tests are fallible to some degree; (d) educational tests can provide valuable information to improve student learning and certify competence; and (e) all uses of educational test scores must be sufficiently justified by validity evidence. The importance of these core values for improving the science and practice of educational measurement to benefit society is discussed.     

Early Education Essentials: Validation of Surveys Measuring Early Education Organizational Conditions

Research Findings: The Early Education Essentials surveys use teacher and parent perceptions to measure 6 organizational conditions of early childhood education programs, extending and complementing existing measures of early childhood care and education (ECE) quality constructs. This study tests the reliability and concurrent validity of the Early Education Essentials in 81 school- and community-based ECE sites in a large Midwestern city selected using a stratified random sampling method. Using a Rasch item response theory model, scales were created; theory and exploratory factor analyses combined scales into higher level constructs called essentials. Multilevel models took into account individual measurement error to create site-level essential scores and assessed relationships between programs’ essential scores and site-level teacher–child interactions scores and student attendance. Findings suggest that the Early Education Essentials is reliable in multiple ECE settings; it is sensitive to site-level differences; and some, but not all, organizational conditions measured are associated in expected directions with site-level metrics indicative of center-based ECE quality. Practice or Policy: The Early Education Essentials has the potential to provide leaders and practitioners with actionable data about organizational supports that enable ECE practitioners to be more effective in their daily work with children and families.     

Use of Adjustment by Minimum Discriminant Information in Linking Constructed‐Response Test Scores in the Absence of Common Items

In many educational tests, both multiple‐choice (MC) and constructed‐response (CR) sections are used to measure different constructs. In many common cases, security concerns lead to the use of form‐specific CR items that cannot be used for equating test scores, along with MC sections that can be linked to previous test forms via common items. In such cases, adjustment by minimum discriminant information may be used to link CR section scores and composite scores based on both MC and CR sections. This approach is an innovative extension that addresses the long‐standing issue of linking CR test scores across test forms in the absence of common items in educational measurement. It is applied to a series of administrations from an international language assessment with MC sections for receptive skills and CR sections for productive skills. To assess the linking results, harmonic regression is applied to examine the effects of the proposed linking method on score stability, among several analyses for evaluation.     

Problem solving in schools and beyond: Transitions from the naive to the neophyte to the master

Techniques emerging from the considerable research on cognitive aspects of survey methodology include various forms of probing and cognitive interviewing. These techniques are used to examine whether respondents' interpretations of self-report items are consistent with researchers' assumptions and intended meanings given the constructs the items are designed to measure. However, although informal procedures are common, such developments have not been systematically applied in educational research. We describe how information derived from the systematic application of cognitive pretesting can contribute to determining the validity—designated cognitive validity—of self-report items. Examples are presented from prominent motivation-related instruments that assess real-world instructional practices, mastery classroom goal structure, and student self-efficacy. The implications and pragmatics of adopting this approach are discussed.     

Identifying national cultures of mathematics education: Analysis of cognitive demands and differential item functioning in TIMSS

Large-scale assessments of student competencies address rather broad constructs and use parsimonious, unidimensional measurement models. Differential item functioning (DIF) in certain subpopulations usually has been interpreted as error or bias. Recent work in educational measurement, however, assumes that DIF reflects the multidimensionality that is inherent in broad competency constructs and leads to differential achievement profiles. Thus, DIF parameters can be used to identify the relative strengths and weaknesses of certain student subpopulations. The present paper explores profiles of mathematical competencies in upper secondary students from six countries (Austria, France, Germany, Sweden, Switzerland, the US). DIF analyses are combined with analyses of the cognitive demands of test items based on psychological conceptualisations of mathematical problem solving. Experts judged the cognitive demands of TIMSS test items, and these demand ratings were correlated with DIF parameters. We expected that cultural framings and instructional traditions would lead to specific aspects of mathematical problem solving being fostered in classroom instruction, which should be reflected in differential item functioning in international comparative assessments. Results for the TIMSS mathematics test were in line with expectations about cultural and instructional traditions in mathematics education of the six countries.     

Two Models of Educational Assessment

Many educational developments in recent decades pose a serious challenge to the traditional scientific measurement model that has dominated assessment practices. The scientific measurement model has led to an over‐emphasis on statistical tests and the reification of single measure test scores. The educational developments that challenge the scientific measurement model include problem‐based learning, newer understandings of cognition, and the rise of performance assessment. These developments reflect widespread attempts by educators to reform assessment practices so as to encourage more effective learning. As a result, a new model of educational assessment, which we call the judgemental model, is emerging. The basic assumptions, features and appropriate uses of these two assessment models are compared and contrasted by referring them to a three‐level conceptual model of education, training and assessment for workplace performance.     

学前教育实习生实习前后心理健康状况调查研究

教育实习是学前教育学生第一次承担幼儿教师职责,角色的转换会造成较大的心理压力。及时了解、掌握实习生的心理健康状况,进行有针对性的培训、辅导是使实习生顺利渡过实习期的关键。本研究采用症状自评量表对学前教育专业实习生的心理健康水平进行测查,发现:(1)与国内大学生常模相比,我院学前教育专业实习生的心理健康水平偏低,其中人际关系敏感、抑郁、焦虑、恐怖等4个因子得分显著高于常模;(2)学前实习生的症状自评量表中各因子得分在实习前与实习后未出现显著差异,这与其他专业实习生有所区别,究其原因主要是实习工作适应难度大及从业前景不乐观所致。     

Decomposing inequalities in performance scores: the role of student background, peer effects and school characteristics

This paper analyses the mechanisms of stratification and inequalities in educational achievements. The main objective is to determine how stratification leads to unequal educational outcomes and how inequalities are channelled through student characteristics, school characteristics and peer effects. This analysis is undertaken in five countries differentiated by their schooling systems. The countries are Japan, Finland, Germany, Italy and the UK, and the dataset used is PISA 2003. The analysis consists of a multilevel econometric model used to explain variations in performance scores. The explanatory variables are student, school and peer characteristics. The institutional context of each education system is used to interpret the results and to describe how inequalities arise. In the last section, policy implications, based on the regression results, are derived.     

关于大学生心理资本现状的调查与思考——以陕西省四所大学为例

采用问卷调查方法,运用大学生心理资本量表对陕西省四所高校740名学生的心理资本进行施测,并使用SPSS17.0软件做数据统计。结果表明,大学生心理资本总体水平较高;大学生心理资本得分在不同性别、年级、专业类别以及是否担任学生干部、有无实习经历等因素上有显著差异。     

Decoding ClassDojo: psycho-policy,social-emotional learning and persuasive educational technologies

ClassDojo is one of the world’s most successful educational technologies, currently used by over 3 million teachers and 35 million children globally. It reinforces and enacts emerging governmental ‘psycho-policies’ around the measurement and modification of children’s social and emotional learning in schools. This article focuses specifically on the ways ClassDojo facilitates psychological surveillance through gamification techniques, its links to new psychological concepts of ‘character development,’ ‘growth mindsets’ and ‘personal qualities,’ and its connections to the psychological techniques of Silicon Valley designers. Methodologically, the research mobilizes network analysis to trace the organizational, technical, governmental and scientific relations that are translated together and encoded in the ClassDojo app. Through its alignment with emerging education psycho-policy agendas around the measurement of non-cognitive learning, ClassDojo is a key technology of ‘fast policy’ that functions as a ‘persuasive technology’ of ‘psycho-compulsion’ to reinforce and reward student behaviours that are aligned with governmental strategies around social-emotional learning.     

大学新生心理健康UPI调查分析与教育措施

使用UPI问卷对保定学院2007级新生进行心理健康普查,了解大学新生心理健康现状,对存在的问题进行分析,提出相应的教育措施。     

Assessing Teacher Appraisals and Stress in the Classroom: Review of the Classroom Appraisal of Resources and Demands

Stress research increasingly emphasizes the role of appraisal in determining which events are perceived as stressful. The Classroom Appraisal of Resources and Demands (CARD) was developed to measure teachers’ appraisals of their classroom demands and resources in order to assess their risk for experiencing occupational stress. The present purposes are to review the literature identifying appraisals as a key determinant of stress, to describe the development of the CARD, and to provide meta-analytic results from 18 studies comparing CARD scores to the following variables: teacher’s job satisfaction and occupational commitment, burnout symptoms, stress prevention resources, and challenging student demands. Results suggest moderate effects for associations between the CARD and these constructs, and implications for educational policy aimed at reducing turnover and increasing teacher and student welfare are discussed.     

Commentary: Student Cognition, the Situated Learning Context, and Test Score Interpretation

Although it is assumed that student cognition contributes to student performance on achievement tests, it may be that current testing models lack the degree of specification necessary to warrant such inferences. With test score interpretations as the referent, the authors in this special issue address the role of student cognition in learning and educational measurement. The practicality of their recommendations is viewed through the filter of No Child Left Behind Act (NCLB) testing requirements. Commentary on the broader learning context is also shared. Although a focus on student cognition is an important one, the situational context within which learning occurs and tests are administered warrants consideration of a variety of social factors in order to fully specify the measurement construct and to interpret test scores meaningfully.     

Comparability of educational achievement and learning attitudes across nations

We estimated the invariance of educational achievement (EA) and learning attitudes (LA) measures across nations. A multi-group confirmatory factor analysis was used to estimate the invariance of educational achievement and learning attitudes across 55 nations (Programme for International Student Assessment [PISA] 2006 data, N?=?354,203). The constructs had the same meaning (factor loadings) but different scales (intercepts). Our conclusion is that comparisons of the relationships between educational achievement and learning attitudes across countries need to take into consideration two sources of variability: individual differences of students and group differences of educational systems. The lack of scalar invariance in EA and LA measures means that the relationships between EA and LA may have a different meaning at the level of nations and at the student level within countries. In other words, as PISA measures are not invariant in scalar sense, the comparisons across countries with nationally aggregated scores are not justified.     

Limitations of using students' self-reports of academic development as proxies for traditional achievement measures

An important issue in national assessment efforts is how best to measure the outcomes of college. While initial discussions about a national collegiate assessment focused on the reliability, validity, and feasibility of using achievement tests to measure student learning, subsequent discussions have raised the possibility of using students' self-reports of academic development as proxies for achievement test scores. The present study examines the stability of the relationships among self-reports and test scores across samples of two- and four-year colleges and universities. Multitrait-multimethod analyses indicated that self-reports and test scores developed from the same set of test specifications do measure the same constructs, although the scores from one type of measurement may not be substitutable for scores from the other type of measurement. In addition, the analyses produced ambiguous results concerning the stability of relationships across different types of institutions.Paper presented at the annual meeting of the Association for Institutional Research, Boston, May 29, 1995.     

Self-beliefs in organic chemistry: Evaluation of a reciprocal causation,cross-lagged model

This study is designed to test a reciprocal causation, cross-lagged model of self-concept, self-efficacy, and achievement in a postsecondary STEM course. Both self-efficacy and self-concept are known to be related to achievement; however, there is a need to untangle the relationship between the two constructs as well as their association to achievement across time to best direct future research efforts. To achieve this research interest, a longitudinal measurement strategy was used to measure chemistry self-concept and self-efficacy for learning and performance before and after achievement measures (i.e., two term examinations) in a postsecondary organic chemistry course context. A reciprocal causation, cross-lagged model best fits the data as a representation of the relationships between these three measures over time as compared to autoregressive, performance effects, and self-belief effects models. Significant paths in the reciprocal causation, cross-lagged model include the first self-concept measure to the first achievement measure as well as from the second self-concept measure to the third self-efficacy measure. Relationships from achievement to each subsequent self-belief measure were also significant. This study demonstrates the ability of longitudinal measurements of multiple constructs in postsecondary STEM educational research to collect nuanced information that is overlooked when pre-measure designs of single constructs are used. In the classroom, an initial measure of self-concept can inform instructors of the likelihood of students to succeed on an initial achievement measure, at which point they may choose to implement some of the targeted intervention strategies from literature.     

The development and validation of the Attitudinal Learning Inventory (ALI): a measure of attitudinal learning and instruction

In this paper, we present the development and validation of a new measure of attitudinal learning—the Attitudinal Learning Inventory (ALI). While specific scales are available for measuring attitudes, they largely focus on established attitudes, not the impact of instruction on those attitudes. We developed the inventory with two explicit objectives: (1) to measure a broad range of attitude constructs representing a holistic view of attitudinal learning and instruction; and (2) to facilitate the measurement of attitudinal learning that can be useful for educational researchers beyond traditional metrics. The ALI was developed and validated across two samples of a total of 1009 participants with diverse demographics. The ALI comprises 15 scale items and exhibited good psychometric properties and conformed to the theoretical four-dimensional structure of attitudinal learning: cognitive, affective, behavioral, and social. The ALI was also shown to correlate with behavioral metrics of class engagement. Future uses of the new measure are discussed. Participants were taken from entirely online populations, and while demographically diverse, implementation of the scale with face-to-face instruction, in varied settings, and across different groups of learners is needed to provide additional evidence of its intended generalizability and consider possible biases.     

A preliminary study of multiple college admission criteria in Taiwan: the relationship among motivation,standardized tests,high school achievements,and college success

ABSTRACT

A new college admission policy will be implemented in Taiwan in 2022. The purpose of this study was to understand the relationship between admission criteria and college success. Data was obtained from the Taiwan Higher Education Database; a sample size of 8443 students from 156 universities was used in this study. By using the structural equation model, this study tested a research model that included factors such as motivation, standardized test scores, high school achievements, and college success. The findings revealed that the General Scholastic Ability Test scores (in Chinese, English, Social Studies) and high school average academic grades are significantly associated with college success. A student’s motivation to complete a certain major can significantly predict the quality of student effort and influence college success. These findings highlight the importance of some admission criteria and provide practical implications for educational policy-makers, school administrators, students, and parents.