Explaining calibration accuracy in classroom contexts: the effects of incentives, reflection, and explanatory style

A 2 × 2 quasi-experimental design was used to investigate the impact of extrinsic incentives and reflection on students’ calibration of exam performance. We further examined the relationships among attributional style, performance, and calibration judgments. Participants were 137 college students enrolled in an educational psychology course. Results differed as a function of exam performance. Higher-performing students were very accurate in their calibration and did not show significant improvements across a semester-length course. Attributional style did not significantly contribute to their calibration judgments. Lower-performing students, however, were less accurate in their calibration, and students in the incentives condition showed significant increases in calibration. Beyond exam scores, attributional style constructs were significant predictors of calibration judgments for these students. The constructs targeting study and social variables accounted for most of the additional explained variance. The qualitative data also revealed differences by performance level in open-ended explanations for calibration judgments.     

Identification of educationally at-risk children during the kindergarten year: A four-year follow-up study of group test performance

Group test performance of children identified during the kindergarten year as educationally high, moderate, and low risk was investigated by following a group of 472 children from grades one through four. End of kindergarten predictive measures were the Kindergarten Evaluation of Learning Potential, the Bender-Gestalt Test, and the Slosson Intelligence Test; follow-up measures were group achievement tests administered in April of each school year. Significant differences in achievement performance were found between the high and low risk groups. Significant correlations were found between risk group designation and achievement performance in the first four grades. (No significant differences in group test performance were found for risk groups or individuals between grade levels.) Findings support the predictive validity of the present screening procedures for group test performance through grade four. Further, the findings show that students appear to perform consistently at the same level year to year in a regular class instructional program.     

Knowing what you know: improving metacomprehension and calibration accuracy in digital text

This paper presents results from an experimental study that examined embedded strategy prompts in digital text and their effects on calibration and metacomprehension accuracies. A sample population of 80 college undergraduates read a digital expository text on the basics of photography. The most robust treatment (mixed) read the text, generated a summary for each page of text, and then was prompted with a metacognitive strategy. The metacognitive treatment received metacognitive strategy prompts only, and the cognitive group implemented the cognitive strategy (summarization) only. A control group read the text with no embedded support. Groups were compared on measures of achievement, attitudes, cognitive load, and metacomprehension and calibration accuracy. Results indicated that a combination of embedded cognitive and metacognitive strategies in digital text improved learner achievement on application-level questions, yielded more accurate predictive calibration, and strengthened the relationship between metacomprehension and performance, all of which are common attributes of an academically successful learner.     

Psychiatric,business, and psychological applications of fundamental measurement models

This chapter is a collection of four studies applying probabilistic models to problems outside of education. Denmark's Allerup, Bech, and Loldrup present the calibration of the Visual Analogue Scale, used for measuring pain, and the use of this scale for measuring the reductions in pain brought about via the administration of three different treatments. Alvarez and Bañegil's study is a calibration of the criteria by which implementation of Just In Time and Total Quality Management techniques are recognized in Spain. Styles tests the assertion of sociologist O.D. Duncan that the reason for low correlations between measures of attitude and behaviour may be because these variables may not involve separate constructs but simply differing amounts of the same construct. Her conjoint measurement study of data from an Australian survey shows just how much harder it is to do than to say. Finally, from Israel, Tenenbaum presents the results of two studies involving the relationship of anxiety to performance among athletes, showing that commonly used instruments do not produce data of the consistency needed for fit to probabilistic conjoint measurement models in these applications.     

Conceptual and empirical differences among various value-added models for accountability

Accountability systems in education generally include indicators of student performance. However, these indicators often differ considerably among the various systems. More and more countries try to include value-added measures, mainly because they do not want to hold schools accountable for differences in their initial intake of students. This study presents a conceptual framework of these value-added measures, resulting in an overview of 5 different types. Using data from Dutch secondary schools, we empirically provide estimates of these different measures. Our analyses show that the correlation between the different types of school effects estimated is rather high, but that the different models implicate different results for individual schools. Based on theoretical considerations, arguments are given to use the following indicators in the value-added accountability models: prior achievement, student-level background characteristics, and compositional characteristics of the student population.     

Multivariate prediction of academic performance by Hong Kong University students

Verbal intelligence, English-language skills, personality, and attitude scales were used as predictors of academic performance in 230 male and female arts students at the University of Hong Kong. A series of bivariate, multiple, and canonical correlation analyses were performed. The results showed that verbal intelligence and attitudes, excepting study orientation, were not predictive of performance. English-language skills had the most predictive value, accounting for about 10% of the variance of performance measures. Personality variables failed to predict performance when composite criterion measures (GPAs) were used; however, they proved to be of predictive value when results of individual academic subjects were used as criterion measures. The study points to the importance of using noncomposite criterion measures in prediction and of considering the cultural context of achievement.     

Peer Reputation in Middle Childhood as a Predictor of Adaptation in Adolescence: A Seven-Year Follow-up

This investigation examines the predictive significance of peer reputation in elementary school for the quality of adaptation in adolescence. A normative sample (N = 207) of third to sixth graders was administered the Revised Class Play (RCP). Each received 3 scores (Sociability-Leadership, Aggression-Disruption, and Sensitivity-Isolation). 7 years later, 88% of these children and their parents participated in a questionnaire follow-up study utilizing a broad range of adolescent outcome measures (e.g., social and athletic competence, academic performance, behavioral symptoms, well-being). The 3 RCP scores were significantly related to both adolescent competence and psychopathology, supporting the predictive validity of the RCP as well as the continuity of general adaptation. Each dimension of peer reputation had a different pattern of prediction depending on the outcome criteria under consideration, suggesting the importance of a multidimensional approach to peer reputation. Positive peer reputation proved to be an important predictor of later adjustment. Sex differences were examined; results suggested somewhat different patterns of prediction for boys and girls, especially for the sensitive-isolated dimension.     

Uncontrolled Destinies: Improving Opportunity for Low-Income Students in American Higher Education

Recognizing the current administration’s focus on improving postsecondary access, this study examines resource allocation patterns and the predictive power of these resources in increasing the likelihood of 2- and 4-year college enrollment among low-income students. Using data from the Educational Longitudinal Study, college choice decision-making is conceptualized in relation to an individual’s habitus and capital deficiency theory. The results reveal significant differences in the availability and predictive power of various forms of capital across enrollment classifications. The study concludes with implications for administrators and policymakers interested in improving access for low-income students.     

Administrative Provisions for Guidance in Negro Colleges and Universities

Abstract

This study investigated the empirical validity of the Whitman School In-basket Test for elementary principals, Correlations were computed among scores from a predictive test, a concurrent test, and corresponding measures of observed on-the-job behavior. The results were: (1) there was evidence of stability of in-basket performance; (2) stylistic measures of principals’ in-basket performance were not substantially related to their on-the-job behavior; and (3) there was evidence that impressionistic measures of in-basket performance bore some relationship to impressions of on-the-job behavior.     

The utility of kindergarten teacher ratings for predicting low academic achievement in first grade

The purpose of this study was to assess the predictive value of kindergarten teachers' ratings of pupils for later first-grade academic achievement. Kindergarten students were rated by their teachers on a variety of variables, including math and reading performance, teacher concerns, and amount of learning relative to peers. These variables were then analyzed with respect to outcome measures for math and reading ability administered in the first grade. The teachers' ratings of academic performance were significantly correlated with scores on the outcome measures. Analyses were also carried out to determine sensitivity, specificity, and predictive values of the different teacher ratings. The results indicated high overall accuracy, sensitivity, specificity, and negative predictive value for the ratings. Positive predictive value tended to be lower. A recommendation to follow from these results is that teacher ratings of this sort be used to determine which children should receive cognitive screening measures to further enhance identification of children at risk for learning disability. However, this recommendation is limited by the lack of empirically supported screening measures for math disability versus well-supported screening tools for reading disability.     

Within and between person associations of calibration and achievement

Self-regulated learning (SRL), the ability to set goals and monitor and control progress toward these goals, is an important part of a positive mathematical disposition. Within SRL, accurate metacognitive monitoring is necessary to drive control processes. Students who display this accuracy are said to be calibrated, and although calibration is a growing area of research within Educational Psychology, unanswered questions remain about calibration's role as an aspect of metacognition, including the unique association between calibration and academic performance. In this study, calibration is characterized as part of a dynamic system that varies across tasks within the same person; variance in calibration is associated with variance in performance gain for the same student across tasks (quizzes within a year-long mathematics curriculum, ST Math). Both accurate determinations of certainty (Sensitivity) and uncertainty (Specificity) have unique small, yet statistically significant, associations with performance gains from pre to posttest in ST Math. For Specificity, there also remains a contextual association with performance at the Person level. Results are discussed in light of prior research on calibration and of theories of SRL; the data and analyses present a novel approach to studying calibration within a dynamic system and offer insights for future work.     

Psychological capital and performance among undergraduate students: the role of meaning-focused coping and satisfaction

This study explores the predictive relationships between psychological capital (PsyCap), meaning-focused coping, satisfaction and performance among undergraduate students. Six hundred and eighty two (n?=?682) college students from 29 different academic programmes completed an academic well-being survey, which included measures of PsyCap, coping strategies, and academic satisfaction (time 1). Performance data was collected five months later (time 2), at the end of the year. Path analysis results provided support for a sequential mediation model where PsyCap was directly related to performance, and indirectly related to performance through meaning-focus coping and satisfaction. In addition, PsyCap was directly associated with satisfaction, highlighting the importance of this psychological construct in academic settings. Understanding the role that meaning-focused coping and satisfaction play in the relationship between psychological capital and performance may be useful for scholars and lecturers to design optimal evidenced-based interventions to increase both well-being and academic achievement.     

Achievements and attitudinal patterns of boys and girls in science

Data from the 1983–84 Israeli IEA (International Association for the Evaluation of Education Achievement) science study were used to explore gender-related differences (and their determinants) in the learning of science. The sample was composed of 1,934 ninth-grade students. The study involved several measures of science learning, ten attitudinal measures, and items and errors classification. Differences between boys and girls were observed in some measures of science performance–-particularly in the physical sciences, in items with lower estimates of “opportunity to learn,” and in specific kinds of errors. Gender-related differences were also observed in the predictive model of achievement, using science-specific affective measures. The discussion raises the cognitive and affective readiness of boys and girls for learning science.     

Comparing students’ self-discipline and self-regulation measures and their prediction of academic achievement

Using a multi-source, multi-measure research design involving 507 high school students and their teachers, we compared prediction of these students’ academic achievement by a composite of students’ and teachers’ measures of students’ self-regulation (SR) with a composite of students’ and teachers’ measures of students’ self-discipline (SD). Hierarchical regression analyses revealed that the SR composite was more predictive of students’ grade point average and performance on a state-wide achievement test than the SD composite. Confirmatory factor analyses showed that, although SD and SR latent factors correlated significantly, a two-factor solution provided an acceptable fit for the results. Structural Equation Modeling analyses indicated that the SR latent factor predicted both measures of students’ achievement significantly, but the SD factor did not predict either achievement measure significantly. No significant gender differences were found with students’ SD, SR, or achievement measures. These results suggest a path for integrating two relatively separate streams in self-regulation research on the basis of a well-established distinction between learning and performance processes.     

Elementary students’ metacognitive processes and post-performance calibration on mathematical problem-solving tasks

Calibration, or the correspondence between perceived performance and actual performance, is linked to students’ metacognitive and self-regulatory skills. Making students more aware of the quality of their performance is important in elementary school settings, and more so when math problems are involved. However, many students seem to be poorly calibrated, with a tendency towards over-confidence. The present study analyzes the relationship between post-performance calibration accuracy and the metacognitive process shown by 524 fifth- and sixth-grade students while solving two math problems. After calculating a calibration index and establishing the stability of students’ judgments and actual performance, differences in the metacognitive process exhibited by students with different calibration accuracy (Accurate vs. Inaccurate groups) were analyzed. The emergence of different calibration patterns and differences in the metacognitive process as a function of mathematics achievement and grade level were also examined. Results indicated that: (a) students in the overall sample were little calibrated and over-confident, showing high stability in their judgments and actual performance across problems; (b) inaccurate students reported using information representation sub-processes (drawing/summarizing) less frequently, but writing and reviewing (and also correcting mistakes) more frequently than their accurate peers; and (c) differences in calibration patterns and the metacognitive process were found when achievement level was considered, whereas grade level did not generate any important effect. These findings suggest the usefulness of process-based measures to examine the metacognitive processes involved in making post-performance judgments, considering achievement and its possible mediating role in this relationship.     

Adjusting the Cumulative GPA Using Item Response Theory

In college admissions, the predictive validity of preadmissions measures such as standardized test scores and high school grades is of wide interest. These measures are most often validated against the criterion of the first-year grade point average (GPA). However, neither the first-year GPA nor the four-year cumulative GPA is an adequate indicator of academic performance through four years of college. In this study, Item Response Theory (IRT) is used to develop a more reliable measure of performance, called an IRT-based GPA, which is used to estimate the validity of traditional preadmissions information. The data are preadmissions information and course grades for the Class of 1986 at Stanford University (N = 1564). Principal factor analysis is used as a precursor to determine the dimensionality of the course data and to partition courses into approximately unidimensional subsets, each of which is scaled independently. Results show a substantial increase in predictability when the IRT-based GPA is used instead of the usual GPA.     

The Role of Non-cognitive Variables in Identifying Community College Students in Need of Targeted Supports

Non-academic characteristics and traits, such as academic self-efficacy and conscientiousness, have maintained the interest of higher education researchers for decades. A considerable amount of research has found that these non-cognitive variables (NCVs) are generally predictive of undergraduate success. However, most prior studies have focused on the use of NCVs in 4-year colleges, and understanding if and how these measures predict the academic trajectories of community college students has received less attention. As past work has indicated that NCVs are differentially predictive for students from different backgrounds, such an examination is needed. Drawing on data from two diverse community colleges, our study addresses this need by exploring if seven popular NCVs predict both short and longer-term student outcomes, how these measures are related to help-seeking behaviors, and their utility in identifying students at risk of serious academic failures. We find statistically significant relationships between NCVs and GPA outcomes but conclude that their practical utility appears trivial.

     

Executive functions as predictors of math learning disabilities

In the past years, an increasing number of studies have investigated executive functions as predictors of individual differences in mathematical abilities. The present longitudinal study was designed to investigate whether the executive functions shifting, inhibition, and working memory differ between low achieving and typically achieving children and whether these executive functions can be seen as precursors to math learning disabilities in children. Furthermore, the predictive value of working memory ability compared to preparatory mathematical abilities was examined. Two classifications were made based on (persistent) mathematical ability in first and second grade. Repeated measures analyses and discriminant analyses were used to investigate which functions predicted group membership best. Group differences in performance were found on one inhibition and three working memory tasks. The working memory tasks predicted math learning disabilities, even over and above the predictive value of preparatory mathematical abilities.     

How spatial abilities enhance,and are enhanced by,dental education

In two studies with a total of 324 participants, dentistry students were assessed on psychometric measures of spatial ability, reasoning ability, and on new measures of the ability to infer the appearance of a cross-section of a three-dimensional (3-D) object. We examined how these abilities and skills predict success in dental education programs, and whether dental education enhances an individual's spatial competence. The cross-section tests were correlated with spatial ability measures, even after controlling for reasoning ability, suggesting that they rely specifically on the ability to store and transform spatial representations. Sex differences in these measures indicated a male advantage, as is often found on measures of spatial ability. Spatial ability was somewhat predictive of performance in restorative dentistry practical laboratory classes, but not of learning anatomy in general. Comparisons of the performance of students early and late in their dental education indicated that dentistry students develop spatial mental models of the 3-D structure of teeth, which improves their ability to mentally maintain and manipulate representations of these specific structures, but there is no evidence that dental education improves spatial transformation abilities more generally.     

Individual differences in decision-making and confidence: capturing decision tendencies in a fictitious medical test

Decision-making is a complex process that is largely studied from an experimental perspective or in specific organizational contexts. As such, no generalizable framework exists with which to study decision-making from an individual differences perspective for predictive/selection purposes. By generalising a context-specific decision model proposed by Koriat and Goldsmith (1996), the focus of this research was to therefore test a novel framework for studying individual differences in decision-making tendencies. Utilising this framework within a fictitious Medical Decision-Making Test (MDMT) yielded five novel variables that provided unique insight into individuals’ decision tendencies: Optimal, Realistic, Incompetent, Hesitant and Congruent. Metacognitive confidence and its calibration (bias and CAQ) were used as predictor variables to validate this framework. One hundred ninety-three undergraduate students completed the MDMT and three cognitive ability tests with confidence ratings, a personality questionnaire, and the Need for Closure questionnaire. All decision tendency variables demonstrated excellent internal consistency and were predicted by the metacognitive variables incrementally to the remaining variables as hypothesized. Additionally, the metacognitive indices were found to generalize across the decision-making and cognitive tests. The results imply that this novel framework and MDMT reliably capture individuals’ decision behaviour that shares a meaningful relationship with their general confidence and calibration.