Predicting children’s academic achievement from early assessment scores: a validity generalization study

Although there have been numerous studies investigating the predictive validity of early assessment, observed predictive validity coefficients across studies are not stable. A validity generalization study was conducted in order to answer the question of whether the relationship between early assessment of children and later achievement is generalizable or situation-specific. This study examined 716 predictive correlation coefficients from 44 studies using Hierarchical Linear Modeling (HLM). The findings of this study revealed that predictive validity of early assessment is not generalizable. Additional analyses indicated that predictive validity differ across assessments as a function of test type, specific construct being assessed, length of prediction, and administration procedures. The most impressive finding in this study was the variability of effect sizes across different test administration types. In particular, tests that were scored through ratings were found to be most effective. These findings suggest that instead of addressing a broad predictive validity between a test and a criterion measure, it is necessary to understand early assessment procedures as a whole system by including considerations of various variables related to testing conditions.     

Behavioral self-regulation and relations to emergent academic skills among children in Germany and Iceland

The present study investigated a direct assessment of behavioral self-regulation (the Head-Toes-Knees-Shoulders; HTKS) and its contribution to early academic achievement among young children in Germany and Iceland. The authors examined the psychometric properties and construct validity of the HTKS, investigated gender differences in young children's behavioral self-regulation, and explored relations between the HTKS and a teacher report of behavioral self-regulation (the Child Behavior Rating Scale; CBRS) and emerging academic skills. Findings supported the construct validity of the HTKS when used with young German and Icelandic children. Multilevel analyses revealed gender differences, particularly on the CBRS teacher-rated measure. Finally, higher levels of behavioral self-regulation were related to higher academic skills after important background variables were controlled, although some cross-cultural differences in the predictive utility of the HTKS and CBRS were observed. Overall, these results extend prior psychometric work on the HTKS to samples of young European children and support the importance of understanding of the role behavioral self-regulation in young children's development.     

Investigating the Validity of the Australian Early Development Index

This article aims to contribute to the ongoing evaluation of the Australian Early Development Index (AEDI) by investigating its construct and concurrent validity with a subsample of 642 children aged 4 to 5 years drawn from the Longitudinal Study of Australian Children (LSAC). Construct validity was examined by considering the theoretical consistency of the network of correlations between the AEDI subconstructs and the independently reported multimethod measures of early learning skills and development collected contemporaneously by the LSAC. Concurrent validity was examined by assessing the extent to which children who were “developmentally vulnerable” on the AEDI domains corresponded with the LSAC outcome indices classification of children as “developmentally at risk.” Moderate to large correlations were observed between each of the AEDI domains and subconstructs when compared to analogous teacher-rated LSAC measures, with lower levels of association observed for parent-rated LSAC measures. Concurrent validity was explored; however, with no criterion measure with which to assess the AEDI, findings are inconclusive prior to predictive validity assessment. Future waves of the LSAC will collect information on the children's abilities at school and developmental outcomes, enabling further interpretation of these concurrent and construct validity findings by triangulation and predictive validity analyses.     

Evidence of bias in predictive validity on the Kaufman assessment battery for children in samples of Anglo and Mexican American children

Possible bias in the differential predictive validity of the Kaufman Assessment Battery (K-ABC) was investigated with 76 Anglo and 90 Mexican American fifth- and sixth-grade boys and girls. All children were English-speaking and from similar socioeconomic backgrounds. The criterion variable was the Comprehensive Tests of Basic Skills (CTBS; Language, Reading, Mathematics, and Total Scores). Several statistical techniques were used to investigate test bias (examination of predictive validity coefficients; two methods of examining homogeneity of slopes of the regression lines). The results showed considerable evidence of bias in differential predictive validity, indicating that the global cognitive score of the K-ABC (the Mental Processing Composite) was less effective in predicting CTBS scores for the Mexican American group than for the Anglo group.     

The use of a dynamic screening of phonological awareness to predict risk for reading disabilities in kindergarten children

This study examined the usefulness and predictive validity of a dynamic screening of phonological awareness in two samples of kindergarten children. In one sample (n = 90), the predictive validity of the dynamic assessment was compared to a static version of the same screening measure. In the second sample (n = 96), the dynamic screening measure was compared to a commonly used screening tool, Dynamic Indicators of Basic Early Literacy Skills Initial Sound Fluency. Results showed that the dynamic screening measure uniquely predicted end-of-year reading achievement and outcomes in both samples. These results provide preliminary support for the usefulness of a dynamic screening measure of phonological awareness for kindergarten students.     

Risk assessment of head start children with the Brigance K&1 Screen: differential performance by sex,age, and predictive accuracy for early school achievement and special education placement

Early childhood screening has been a widespread yet controversial practice. Serious concerns have been voiced in the literature about the technical limitations and the inappropriate uses of frequently used screens. Because developmental screening is a requirement set forth by Head Start’s performance standards, there is a need for studies to provide accuracy estimates for the Head Start population on commonly used screens. In response, this study examined sex and age differences in performance as well as reliability and validity indices for a sample of 256 Head Start children who were screened with the Brigance K&1 Screen. Children’s performance on the screen varied by age and sex. While the overall consistency of the test was high, there was considerable variability across subscales. Construct validation of the screen, based on correlations with the K-ABC cognitive battery, yielded moderate coefficients. The screen’s predictive validity was established using correlational and classification analyses. At the end of Head Start, moderate to moderately high validity coefficients were obtained when the Brigance was correlated with teachers’ ratings and with subtests of the K-ABC achievement battery. In addition, the Brigance correlated moderately with the PPVT-R and with several Woodcock-Johnson subtests at the beginning of kindergarten. Classification analyses established that the Brigance had less than optimal accuracy in predicting early school achievement and poor success in predicting assignment to special education at the end of kindergarten.     

Exploring dynamic assessment as a means of identifying children at risk of developing comprehension difficulties

In this study, the authors explore a newly constructed dynamic assessment (DA) intended to tap inference-making skills that they hypothesize will be predictive of future comprehension performance. The authors administered the test to 100 second-grade children using a dynamic format to consider the concurrent validity of the measure. The dynamic portion of the assessment comprised teaching children to be "reading detectives" by using textual clues to solve what was happening in the story. During the DA children listened to short passages and answered three inferential questions (i.e., one setting, two causal). If children were unable to answer a question, they were reminded what a reading detective would do and given a set of increasingly concrete prompts and clues to orient them to the relevant portion of text until they could answer the question correctly. Results showed that the DA correlated significantly with a standardized measure of reading comprehension and explained a small but significant amount of unique variance in reading comprehension above and beyond vocabulary and word identification skills. In addition, results suggest that DA may be better than the standardized measure of reading comprehension at identifying intraindividual differences in young children's reading abilities.     

Are validity coefficients understated due to correctable defects in the GPA?

The predictive validity of preadmissions measures such as standardized test scores and high school grades may be understated because of correctable defects in both the freshman year and cumulative grade point average (GPA). Measurement error in the criterion artificially depresses the size of observed validity coefficients. A study was conducted using item response theory (IRT) to develop a more reliable measure of performance, called an IRT-based GPA, and tested in a predictive validity study using data from Stanford University. Results indicate increased predictability when the IRT-based GPA is compared with the usual GPA.This article is based, in part, on the doctoral dissertation of the author, which was completed at the School of Education at Stanford University.     

Comparing multimethod assessment of approaches to learning among preschool children: Direct measure,teacher report,and parent report

A multi‐informant or multimethod approach has been suggested for use in educational evaluation and children's development assessment. However, in the study field of approaches to learning, most previous studies used one method to measure approaches to learning. In addition, compared with kindergarten and elementary children, younger children have received little attention. This study was dedicated to determining whether a multimethod approach (direct measure, teacher report, and parent report) was needed to assess preschool children's approaches to learning. A total of 713 preschool children were enrolled in this study. Correlations and multiple regressions were conducted to analyze the correlation among the three methods as well as their criterion validity based upon comparisons with an assessment of children's early childhood development. The results revealed significant but weak correlations among the three assessment methods. The direct measure of approaches to learning was more relevant to children's early childhood development than the teacher report and the parent report. The criterion validity of using the direct measure to assess preschool children's approaches to learning was also better than that of the teacher report and the parent report. Therefore, the direct measure was recommended for use in assessing preschool children's approaches to learning, and teacher report can be used as a supplement.     

The Role of Interaction Frequency in Identifying Socially Withdrawn and Socially Isolated Children: a re‐examination of data and concepts

Frequency of child social interaction with peers has been criticised as a measure lacking in concurrent and predictive validity. In the present paper it is suggested that this criticism arises in part from a failure to differentiate the behavioural and conceptual features of social isolation and social withdrawal. Definitions of child social isolation and social withdrawal are here derived from a reinterpretation of published data and it is shown that low rate of interaction with peers is empirically and logically related to the concept of social withdrawal. Suggestions for establishing the concurrent and predictive validity of frequency of peer interaction measures are given and a case made for further research into interventions for children who show low levels of involvement with peers.     

Assessment of attention in school children: teachers' ratings related to tests of attention

The paper provides (1) a teacher-administered rating instrument for inattention without confounding the rating with hyperactivity and conduct disorder, and (2) evidence that the ratings correlate with the scores obtained from cognitive tests of attention. In Study I, the first objective was to investigate the construct validity and the inter-rater reliability of the Attention Checklist (ACL) by factor analysing the teacher ratings of 110 Grade 4 children, obtained by using the ACL. The second objective was to investigate the predictive validity of the ACL by examining the relationship between the scores obtained for the participants from teachers' ratings using the ACL and the scores obtained by participants in the lab-type attention tests. The results of factor analysis showed that a single factor labelled ‘inattention’ underlies the 12 items in the ACL. Examining the differences in performance on attention tests, the ‘low attention’ children as rated by the teachers on the ACL scored lower than the ‘high attention’ children on the objective tests of attention. These findings were replicated in Study II, which was conducted to test further the construct validity and predictive validity of the ACL. This time, only those two tests (Auditory Attention and Visual Attention) that had shown relatively poor discrimination between the high and low attention groups in Study I were, again, administered to another cohort of 97 Grade 4 children, as it was our intention to further challenge the reliability of the ACL. Overall, the results of both studies suggest that comprehensive assessment of attention skills should include both ACL and objective measures of selective attention.     

“I Chose Math Because…”: Cognitive interviews of a motivation measure

The ability to accurately measure academic motivation is important to its value as a predictive variable for learning, achievement, and other outcomes. Although measures of motivation are frequently subject to quantitative validation (e.g., Appleton, Ntoumanis, Quested, Viladrich, & Duda, 2016; Gagné et al., 2015; Pekrun, Goetz, Frenzel, Barchfeld, & Perry, 2011), the establishment of cognitive validity is more rare. By conducting cognitive interviews with a sample of elementary-aged children, we explored the cognitive validity of a novel motivation (expectancy–value and academic emotions) survey embedded in an educational technology. Children were largely able to accurately interpret questions, elaborate on their reasoning for answers, and choose answers congruent with those reasons. Challenges to cognitive validity fell under varied and underdeveloped interpretations of expectancy–value concepts; misunderstandings related to available response choices; and discrepancies between younger and older children’s abilities to judge their perceived competencies and values. Insights from these interviews can be applied to interpretation of the immediate survey, but also to design and interpretation of motivation surveys beyond the current measure.     

Assessment of Language Skills in Young Children with Profound Hearing Loss under Two Years of Age

This article presents an initial evaluation of a technique known as the Diary of Early Language (Di-EL), designed to obtain data about early lexical development in young children with profound hearing loss using cochlear implants, hearing aids, or both. The validity of the Di-EL, a parent report technique, was examined through comparisons with other measures of language development. Lexical data reported by parents using the Di-EL was found to agree with that reported by the same parents for the same children using the MacArthur Communicative Development Inventories (CDI), although some differences in the lexical items were noted. Rate of lexical acquisition on the Di-EL was found to correlate highly with that measured by the CDI and with expressive language skills as measured by the Rossetti Infant Toddler Language Scale, suggesting that the Di-EL is a valid measure of early lexical progress. These results are discussed with reference to other diary studies, along with research and clinical applications of the Di-EL.     

ADHD and achievement: meta-analysis of the child, adolescent, and adult literatures and a concomitant study with college students

This article presents results from two interrelated studies. The first study conducted a meta-analysis of the published literature since 1990 to determine the magnitude of achievement problems associated with attention-deficit/hyperactivity disorder (ADHD). Effect sizes were significantly different between participants with and without ADHD (sample weighted r = .32, sample weighted d = . 71; p = .001). Effects were also examined according to the moderators of age, gender, achievement domain (reading, math, spelling), measurement method (standardized tests vs. grades, parent/teacher ratings, etc.), sample type (clinical vs. nonclinical), and system used to identify ADHD (DSM-III-R vs. DSM-IV). Significant differences emerged from the moderator comparisons. The second study, using averaged effect sizes from the first study as a baseline for comparison, investigated achievement levels for an understudied age group with ADHD, namely, college students. Unlike previous studies at the college level, the sample incorporated both student and parent ratings (N = 380 dyads). The results were comparable to outcomes from the meta-analysis for college students and adults. Analyses demonstrated modest (R = .21) but meaningful predictive validity across 1 year to end-of-first-year grades. However, unlike earlier studies with children and adolescents, student ratings were as predictive as parent ratings. Findings are discussed in terms of the impact of moderator variables on ADHD and achievement.     

Valid assessment of writing and access to academic discourse

One way to improve students' access to and retention in post-secondary degree progams is to assess their readiness for such programs accurately. To place deaf and hard-of-hearing students in preparatory courses and to determine their readiness for degree programs more accurately, a direct measure of writing was developed for deaf and hard-of-hearing students at a large technical university. The purpose of this study was to estimate the concurrent and predictive validity of this measure. The Test of Written English (Educational Testing Service, 1992) served as the criterion in the concurrent validity study, and student success in the university's gateway freshman composition course served as the criterion in the predictive validity study. Results provide evidence of the concurrent and predictive validity of the measure, supporting its use for course placement and early planning purposes.     

Identification of educationally at-risk children during the kindergarten year: A four-year follow-up study of group test performance

Group test performance of children identified during the kindergarten year as educationally high, moderate, and low risk was investigated by following a group of 472 children from grades one through four. End of kindergarten predictive measures were the Kindergarten Evaluation of Learning Potential, the Bender-Gestalt Test, and the Slosson Intelligence Test; follow-up measures were group achievement tests administered in April of each school year. Significant differences in achievement performance were found between the high and low risk groups. Significant correlations were found between risk group designation and achievement performance in the first four grades. (No significant differences in group test performance were found for risk groups or individuals between grade levels.) Findings support the predictive validity of the present screening procedures for group test performance through grade four. Further, the findings show that students appear to perform consistently at the same level year to year in a regular class instructional program.     

EFFECTS OF VIOLATING ITEM CONSTRUCTION PRINCIPLES1

Two forms of a social studies achievement test were constructed with half the items for each form containing a cue, grammar, or length fault. Faults were found to make the items easier, which was supported by confidence intervals for the differences. However, validity coefficients with achievement and intelligence criteria, as well as the reliability coefficients, were virtually unchanged. The results agreed with those of Dunn and Goldstein (1959), even though the methodology differed. A suggested measure of test-wiseness for groups is presented.     

Predictors of academic success in medical school

This study was designed to determine the predictive validity of selected admissions variables used by the University of Kentucky College of Medicine. Data for the study were collected from 586 students admitted to the M.D. program from 1961 to 1968. These students were categorized into one of three groups (Successful [SUCC], Successful with Extended Study [SWES], and Lost to Medicine [LTM]) based upon their performance in medical school. A two-way analysis of variance and the use of a post hoc Scheffé method of multiple comparisons indicated significant differences between the SUCC group and the combined SWES and LTM groups. Therefore, the data revealed the predictive ability of two admissions variables and suggested the need for additional research into the subjective variables used in the admissions process.     

Name writing ability not length of name is predictive of future academic attainment

Background: The Performance Indicators in Primary Schools On Entry Baseline assessment for pupils starting school includes an item which aims to assess how well a pupil writes his or her own name. There is some debate regarding the utility of this measure, on the grounds that name length may constitute bias.

Purpose, method and design: The predictive validity of this item and its link to name length was investigated with a view to using this item in further assessments. Previous modest scale work from the USA, suggests that name writing ability is a robust indicator which correlates substantively with other known indicators of later reading whilst remaining independent of name length. This paper greatly expanded the sample size and geographical coverage and, rather than concurrent measures, the predictive validity of the item is assessed. The sample includes children from England, Scotland and Australia (N = 14932), assessed between 2011 and 2013. Potential confounding factors that are analysed include age, geographical region and ethnicity.

Findings and conclusions: The evidence suggests that the name writing item is a robust measure, with good predictive validity to future academic outcomes in early reading, phonological awareness and mathematics. The length was not related to the ability to write one’s own name nor was it predictive of future outcomes.     

Try a Little Bit of Teaching: A Dynamic Assessment of Word Decoding as a Kindergarten Predictor of Word Reading Difficulties at the End of Grade 1

The present study examined the construct and predictive validity of a dynamic test of decoding. In theory, a dynamic test provides a direct measure of potential for learning. In this study, children were taught 3 novel letters and how to blend the sounds of those into new words, then they were tested on different words comprising the 3 letters. The study followed 171 children from kindergarten to the end of Grade 1. The dynamic test was found to add significantly to the longitudinal prediction of word reading difficulties at the end of Grade 1 even after controlling for a wide range of standard predictors. The dynamic test correlated strongly with concurrent measures of early reading, letter knowledge, and phoneme awareness but less strongly with vocabulary and nonverbal intelligence. It is suggested that the dynamic test taps learning of sublexical units and processing essential for initial reading development.