A COMPARISON OF THE VALIDITIES OF CONVENTIONAL CHOICE TESTING AND VARIOUS CONFIDENCE MARKING PROCEDURES

This study compared the convergent and discriminant validity of two confidence marking techniques with that of conventional choice testing. Achievement in vocabulary, social studies, and science (traits) was measured by a 60-item test containing true-false and 5-alternative items (methods). The test was administered to three randomly assigned groups (one for each response system), totaling 535 Ss. The results indicated very slight differences in convergent and discriminant validity that favored conventional choice testing over confidence marking techniques.     

The effect of a pretest in an interactive,multimodal pretraining system for learning science concepts

Multiple traits of language proficiency as well as test method effects were concurrently analyzed to investigate interrelations of construct validity, convergent validity, and discriminant validity using multitrait-multimethod (MTMM) matrices. A total of 585 test takers' scores were derived from the field test of the Pearson Test of English Academic. An MTMM confirmatory factor analysis model was parameterized using 4 traits and 3 assessment methods. The 4 traits included listening, reading, speaking, and integrated skills, while the 3 methods included prescribed multiple-choice responses, constructed responses, and summarized responses. The trait factor loadings were systematically greater than those of methods, providing evidence that the indicators were strongly related to their latent constructs, after adjusting for the method effects. The results showed robust convergent validity, moderate discriminant validity, and insignificant method effects. Implications are discussed.     

Examining Trait × Method Interactions Using Mixture Distribution Multitrait–Multimethod Models

Multitrait-multimethod (MTMM) analyses are used in psychology to assess convergent and discriminant validity and to study method effects. Most current MTMM approaches assume that measures have equal convergent and discriminant validity across the entire range of trait values and thus do not account for potential trait × method interactions. A novel approach is presented that allows analyzing trait × method interactions using factor mixture modeling. The new MTMM mixture model allows identifying latent classes of individuals who differ with respect to convergent and discriminant validity. The new approach was applied to mother’s and father’s ratings of children’s attention deficit hyperactivity disorder (ADHD) symptoms (N = 618). Results revealed four latent classes: one with no symptom levels, two with low symptom levels, and one with moderate symptom levels. Three classes showed evidence for convergent and discriminant validity, whereas a low symptom class lacked convergent validity for ratings of inattention.

     

Validation of Preschool Teachers’ Recall of Children’s Behavior

A multiple-choice behavioral observation scale designed to be used by preschool teachers (N = 424) was factor analyzed to obtain factor coefficients. The four factors obtained were cognitive skills, self-control, relationship with achievement model, and dependency. The factor coefficients were then applied to the ratings of 153 preschool disadvantaged children by 33 teachers and 64 teacher aides. Three multi-trait, multi-rater matrices were analyzed using Campbell and Fiske’s convergent discriminant validation procedures. The analyses supported convergent and discriminant validity of three of the four traits.     

CONVERGENT AND DISCRIMINANT VALIDITY OF THREE MEASURES OF ABILITY,ASPIRATION-LEVEL,ACHIEVEMENT, ADJUSTMENT AND DOMINANCE1

The convergent and discriminant validity of three measures of the concepts of aspiration level, ability, achievement, adjustment, and dominance were examined in the context of a multitrait-multimethod matrix. Self-reports and peer-reports on 75 Ss were employed as two measures of each trait. In addition, aspiration level was measured by the Edwards Personal Preference Schedule (EPPS) Nach scale, dominance by the EPPS (dom scale), achievement by cumulative college grade point ratio (GPR), ability by the Ohio State Psychological Examination (OSPE), and adjustment by the Bell Adjustment Inventory. Of the paper and pencil instruments, only the OSPE and EPPS (dominance scale) exhibited satisfactory convergent validity. No measure met all the requirements of discriminant validity. The desirability of establishing adequate validational evidence prior to using “trait” measures in studies relating theoretical variables was emphasized.     

THE CONCURRENT VALIDITY OF STANDARDIZED ACHIEVEMENT TESTS BY CONTENT AREA USING TEACHERS' RATINGS AS CRITERIA

To assess the concurrent validity of standardized achievement tests using teachers' ratings (and rankings) of pupils' academic achievement as criteria, 42 teachers evaluated each of their students (n = 1,032) in each of five major curricular areas prior to the administration of a battery of standardized achievement tests. The teachers were directed to rate each student's proficiency disregarding attendance, attitude, deportment, and so on. Within-class correlation coefficients were computed to eliminate rater leniency bias. The standardized achievement tests were found to have substantial concurrent validity in reading, math, language arts, science, and social studies. The normalized teacher ranks yielded significantly higher validity coefficients than did the ratings, although the magnitude of the difference was small. The concurrent validity coefficients for language arts, reading, and math were significantly higher than those in science and social studies.     

A Comparison of Exposure Control Procedures in CAT Systems Based on Different Measurement Models for Testlets

The purposes of the study were twofold: (a) to test for evidence of construct validity related to social and academic (general, English, mathematics) competencies, as measured by self, teachers, parents, and peers, for children in Grades 3 (n = 175), 7 (n = 193), and 11 (n = 159), and (b) to examine the extent to which ratings by significant others were concordant with self-ratings for these four constructs. Based on a comparison of nested multitrait-multimethod covariance structure models, results demonstrated that despite evidence of global convergent and discriminant validities, method effects were instrumental in attenuating trait variances. In particular, self-ratings for early preadolescents and peer ratings for all ages were the most vulnerable to this anomaly. Overall, results suggest that researchers consider making greater use of parent ratings in the measurement of social and academic competencies for elementary children.     

The Academic Diligence Task (ADT): assessing individual differences in effort on tedious but important schoolwork

The current study reports on the development and validation of the Academic Diligence Task (ADT), designed to assess the tendency to expend effort on academic tasks which are tedious in the moment but valued in the long-term. In this novel online task, students allocate their time between solving simple math problems (framed as beneficial for problem solving skills) and, alternatively, playing Tetris or watching entertaining videos. Using a large sample of high school seniors (N = 921), the ADT demonstrated convergent validity with self-report ratings of Big Five conscientiousness and its facets, self-control and grit, as well as discriminant validity from theoretically unrelated constructs, such as Big Five extraversion, openness, and emotional stability, test anxiety, life satisfaction, and positive and negative affect. The ADT also demonstrated incremental predictive validity for objectively measured GPA, standardized math and reading achievement test scores, high school graduation, and college enrollment, over and beyond demographics and intelligence. Collectively, findings suggest the feasibility of online behavioral measures to assess noncognitive individual differences that predict academic outcomes.     

Generalization of the Child Observation Record: a validity study for diverse samples of urban,low-income preschool children

The present investigation addressed the construct validity of the Child Observation Record (COR) with low-income urban preschool children. From two separate samples representing low-income preschool children, COR ratings were analyzed using multivariate techniques. Independent analyses from these two urban sites yielded a three-dimensional structure: Cognitive Skills, Social Engagement, and Coordinated Movement. Further analyses cross-validated this structure for males and females and across ethnic groups. Concurrent assessments provided convergent and discriminant validity for the Social Engagement dimension and convergent validity for Cognitive Skills dimension. Analyses of item distributions of the 5-point developmental sequences represented by the 30 COR items were used to examine the assumption that all the distributions were continuous unimodal distributions. Findings did not universally support this assumption revealing some irregular distributions with troughs in the mid-range of continua. Implications of the findings for early childhood assessment of vulnerable children and future research were discussed.     

Multitrait–Multimethod Assessment of Giftedness: An Application of the Correlated Traits–Correlated (Methods – 1) Model

Although the use of multiple criteria and informants is one of the most universally agreed on practices in the identification of gifted children, few studies to date have examined the convergent validity of multiple informants and objective ability tests in gifted identification. In this study, we illustrate the use of the correlated traits–correlated (methods – 1) or CT–C(M – 1) model (Eid, Lischetzke, Nussbeck, & Trierweiler, 2003) to examine the convergent validity of self, parent, and teacher ratings relative to objective cognitive ability tests in a sample of 145 4th to 6th graders. The CT–C(M – 1) analyses revealed that teacher ratings showed the highest convergence with the objective assessments, whereas self-ratings had the lowest reliabilities and insufficient validity. Parent ratings were more reliable and valid than self-reports, but were outperformed by teacher ratings for most abilities. Overall, the CT–C(M – 1) analyses showed that the convergent validity of the ratings relative to the objective test battery was highest for numerical and lowest for creative abilities. Furthermore, whereas part of the shared variance between parent and teacher ratings reflected true convergent validity, agreement between parent and self-reports was entirely due to a shared rater variance. Our analyses demonstrate the usefulness and proper interpretation of the CT–C(M – 1) approach for examining convergent validity and method effects in multitrait–multimethod data.     

ESTIMATING THE RELIABILITY, VALIDITY, AND INVALIDITY OF ESSAY RATINGS

In an essay rating study multiple ratings may be obtained by having different raters judge essays or by having the same rater(s) repeat the judging of essays. An important question in the analysis of essay ratings is whether multiple ratings, however obtained, may be assumed to represent the same true scores. When different raters judge the same essays only once, it is impossible to answer this question. In this study 16 raters judged 105 essays on two occasions; hence, it was possible to test assumptions about true scores within the framework of linear structural equation models. It emerged that the ratings of a given rater on the two occasions represented the same true scores. However, the ratings of different raters did not represent the same true scores. The estimated intercorrelations of the true scores of different raters ranged from .415 to .910. Parameters of the best fitting model were used to compute coefficients of reliability, validity, and invalidity. The implications of these coefficients are discussed.     

Convergent and discriminant validity of the mental processing scales of the Kaufman Assessment Battery for Children

Convergent and discriminant validity of the Mental Processing Scales of the Kaufman Assessment Battery for Children (K-ABC) were examined using 51 first-grade children. Convergent validity was assessed using the Reading Recognition and Comprehension subtests of the Peabody Individual Achievement Test. Discriminant validity was assessed using a measure of anxiety/self-esteem, the Child Anxiety Scale, and a measure of hyperactive behavior, the Hyperactivity scale of the Achenbach Child Behavior Checklist. Results supported the convergent validity of the K-ABC; correlations with reading achievement were fairly large. The discriminant validity received only partial support. The K-ABC did not correlate with the Child Anxiety Scale, but did show rather large correlations with the measure of hyperactive behavior. Implications for understanding what the K-ABC Mental Processing Scales are measuring are discussed.     

COMPARISON OF PREDICTIVE VALIDITY AND DIAGNOSTIC ACCURACY OF SCREENING MEASURES OF READING SKILLS

Assessment data must be valid for the purpose for which educators use them. Establishing evidence of validity is an ongoing process that must be shared by test developers and test users. This study examined the predictive validity and the diagnostic accuracy of universal screening measures in reading. Scores on three different universal screening tools were compared for nearly 500 second‐ and third‐grade students attending four public schools in a large urban district. Hierarchical regression and receiver operating characteristic curves were used to examine the criterion‐related validity and diagnostic accuracy of students’ oral reading fluency (ORF), Fountas and Pinnell Benchmark Assessment System (BAS) scores, and fall scores from the Measures of Academic Progress for reading (MAP). Results indicated that a combination of all three measures accounted for 65% of the variance in spring MAP scores, whereas a reduced model of ORF and MAP scores predicted 60%. ORF and BAS scores did not meet standards for diagnostic accuracy. Combining the measures improved diagnostic accuracy, depending on how criterion scores were calculated. Implications for practice and future research are discussed.     

Multi-Method Assessment of ADHD Characteristics in Preschool Children: Relations between Measures

Several forms of assessment tools, including behavioral rating scales and objective tests such as the Continuous Performance Test (CPT), can be used to measure inattentive and hyperactive/impulsive behaviors associated with Attention-Deficit/Hyperactivity Disorder (ADHD). However, research with school-age children has shown that the correlations between parent ratings, teacher ratings, and scores on objective measures of ADHD-characteristic behaviors are modest at best. In this study, we examined the relations between parent and teacher ratings of ADHD and CPT scores in a sample of 65 preschoolers ranging from 50 to 72 months of age. No significant associations between teacher and parent ratings of ADHD were found. Parent-ratings of both inattention and hyperactivity/impulsivity accounted for variance in CPT omission errors but not CPT commission errors. Teacher ratings showed evidence of convergent and discriminant validity when entered simultaneously in a hierarchical regression. These tools may be measuring different aspects of inattention and hyperactivity/impulsivity.     

Validation of the Metacomprehension Scale

Evidence for the factorial, convergent and discriminant, and criterion-related validity of the Metacomprehension Scale (MCS) was examined in a sample of 237 young adults. The instrument was factorially heterogeneous but exhibited homogeneity within each of the seven subscales. Evidence for the convergent and discriminant validity of the MCS was examined by correlating the subscales from the MCS with subscales from metacognitive questionnaires measuring similar constructs from related domains. In general, correlations within constructs were larger than correlations between constructs, providing preliminary evidence of the convergent and discriminant validity of the MCS. The criterion-related validity of the MCS relative to other metacognitive measures was examined by using the metacognitive measures and the MCS to predict comprehension performance. The MCS predicted performance better than the other measures of metacognition and accounted for variance in performance not accounted for by the other measures. These results show promise for the value of self-assessments of metacomprehension.     

Can We Learn From Student Mistakes in a Formative,Reading Comprehension Assessment?

This article describes an ongoing project to develop a formative, inferential reading comprehension assessment of causal story comprehension. It has three features to enhance classroom use: equated scale scores for progress monitoring within and across grades, a scale score to distinguish among low‐scoring students based on patterns of mistakes, and a reading efficiency index. Instead of two response types for each multiple‐choice item, correct and incorrect, each item has three response types: correct and two incorrect response types. Prior results on reliability, convergent and discriminant validity, and predictive utility of mistake subscores are briefly described. The three‐response‐type structure of items required rethinking the item response theory (IRT) modeling. IRT‐modeling results are presented, and implications for formative assessments and instructional use are discussed.     

STUDENT RATINGS OF TERTIARY COURSES FOR ‘ALTERNATIVE CALENDAR’ PURPOSES

Surveys of student opinion of tertiary courses often constitute a major source of information for prospective students. Yet the reliability and validity of such surveys have not previously been investigated. In this paper, data from a survey of some 20,000 individual ratings of 224 undergraduate courses were analysed. The purpose was to explore the relationship between students’ ratings and the characteristics of their courses. It was found that these ratings were stable and significantly related to the academic area of the course, the size of class, the percentage of full‐time students, the academic year of the course, and the grades awarded in the course. Implications for the validity of such ratings are then discussed.     

Relationships Among Early Childhood Behavior Rating Scales: Convergent and Discriminant Construct Validity of the Preschool and Kindergarten Behavior Scales

Research Findings: Data that serve to establish the convergent and discriminant construct validity of a new behavior rating scale for use with the early childhood preschool population-the Preschool and Kindergarten Behavior Scales (PKBS)-are presented. The results of four different studies are presented wherein PKBS ratings of preschool or kindergarten age children were correlated with established comparison measures: the Social Skills Rating System, Mattson Evaluation of Social Skills with Youngsters, Conners Teacher Rating Scale, and School Social Behavior Scales. Correlations were in the desired directed for demonstrating convergent and discriminant construct validity of the PKBS. Practice Implications: The PKBS appears to adequately measure the constructs of social skills and both internalizing and externalizing problem behavior in early childhood. Although additional validation research for this instrument is needed, the PKBS appears to show promise as a research tool, screening device, and assessment instrument for assessing social-emotional behavior of children ages 3–6. Given the increasing importance of early detection of social-emotional problems as part of a comprehensive system of prevention and early intervention, future efforts at linking assessment tools to specific and effective intervention techniques appear to be a much needed and significant endeavor.     

Relationships Among Early Childhood Behavior Rating Scales: Convergent and Discriminant Construct Validity of the Preschool and Kindergarten Behavior Scales

Research Findings: Data that serve to establish the convergent and discriminant construct validity of a new behavior rating scale for use with the early childhood preschool population-the Preschool and Kindergarten Behavior Scales (PKBS)-are presented. The results of four different studies are presented wherein PKBS ratings of preschool or kindergarten age children were correlated with established comparison measures: the Social Skills Rating System, Mattson Evaluation of Social Skills with Youngsters, Conners Teacher Rating Scale, and School Social Behavior Scales. Correlations were in the desired directed for demonstrating convergent and discriminant construct validity of the PKBS. Practice Implications: The PKBS appears to adequately measure the constructs of social skills and both internalizing and externalizing problem behavior in early childhood. Although additional validation research for this instrument is needed, the PKBS appears to show promise as a research tool, screening device, and assessment instrument for assessing social-emotional behavior of children ages 3-6. Given the increasing importance of early detection of social-emotional problems as part of a comprehensive system of prevention and early intervention, future efforts at linking assessment tools to specific and effective intervention techniques appear to be a much needed and significant endeavor.     

Attention deficit disorder and specific reading disability: separate but often overlapping disorders

Within a large (N = 182) heterogeneous sample of clinic-referred children with DSM-III-diagnosed attention deficit disorder (ADD), three behavioral subgroups were identified via cluster analysis of teacher ratings: 40% of the children had ADD with hyperactivity (ADDH), 30% had ADD with hyperactivity and aggressivity (ADDHA), and 31% had ADD without hyperactivity or aggressivity. Proportionally more girls were in the ADD-only subgroup. Over half the sample (n = 94) were poor readers, with 82 meeting discrepancy criteria for specific reading disability (RD). Proportionately more boys than girls met the RD criteria (9.2:1.0), whereas the sex ratio of males to females for the whole sample was 5.1 to 1.0. Focusing just on white males, the three behavioral subgroups were significantly different on convergent validity measures, such as other teacher ratings, parent ratings, and interview-elicited ratings of externalizing behavior, but were not different on such divergent validity measures as IQ and achievement scores, self-ratings, and laboratory performance tasks. Boys in the ADD sample who did not meet criteria for RD had significantly higher IQs than those who did, but subgroups with and without RD still differed significantly on WRAT-R reading and spelling scores with IQ covaried out. Both groups with and without RD could be differentiated from a control group on laboratory measures of sustained attention and impulse control. Methylphenidate benefited all subgroups equally, whether RD or not, and whether given a low (0.3 mg/kg) or high (0.6 mg/kg) dose.