EFFECTS OF EMPIRICAL OPTION WEIGHTING ON RELIABILITY AND VALIDITY OF AN ACADEMIC APTITUDE TEST1

Item options of shortened forms of the GRE Verbal and Quantitative tests were empirically weighted by two variants of a method originally attributed to Guttman (1941). When compared with formula scores, it was found that tests scored with the empirical weights were more reliable but less valid when correlated with undergraduate GPA. A factor analysis revealed large increases in variance accounted for by the first factor. It was suggested that the weighting procedures used tended to capitalize on omitting behavior which, although a highly reliable tendency, may be invalid.     

Technical characteristics of the peabody individual achievement test as a function of item arrangement and basal and ceiling rules

The effects that item order and basal and ceiling rules have on test means, variances, and internal consistency estimates for the PIAT mathematics and reading recognition subtests were examined. Seven items on the math subtest and one item on the reading recognition subtest were significantly easier or harder than their test placement indicated. The use of basal and ceiling rules had a pronounced effect on the means, variances, and reliabilities on the multiple choice math subtest, while the rules' effects on the reading recognition subtests were minor. Item order also affected scores on the math subtest.     

The WISC-R and evidence of item bias for native-American Navajos

The study investigated cultural bias in the 79 items of the three verbal tests of the Wechsler Intelligence Scale for Children-Revised (WISC-R). The Information, Similarities, and Vocabulary subtests were administered to 40 Anglo and 40 Native- American Navajo subjects matched for grade level. The responses of the two groups of subjects on individual items were analyzed by log-linear technique using the likelihood ratio chi-square statistic. The findings revealed that performance of subjects was homogeneous across groups on most of the items of three verbal subtests of the WISC-R. Only 15 (19%) of the 79 items comprising Information, Similarities, and Vocabulary subtests were found to be biased against the Navajo sample. Five of these items are from the Information, four from the Similarities, and the remaining six items are from the Vocabulary subtest. Implications of these findings for the psychoeducational assessment of minority children were discussed.     

Oral language facility,verbal intelligence and performance on tests of psycholinguistic ability

The performance of 152 children in the age range 7 years 5 months to 7 years 11 months on a battery comprising the Illinois Test of Psycholinguistic Abilities (ITPA), the English Picture Vocabulary Test 2 (EPVT), a test of auditory discrimination, a sentence repetition test and an orally administered verbal intelligence test was compared with the complexity and fluency of spoken language recorded from each child. Only a moderate correlation was found between the ITPA total score and the length‐complexity index (LCI) from the language samples. Moderate correlations were found between the LCI and several auditory‐vocal channel subtests from the ITPA; however, with the influence of verbal intelligence partialled out most of these were reduced to low or nonsignificant levels. The scores from the Manual Expression subtest in the visual‐motor channel were found to correlate as highly with LCI as did the ITPA total score.

This relationship was maintained to two situations, (i) when the factor of verbal intelligence was controlled statistically, (ii) when a sub‐sample of subjects within a narrow IQ range was selected for separate analysis. The Verbal Expression subtest also showed moderate correlation with LCI in four out of five analyses, thus lending support to the “process” construct in the ITPA model. No connection was found between any of the psycholinguistic subtests and fluency of language production. The validity of the ITPA as a test of oral language performance was questioned.     

Gender Differences in Multiple-Choice Tests: The Role of Differential Guessing Tendencies

The present study focused on gender differences in the tendency to omit items and to guess in multiple-choice tests. It was hypothesized that males would show greater guessing tendencies than females and that the use of formula scoring rather than the use of number of correct answers would result in a relative advantage for females. Two samples were examined: ninth graders and applicants to Israeli universities. The teenagers took a battery of five or six aptitude tests used to place them in various high schools, and the adults took a battery of five tests designed to select candidates to the various faculties of the Israeli universities. The results revealed a clear male advantage in most subtests of both batteries. Four measures of item-omission tendencies were computed for each subtest, and a consistent pattern of greater omission rates among females was revealed by all measures in most subtests of the two batteries. This pattern was observed even in the few subtests that did not show male superiority and even when permissive instructions were used. Correcting the raw scores for guessing reduced the male advantage in all cases (and in the few subtests that showed female advantage the difference increased as a result of this correction), but this effect was small. It was concluded that although gender differences in guessing tendencies are robust they account for only a small fraction of the observed gender differences in multiple-choice tests. The results were discussed, focusing on practical implications.     

Cross-validation of two creativity tests designed for preschool children

Two tests, designed to assess ideational fluency through different response modes, were administered to 24 middle-class preschool children. The Multidimensional Stimulus Fluency Measure (MSFM) requires verbal responses to verbal and visual-tactile stimuli. Thinking Creatively in Action and Movement (TCAM) employs a kinesthetic, nonverbal response mode to verbal and visual-tactile stimuli. Concurrent and construct validity were established for these two instruments. Significant intercorrelations among the subtests of each instrument demonstrated construct validity, and concurrent validity was established with Spearman rank-order correlations between the scores of the two tests, r=.55, p<.01. The results showed construct validity in that ideational fluency assessed via the two instruments was not related to IQ. The low correlations found between subtest D of the TCAM and all of the subtests of the MSFM were discussed with regard to differences in response modality and stimulus specificity.     

Sensory, cognitive, and linguistic factors in the early academic performance of elementary school children: The Benton-IU project

Standardized sensory, perceptual, linguistic, intellectual, and cognitive tests were administered to 470 children, approximately 96% of the students entering the first grade in the four elementary schools of Benton County, Indiana, over a 3-year period (1995--1997). The results of 36 tests and subtests administered to entering first graders were well described by a 4-factor solution. These factors and the tests that loaded most heavily on them were reading-related skills (phonological awareness, letter and word identification); visual cognition (visual perceptual abilities, spatial perception, visual memory); verbal cognition (language development, vocabulary, verbal concepts); and speech processing (the ability to understand speech under difficult listening conditions). A cluster analysis identified 9 groups of children, each with a different profile of scores on the 4 factors. Within these groups, the proportion of students with unsatisfactory reading achievement in the first 2 years of elementary school (as reflected in teacher-assigned grades) varied from 3% to 40%. The profiles of factor scores demonstrated the primary influence of the reading-related skills factor on reading achievement and also on other areas of academic performance. The second strongest predictor of reading and mathematics grades was the visual cognition factor, followed by the verbal cognition factor. The speech processing factor was the weakest predictor of academic achievement, accounting for less than 1% of the variance in reading achievement. This project was a collaborative effort of the Benton Community School Corporation and a multidisciplinary group of investigators from Indiana University.     

Working memory and mathematics in primary school children: A meta-analysis

Working memory, including central executive functions (inhibition, shifting and updating) are factors thought to play a central role in mathematical skill development. However, results reported with regard to the associations between mathematics and working memory components are inconsistent. The aim of this meta-analysis is twofold: to investigate the strength of this relation, and to establish whether the variation in the association is caused by tests, sample characteristics and study and other methodological characteristics. Results indicate that all working memory components are associated with mathematical performance, with the highest correlation between mathematics and verbal updating. Variation in the strength of the associations can consistently be explained by the type of mathematics measure used: general tests yield stronger correlations than more specific tests. Furthermore, characteristics of working memory measures, age and sample explain variance in correlations in some analyses. Interpretations of the contribution of moderator variables to various models are discussed.     

Validity of the Differential Aptitude Test for the Assessment of Immigrant Children

This paper addresses the construct as well as the criterion validity of the Differential Aptitude Test (DAT) for the assessment of secondary school minority group students ( N = 111) as compared to majority group students ( N = 318) in The Netherlands. Comparison of the test dimensions with the structural equation modelling program EQS showed that construct validity was good for both groups. With one exception, the subtests of the DAT measured the cognitive abilities of minority and majority group students equally well. The estimate of g as computed with the DAT showed strong predictive validity with little bias for various school subjects and achievement tests for mathematics and Dutch. Although some criteria revealed prediction bias to the disadvantage of the minority group, these differences concerned very small changes in R 2 . Conversely, the predictive value decreased substantially when an estimate of g was used excluding subtests that measure aspects of crystallised intelligence. Spearman's hypothesis tested with DAT subtest scores and criterion scores showed that g explained most of the group differences. Professional test users can safely draw conclusions from the DAT regardless of the students' ethnicity.     

The influence of examiners' ethnic attributes on intelligence test scores

The study investigated the effect of examiners' ethnicity on the intelligence test performance of Anglo and Mexican-American subjects. Two verbal subtests of the WISC and the Raven Progressive Matrices, a nonverbal intelligence test, were administered by two Anglo and two Mexican-American examiners. All subjects (N=96) were given half the items of each test by an Anglo and half by a Mexican-American examiner. On two of the three tests, the subjects' performance was unaffected by examiner's ethnic attributes. On the WISC Vocabulary, however, Mexican-American subjects scored significantly higher when the test was administered by Mexican-American examiners. The generalizability of these results and their implications for learning in schools are discussed.     

Item Difficulty and Interviewer Knowledge Effects on the Accuracy and Consistency of Examinee Response Processes in Verbal Reports

The Standards for Educational and Psychological Testing indicate that multiple sources of validity evidence should be used to support the interpretation of test scores. In the past decade, examinee response processes, as a source of validity evidence, have received increased attention. However, there have been relatively few methodological studies of the accuracy and consistency of examinee response processes as measured by verbal reports in the context of educational measurement. The objective of the current study was to investigate the accuracy and consistency of examinee response processes—as measured by verbal reports—as a function of varying interviewer and item variables in a think aloud interview within an educational measurement context. Results indicate that the accuracy of responses may be undermined when students perceive the interviewer to be an expert in the domain. Further, the consistency of response processes may be undermined when items that are too easy or difficult are used to elicit reports. The implications of these results for conducting think-aloud studies are explored.     

Using the Partial Credit Model to Investigate Responses to Structured Subtests

This article applies the partial credit model to an assessment situation in which items (termed "superitems" by the developers) have been constructed as subtests sharing a particular set of stimulus materials and according to a neo-Piagetian theory of learning (SOLO). Comparison is made with Guttman scaling, a nonprobabilistic approach. The complementary nature of results from a dichotomous Rasch analysis and the polytomous partial-credit analysis is explored.     

Stability reliability for elementary-age students on the Woodcock-Johnson psychoeducational battery-Revised (achievement section) and the Kaufman Test of Educational Achievement

These two studies examined the stability reliability for the Woodcock-Johnson-Revised (WJ-R; Woodcock & Johnson, 1989) and the Kaufman Test of Educational Achievment (KTEA; Kaufman & Kaufman, 1985) with approximately a 2-week retest interval for elementary-age students. Results indicated that across grade levels, the Broad Reading Cluster for the WJ-R remained stable. Most correlations for the clusters for mathematics and written language as well as the subtests for reading, mathematics, and written language were less than .90. Correlations for all composites and subtests for the KTEA exceeded .90. These data illustrate the need for more specific information in test manuals on test-retest reliability in order to enable examiners to select the most reliable measures.     

Learning environment, attitudes and conceptual development associated with innovative strategies in middle-school mathematics

This study of middle-school students in California focused on the effectiveness of using innovative teaching strategies for enhancing the classroom environment, students’ attitudes and conceptual development. A sample of 661 students from 22 classrooms in four inner city schools completed modified forms of the Constructivist Learning Environment Survey (CLES), What Is Happening In this Class? (WIHIC) questionnaire and Test of Mathematics Related Attitudes (TOMRA). Data analyses supported the factor structure, internal consistency reliability, discriminant validity and the ability to distinguish between different classes for these questionnaires when used with middle-school mathematics students in California. The effectiveness of the innovative instructional strategy was evaluated in terms of classroom environment and attitudes to mathematics for the whole sample, as well as for mathematics achievement for a subgroup of 101 students. A comparison of an experimental group which experienced the innovative strategy with a control group supported the efficacy of the innovative teaching methods in terms of learning environment, attitudes and mathematics concept development. Also associations were found between perceptions of classroom learning environment and students’ attitudes to mathematics and conceptual development.     

A comparison of cognitive skills between completes and dropouts in a college physics course

Separate tests of mathematics skills, proportions and translations between words, and mathematical expression given the first week of class were correlated with performance for students who completed a college physics course (completes) and students who dropped the course (drops). None of the measures used discriminated between completes and drops as groups. However, the correlations between score on the test of math skills and on both of the measures involving mathematical reasoning (proportions, and translations) were dramatically different for the two groups. For the completes, these correlations were slightly negative, but not significant. For the drops, the correlation was positive and signficant at the p < 0.01 level. This suggests the possibility that the students who complete the course tend to have independent cognitive skills for the “mechanical” mathematical operations and for questions requiring some degree of reasoning, while, in contrast, the same skills for students at high risk for dropping overlap significantly. The study also found that when students are given the results of mathematics skills tests in a diagnostic mode, with feedback on specific areas of weakness and time to remediate with self study, the correlation between mathematics and physics is lower than previously reported values.     

WISC-R examiner errors: Cause for concern

While errors on the WISC-R are conceived primarily in terms of internal consistency and stability over time, examiners make mistakes that contribute to the inaccuracy of test scores. Studies to date mainly have investigated general scoring errors, rather than specific items most prone to error. Investigation of graduate students' test protocols indicated numerous scoring and mechanical errors that influenced the Full Scale IQ scores on two-thirds of the protocols. Particularly prone to error were Verbal subtests of Vocabulary, Comprehension, and Similarities. More importantly, specific items on subtests in which numerous mistakes occurred were noted, as well as the most likely type of error for each item. These findings have implications for the education and training of assessment specialists.     

The influence of supplementary subtests on WISC IQ scores for high achieving students

The validity of Wechsler's (1949) comments concerning the addition of the supplementary WISC subtests was investigated for a sample of 20 fifth-grade students. The study was designed like that of Engin (1974) which investigated whether or not the addition of one or both of the supplementary WISC sub-tests, Digit Span and Mazes, materially affected the obtained IQs of high achieving fifth-grade subjects. All 12 subtests of the WISC were individually administered to the students, and IQs were then calculated in such a manner that specific comparisons could be made. These comparisons were between verbal, Performance and Full Scale IQs composed of the maximum number of subtests, and verbal, Performance, and Full Scale IQs exclusive of Digit Span, Mazes, or both subtests. T-tests for correlated means were employed and revealed highly significant differences. The addition of Digit Span and Mazes in the WISC battery served to depress the verbal, Performance and Full Scale IQs of the high achieving students. The study serves to validate the previous investigation by Engin (1974).     

Field Independence Associates with Mathematics and Science Performance in 5‐ to 10‐Year‐Olds after Accounting for Domain‐General Factors

Field independence describes the extent to which individuals are influenced by context when trying to identify embedded targets. It associates with cognitive functioning and is a predictor of academic achievement. However, little is known about the neural and cognitive underpinnings of field independence that lead to these associations. Here, we investigated behavioral associations between two measures of field independence (Children's Embedded Figures Test [CEFT] and Design Organization Test [DOT]) and performance on tests of mathematics (reasoning and written arithmetic) and science (reasoning and scientific inquiry) in 135 children aged 5–10 years. There were strong associations between field independence and mathematics and science, which were largely explained by individual differences in age, intelligence, and verbal working memory. However, regression analyses indicated that after controlling for these variables, the CEFT explained additional variance on the mathematical reasoning and science tests, whereas the DOT predicted unique variance on the written arithmetic test.     

A comparison of the Visual Aural Digit Span and the Bender Gestalt as discriminators of low achievement in the primary grades

The Visual Aural Digit Span (VADS) and the Bender Visual Motor Gestalt Test (Bender) were studied with regard to their ability to discriminate low from average achievers in reading and arithmetic skills, as identified by the Iowa Test of Basic Skills. A sample of 78 normal children aged 6 through 9 were administered a battery of tests, including the verbal section of the WISC-R. Analysis of covariance with IQ controlled showed that the Bender and the VADS were able to discriminate between achievement groups for vocabulary and math concepts. The Bender discriminated between math problem-solving groups, but neither test could discriminate between reading comprehension groups. Age was a significant variable for the Bender and all VADS subtests except Aural-Written. Correlational analysis indicated that although the VADS was related to Verbal IQ, it is related only minimally to the Bender when age is controlled.