Negative Keying Effects in the Factor Structure of TIMSS 2011 Motivation Scales and Associations with Reading Achievement

ABSTRACT

The Student Background survey administered along with achievement tests in studies of the International Association for the Evaluation of Educational Achievement includes scales of student motivation, competence, and attitudes toward mathematics and science. The scales consist of positively- and negatively keyed items. The current research examined the factorial structure of the 18-item motivational scales in fourth-grade mathematics in the 2011 Trends in International Mathematics and Science Study (TIMSS). Survey data from six European countries were analyzed. In comparisons of alternative models, the fit was adequate when three correlated factors were specified and negative keying was taken into account as a latent factor, or with correlated uniquenesses among negatively keyed items. Participants reading achievement scores correlated systematically to negative keying with coefficients ranging from .254 to .395 in the six samples. Unlike their higher-scoring peers, fourth-graders with lower reading achievement responded differentially to similar items depending on the direction of item keying, in such a way that their motivation scores were biased downward. Implications about the use of reverse keying in surveys for young students are discussed.     

The Impact of Vertical Scaling Decisions on Growth Interpretations

Most growth models implicitly assume that test scores have been vertically scaled. What may not be widely appreciated are the different choices that must be made when creating a vertical score scale. In this paper empirical patterns of growth in student achievement are compared as a function of different approaches to creating a vertical scale. Longitudinal item‐level data from a standardized reading test are analyzed for two cohorts of students between Grades 3 and 6 and Grades 4 and 7 for the entire state of Colorado from 2003 to 2006. Eight different vertical scales were established on the basis of choices made for three key variables: Item Response Theory modeling approach, linking approach, and ability estimation approach. It is shown that interpretations of empirical growth patterns appear to depend upon the extent to which a vertical scale has been effectively “stretched” or “compressed” by the psychometric decisions made to establish it. While all of the vertical scales considered show patterns of decelerating growth across grade levels, there is little evidence of scale shrinkage.     

A Note on the Problem Solving Style Questionnaire: An alternative to Kolb's Learning Style Inventory?

Given the psychometric limitations of existing measures of Kolb's experiential learning model (ELM), two new scales of learning styles have been developed. The validity of these scales has been supported in samples of undergraduate and MBA students in the USA. This paper provides evidence of some psychometric properties of scores yielded by these scales using samples of undergraduate students in the UK. Only limited support is found for the internal consistency reliability and construct validity of scores produced by the scales. However, an item attrition exercise identifies a two‐factor solution providing an acceptable fit to the data. The scales are reported as being positively correlated with academic performance and prior academic achievement. Despite the mixed evidence, we suggest further development of the scales is warranted to create a psychometrically sound measure of the ELM.     

Conceptions of learning in adult students embarking on distance education

A 60-item questionnaire on conceptions of learning was mailed to students taking preparatory courses by distance learning with The Open University in the United Kingdom. Complete data were provided by 372 respondents. Their scores on six factor-based scales showed satisfactory internal consistency, cluster analysis identified five groups of students with distinct patterns of scores, and discriminant analysis identified the scales that served to distinguish among the clusters. Three groups had conceptions of learning based on critical thinking, personal development, and personal change, but the other groups had conceptions that were defined in largely negative terms. Adult learners embarking on distance education seem to hold distinctive conceptions of learning, which suggests that conceptions of learning are culturally and contextually dependent.     

National survey of problems and competencies among four- to sixteen-year-olds: parents' reports for normative and clinical samples.

We compared parent-reported problems and competencies for national samples of 2,600 4-16-year-olds assessed at intake into mental health services and 2,600 demographically matched nonreferred children assessed in a home interview survey. Parents responded to the ACQ Behavior Checklist, which includes 23 competence items, three competence scales, 216 problem items, eight syndrome scales, Internalizing, Externalizing, and total competence and problem scores. Most items and scales discriminated significantly (p less than .01) between referred and nonreferred samples. There were important sex and age differences in problem patterns, but regional and ethnic differences were minimal. Somewhat more problems and fewer competencies were reported for lower- than upper-socioeconomic-status children. Referral rates were similar in the most urban and rural areas, but they were significantly higher in areas of intermediate urbanization. Correlations of problem scores with those obtained 10 years earlier in a regional survey and with surveys in other countries showed considerable consistency in the rank order of prevalence rates among specific problems. Apparently owing to its more differentiated response scales, the ACQ was susceptible to respondent characteristics that reduced its discriminative power below that of the Child Behavior Checklist. Comparisons of procedures for discriminating between the normal and the clinical range supported the value of a borderline category for children who are neither clearly normal nor clearly deviant. Interview data from the survey sample yielded significantly higher ACQ problem scores for children who had fewer related adults in their homes, those who had more unrelated adults in their homes, those whose biological parents were unmarried, separated, or divorced, those whose families received public assistance, and those whose household or family members had received mental health services. Children who scored higher on Externalizing than Internalizing problems tended to have unmarried, separated, or divorced parents and to come from families receiving public assistance. However, among children whose household or family members had received mental health services, there were greater proportions of both Externalizing and Internalizing patterns than among other children.     

Student and tutor perceptions of effective tutoring in distance education

Questionnaire responses of 457 students and 602 tutors were used to investigate conceptions of a ‘good tutor.’ In each case, factor analysis identified scales that reflected key constructs; cluster analysis identified subgroups with different patterns of scale scores; and discriminant analysis determined the scales that contributed the most to differences among the clusters. Both sets of data yielded conceptions of tutoring that were described as task‐oriented and student‐oriented, respectively. The students’ data yielded an additional, career‐oriented conception. The tutors’ data yielded two additional conceptions that were described as knowledge‐oriented and impersonal, respectively. The distribution of the tutors’ conceptions (but not that of the students’ conceptions) varied across different faculties, suggesting that tutors from different disciplines have different beliefs about effective tutoring. The study suggests that both tutors and students would benefit from having a better appreciation of the importance of support in facilitating learning.     

Utility of the PASS theory and cognitive assessment system for Dutch children with and without ADHD

This study examined the utility of the Planning, Attention, Simultaneous, Successive (PASS) theory of intelligence as measured by the Cognitive Assessment System (CAS) for evaluation of children with attention-deficit/hyperactivity disorder (ADHD). The CAS scores of 51 Dutch children without ADHD were compared to the scores of a group of 20 Dutch children with ADHD. The scores of the Dutch children were also compared to American standardization samples of children with and without ADHD. The findings showed that children with ADHD in both countries demonstrated relatively low scores on the Planning and Attention scales of the CAS, but average scores on the Simultaneous and Successive scales. These findings are similar to previously published research suggesting that the PASS theory, as operationalized by the CAS, has sensitivity to the cognitive processing difficulties found in some children with ADHD.     

Verbal/visual processing for males disabled in print acquisition

Fourteen males disabled in print acquisition were studied over time to determine their patterns of verbal and spatial processing. Comparing results of WISC-R and WAIS scales using a Bonferroni correction for t-values, it was found that the Similarities subtest scores significantly increased while the Vocabulary subtest scores and Acquired Knowledge category significantly decreased. Hemispheric specialization was suggested as a possible cause for differences in verbal and nonverbal processing.     

Approaches to Studying and Perceptions of the Academic Environment in Students Following Problem-Based and Subject-Based Curricula

To investigate the effects of different kinds of curriculum, the Course Experience Questionnaire (CEQ) and a short form of the Approaches to Studying Inventory (ASI) were administered to 225 2nd-year students at six different schools of occupational therapy. Their curricula were classified as problem-based, subject-based or hybrid. Their scores on the scales of the CEQ and ASI were closely related, insofar as they shared more than half of their respective variance. Problem-based curricula were associated with higher scores on the scales of the CEQ concerned with appropriate assessment and emphasis on independence. With the ASI, problem-based curricula were associated with lower scores on all of the scales concerned with a reproducing orientation, and with higher scores on the scale concerned with a deep approach. These findings suggest that the implementation of a problem-based curriculum has desirable effects on the quality of learning, and these are at least in part mediated by students' perceptions of their academic environment.     

Recentering and Realigning the SAT Score Distributions: How and Why

The process employed to produce the conversions that take scores from the original SAT scales to recentered scales, in which reference group scores are centered near the midpoint of the score-reporting range, is laid out. For the purposes of this article, SAT Verbal and SAT Mathematical scores were placed on recentered scales, which have reporting ranges of 920 to 980, means of 950, and standard deviations of 11. (The 920-to-980 scale is used in this article to highlight the distinction between it and the old 200-to-800 scale. In actuality, recentered scores were reported on a 200-to-800 scale.) Recentering was accomplished via a linear transformation of normally distributed scores that were obtained from a continuized, smoothed frequency distribution of original SAT scores that were originally on augmented two-digit scales (i.e., discrete scores rounded to either 0 or 5 in the third decimal place). These discrete scores were obtained for all students in the 1990 Reference Group using 35 different editions of the SAT spanning October 1988 to June 1990. The performance of this 1990 Reference Group on the original and recentered scales is described. The effects of recentering on scores of individuals and the 1990 Reference Group are also examined. Finally, recentering did not occur solely on the basis of its technical merit. Issues associated with converting recentering from a possibility into a reality are discussed.     

Development of the Preferences for College Counseling Inventory

This study describes the development of the Preferences for College Counseling Inventory (PCCI). In addition to providing data about college students' preferences for the logistics of counseling and the demographic/professional background of a potential therapist, the PCCI also provides numerical scores on 3 scales (Therapist Expertise, Therapist Warmth, and Therapist Directiveness) that measure preferences for therapist dispositions and 2 scales (Task‐Oriented Activities and Experiential/Insight‐Oriented Activities) that measure preferences for counseling activities.     

Reducing the length of questionnaires through structurally incomplete designs: An illustration

This paper serves as an illustration of the usefulness of structurally incomplete designs as an approach to reduce the length of educational questionnaires. In structurally incomplete test designs, respondents only fill out a subset of the total item set, while all items are still provided to the whole sample. The scores on the unadministered items are subsequently dealt with by using methods for the estimation of missing data. Two structurally incomplete test designs — one recording two thirds, and the other recording a half of the potentially complete data — were applied to the complete item scores on 8 educational psychology scales. The incomplete item scores were estimated with missing data method Data Augmentation. Complete and estimated test data were compared at the estimates of total scores, reliability, and predictive validity of an external criterion. The reconstructed data yielded estimates that were very close to the values in the complete data. As expected the statistical uncertainty was higher in the design that recorded fewer item scores. It was concluded that the procedure of applying incomplete test designs and subsequently dealing with the missing values is very fruitful for reducing questionnaire length.     

4-6年级小学生心理健康状况的调查分析

本采用MHT对449名小学生的心理健康状况进行了调查，通过统计分析发现：（1）全体学生在八个分量表的平均分上有显性差异。（2）总量表和分量表（恐怖倾向量表除外）的平均分均无显性性别差异（3）总量表的平均分随年级升高而增高，四年级学生和六年级学生存在显性差异。     

Children's agency beliefs, cognitive performance, and conceptions of effort and ability: individual and developmental differences

In order to understand how children's beliefs and cognitive performance vary according to development in their conceptions of effort and ability, 120 fourth and sixth graders were given the following assessments: (a) a measure of agency beliefs, defined as the extent to which persons believe they have access to certain classes of potential means, including effort, ability, powerful others, and luck; (b) a battery of intelligence test scales, including figural patterns, letter series, arithmetic, and spelling; (c) an interview designed to assess developmental levels in conceptions of effort and ability. Correlations between intelligence scores and agency beliefs for effort decreased with increasing levels of reasoning about effort and ability, but correlations between intelligence scores and agency beliefs for ability increased with increasing levels of reasoning. No such trends were found in correlations between performance and agency beliefs for luck or for unknown causes. The results are discussed in terms of the interaction between individual differences and developmental change.     

Felder and Silverman's Index of Learning Styles and Honey and Mumford's Learning Styles Questionnaire: How do they compare and do they predict academic performance?

Samples of engineering and business students at undergraduate, postgraduate and post-experience levels at two UK universities completed the Index of Learning Styles (ILS) (N = 284) or the (Learning Styles Questionnaire LSQ) (N = 182) and a biographical data questionnaire. Broad psychological aspects of the two learning style instruments are examined and compared. Psychometric properties of the instruments, including factor structure, internal reliability and inter-scale correlation are analysed. Potential limitations are commented on, in particular those related to the construct validity and relatively low internal reliability of the ILS scales (alpha = 0.41 to 0.65). These compare with alphas of 0.59 to 0.74 for the LSQ. Relationships between the scales are discussed and a circumplex arrangement of the LSQ is proposed. Proposals for augmenting the circumplex are made. Academic performance results and scores on each of the two instruments are compared. The general lack of significant correlations between learning style scores and performance in these samples is discussed. Conclusions are drawn about the disappointing psychometric robustness of the measures, the activity-centred nature of learning styles and the advantages of viewing styles as a circumplex.     

Extending Big‐Five theory into childhood: A preliminary investigation into the relationship between Big‐Five personality traits and behavior problems in children

An investigation was conducted to link Big‐Five personality traits with behavior problems identified in childhood. Eighty‐six children ranging in age from 9 to 13 were rated by their respective classroom teacher using an experimental ratings instrument developed to measure Big‐Five personality constructs and behavior concurrently. Big‐Five Personality and Behavior Problem scales were correlated. Results show distinct patterns of behavior problems associated with various personality characteristics. Children with low scores in Agreeableness and Conscientiousness exhibit social problems, conduct problems, attention deficits, and hyperactivity. Children with low scores on the scale, Openness to Experience, exhibit problems in social behavior, conduct, and attention. The Neuroticism trait was associated with anxiety and depression. Potential implication for clinical practice and future research study was provided. These preliminary data indicate that identifying Big‐Five personality trait patterns may be a useful dimension of assessment for understanding underlying motives and predispositions associated with children's problem behavior. © 1999 John Wiley & Sons, Inc.     

Psychometric Properties of Raw and Scale Scores on Mixed‐Format Tests

This paper illustrates that the psychometric properties of scores and scales that are used with mixed‐format educational tests can impact the use and interpretation of the scores that are reported to examinees. Psychometric properties that include reliability and conditional standard errors of measurement are considered in this paper. The focus is on mixed‐format tests in situations for which raw scores are integer‐weighted sums of item scores. Four associated real‐data examples include (a) effects of weights associated with each item type on reliability, (b) comparison of psychometric properties of different scale scores, (c) evaluation of the equity property of equating, and (d) comparison of the use of unidimensional and multidimensional procedures for evaluating psychometric properties. Throughout the paper, and especially in the conclusion section, the examples are related to issues associated with test interpretation and test use.     

THE CHOICE OF SCALE FOR EDUCATIONAL MEASUREMENT: AN IRT PERSPECTIVE

Two methods of constructing equal-interval scales for educational achievement are discussed: Thurstone's absolute scaling method and Item Response Theory (IRT). Alternative criteria for choosing a scale are contrasted. It is argued that clearer criteria are needed for judging the appropriateness and usefulness of alternative scaling procedures, and more information is needed about the qualities of the different scales that are available. In answer to this second need, some examples are presented of how IRT can be used to examine the properties of scales: It is demonstrated that for observed score scales in common use (i.e., any scores that are influenced by measurement error), (a) systematic errors can be introduced when comparing growth at selected percentiles, and (b) normalizing observed scores will not necessarily produce a scale that is linearly related to an underlying normally distributed true trait.     

The Impact of Anonymization for Automated Essay Scoring

This study investigated the impact of anonymizing text on predicted scores made by two kinds of automated scoring engines: one that incorporates elements of natural language processing (NLP) and one that does not. Eight data sets (N = 22,029) were used to form both training and test sets in which the scoring engines had access to both text and human rater scores for training, but only the text for the test set. Machine ratings were applied under three conditions: (a) both the training and test were conducted with the original data, (b) the training was modeled on the anonymized data, but the predictions were made on the original data, and (c) both the training and test were conducted on the anonymized text. The first condition served as the baseline for subsequent comparisons on the mean, standard deviation, and quadratic weighted kappa. With one exception, results on scoring scales in the range of 1–6 were not significantly different. The results on scales that were much wider did show significant differences. The conclusion was that anonymizing text for operational use may have a differential impact on machine score predictions for both NLP and non‐NLP applications.