An empirical examination of the construct validity of goal commitment in the persistence process

This study represents the first published investigation into the construct validity of goal commitment as it affects the persistence process. Confirmatory factor analyses revealed that goal commitment could be decomposed into multiple indicators of the same latent construct: a special factor called goal commitment that groups items related to goal importance, specificity of goals, and situational influence; a second factor represented by items indicating certainty of purpose; and a third factor consisting of items related to goals in general. The predictive validity of each subcomponent on different outcomes related to student persistence was established. While goal commitment was found to have a significant direct effect on both students' intents to persist and actual persistence behavior, neither of the other two factors were as equally predictive as measures of student retention.     

MULTIPLE PROCESSING STRATEGIES AND THE CONSTRUCT VALIDITY OF VERBAL REASONING TESTS

This study examines the influence of processing strategies, and the associated metacomponents that determine when to apply them, on the construct validity of a verbal reasoning test. Three strategies for solving verbal analogy items were examined: a rule-oriented strategy, an association strategy, and a partial rule strategy. Construct validity was studied in two separate stages: construct representation and nomothetic span. For construct representation, evidence was obtained that all three strategies, and their related metacomponents, are associated with performance on analogy items. For nomothetic span, the current study found that all three strategies contribute to individual differences in verbal reasoning and to the predictive validity of the test. The results of this study also point to the utility of metacomponents as constructs for describing and understanding test performance. Implications of the results for test development and theories of aptitude are elaborated.     

Construction and validation of a classroom climate scale: a mixed methods approach

Students’ perceptions of their classroom climate have been found to relate significantly to students’ learning outcomes. The purpose of the present study was to construct an instrument for assessing elementary-school students’ perceptions of classroom climate, based on a previous instrument that was being used in Chile by a public national school mental health program as a tool for aiding teachers in improving classroom management, but which showed poor psychometric properties. We used a six-staged mixed-methods approach to construct relevant items and dimensions based on this measure and by adapting previously-existing scales. Item development included participatory construction of items involving program officials, focus groups with students, and a pilot study. The final version was administered to a sample of 6813 elementary-school students. Results showed adequate reliability and construct validity, convergent validity with school climate, and divergent validity with peer victimisation. When consequential validity was explored through semi-structured interviews with program officials and school administrators, we found that the instrument was being used as a tool for helping teachers to improve their school climate and management skills. We discuss the importance of constructing instruments using a mixed-methods approach.     

Development and validation of the elder’s spiritual health scale

ABSTRACT

Spiritual health is one of the most important aspects of the elders’ health. The aim of this study was to develop and evaluate psychometric properties of a scale for evaluating spiritual health of older adults in Iran. This is a mixed research, consisted of two phases. First, the perception of elder people regarding the spiritual health was explored, using directed content analysis, and the scale items generated based on the results. Second, the content, face and construct validity were determined. Exploratory factor analysis was used for the construct validity. To determine the reliability, Cronbach’s alpha and test-retest were used. Preliminary designed questionnaire included 94 items, which were reduced to 38 following the content and face validity processes. Exploratory factor analysis demonstrated that 20 items loaded on five factors determined about 66% of variance. The total internal consistency of the scale was 0.89. Results of test-retest indicated a Pearson correlation coefficient of 0.85 while intra-class correlation coefficient of scale was 0.92. The ESHS is a short, user-friendly valid and reliable tool which can be used for assessing the spiritual health of older adults.     

The measurement of classroom management self‐efficacy: a review of measurement instrument development and influences

Teachers' self‐efficacy (SE) in their classroom management capabilities is thought to be an important factor in teachers' overall judgements of their teaching SE. Low SE in classroom management has been linked to teacher attrition and burnout, and reduced student learning outcomes. This article provides the first comprehensive review of classroom management as a factor in the construct of teacher SE. Twenty‐five peer‐reviewed articles published from 1984 to 2009 that reported on the use of SE scales containing at least one novel classroom management self‐efficacy (CMSE) item were reviewed. The validity and reliability of CMSE scales and items were found to be very good, with classroom management items pertaining to maintaining order and control the most frequent category included. Approximately one in four items in the SE scales reviewed was CMSE item, and, in general, CMSE items were not linked explicitly to classroom management research or contemporary psychological or philosophical approaches.     

MATHEMATICS EDUCATION VALUES QUESTIONNAIRE FOR TURKISH PRESERVICE MATHEMATICS TEACHERS: DESIGN, VALIDATION, AND RESULTS

The purpose of this study was to develop a questionnaire that could measure preservice mathematics teachers' mathematics educational values. Development and validation of the questionnaire involved a sequential inquiry in which design principles were established from the existing literature and a pool of items was constructed then submitted to experts for consideration of the construct validity. Alterations to the items based on their suggestions were made to produce a trial version of the questionnaire. A pilot study involving preservice mathematics teachers explored the validity and usefulness of the questionnaire. The pilot results were used to revise the questionnaire that was administered to a sample of preservice mathematics teachers attending Cumhuriyet University, Sivas, Turkey. Further explorations of the construct and structural validity, item contributions, and reliability were achieved by using a factor analysis and two different item analysis methods. Results revealed that the questionnaire included four factors, satisfactory item contributions, and acceptable internal consistency. One result obtained in this study suggested that some mathematics education values based on Western culture (e.g., accessibility–special) have not been accepted by Turkish preservice mathematics teachers.     

Measuring ocean literacy of high school students: psychometric properties of a Chinese version of the ocean literacy scale

This study established a Chinese scale for measuring high school students’ ocean literacy. This included testing its reliability, validity, and differential item functioning (DIF) with the aim of compensating for the lack of DIF tests focusing on current scales. The construct validity and reliability were verified and tested by analyzing the established scale’s items using the Rasch model, and a gender DIF test was conducted to ensure the test results’ fairness when distinct groups were compared simultaneously. The results indicated that the scale established in this study is unidimensional and possesses favorable internal consistency and construct validity. The gender DIF test results indicated that several items were difficult for either female or male students to correctly answer; however, the experts and scholars discussed these items individually and suggested retaining them. The final Chinese version of the ocean literacy scale developed here comprises 48 items that can reflect high school students’ understanding of ocean literacy—which helps students understand the topics of marine science encountered in real life.     

Construct Validity of Self-Reported Metacognitive Learning Strategies

Despite their significant contributions to research on self-regulated learning, those favoring online and trace approaches have questioned the use of self-report to assess learners' use of learning strategies. An important rejoinder to such criticisms consists of examining the validity of self-report items. The present study was designed to assess the validity of items to assess 9th-grade students' use of planning, monitoring, and regulation when studying math. To establish response process evidence of construct validity, cognitive interviews were coded to determine whether students' interpretations of the items were consistent with their intended meaning and whether their response choices were congruent with those interpretations. Evidence supported the construct validity of monitoring and regulation items, but to a lesser degree those designed to assess planning. We discuss implications of the evidence for the self-report assessment of learners' use of metacognitive strategies.     

An Empirical Investigation Demonstrating the Multidimensional DIF Paradigm: A Cognitive Explanation for DIF

Differential Item Functioning (DIF) is traditionally used to identify different item performance patterns between intact groups, most commonly involving race or sex comparisons. This study advocates expanding the utility of DIF as a step in construct validation. Rather than grouping examinees based on cultural differences, the reference and focal groups are chosen from two extremes along a distinct cognitive dimension that is hypothesized to supplement the dominant latent trait being measured. Specifically, this study investigates DIF between proficient and non-proficient fourth- and seventh-grade writers on open-ended mathematics test items that require students to communicate about mathematics. It is suggested that the occurrence of DIF in this situation actually enhances, rather than detracts from, the construct validity of the test because, according to the National Council of Teachers of Mathematics (NCTM), mathematical communication is an important component of mathematical ability, the dominant construct being assessed. However, the presence of DIF influences the validity of inferences that can be made from test scores and suggests that two scores should be reported, one for general mathematical ability and one for mathematical communication. The fact that currently only one test score is reported, a simple composite of scores on multiple-choice and open-ended items, may lead to incorrect decisions being made about examinees.     

Validation of the perceived school bullying severity scale

Research on school bullying has tended to focus on its prevalence or frequency while ignoring its perceived severity. This study attempted to construct a perceived School Bullying Severity Scale (SBSS). The original 24-item instrument, revised from the Victim Scale of the School Bullying Scales, covered the four categories of physical, verbal, relational and cyber bullying. The partial credit model was used to conduct Rasch analysis with ConQuest software on data derived from two samples of Taiwanese secondary school students. Sample 1 and sample 2 consisted of 605 and 869 students, respectively. Three items were deleted after examining the quality of the data from sample 1. The reliability and validity of the 21 items on the final scale were verified using data from sample 2. Results demonstrated the reliability and validity of information collected by the SBSS. This study also found that secondary school students rated relational and cyber bullying as more severe than physical and verbal bullying. Differences between teachers’ and students’ perspectives on the perceived severity of various bullying behaviours as well as implications for preventing and intervening in bullying are discussed.     

Scientific reasoning in elementary school children: Assessment and relations with cognitive abilities

The primary goal of this study was the broad assessment and modeling of scientific reasoning in elementary school age. One hundred fifty-five fourth graders were tested on 20 recently developed paper-and-pencil items tapping four different components of scientific reasoning (understanding the nature of science, understanding theories, designing experiments, and interpreting data). As confirmed by Rasch analyses, the scientific reasoning items formed a reliable scale. Model comparisons differentiated scientific reasoning as a separate construct from measures of intelligence and reading skills and revealed discriminant validity. Furthermore, we explored the relationship between scientific reasoning and the postulated prerequisites inhibitory control, spatial abilities and problem-solving skills. As shown by correlation and regression analyses, beside general cognitive abilities (intelligence, reading skills) problem-solving skills and spatial abilities predicted performance in scientific reasoning items and thus contributed to explaining individual differences in elementary school children's scientific reasoning competencies.     

Validating Measurement of Knowledge Integration in Science Using Multiple-Choice and Explanation Items

This study explores measurement of a construct called knowledge integration in science using multiple-choice and explanation items. We use construct and instructional validity evidence to examine the role multiple-choice and explanation items plays in measuring students' knowledge integration ability. For construct validity, we analyze item properties such as alignment, discrimination, and target range on the knowledge integration scale using a Rasch Partial Credit Model analysis. For instructional validity, we test the sensitivity of multiple-choice and explanation items to knowledge integration instruction using a cohort comparison design. Results show that (1) one third of correct multiple-choice responses are aligned with higher levels of knowledge integration while three quarters of incorrect multiple-choice responses are aligned with lower levels of knowledge integration, (2) explanation items discriminate between high and low knowledge integration ability students much more effectively than multiple-choice items, (3) explanation items measure a wider range of knowledge integration levels than multiple-choice items, and (4) explanation items are more sensitive to knowledge integration instruction than multiple-choice items.     

Using a Multidimensional Differential Item Functioning Framework to Determine if Reading Ability Affects Student Performance in Mathematics

Many teachers and curriculum specialists claim that the reading demand of many mathematics items is so great that students do not perform well on mathematics tests, even though they have a good understanding of mathematics. The purpose of this research was to test this claim empirically. This analysis was accomplished by considering examinees that differed in reading ability within the context of a multidimensional DIF framework. Results indicated that student performance on some mathematics items was influenced by their level of reading ability so that examinees with lower proficiency classifications in reading were less likely to obtain correct answers to these items. This finding suggests that incorrect proficiency classifications may have occurred for some examinees. However, it is argued that rather than eliminating these mathematics items from the test, which would seem to decrease the construct validity of the test, attempts should be made to control the confounding effect of reading that is measured by some of the mathematics items.     

The psychological sense of school membership among adolescents: Scale development and educational correlates

This article discusses the development and validation of a measure of adolescent students' perceived belonging or psychological membership in the school environment. An initial set of items was administered to early adolescent students in one suburban middle school (N = 454) and two multi-ethnic urban junior high schools (N = 301). Items with low variability and items detracting from scale reliability were dropped, resulting in a final 18-item Psychological Sense of School Membership (PSSM) scale, which had good internal consistency reliability with both urban and suburban students and in both English and Spanish versions. Significant findings of several hypothesized subgroup differences in psychological school membership supported scale construct validity. The quality of psychological membership in school was found to be substantially correlated with self-reported school motivation, and to a lesser degree with grades and with teacher-rated effort in the cross-sectional scale development studies and in a subsequent longitudinal project. Implications for research and for educational practice, especially with at-risk students, are discussed.     

Development of an item bank for assessing generic competences in a higher-education institute: a Rasch modelling approach

This paper describes the development and validation of an item bank designed for students to assess their own achievements across an undergraduate-degree programme in seven generic competences (i.e., problem-solving skills, critical-thinking skills, creative-thinking skills, ethical decision-making skills, effective communication skills, social interaction skills and global perspective). The Rasch modelling approach was adopted for instrument development and validation. A total of 425 items were developed. The content validity of these items was examined via six focus group interviews with target students, and the construct validity was verified against data collected from a large student sample (N?=?1151). A matrix design was adopted to assemble the items in 26 test forms, which were distributed at random in each administration session. The results demonstrated that the item bank had high reliability and good construct validity. Cross-sectional comparisons of Years 1–4 students revealed patterns of changes over the years. Correlation analyses shed light on the relationships between the constructs. Implications are drawn to inform future efforts to develop the instrument, and suggestions are made regarding ways to use the instrument to enhance the teaching and learning of generic skills.     

Assessing Teachers' Beliefs about Their Science Teaching Context

The primary purpose of this study was to develop and apply a method for assessing teachers' context beliefs about their science teaching environment. Interviews with 130 purposefully selected teachers resulted in 28 categories of environmental factors and/or people who were perceived to influence science teaching. These categories were used to develop items for the Context Beliefs about Teaching Science instrument and provided evidence for content validity. Construct validity was partially confirmed through factor analysis that resulted in 26 items and two subscales on the final instrument. Using Ford's Motivation Systems Theory and Bandura's Theory of Collective Efficacy, additional evidence for construct validity was found in the modest correlation of context beliefs with outcome expectancy beliefs and the low correlation with science teaching self‐efficacy beliefs. The instrument was tested using 262 teachers participating in long‐term science professional development programs. These teachers possessed fairly positive context beliefs and, according to Ford's theory, should be capable of effective functioning in the classroom. It was concluded that the assessment of context beliefs would complement current science teacher self‐efficacy measures, thereby allowing researchers to develop profiles of science teachers' personal agency belief patterns. It could also be used to determine the factors which predict particular personal agency belief patterns, and assess teachers' perceptions of the strengths and weaknesses of school science programs, and could be used in planning and monitoring professional development experiences for science teachers. © 2000 John Wiley & Sons, Inc. J Res Sci Teach 37: 275–292, 2000.     

A technique for evaluating skills in high school science

This report sets out the procedures followed in developing a set of science items to test a variety of intellectual skills deemed important in secondary school science and then analyzing them in order to examine their construct validity in relation to a technique of evaluation which analyzes the way in which individuals structure their responses to previously learned material (Biggs & Collis, 1982). The items covered the four sciences commonly taught in Australian schools, Geology, Biology, Physics, and Chemistry. Each item followed the superitem format devised by Cureton (1965) and consisted of a stem followed by four questions. Each group of four questions was devised so that they formed a hierarchy of difficulty levels. Nineteen of the items finally accepted as meeting the initial criteria were arranged for group testing to enable a validation trial to be carried out. The analysis showed that the items had construct validity in terms of the theory and were viable for testing certain science skills at the High School level. Implications of the study point to a need for further investigations in both the curriculum and teaching areas of school science.     

A Data-based Analysis of the Psychometric Performance of the Differential Emotions Scale

This Differential Emotions Scale (DES) is an objective pencil-and-paper instrument designed to measure the subjective-experience components of the fundamental emotions, based on the assumption that mood states involved a characteristic pattern. Following Boyle (Boyle, G.J. Reliability and validity of Izard's Differential Emotions Scale, Personality, 56, pp. 747-750, 1984), the present paper reports a repeated-measure multiple discriminant function analysis for individual items across raters. At least, two-thirds of the DES items are sensitive indicators of the different mood states, however, the construct validity of the subscales is not clear. In particular, the profiles for the South African participants indicated that there is need for improving the construct validity and retest reliability of DES in order to be useful in applied psychological contexts in non-Western nations.     

Generalization of the Child Observation Record: a validity study for diverse samples of urban,low-income preschool children

The present investigation addressed the construct validity of the Child Observation Record (COR) with low-income urban preschool children. From two separate samples representing low-income preschool children, COR ratings were analyzed using multivariate techniques. Independent analyses from these two urban sites yielded a three-dimensional structure: Cognitive Skills, Social Engagement, and Coordinated Movement. Further analyses cross-validated this structure for males and females and across ethnic groups. Concurrent assessments provided convergent and discriminant validity for the Social Engagement dimension and convergent validity for Cognitive Skills dimension. Analyses of item distributions of the 5-point developmental sequences represented by the 30 COR items were used to examine the assumption that all the distributions were continuous unimodal distributions. Findings did not universally support this assumption revealing some irregular distributions with troughs in the mid-range of continua. Implications of the findings for early childhood assessment of vulnerable children and future research were discussed.     

Using multidimensional Rasch analysis to validate the Chinese version of the Motivated Strategies for Learning Questionnaire (MSLQ-CV)

This article used the multidimensional random coefficients multinomial logit model to examine the construct validity and detect the substantial differential item functioning (DIF) of the Chinese version of motivated strategies for learning questionnaire (MSLQ-CV). A total of 1,354 Hong Kong junior high school students were administered the MSLQ-CV. Partial credit model was suggested to have a better goodness of fit than that of the rating scale model. Five items with substantial gender or grade DIF were removed from the questionnaire, and the correlations between the subscales indicated that factors of cognitive strategy use and self-regulation had a very high correlation which resulted in a possible combination of the two factors. The test reliability analysis showed that the subscale of test anxiety had a lower reliability compared with the other factors. Finally, the item difficulty and step parameters for the modified 39-item questionnaire were displayed. The order of the step difficulty estimates for some items implied that some grouping of categories might be required in the case of overlapping. Based on these findings, the directions for future research were discussed.