Negative Keying Effects in the Factor Structure of TIMSS 2011 Motivation Scales and Associations with Reading Achievement

ABSTRACT

The Student Background survey administered along with achievement tests in studies of the International Association for the Evaluation of Educational Achievement includes scales of student motivation, competence, and attitudes toward mathematics and science. The scales consist of positively- and negatively keyed items. The current research examined the factorial structure of the 18-item motivational scales in fourth-grade mathematics in the 2011 Trends in International Mathematics and Science Study (TIMSS). Survey data from six European countries were analyzed. In comparisons of alternative models, the fit was adequate when three correlated factors were specified and negative keying was taken into account as a latent factor, or with correlated uniquenesses among negatively keyed items. Participants reading achievement scores correlated systematically to negative keying with coefficients ranging from .254 to .395 in the six samples. Unlike their higher-scoring peers, fourth-graders with lower reading achievement responded differentially to similar items depending on the direction of item keying, in such a way that their motivation scores were biased downward. Implications about the use of reverse keying in surveys for young students are discussed.     

高中学业水平考试研究(二):考试质量评价

学业水平考试试题和试卷的质量直接影响学业评价和诊断结果的有效性和可靠性。本文以上海市高中思想政治学科学业水平考试为例,从试题和题组功能偏差、试题得分与总分的相关系数、识别指数分析及分类一致性和准确性四个方面对考试的质量进行了定量评价,以介绍学业水平考试质量评价的方法。     

Achievement as a Function of Test Item Complexity and Difficulty

The effects of training tests on subsequent achievement were studied using 2-test item characteristics: item difficulty and item complexity. Ninety Ss were randomly assigned to treatment conditions having easy or difficult items and calling for rote or complex skills. Each S was administered two training tests during the quarter containing only items defined by his treatment condition. The dependent measure was a sixty item final examination with fifteen items reflecting each of the four treatment condition item types. The results showed greater achievement for those trained with difficult items and with rote items. In addition, two interaction of treatment conditions with type of test items were found. The results are discussed as supporting a hierarchical model rather than a “similarity” transfer model of learning.     

A FACET-FACTORIAL APPROACH TO THE CONSTRUCTION OF RATING SCALES TO MEASURE COMPLEX BEHAVIORS

The purpose of this study was to examine a technique for the development of performance rating scales to measure achievement in courses whose objectives require complex behaviors not easily measurable with paper and pencil achievement tests. A facet-factorial approach to rating scale construction was employed (i.e. the behavior was conceptualized as multidimensional and items for the scales were selected by employing factor analytical techniques) to construct scales to measure clarinet music performance. The three major results of the study were: 1) a thirty-item rating scale based on a six factor structure of clarinet music performance; 2) high inter-judge reliability estimates for both the total score (above .90) and the scale scores (above .60); and, 3) criterion-related validity coefficients greater than .80. Results of the investigation suggest that the facet-factorial approach can be an effective technique for the construction of rating scales to measure complex behavior such as music performance.     

MATCHED PAIR TRUE-FALSE SCORING: EFFECT ON RELIABILITY AND VALIDITY1

The matched pair technique for writing and scoring true-false items was designed to compensate for the acquiescence response set of primary grade children. The claim that this technique increases reliability to an appreciable extent over traditional true-false scoring was investigated by comparing alpha internal consistency coefficients computed for the matched pair true-false, traditional true-false, and three other scoring schemes. Both the total sample coefficients and individual classroom coefficients were computed from the standardization sample of a primary grade economics achievement test (Primary Test of Economic Understanding). Classroom reliability coefficients computed from the matched pair scores were found to be higher than those from scores computed by the other methods. Total sample coefficients obtained from four of the five methods were nearly equal. Evidence of the effects of each scoring technique on concurrent validity is also presented. Contrary to expectations, the correlations of traditional and matched pair scores with Iowa Test of Basic Skills (ITBS) subtests (when adjusted for differing reliabilities) were approximately equal.     

Recalled test anxiety in relation to achievement,in the context of general academic self-concept,study habits,parental involvement and socio-economic status among Grade 6 Ethiopian students

The study investigated the predictive nature of test anxiety on achievement in the presence of perceived general academic self-concept, study habits, parental involvement in children's learning and socio-economic status. From a population of 2482 Grade 6 students from seven government primary schools of a sub-city in Addis Ababa, 497 participants were randomly selected, namely 248 boys and 249 girls. The mean age of the participants was 12.98 years. An adapted version of Sarason's Test Anxiety Scale (28 items), plus the General Academic Self-Concept Scale (18 items), and Parental Involvement (10 items), Study Habits (10 items) and Socio-Economic Status (10 items) scales developed by the authors were the instruments of the study. The findings of the study indicated: (a) test anxiety correlated with achievement with a weak correlation of ?0.186; and (b) perceived general academic self-concept and study habits were positively and significantly related to achievement. Stepwise multiple regression on achievement resulted in the selection of general academic self-concept, study habits and parental involvement as significant contributors to achievement in that order. Test anxiety was found to be a non-predictor of achievement in the presence of other variables.     

Development of an integrated process skill test: TIPS II

The purpose of this project was to develop a valid and reliable science process skill test for middle and high school students. Multiple-choice items were generated for each of five objectives. Following pilot testing and revision, the test was administered to middle and high school students in the northeastern United States. The 36-item test can be completed in a normal class period. Results yielded a mean score of 19.14 and a total test reliability of 0.86. Mean difficulty and discrimination indices were 0.53 and 0.35, respectively. Split-test correlations coefficients between TIPS II and the original TIPS items were 0.86 and 0.90. TIPS II provides another reliable instrument for measuring process skill achievement. Additionally, it increases the available item pool for measuring these skills.     

Educational Assessment as a Promising Area for Psychometric Research

Tests of educational achievement typically present items in the multiple-choice format. Some achievement test items may be so "saturated with aptitude" (Willingham, 1980) as to be insensitive to skills acquired through education. Multiple-choice tests are ill-suited for assessing productive thinking and problem-solving skills, skills that often constitute important objectives of education. Viewed as incentives for learning, multiple-choice tests may impede student progress toward these objectives. There is need for accelerated research to develop alternatives to multiple-choice achievement tests, with content selected to match the specified educational objectives.     

Predicting science achievement: The role of developmental level,disembedding ability,mental capacity,prior knowledge,and beliefs

Controversy in the literature exists concerning the relative importance of cognitive characteristics such as general developmental level and specific prior knowledge for science achievement. Thus the relative ability of developmental level and prior knowledge to predict achievement of concepts of evolution and natural selection was assessed by means by three commonly used types of examination items: multiple choice items, a computational item, and an essay item. The effects of students' disembedding ability, mental capacity, and belief in evolution were also assessed. Disembedding ability, prior knowledge, and belief in evolution were found to be significantly related to overall achievement, while developmental level and mental capacity were not. However, developmental level was found to be the best predictor of performance on the computational item while belief in evolution and prior knowledge were found to be the best predictors of performance on the multiple-choice items. Mental capacity was found to be a predictor of performance on the essay item. It is argued that the cognitive variables influence different aspects of science achievement. Implications are discussed.     

Testing a motivational model of achievement: How students’ mathematical beliefs and interests are related to their achievement

Blackwell et al. (Child Development 78(1):246–263, 2007) tested a motivational model of achievement in which an incremental theory of intelligence leads to learning goals and positive effort beliefs, which leads to fewer ability-based, helpless attributions, and more positive strategies, which leads to improved grades. In the present study, we tested this model by using a different population, using domain-specific items, comparing the results across gender, including “interest” as another variable in the model, and assessing the impact of achievement as an antecedent and outcome variable in the model. Participants included 163, mostly White, ninth-grade Algebra students from a suburban school in southwest Virginia, USA. We established that the model had a relatively good fit to the data and all paths were statistically significant when achievement was used as both an antecedent and outcome variable. The standardized path coefficients were consistent with those reported by Blackwell et al. and the model was invariant across genders. Furthermore, we documented that interest could be included as another mediating variable in the model. The results of this study provide evidence to support the validity of the relationships between the constructs in the Blackwell et al. model for the population examined.     

Relationship between Science Teaching Practices and Students’ Achievement in Singapore,Chinese Taipei,and the US: An Analysis Using TIMSS 2011 Data

The assumption that inquiry-based instruction is more effective in influencing student science achievement than traditional didactic teaching has been the driving force of science education reform in recent decades and in many countries. However, the empirical relationship between these two kinds of science teaching and student science performance is not soundly established, which is worth a careful examination. Framed through the theoretical perspectives of inquiry-based instruction and culturally relevant pedagogy, using a two-level hierarchical linear modeling (HLM) approach and simultaneous multiple regression, this study examines the above relationship using the Trends in International Mathematics and Science Study (TIMSS) 2011 8th grade dataset from Singapore, Chinese Taipei, and the US. The study found that for the low-performing students, none of the inquiry-based teaching practice items measured had a significant relationship with the science achievements at any performance levels of students in any country/region except for the case of two inquiry-based teaching practice items that were positively related to Chinese Taipei students’ achievements. No didactic teaching practice items were associated with the Singapore students’ science achievement, three of these practice items were found negatively related to Chinese Taipei students’ science achievement, and one traditional didactic teaching practice was negatively related to the science achievement of U.S. students. However, for medium- and high-performing students, none of these inquiry-based or traditional didactic science-teaching practices were found to be positive predictors of science performance in all three countries/regions. However, in the case of Chinese Taipei, one didactic teaching practice item was negatively related with the medium level performing students’ achievement and two didactic teaching practices were found to hinder high-performing students’ science achievements.     

THE CONCURRENT VALIDITY OF STANDARDIZED ACHIEVEMENT TESTS BY CONTENT AREA USING TEACHERS' RATINGS AS CRITERIA

To assess the concurrent validity of standardized achievement tests using teachers' ratings (and rankings) of pupils' academic achievement as criteria, 42 teachers evaluated each of their students (n = 1,032) in each of five major curricular areas prior to the administration of a battery of standardized achievement tests. The teachers were directed to rate each student's proficiency disregarding attendance, attitude, deportment, and so on. Within-class correlation coefficients were computed to eliminate rater leniency bias. The standardized achievement tests were found to have substantial concurrent validity in reading, math, language arts, science, and social studies. The normalized teacher ranks yielded significantly higher validity coefficients than did the ratings, although the magnitude of the difference was small. The concurrent validity coefficients for language arts, reading, and math were significantly higher than those in science and social studies.     

Three Options Are Optimal for Multiple-Choice Items: A Meta-Analysis of 80 Years of Research

Multiple-choice items are a mainstay of achievement testing. The need to adequately cover the content domain to certify achievement proficiency by producing meaningful precise scores requires many high-quality items. More 3-option items can be administered than 4- or 5-option items per testing time while improving content coverage, without detrimental effects on psychometric quality of test scores. Researchers have endorsed 3-option items for over 80 years with empirical evidence—the results of which have been synthesized in an effort to unify this endorsement and encourage its adoption.     

An Experiment in Developing Critical Thinking in Children

A three week experiment was conducted comparing the academic achievement of pupils in five classrooms (N = 108) taught in small cooperative groups against that of pupils from five classes (N = 109) taught in the traditional whole-class approach. Special achievement tests were prepared for each grade level, two through six. These tests were constructed with items requiring responses at low and high levels of cognitive functioning. Pupils in grades two, four, and six from small-group classrooms excelled on high level items as predicted. Pupils in the fifth grade produced superior answers on questions requiring original contributions. Achievement scores of both groups did not differ on items measuring low level cognitive functioning.     

Student perceptions on the conditions of learning the proportion concept in canadian upper secondary science (physics,chemistry, biology) classes; results from the second international science study (S.I.S.S.) in Canada

Correlates of Canadian upper secondary science students' achievement in the proportion concept were studied from data collected as part of Canada's participation in the Second IEA Science Study. Average class achievement on 14 items concerning the proportion concept was calculated and related to students' reports of teachers' instructional approaches. The study analysis revealed that classes where the proportion content was well covered by the teachers demonstrated higher achievements. Students whose teachers perceived as making science relevant and interesting did well on the test items, and the use of lab work and being task oriented were also related to students' higher achievement.     

The episodic/semantic memory distinction as an heuristic in the study of instructional effects on cognitive structure

Previous attempts to measure the effects of instruction on students' cognitive structure have produced inconsistent and ambiguous findings. One reason may be that researchers have not distinguished between well-learned, abstracted information and memorized facts or formulas. Using the distinction between episodic and semantic memory as an heuristic, a procedure was designed to increase the likelihood that student performance on structure tasks was indicative of semantic rather than episodic memory. Significant correlations between measures of cognitive structure and performance on achievement test items were observed. In addition, partial correlations revealed that the measures of cognitive structure are correlated with the achievement items tapping higher-order cognitive processes and not with items tapping lower-order processes. These results are interpreted as supporting the argument that related research needs to be guided by theory-based definitions of cognitive structure.     

Neo-piagetian predictors of achievement in physical science

This article examines the predictive value of the cognitive variables of developmental level, mental capacity, cognitive style, short-term storage space, and numerical inductive reasoning for student achievement in college science. Achievement was analyzed for each of the categories of pure recall, computational, complex items, and total score of a midterm exam as well as for the composite score on a final exam. The sample for this study consisted of a class of 32 nonscience majors enrolled in Physical Science I at the University of Southern Mississippi. The results showed that developmental level was the single best predictor of achievement. Short-term storage space and mental capacity were significant predictors of achievement for computational and complex items, but, as expected from theoretical considerations, not for pure recall items. The degree of field dependence did not well predict performance on pure recall or computational items. The results also indicate that mental capacity and field dependence do not contribute significantly to the variance if developmental level is held constant. The pattern of the predictive power of numerical inductive reasoning parallels, in magnitude, that of mental capacity. The results of this study and its implications indicate that the construct of short-term storage space has great potential to guide classroom practice and the development of instructional materials. A strategy is outlined that would guide curriculum planners and classroom teachers in the development of materials which would allow students to develop complex problem-solving behaviors.     

Mathematics achievement of vietnamese grade 5 pupils

This article reports the results of a national survey of mathematics achievement at the end of primary school in Vietnam. A sample of more than 72000 students were assessed from 61 provinces. The items were matched to the Vietnam Mathematics curriculum for Year 5 students. Using a skills audit of the items, a variable of Vietnamese mathematics development was defined following an item response analysis of the data. Findings reveal that the levels of mathematics achievement were relatively high and that the mathematics curriculum was closely aligned with international trends in mathematics for primary school. There were some areas of concern in that lower achievement levels were persistently aligned with low socio economic groups and these were typically located in three regions of Vietnam, heavily populated by ethnic sub-groups. The views expressed in this paper are those of the authors and do not claim to represent the views of the Vietnamese Government in any way. The article is based on a World Bank project (in Vietnam) on achievement at the end of primary school.     

Investigating the relationship between instructional practices and science achievement in an inquiry-based learning environment

ABSTRACT

This study investigates the discrete effects of inquiry-based instructional practices that described the PISA 2015 construct ‘inquiry-based instruction’ and how each practice, and the frequency of each practice, is related to science achievement across 69 countries. The data for this study were drawn from the PISA 2015 database and analysed using hierarchical linear modelling (HLM). HLMs were estimated to test the contribution of each item to students’ science achievement scores. Some inquiry practices demonstrated a significant, linear, positive relationship to science achievement (particularly items involving contextualising science learning). Two of the negatively associated items (explaining their ideas and doing experiments) were found to have a curvilinear relationship to science achievement. All nine items were dummy coded by the reported frequency of use and an optimum frequency was determined using the categorical model and by calculating the inflection point of the curvilinear associations in the previous model e.g. students that carry out experiments in the lab in some lessons have higher achievement scores than students who perform experiments in all lessons. These findings, accompanied by detailed analyses of the items and their relationships to science outcomes, give stakeholders clear guidance regarding the effective use of inquiry-based approaches in the classroom.     

The dimensions of reading attitude for children in the intermediate grades

This study represents the first phase of an ongoing research program aimed at identifying the dimensions of reading attitude and determining how these dimensions relate to a broader range of variables such as reading achievement, scholastic aptitude, classroom behavior, and cognitive styles. The 150 items which comprise the experimental edition of A Survey of Reading Altitudes (Wallbrown, Brown, & Engin, 1974) were factor analyzed along with measures of scholastic aptitude, reading vocabulary, and reading comprehension. The analysis was based on the responses of 853 intermediate grade students. The attitude items were presented orally to control for reading problems, but the students also had booklets so they could read the items to themselves. Seven independent dimensions of reading attitude and one reading achievement factor were identified, interpreted, and discussed. The findings are encouraging, since they suggest that investigating the expressed reading attitudes of intermediate grade children constitutes a promising line of educational research.