Monitoring Student Achievement for Accountability: The Demonstration of a Model

Abstract

This study demonstrated a procedural model that can be applied by any school to assess, guide, and account for the progress of its students as well as to analyze its own effectiveness. The model uses equivalent achievement tests to monitor student achievement in subject areas at grade levels, between grade levels, and across subgroups of students. Multiple regression analyses of test scores between grades identify factors associated with achievement Using sixth and eighth grade Comprehensive Tests of Basic Skills scores in a matched longitudinal sample of 208 students, the study found small differences in average achievement between boys and girls. Differences between corresponding sixth and eighth grade test means were higher in mathematics than in language. From the sixth grade to the eighth, there was a widening gap in average achievement between high and low I.Q. groups. In multiple regressions of eighth grade test scores on sixth grade measures, I.Q., study skills, and reading were prevalent in the regression equations, but clusters of measures associated with achievement differed between high and low’ LQ. groups. The results of the study have implications for developing and evaluating the achievement of students with varying mental abilities.     

Using student notebooks to measure opportunity to learn in Botswana and South African classrooms

This article describes how a detailed notebook analysis was used to assess and compare the opportunity to learn of a sample of grade 6 students from 126 classes in South East Botswana and North West Province, South Africa. Students’ mathematics notebooks provided the main data source for estimating how much time is spent on the subject during the 6th grade year, and what mathematics is covered relative to what the study’s student test asked the sample of students. The notebook analysis shows that in sample classes in both countries, most teachers were giving an average number of mathematics lessons that was considerably lower than the number officially programmed under each country’s ministry guidelines. It also reveals that, on average, teachers in both samples of classes covered just over a third of the possible topics related to the test items between the pre-test and post-test. Nevertheless, substantial differences in students’ opportunities to learn in the two countries are evident.     

Exploring the factors teachers consider in determining students’ grades

The purpose of this study was to investigate the specific factors teachers consider when assigning students’ report card grades. Data were gathered from 943 K-12 teachers from five school districts in a southeastern state in the United States who completed the Teachers’ Grading Practices Survey. Analyses focused on how teachers weigh different factors in determining report card grades, and if these factors and weights differ among teachers who teach at different grade levels and have different amounts of classroom experience. Results revealed statistically significant differences among teachers at different grade levels but no differences associated with teachers’ years of experience and no interaction effect. Differences by grade level were evident in teachers’ consideration of both cognitive and non-cognitive factors of students’ performance. Implications are discussed for improving grading policies and practices, teacher education and teacher professional development.     

Primary school students’ strategies in early algebra problem solving supported by an online game

In this study we investigated the role of a dynamic online game on students’ early algebra problem solving. In total 253 students from grades 4, 5, and 6 (10–12 years old) used the game at home to solve a sequence of early algebra problems consisting of contextual problems addressing covarying quantities. Special software monitored the students’ online working when solving the problems. Before and after the intervention a paper-and-pencil test on early algebra was administered. The data analysis revealed that the online working contributed to the students’ early algebra performance. There was a significant gain in performance across all grades. The highest effect was found in grade 6. Out of the three strategy profile clusters that could be distinguished in the whole sample, the cluster dominated by using extreme values and the cluster characterized by the trial-and-error strategy were most influential on the gain in early algebra performance. The students’ level of online working, which was defined as a combination of online involvement and strategy use, appeared to have a marginally significant effect on the gain score for the total sample. Per grade there was no significant effect, yet the levels of online working were significantly related to grade. Free playing was mostly performed in grade 4, looking for answers in grade 5, and exploring relations slightly more in grades 5 and 6. About 17 % of the effect of grade on the gain score was mediated by the level of online working.     

USE OF OPTICALLY SCORED TEST ANSWER SHEETS WITH YOUNG CHILDREN

By testing a sample of 840 school children from an urban school district, the authors examined the practical limitations of the use of an optically scored answer sheet. The independent variables were grade (two through nine) and degree of instruction and practice (three levels). The dependent variable was the number of easy items (embedded within a set of ordinary test items) that were correctly answered. The easy items, which were different for the various grades, were selected because other children, comparable to those used in the experiment, were able to answer them virtually without error when answering on a test booklet rather than on the answer sheet. The findings indicated that students in grades two and three, in the fall of the year, were unable to make valid responses. Students in grades four and five seemed able to make valid responses only after receiving specific instructions and a practice session. Above grade five, students made valid responses regardless of the types of instructions they received.     

Measurement Characteristics of the lEA Reading Literacy Scales for 9- and 10-Year-Olds at Country and Individual Levels

This article presents an approach for studying measurement characteristics of instruments designed for comparative studies of educational achievement. It presents a reanalysis of data from 22 countries and 73,818 9- and 10-year-old students from the IEA reading literacy study. Using 2-level confirmatory factor analysis procedures derived by Muthén (1990, 1994), it is demonstrated how the overall variance in performance on items from 3 domains—-Documents, Narrative Prose, and Expository Text—may be decomposed into multiple latent sources of variance at country and individual levels. For the Documents items, a 1-factor model was fitted at both levels. For Narrative Prose and Expository Text items, 2 latent variables (a general dimension of reading performance and a factor capturing high versus low performance on passages towards the end of the tests) were fitted at both country and individual levels. For the Documents dimension, estimates of country scores were close to those of the original analysis; however, for performance in the Narrative Prose and Expository Text domains, differences between the original analysis and the reanalysis were noted for many countries.     

A cross-national survey of cultural expectations and sex-role standards in reading

Cultural expectations and sex-role standards about reading in Canada, Denmark, England, Finland, Israel, Japan, and U.S.A. were investigated. A picture test showing various activities and objects including reading and books was administered in each country to six sub-samples that approximated to the levels: students in grades I, IV, VIII, and XII, college students, and adults. Subjects had to ascribe the activity or object as being appropriate for either a boy or a girl. The results were consistent with the hypothesis that cultural expectations and sex-role standards about reading are congruent within one country but that they differ between countries. In particular, boys in Canada and the U.S.A. rapidly learn to perceive reading as a feminine activity, whereas Danish and Japanese males at all ages consistently view reading as acceptable masculine behaviour.     

Relationship between Science Teaching Practices and Students’ Achievement in Singapore,Chinese Taipei,and the US: An Analysis Using TIMSS 2011 Data

The assumption that inquiry-based instruction is more effective in influencing student science achievement than traditional didactic teaching has been the driving force of science education reform in recent decades and in many countries. However, the empirical relationship between these two kinds of science teaching and student science performance is not soundly established, which is worth a careful examination. Framed through the theoretical perspectives of inquiry-based instruction and culturally relevant pedagogy, using a two-level hierarchical linear modeling (HLM) approach and simultaneous multiple regression, this study examines the above relationship using the Trends in International Mathematics and Science Study (TIMSS) 2011 8th grade dataset from Singapore, Chinese Taipei, and the US. The study found that for the low-performing students, none of the inquiry-based teaching practice items measured had a significant relationship with the science achievements at any performance levels of students in any country/region except for the case of two inquiry-based teaching practice items that were positively related to Chinese Taipei students’ achievements. No didactic teaching practice items were associated with the Singapore students’ science achievement, three of these practice items were found negatively related to Chinese Taipei students’ science achievement, and one traditional didactic teaching practice was negatively related to the science achievement of U.S. students. However, for medium- and high-performing students, none of these inquiry-based or traditional didactic science-teaching practices were found to be positive predictors of science performance in all three countries/regions. However, in the case of Chinese Taipei, one didactic teaching practice item was negatively related with the medium level performing students’ achievement and two didactic teaching practices were found to hinder high-performing students’ science achievements.     

Relationships between writing motivation, writing activity, and writing performance: effects of grade, sex, and ability

A convenience sample of 618 children and adolescents in grades 4 through 10, excluding grade 8, were asked to complete a writing motivation and activity scale and to provide a timed narrative writing sample to permit an examination of the relationships between writing motivation, writing activity, writing performance, and the student characteristics of grade, sex, and teacher judgment of writing ability. Female students and older students wrote qualitatively better fictional stories, as did students with higher levels of writing ability based on teacher judgment. With respect to writing activity, more frequent writing in and out of school was reported by girls, better writers, and younger students. In a path analysis, grade and sex directly influenced writing activity, while sex, teacher judgment of writing ability, and writing activity directly influenced some aspects of writing motivation. Overall, teacher judgment of writing ability, grade level, and motivational beliefs each exerted a significant direct positive influence on narrative quality, whereas performance goals exerted a significant direct negative impact on quality.     

The Development and Validation of a Formula for Measuring Single-Sentence Test Item Readability

This study describes the development and validation of the Homan-Hewitt Readability Formula. This formula estimates the readability level of single-sentence test items. Its initial development is based on the assumption that differences in readability level will affect item difficulty. The validation of the formula is achieved by (a) estimating the readability levels of sets of test items predicted to be written at 2nd- through 8th-grade levels; (b) administering the tests to 782 students in grades 2 through 5; (3) using the class means as the unit of analyses and subjecting the data to a two-factor repeated measures ANOVA. Significant differences were found on class mean performance scores across the levels of readability. These results indicated that a relationship exists between students'reading grade levels and their responses to test items written at higher readability levels.     

Klassenwiederholen in PISA-I-Plus: Was lernen Sitzenbleiber in Mathematik dazu?

This study analyses the effects which repeating a class has on ninth grade students’ development of mathematical competency. The following research questions were addressed: How many students repeat grades in the different types of schools? How do students who repeat a grade differ from those who do not in their performance and background characteristics? How much extra mathematics do students repeating a grade learn in one school year? What are the differences between various types of school? Can students with poor mathematics grades in particular profit from repeating a grade? The sample is a sub-sample of the PISA-I-Plus study and comprises N = 360 ninth grade students. The total sample of PISA-I-Plus is representative for all ninth/tenth grade students from the different school types in Germany. The data survey was carried out in the ninth grade and then repeated after the students had repeated a year. The results document differences in the amount of grade repeat quotas between types of school. Furthermore, compared to students not repeating, those repeating a grade had lower mathematics (d = 1.02) and german (d = 1.14) grades, a lower level of mathematical literacy (d = 0.51), and lower test results with regard to basic cognitive abilities (d = 0.32). In terms of the development of mathematical literacy, the students repeating a grade could improve by an average of 23 points (d = 0.27) on the PISA mathematics scale. However, the results identify 38 percent of students repeating a grade who do not make any significant improvement in mathematics or even get worse. A differentiation according to school types shows that students repeating a grade in integrated comprehensive secondary schools and in schools with several educational levels in particular do not, on average, show any noteworthy improvement in their mathematical literacy. The analysis of the school grades received in mathematics shows that students whose mathematics grades are unsatisfactory do not benefit more from repeating a grade than students whose mathematics performance has been rated as being “satisfactory” or better. The article concludes with a discussion of the possible consequences of changing the way in which repetitions of grades are dealt with.     

Visual Chunking Skills of Hong Kong Children

This study investigated the development of visual chunking skills in the processing of Chinese characters among Hong Kong pupils. One-hundred-seventy-nine primary school students from first, second and fourth grades were administered a character copying task. Children as young as 6 years of age were aware of character units and were able to apply visual chunking strategies when processing characters. Children in higher grades performed better than those in lower grades on every character type, and the types of errors they made revealed that their chunking level was higher than that of younger children. Differences between ability groups emerged in second grade and vanished in fourth grade, suggesting that children with a lower reading ability are slower to develop advanced chunking skills.     

The association of precollege use of calculators with student performance in college calculus

This study investigates how the use of calculators during high school mathematics courses is associated with student performance in introductory college calculus courses in the USA. Data were drawn from a nationally representative sample of 7087 students enrolled in college calculus at 134 colleges and universities. They included information about students’ demographics, standardized test scores, and high school mathematics course enrollment and performance. Factor analysis reduced ten items describing high school calculator usage to two composites: how extensively calculators were employed and teacher-imposed restrictions on their use. Hierarchical linear models predicted students’ college calculus grades, reported by their professor, while controlling for differences between colleges and student backgrounds. The more extensively students had used calculators in high school, the lower their grade in college calculus. However, students earned higher college calculus grades to the extent that their high school teachers had limited calculator use on quizzes and exams and had restricted calculator use until paper-and-pencil methods had been mastered, which offset the negative association of extensive calculator use with grades. The effect sizes of both calculator composites were very small. Overall, the findings raise doubts about any substantial long-term effects on college mathematics performance of calculator use in high school.     

Trends of progression of student level of reasoning and generalization in numerical and figural reasoning approaches in pattern generalization

This study explored progression of students’ level of reasoning and generalization in numerical and figural reasoning approaches across grades and in different pattern generalization types. An instrument that included four figural patterns was administered to a sample of 1232 students from grades 4 to 11 from five private schools. The findings suggest that there was progressive development in the level of reasoning and generalization in each reasoning approach across clusters of grades. The level of reasoning and generalization in figural approach was higher than that for numerical approach in each grade. In addition, the level of reasoning and generalization for each approach and in each grade was not limited to one level but to several levels. The type of generalization influenced the progression of students’ level of reasoning and generalization in each approach.     

Grading as a Reform Effort: Do Standards‐Based Grades Converge With Test Scores?

Standards‐based progress reports (SBPRs) require teachers to grade students using the performance levels reported by state tests and are an increasingly popular report card format. They may help to increase teacher familiarity with state standards, encourage teachers to exclude nonacademic factors from grades, and/or improve communication with parents. The current study examines the SBPR grade–state test score correspondence observed across 2 years in 125 third and fifth grade classrooms located in one school district to examine the degree of consistency between grades and state test results. It also examines the grading practices of a subset of 37 teachers to determine whether there is an association between teacher appraisal style and convergence rates. A moderate degree of grade–test score convergence was observed using three agreement estimates (coefficient kappa, tau‐b correlations, and classroom‐level mean differences between grades and test scores). In addition, only small amounts of grade–test score convergence were observed between teachers; a much greater proportion of variance lay within classrooms and subjects. Appraisal style correlated weakly with convergence rates, but was most strongly related to assigning students to the same performance level as the test. Therefore using recommended grading practices may improve the quality of SBPR grades to some extent.     

Applying the many‐facet Rasch model to evaluate PowerPoint presentation performance in higher education

This study investigated the usefulness of the many‐facet Rasch model (MFRM) in evaluating the quality of performance related to PowerPoint presentations in higher education. The Rasch Model utilizes item response theory stating that the probability of a correct response to a test item/task depends largely on a single parameter, the ability of the person. MFRM extends this one‐parameter model to other facets of task difficulty, for example, rater severity, rating scale format, task difficulty levels. This paper specifically investigated presentation ability in terms of items/task difficulty and rater severity/leniency. First‐year science education students prepared and used the PowerPoint presentation software program during the autumn semester of the 2005–2006 school year in the ‘Introduction to the Teaching Profession’ course. The students were divided into six sub‐groups and each sub‐group was given an instructional topic, based on the content and objectives of the course, to prepare a PowerPoint presentation. Seven judges, including the course instructor, evaluated each group’s PowerPoint presentation performance using ‘A+ PowerPoint Rubric’. The results of this study show that the MFRM technique is a powerful tool for handling polytomous data in performance and peer assessment in higher education.     

Differences in science students' view of ideal and actual role behavior according to success and gender

Information about how success and gender affect students' views of ideal and actual classroom role behavior can help both researchers and teachers better understand classroom components such as achievement and curriculum. A 20-item double Q sort was used to measure differences in perceptions of high school science students according to letter grades and gender. Individual Q sort item rankings of 160 students were tested for significant differences according to letter grade received; item ratings were compared according to gender for 215 students. Differences in perception according to success were found for both ideal and actual behavior; 8 and 5 items, respectively, out of each 20-item sort were found to be significant at the p = < 0.05 level. No such overall patterns of difference in view were found between boys and girls, although three ideal student items and one actual self-report item were found to be significantly different at the p = < 0.05 level.     

The role of engagement and academic behavioral skills on young students’ academic performance—A validation across four countries

The aim of this study is to validate an instrument measuring students’ academic behavioral skills and engagement—skills identified as vital for student achievement. We inspect the reliability and validity of the survey with respect to item fit, factorial structure, relations with academic performance, and the fairness of the items across student groups. The fairness analyses are critical to making valid comparisons between groups and across countries. Data comprising 8520 grade 10 students from four countries were analysed using item response theory. We found that both scales were multidimensional, acted fairly across students’ gender, country, immigrant-, and socio-economic background (after removing four items), and were positively and significantly correlated with self-reported and performance-based academic performance.     

Relation of field independence and test-item format to student performance on written piagetian tests

In this study we have investigated the relationship between the field-dependence-independence (FDI) dimension as measured by the Group Embedded Figures Test (GEFT) and subject performance on the Longeot test, a pencil-and-paper Piagetian test, through the open or closed format of its items. The sample consisted of 141 high school students. Correlation and variance analysis show that the FDI dimension and GEFT correlate significantly on only those items on the Longeot test that require formal reasoning. The effect of open- or closed-item format is found exclusively for formal items; only the open format discriminates significantly (at the 0.01 level) between the field-dependent and -independent subjects performing on this type of item. Some implications of these results for science education are discussed.