The failure of a decision support system: inconsistency in test grading by teachers

This paper examines the interaction between the contribution of established criteria to objective assessment of examinations, and the effect of mental fatigue induced during the process. Copies of 31 compositions, previously awarded a grade of 80 percent by specialists, were stacked in three randomly arranged sequences. Sixty teachers graded the compositions in sequential order. Thereafter, they re-evaluated every essay according to each of six criteria. These scores were used to generate a series of accumulative grades. In both scoring procedures, the grades rose in a time-dependent fashion, indicating impairment of judgment, apparently as a result of mental fatigue. Differences between the two assessments suggest that the criteria were not used efficiently in the original evaluation of the teachers. This may be explained by the phenomenon of bounded rationality.     

Elementary Teachers' Classroom Assessment and Grading Practices

Abstract

The authors investigated the assessment and grading practices of over 900 Grades 3-5 teachers representing urban, suburban, and rural schools. Teachers indicated the extent to which they used various factors to grade students, the types of assessments used, the cognitive level of assessments, and the grades awarded. Teachers appeared to conceptualize 6 major factors when they graded students; they placed the greatest weight on academic performance and academic-enabling behaviors, such as effort and improvement, and much less emphasis on homework, comparisons with other students, grade distributions of other teachers, and borderline cases. The teachers used 3 types of assessments—constructed-response, objective, and teacher-made major examinations; they differentiated between recall and higher level cognitive skills. However, there were few relationships between assessment and grade level, subject matter assessed, and grades awarded. Results are discussed in light of other research, indicating that teachers use a “hodgepodge” of factors when assessing and grading students.     

Beliefs and Influences About Grading Elicited From Student Performance Sketches

Undergraduate education majors, student teachers, and experienced teachers (N = 326) responded to two written sketches depicting different levels of student effort. They decided on a grade, rated the importance of achievement and nonachievement factors, and wrote a defense of their grading decision. Overall, experienced teachers tended to give lower grades. Lower grades were awarded more when a female with high effort and low aptitude was portrayed or when a male with low effort and high aptitude was portrayed. Preservice and experienced teachers considered cognitive, nonachievement dispositions to be valid educational outcomes contributing to grade variation. There was a dichotomy of beliefs about borderline grading: Participants were struggling with being a judge or an advocate when the grade was used to punish inadequate effort or to reward improvement. An understanding of the interplay of beliefs, attitude, and social judgment with assessment is essential if teachers are to develop grading strategies that communicate the diverse learning outcomes expected of students.     

Grading as a Reform Effort: Do Standards‐Based Grades Converge With Test Scores?

Standards‐based progress reports (SBPRs) require teachers to grade students using the performance levels reported by state tests and are an increasingly popular report card format. They may help to increase teacher familiarity with state standards, encourage teachers to exclude nonacademic factors from grades, and/or improve communication with parents. The current study examines the SBPR grade–state test score correspondence observed across 2 years in 125 third and fifth grade classrooms located in one school district to examine the degree of consistency between grades and state test results. It also examines the grading practices of a subset of 37 teachers to determine whether there is an association between teacher appraisal style and convergence rates. A moderate degree of grade–test score convergence was observed using three agreement estimates (coefficient kappa, tau‐b correlations, and classroom‐level mean differences between grades and test scores). In addition, only small amounts of grade–test score convergence were observed between teachers; a much greater proportion of variance lay within classrooms and subjects. Appraisal style correlated weakly with convergence rates, but was most strongly related to assigning students to the same performance level as the test. Therefore using recommended grading practices may improve the quality of SBPR grades to some extent.     

Student Grades and Average Ratings of Instructional Quality: The Need for Adjustment

Abstract

The authors compared the average grades given in 165 behavioral and social science courses with the average ratings given by students to the instructors who taught the courses. Significant positive correlations were found between the average ratings for instructional quality and the average grades received by students. The courses in which the average grades were the highest were also those in which students gave teachers the highest ratings. Among possible reasons for the correlations are that better teachers attracted better students or that quality teachers provided more effective instruction, resulting in more student learning and, thus, higher average grades. Another explanation is that most college students tend to bias their ratings of instructional quality in favor of teachers who grade leniently (I. Neath, 1996). If correct, the latter reasoning begins to explain why the widespread use of student evaluations in the United States in recent decades has been accompanied by increases in the average grades that university students received. To prevent grade inflation, and particularly to avoid rewarding and promoting instructors who use increasingly lax grading standards, administrators should adjust student ratings of instructional quality for the average grades given for a course. In general, only courses near the extremely high and low ends in terms of students' average grades were significantly affected by the statistical adjustment.     

Examining the grading practices of teachers

Despite the recommendations of some measurement specialists, teachers do not always assign grades based on achievement only. The primary purpose of this study is to clarify the meaning of grades, and to examine some of the factors teachers consider when assigning final grades with a focus on borderline cases. The sample consisted of 516 American public school teachers, selected via stratified random sample in a major metropolitan school district in the Southeast. A 53-item survey using Guttman’s mapping sentences, previously piloted in a separate school district, was created and distributed. Teachers were provided with scenarios that described student ability, achievement, behavior, and effort and the teacher was asked to assign both a numerical and letter grade. A four-way between-subjects ANOVA was conducted with the student characteristics ability, achievement, behavior, and effort as independent variables and final grade as the dependent variable. Findings demonstrate that teachers abided by the official grading policy of the participating school district assigning grades based primarily on achievement under most circumstances, however, in some borderline cases teachers report considering non-achievement factors. Implications for pre-service and in-service professional development are discussed.     

Making a Difference: Using peers to assess individual students' contributions to a group project

Overcoming the potential dilemma of awarding the same grade to a group of students for group work assignments, regardless of the contribution made by each group member, is a problem facing teachers who ask their students to work collaboratively together on assessed group tasks. In this paper, we report on the procedures to factor in the contributions of individual group members engaged in an integrated group project using peer assessment procedures. Our findings demonstrate that the method we used resulted in a substantially wider spread of marks being given to individual students. Almost every student was awarded a numerical score which was higher or lower than a simple group project mark would have been. When these numerical scores were converted into the final letter grades, approximately one-third of the students received a grade for the group project that was different from the grade that they would have received if the same grade had been awarded to all group members. Based on these preliminary findings we conclude that peer assessment can be usefully and meaningfully employed to factor individual contributions into the grades awarded to students engaged in collaborative group work.     

Grading and the Ethos of Effort

Using quantitative and qualitative data from a project to improve middle-school girls' engagement with science, mathematics and technology, this study examined the ways in which grading practices worked to augment the messages about achievement and compliance conveyed in more and less troubled schools. In particular, the study revealed differences in the extent to which different kinds of schools embedded achievement and non-achievement factors in the grades awarded to middle school girls. The study linked these differences to school-level differences in teachers' overall orientation toward grading. Grading in the more troubled schools - those with a more custodial orientation toward control - appeared to reflect teachers' tendency to subscribe to an 'ethos of effort'. Acting on the basis of this 'ethos', teachers in the troubled schools tended to confound achievement and non-achievement factors in the determination of report-card grades. In the less troubled school, by contrast, teachers did not rely as heavily on non-achievement factors when they determined students' grades. Given limitations associated with sampling, however, findings from the study were viewed cautiously. Nevertheless, the research appeared to support the claim that there could be systematic associations between school-level climate characteristics and particular instructional practices. This revised version was published online in July 2006 with corrections to the Cover Date.     

Does Student Language Ability Affect the Assessment of Teacher Language Ability?

This article arises from an investigation into the classroom language assessment (CLA) component of the Hong Kong English language benchmark test for lower secondary (Grades 7–9) teachers of English as a second language (ESL). After some background to the benchmarking initiative in Hong Kong and to the CLA test, the article describes a small-scale study into the effect of student levels of ability on grades awarded to teachers working in schools with differing ability intakes. In the study, a group of ESL teachers were videotaped teaching ESL classes at two different student ability levels, with the two sets of videos assessed by trained CLA assessors. Results indicate that, although there is some variety in the grade awarded to the two different classes taught, variation in grades may well be attributable to individual teacher performance rather than to external factors such as the ability levels of the students being taught. The grades awarded were generally consistent, irrespective of whether classes of high-ability students or low-ability students were being taught.     

Teachers' Assessment Practices: Preparation,Isolation, and the Kitchen Sink

A sample of 143 midwestern elementary and secondary school teachers from a variety of practice settings responded to a survey and provided comments regarding their assessment practices The purpose of the survey was to collect background (demographic) information on the teachers and information on several assessment-related practices, including frequency with which teachers assign routine class assignments, types of marks used to report student performance, frequency and grading of major assignments and tests, source of classroom tests, kinds of marks used, methods used to combine marks, meaning of grades, teachers' knowledge and perceptions regarding district grading policies, and teachers' awareness of the grading policies of their peers. Interviews with the teachers provided additional insights into their practices. Results indicated that teachers' assessment practices were highly variable and unpredictable from characteristics such as practice setting, gender, years of experience, grade level, or familiarity with assessment policies in their school district. Teachers generally claim to consider and incorporate a variety of objective and subjective factors when assigning grades on assignments, assessments, and report cards, synthesizing diverse kinds of information about achievement in ways that tend to maximize the likelihood that students will achieve high grades. Only about one half of the teachers surveyed indicated that they were aware of their districts' policies on grading; most were not aware of the assessment practices of their colleagues. Many teachers seemed to have individual assessment policies that reflected their own individualistic values and beliefs about teaching. Recommendations for making grades more meaningful ways of communicating about student performance are suggested.     

Vergleichbarkeit von Abiturleistungen

This article examines differences in the mathematics and English proficiency of academic-track students in Baden-Württemberg (N?=?3526) and Hamburg (N?=?3734), investigating whether and to what extent these differences are reflected in the Abitur grades the students are awarded. The article also examines the extent to which scores in centrally conducted examinations provide better comparability than do coursework grades. Multilevel analyses predicting coursework grades in mathematics revealed clear between-state differences, with students in Hamburg being awarded higher mathematics grades than comparably able students in Baden-Württemberg. These differences are partly attributable to frame of reference effects and their impact on teachers’ grading practices. No corresponding between-state differences were found for English. Mathematics examination scores provided a much better measure of student achievement than mathematics coursework grades. The findings are discussed in terms of meritocratic access to sought-after university and training places.     

A study of the relationships between faculty characteristics,subject field,and course grading patterns

This paper describes an analysis of faculty grading patterns at a large public university. It introduces a methodology to both normalize and stabilize grade data by courses. Using this method, we report on grading patterns for over 1,000 teachers, giving more than 40,000 grades in approximately 2,000 courses. The findings indicate that the academic field of the course is strongly related to the types of grades assigned. Courses emphasizing quantitative and factual learning tend to have assigned lower grades. The higher grades are found in career-oriented courses, such as teacher training. This was shown in separate analyses for undergraduate and graduate courses. For undergraduate courses, lower grades were found in freshman and sophomore courses as compared with junior and senior courses. The academic credentials and personal characteristics of the teachers were only moderately related to grades. Temporary teachers had higher grades in their courses, but such personal characteristics as sex, marital, and minority status were not systematically related to grades given. There was a tendency in undergraduate courses that as class size increased, the proportion of higher grades decreased.     

How Quality of Writing Instruction Impacts High-risk Fourth Graders’ Writing

From a larger longitudinal study of 610 fourth graders in 17 inner city schools, 40 students were randomly selected from 10 classrooms rated high (i.e., top quartile) or low (i.e., bottom quartile) in quality of writing instruction in grades 3 and 4. The written compositions of these students were scored in three ways: (1) according to a rating scale within a reliable scoring rubric, (2) according to countable surface features such as words correctly sequenced, and (3) according to the frequency of specific phonological, morphosyntactic, and orthographic errors in the children’s writing. A multivariate analysis of variance was conducted to examine whether quality of writing instruction in grades 3 and 4 predicted students’ writing performance at the end of grade 4. Students who received high quality instruction in fourth grade wrote longer compositions with more correctly spelled words than those who had poor quality writing instruction. There was a tendency for two years of quality instruction to be better than one, and, among students who had poor quality instruction in grade 3, compositions were longer in grade 4 when they received quality instruction in fourth grade. Foundational problems of language formulation, production and representation, however, were ubiquitous across the sample. Although these students were within the average range on standardized reading tests, spelling and writing were not developing at average levels. The study confirms the urgent need for more and better writing instruction for high risk, minority children.     

Exploring the factors teachers consider in determining students’ grades

The purpose of this study was to investigate the specific factors teachers consider when assigning students’ report card grades. Data were gathered from 943 K-12 teachers from five school districts in a southeastern state in the United States who completed the Teachers’ Grading Practices Survey. Analyses focused on how teachers weigh different factors in determining report card grades, and if these factors and weights differ among teachers who teach at different grade levels and have different amounts of classroom experience. Results revealed statistically significant differences among teachers at different grade levels but no differences associated with teachers’ years of experience and no interaction effect. Differences by grade level were evident in teachers’ consideration of both cognitive and non-cognitive factors of students’ performance. Implications are discussed for improving grading policies and practices, teacher education and teacher professional development.     

Dimensions of Norm-Referenced Compulsory School Grades and their Relative Importance for the Prediction of Upper Secondary School Grades

Irrespective of the grading system, grades are the most valid instrument for predicting educational success. Previous studies have shown that criterion-referenced compulsory school grades are multidimensional, reflecting subject-specific dimensions and a common grade dimension, both of which contribute to the predictive validity of grades. This suggests that in addition to knowledge and skills, grades reflect other aspects which might have importance for the prediction of educational success. The purpose of this study was to investigate, using structured equation modeling, whether norm-referenced compulsory school grades display similar patterns of dimensionality and predictive validity to criterion-referenced grades. Possible differences due to gender and parents' education were considered. Participants were 3855 students born in 1972. The results showed that norm-referenced grades are multidimensional, and that both the subject-specific and common grade dimensions contribute to predicting educational success. In the common grade dimension, girls and students with higher educational backgrounds were favored.     

Secondary Teachers' Classroom Assessment and Grading Practices

What variables do secondary teachers use to assign grades? What is the relationship between classroom assessment practices and grading? Is the form of classroom assessment related to grade level, subject area, and student ability?     

Do central examinations lead to greater grading comparability? A study of frame-of-reference effects on the University entrance qualification in Germany

In Germany, the Abitur grades awarded at the end of upper secondary education are critical in the allocation of sought-after university places. Drawing on a representative sample of 3526 grade 13 Abitur students in the German state of Baden-Württemberg, this article examines whether and to what extent grading is affected by the mean achievement of the school serving as frame of reference (“group-referenced grading”), and to what extent this influence differs for coursework and examination grades in mathematics and English as a foreign language. Overall, the results indicate that the higher level of standardization of the central Abitur examinations makes examination grades less susceptible to frame-of-reference effects than are coursework grades.     

Teachers' Grading Practices: Meaning and Values

Classroom teachers do not always follow recommended grading practices. Why not? It is possible to conceptualize this question as a validity issue and ask whether teachers' concerns over the many uses of grades outweigh concerns about the interpretation of grades. The purpose of this study was to investigate the meaning classroom teachers associate with grades, the value judgments they make when considering grades, and whether the meaning or values associated with grades differed by whether teachers had measurement instruction. A sample of 84 teachers, 40 with and 44 without measurement instruction, responded to classroom grading scenarios in two ways–with multiple-choice responses indicating what they would do and with written responses to the question, “Why did you make this choice?” A coding scheme based on Messick's (1989a, 1989b) progressive matrix of facets of validity was used for quantitative and qualitative analyses of written responses. The meaning of grades is closely related to the idea of student work; grades are pay students earn for activities they perform. The relationship of this notion to classroom management should be investigated. Teachers do make value judgments when assigning grades and are especially concerned about being fair. Teachers also are concerned about the consequences of grade use, especially for developing student self-esteem and good attitudes toward future school work. Measurement instruction made very little difference, although it did reduce the amount of self-referenced grading reported.     

Announced Repetitive Tests as a Basis for Self-Directed Study and Evaluation

College students voluntarily took all their courses or one course on a pass-fail basis. The mean grade point average (GPA) before conversion to pass-fail for freshmen taking all their courses on a pass-fail basis was 1.67 (C-), which is significantly lower than the 2.26 (C+) for controls who wanted but were denied pass-fail grading. Even after returning to conventional grading the former pass-fail students continued to get significantly lower grades than controls. Juniors taking one course on a pass-fail basis received significantly lower grades, before conversion, in their pass-fail course (mean 2.07) than did controls who wanted but were denied pass-fail grading (mean 2.40). There was no compensatory improvement in the grades received in non-pass-fail courses.     

Teachers’ grading decisions and practices across cultures: Exploring the value,consistency, and construction of grades across Canadian and Chinese secondary schools

Grades are the dominant currency that enables student migration patterns; in particular, the recent upsurge of Chinese students studying and settling in Canada. Given the use of grades for student promotion, mobilization, and admission into educational programs internationally, there is an urgent need to understand the validity of grades across learning contexts. This study explored 35 Canadian and Chinese secondary school teachers’ grading decisions and practices through nine focus groups. Following inductive analyses, findings indicated that teachers primarily valued fairness as an overarching driver of decision-making when generating and considering grades. Teachers’ considerations of fairness centred around four emergent themes: (a) context and classroom management, (b) learning values: grades as academic enablers, (c) policy and external pressures, and (d) consequences of grade use. The overarching thread common among both Canadian and Chinese teachers was their reported challenge in maintaining fair grading practices.