Does calibration reduce variability in the assessment of accounting learning outcomes?

Reliable, consistent assessment process that produces comparable assessment grades between assessors and institutions is a core activity and an ongoing challenge with which universities have failed to come to terms. In this paper, we report results from an experiment that tests the impact of an intervention designed to reduce grader variability and develop a shared understanding of national threshold learning standards by a cohort of reviewers. The intervention involved consensus moderation of samples of accounting students’ work, with a focus on three research questions. First, what is the quantifiable difference in grader variability on the assessment of learning outcomes in ‘application skills’ and ‘judgement’? Second, does participation in the workshops lead to reduced disparity in the assessment of the students’ learning outcomes in ‘application skills’ and ‘judgement’? Third, does participation in the workshops lead to greater confidence by reviewers in their ability to assess students’ skills in application skills and judgement? Our findings suggest consensus moderation does reduce variability across graders and also builds grader confidence.     

Separating arguments from conclusions: the mistaken role of effect size in educational policy research

ABSTRACT

Effect size is the basis of much evidence-based education policymaking. In particular, it is assumed to measure the educational effectiveness of interventions. Policy is being driven by the influential work of John Hattie, the Education Endowment Foundation, and others, which is grounded in this assumption. This article demonstrates the assumption is false and notes that, when criticized, proponents either attempt to inoculate themselves by listing (without checking) assumptions or use the specious reasoning that, however flawed their argument, no-one has disproved their conclusions.     

Silicon Valley versus Silicon Glen: the impact of computers upon teaching and learning: a comparative study

Drawing on data from a major survey on the impact of information technology initiatives on Scottish schools, we assess the extent to which the outcomes resemble those that have already been reported by Larry Cuban in his study of schools in Silicon Valley. We find a large measure of agreement. Scottish teachers and school students mostly have access to computers in their homes and classrooms. The home computers appear to be frequently used. The classroom computers, however, are encountered by students only seldom and when students do use computers in the classroom the activity is often peripheral to the learning process, such as word processing of essays. We discuss some possible explanations for this state of affairs and suggest some implications for policy-making.     

Practical Considerations When Using Benchmarking for Accountability in Higher Education

The qualitative study on which this article is based examined key individuals’ perceptions, both within a research university community and beyond in its external governing board, of how to improve benchmarking as an accountability method in higher education. Differing understanding of benchmarking revealed practical implications for using it as an accountability tool. A change model is presented for enhancing the effectiveness of benchmarking in higher education communities.Sue D. Achtemeier received the B.A. in mathematics and computer science with High Honors and the M.S. in mathematics from Florida State University and the Ph.D. in Higher Education from the University of Georgia. She has held teaching positions in Illinois and Georgia and is now Assistant Director for Institutional Effectiveness for the University of Georgia with particular interest in accreditation and accountability. Ronald D. Simpson is Professor Emeritus of Higher Education and Science Education and Director Emeritus of the Office of Instructional Support and Development at the University of Georgia. He holds degrees in the biological sciences and in science education from the University of Tennessee and the University of Georgia and continues to teach and advise doctoral students in UGA’s Institute of Higher Education.     

The science support scale: Its appropriateness with high school students

Implementing the idea that more emphasis should be placed on student achievement in the affective domain is contingent upon the concurrent development of suitable instruments for the assessment of prescribed criteria. One such instrument, the Schwirian Science Support Scale (Tri-S scale), was reported in a recent NSTA publication as a promising tool for measuring student science support. Recent research using the Tri-S scale with high school pupils showed that scores on this instrument did not increase after the students had taken a tenth grade introductory course in biology. Further analysis indicated students of teachers scoring “high” in science support did not produce higher scores on the Tri-S scale than students studying biology from teachers “low” in science support. Reliability estimates using high school student scores were well below previous estimates using scores from college undergraduates. Factor analysis of inter-item correlations indicated that student interpretation of item meaning did not correspond to the five subtest structure of the Tri-S scale. Findings from this study demonstrate that the Tri-S scale is not an appropriate instrument for measuring attitudinal changes of tenth grade high school students. This study is suggestive of the fact that went and future instruments that purport to measure achievement in noncognitive areas should be carefully analyzed before they are recommended for use with specific populations.     

Apprehending Mathematical Structure: A Case Study of Coming to Understand a Commutative Ring

Abstract algebra courses tend to take one of two pedagogical routes: from examples of mathematics structures through definitions to general theorems, or directly from definitions to general theorems. The former route seems to be based on the implicit pedagogical intention that students will use their understanding of particular examples of an algebraic structure to get a sense of those properties which form the basis of the fundamental definitions. We will explain the transition from examples to abstract algebra as a series of shifts of attention and in this paper we will use a case study to examine the initial shift, which we will call apprehending a structure, and examine how one student came to apprehend the structure of the commutative ring Z₉₉.     

Salary comparisons: New methods for correcting old fallacies

Institutions of higher education place high priority on ensuring that their faculties are adequately paid. Since gauging the adequacy of compensation is a problem of relative measure, this assessment is usually performed by comparing university-level average salaries for a group of peer institutions. Often the potential distortions caused by differences in the rank mix of the faculty are forestalled by comparing average salaries for each of the four ranks. However, few if any institutions take similar steps to prevent the skewing of averages owing to differences in cost of living, differences in distribution of faculty in academic year and twelve-month appointments, and differences in the configuration of the faculty over disciplines. This paper shows that salary comparisons that do not compensate for these variation in staff mix are seriously flawed. A recommended methodology is demonstrated.