Separating arguments from conclusions: the mistaken role of effect size in educational policy research

ABSTRACT

Effect size is the basis of much evidence-based education policymaking. In particular, it is assumed to measure the educational effectiveness of interventions. Policy is being driven by the influential work of John Hattie, the Education Endowment Foundation, and others, which is grounded in this assumption. This article demonstrates the assumption is false and notes that, when criticized, proponents either attempt to inoculate themselves by listing (without checking) assumptions or use the specious reasoning that, however flawed their argument, no-one has disproved their conclusions.     

The instructional design portfolio:

Conclusion In order to make PIHnet forum more usable and useful, we recently redesigned the forum tool based on findings from a survey, interviews and observations of user posting patterns. We also developed a journal tool to assist teachers with their personal reflection. Interviews with four users showed that the new forum was easy to use and effectively helped them communicate with other members. We are still developing the journal tool, so user experiences with the new journal tool have not been investigated yet. We will continue to improve the tools based on critical feedback from actual users.     

The Connection Between Teaching Methods and Attribution Errors

We collected data at a large, very selective public university on what math and science instructors felt was the biggest barrier to their students’ learning. We also determined the extent of each instructor’s use of research-based effective teaching methods. Instructors using fewer effective methods were more likely to say the greatest barrier to student learning was the internal deficiencies of the students (the “fundamental attribution error”). They listed deficiencies such as poor preparation and work ethic. In total, 37 % of the instructor attributions were to student deficiencies, but this fraction varied dramatically between departments.     

Silicon Valley versus Silicon Glen: the impact of computers upon teaching and learning: a comparative study

Drawing on data from a major survey on the impact of information technology initiatives on Scottish schools, we assess the extent to which the outcomes resemble those that have already been reported by Larry Cuban in his study of schools in Silicon Valley. We find a large measure of agreement. Scottish teachers and school students mostly have access to computers in their homes and classrooms. The home computers appear to be frequently used. The classroom computers, however, are encountered by students only seldom and when students do use computers in the classroom the activity is often peripheral to the learning process, such as word processing of essays. We discuss some possible explanations for this state of affairs and suggest some implications for policy-making.     

Practical Considerations When Using Benchmarking for Accountability in Higher Education

The qualitative study on which this article is based examined key individuals’ perceptions, both within a research university community and beyond in its external governing board, of how to improve benchmarking as an accountability method in higher education. Differing understanding of benchmarking revealed practical implications for using it as an accountability tool. A change model is presented for enhancing the effectiveness of benchmarking in higher education communities.Sue D. Achtemeier received the B.A. in mathematics and computer science with High Honors and the M.S. in mathematics from Florida State University and the Ph.D. in Higher Education from the University of Georgia. She has held teaching positions in Illinois and Georgia and is now Assistant Director for Institutional Effectiveness for the University of Georgia with particular interest in accreditation and accountability. Ronald D. Simpson is Professor Emeritus of Higher Education and Science Education and Director Emeritus of the Office of Instructional Support and Development at the University of Georgia. He holds degrees in the biological sciences and in science education from the University of Tennessee and the University of Georgia and continues to teach and advise doctoral students in UGA’s Institute of Higher Education.     

Is the Door-in-the-Face a Concession?

The Door-in-the-Face (DITF) sequential message strategy was investigated in a three-study analysis of existing experimental findings. The current study predicted there would be a positive relationship between concession size and compliance rates in DITF studies. Study 1 included 25 comparisons where size of concession was quantifiable as measured by percentage reduction from initial to target request in the DITF condition. Study 2 data relied on a panel of undergraduate students to provide an index of concession size in 12 additional observations. A third study validated the panel procedure of rating concession size and also provided 9 additional independent observations from the pool of published studies on DITF. Results from each study indicated a positive relationship between concession size and effect size (r = 0.35, 0.55, 0.68, respectively). Study findings provide support for reciprocal concessions explanation for DITF effects.     

The science support scale: Its appropriateness with high school students

Implementing the idea that more emphasis should be placed on student achievement in the affective domain is contingent upon the concurrent development of suitable instruments for the assessment of prescribed criteria. One such instrument, the Schwirian Science Support Scale (Tri-S scale), was reported in a recent NSTA publication as a promising tool for measuring student science support. Recent research using the Tri-S scale with high school pupils showed that scores on this instrument did not increase after the students had taken a tenth grade introductory course in biology. Further analysis indicated students of teachers scoring “high” in science support did not produce higher scores on the Tri-S scale than students studying biology from teachers “low” in science support. Reliability estimates using high school student scores were well below previous estimates using scores from college undergraduates. Factor analysis of inter-item correlations indicated that student interpretation of item meaning did not correspond to the five subtest structure of the Tri-S scale. Findings from this study demonstrate that the Tri-S scale is not an appropriate instrument for measuring attitudinal changes of tenth grade high school students. This study is suggestive of the fact that went and future instruments that purport to measure achievement in noncognitive areas should be carefully analyzed before they are recommended for use with specific populations.     

Apprehending Mathematical Structure: A Case Study of Coming to Understand a Commutative Ring

Abstract algebra courses tend to take one of two pedagogical routes: from examples of mathematics structures through definitions to general theorems, or directly from definitions to general theorems. The former route seems to be based on the implicit pedagogical intention that students will use their understanding of particular examples of an algebraic structure to get a sense of those properties which form the basis of the fundamental definitions. We will explain the transition from examples to abstract algebra as a series of shifts of attention and in this paper we will use a case study to examine the initial shift, which we will call apprehending a structure, and examine how one student came to apprehend the structure of the commutative ring Z₉₉.