Use of Generalizability Theory Within K–12 School-Based Assessment: A Critical Review and Analysis of the Empirical Literature

ABSTRACT

Generalizability Theory (GT) offers several advantages relevant to educational contexts, including the fact that it can be used to estimate multiple sources of variance (e.g., raters, forms) simultaneously and to derive coefficients for the purposes of both relative and absolute decision making. Although GT has been increasingly applied in recent years to assessment data in K–12 school settings, analysis and critique of its application has not yet taken place. The goals of the current article were therefore to (1) undertake a systematic review of the school-based assessment literature in order to identify relevant applications of this statistical framework and (2) assess the degree of consistency between methodological recommendations for use of GT and reporting practices within this literature. In addition to describing the current state of this literature, suggestions for strengthening future applications are provided.     

Comparing effects of different writing activities on reading comprehension: A meta-analysis

The purposes of this review were to determine: (1) if different writing activities were more effective than others in improving students’ reading comprehension, and (2) if obtained differences among writing activities was related to how reading comprehension was measured? Meta-analysis was used to examine these questions across studies involving students in grades 1–12. Nineteen studies were located that met inclusion criteria, resulting in 4 writing activities comparisons with 4 or more studies per comparison: summary writing versus answering questions (k = 5), summary writing versus note taking (k = 7), answering questions versus note taking (k = 4), and answering questions versus extended writing activities (k = 6). Effect sizes calculated for each writing activities comparison indicated there were no statistically significant differences for any of these comparisons when effects were averaged over all reading comprehension measures, excluding treatment-inherent measures. However, statistically significant differences were found for two of the comparisons on specific measures. Extended writing enhanced reading comprehension better than question answering on measures where comprehension was assessed via an extended writing activity, whereas summary writing enhanced reading comprehension better than question answering on a free recall measure. The results provide limited support for the theoretical viewpoint that writing activities are differentially effective in improving reading comprehension based on how closely the writing activities are aligned with a particular measure.     

Estimating Linear Size and Scale: Body rulers

The National Science Education Standards emphasise the use of concepts and skills that cut across the science domains. One of these cross‐cutting areas is measurement. Students should know measurement systems, units of measurement, tools and error in measurement as well as the importance of measurement to scientific endeavours. Even though measurement is an essential skill, little is known about how students estimate and use measurement in different contexts. This study examines the impact of teaching students to use their bodies as rough measurement tools (body rulers) on their ability to estimate linear measurements. Nineteen middle school students participated in metric intervention tasks and completed a pre‐instruction and a post‐instruction Linear Measurement Assessment. Results showed that teaching students to use rough body measures as tools (a body ruler) for estimation had a significant influence on their estimation accuracy. After instruction, students were better able to estimate the sizes of objects, use their body in making estimations of size, and estimate while touching an object or pacing a distance. Furthermore, proportional reasoning was significantly correlated with students’ post‐instruction scores on the Linear Measurement Assessment.     

Cross‐national Comparisons in the International Successful School Principalship Project (ISSPP): The USA,Norway and China

This article provides a cross‐national perspective on successful school principalship in three countries derived from an analysis of case studies in the International Successful School Principalship Project (ISSPP). The ISSPP aims to investigate the characteristics, processes and effects of successful school leadership across eight countries (i.e. Australia, Canada, England, the United States, China, Denmark, Sweden and Norway). Examples from the USA, Norway and China (Shanghai) were selected to illustrate cross‐national differences related to the societal purposes of education, the structure and funding of different national educational systems and the influence of particular governmental educational policies (i.e. accountability‐oriented policies) on the leadership practices of individual school principals. Variations in selection criteria and research procedures were also noted. Recommendations for further research using a cultural framework include analysing multiethnic schools to identify culturally specific leadership practices as well as developing further ISSPP case studies in non‐Western contexts.     

A Matter of Friendship: Educational Interventions into Culture and Poverty

Contemporary educational reformers have claimed that research on social class differences in child raising justifies programs that aim to lift children out of poverty by means of cultural interventions. Focusing on the Knowledge Is Power Program (KIPP), Ruby Payne's “aha! Process,” and the Harlem Children's Zone as examples, Amy Shuffelton argues that such programs, besides overstepping the social science research, are ethically illegitimate insofar as they undermine the equitable development of civic agency. Shuffelton invokes Aristotelian civic friendship, particularly as interpreted by Danielle Allen and Sibyl Schwarzenbach, as key to a politics that avoids relations of domination and subordination. She concludes that social justice requires that educators involved with culturally interventionist programs recognize the workings of power within schooling and society, that they accept the limits of their own perspectives, and that they remain open to what is of value in child‐raising practices other than those associated with the contemporary middle class.     

Raising Activity Levels: A Multidimensional Analysis of Curriculum Change

Using Fullan's (1982) The Meaning of Educational Change (Toronto, OISE Press) multidimensional framework, this paper discusses the problems of curriculum change by examining the subjective experiences of physical education teachers. The discussion was based on interviews with teachers who were involved in a study to change health behaviors of high school students. Although no discernable changes in behavior were observed, the explanations for the results, as reported by the teachers, have provided insights of greater importance. Two physical education classes consisting of boys and girls aged between 11 and 13 years from 20 secondary schools were selected for pre-test and post-test measurements. The data collected included a survey, direct observation, heart rate monitoring and motion sensoring. A teacher-designed modification of the physical education curriculum was used as an intervention as it was specifically aimed at increasing activity. A debriefing of teachers in the experimental group indicated that they considered the teacher-designed lesson guide to be valuable. However, when the teachers were faced with implementing change in their physical classes, they (like the researchers) underestimated the situational and personal-social factors that prevented untested ideas from being transformed into firm beliefs and values. The findings confirmed the predictions of a multidimensional model of educational change, which indicated that, when challenged, technological and material innovations are more easily changed than attitudes, beliefs and values held by teachers.     

Understanding teachers’ will and capacity to accomplish physical education curriculum reform: the implications for teacher development

Drawing on data generated by a two-year ethnographic study in a high security Spanish prison, this article explores the multiple meanings given to the social practices of sport and physical activity. We provide details of the following key themes that emerged from the analysis: (a) escaping time; (b) perceived therapeutic benefits; (c) social control; (d) gendered dimensions; and (e) performing masculinity. The findings suggest that a diverse and contradictory set of meanings are associated with sport and physical activity within this particular prison culture, and that the performance of specific kinds of masculinity is both a process and product that shapes the construction of experience in powerful ways.     

Continuing Education Module Transforming Maternity Care: Implementing the Blueprint for Action

In January 2010, Women’s Health Issues published two direction-setting reports from the Transforming Maternity Care (TMC) Project: “2020 Vision for a High-Quality, High-Value Maternity Care System” and “Blueprint for Action: Steps Toward a High-Quality, High-Value Maternity Care System.” This guest editorial summarizes highlights of the implementation phase of what is now known as the TMC Partnership. Major progress has been made in elevating maternity care quality to a national policy priority, increasing the availability and use of maternity care performance measures, and developing shared decision making tools for childbearing women.     

An audience response system may influence student performance on anatomy examination questions

This study integrated an in-house audience response system (ARS) in the human anatomy course over two years to determine whether students performed better on high-stakes examinations following exposure to similar interactive questions in a large lecture format. Questions in an interactive ARS format were presented in lectures via PowerPoint presentations. Students who chose to participate in the anonymous ARS sessions submitted answers via their personal wireless devices (e.g., laptops, smartphones, PDAs, etc). Students were surveyed for feedback. Student participation in ARS activities was greatest (65-80%) in the first lecture. The number of students who actively participated in ARS activities decreased over the next four sessions, and then slightly increased in the last two sessions. This trend was the same for both years. Use of the ARS did not dramatically enhance overall student performance on examination questions that dealt with content similar to content presented in the ARS sessions. However, students who scored in the lower quartile of the examination performed better on the examination questions after the ARS was implemented. Accordingly, our findings suggest that the effect of ARS to improve student performance on examinations was not uniform. The overall benefit of an ARS to enhance the lecture experience was confirmed by student surveys.     

Maternal work conditions and child development

How do maternal work conditions, such as psychological stress and physical hazards, affect children's development? Combining data from the Child Development Supplement of the Panel Study of Income Dynamics and the Occupational Information Network allows us to shed some light on this question. We employ various techniques including OLS with extensive controls, a value added approach and individual fixed effects in order to address potential endogeneity problems. Our results reveal that mothers’ exposure to work-related hazards negatively affects children's cognitive development and to work-related stress negatively affects children's behavioral development. While maternal time investments play a small but significant role in mediating these negative associations, paternal time investments neither reinforce nor compensate these associations.