Use of Generalizability Theory Within K–12 School-Based Assessment: A Critical Review and Analysis of the Empirical Literature

ABSTRACT

Generalizability Theory (GT) offers several advantages relevant to educational contexts, including the fact that it can be used to estimate multiple sources of variance (e.g., raters, forms) simultaneously and to derive coefficients for the purposes of both relative and absolute decision making. Although GT has been increasingly applied in recent years to assessment data in K–12 school settings, analysis and critique of its application has not yet taken place. The goals of the current article were therefore to (1) undertake a systematic review of the school-based assessment literature in order to identify relevant applications of this statistical framework and (2) assess the degree of consistency between methodological recommendations for use of GT and reporting practices within this literature. In addition to describing the current state of this literature, suggestions for strengthening future applications are provided.     

Looking Back on Lessons Learned: Gifted Adults Reflect on Their Experiences in Advanced Classes

The purpose of this study was to learn about gifted adults' experiences in advanced classes and attitudes about advanced classes or gifted programs for their children. Participants were 88 adults (33 men and 55 women) who have been participating in a longitudinal study of academically talented individuals since their high-school graduation in 1988. Participants responded to open-ended questions via mailed surveys. Eighty-five percent of participants described their academic experiences in advanced classes as positive, whereas slightly fewer participants (59%) described their interpersonal experiences in advanced classes as positive. Seventy-five percent of participants with children described signs of giftedness in their children, and 88% of participants indicated that they would support advanced placement for their children if it were recommended by the school. Implications of the findings were discussed and directions for future research were provided.     

Comparing effects of different writing activities on reading comprehension: A meta-analysis

The purposes of this review were to determine: (1) if different writing activities were more effective than others in improving students’ reading comprehension, and (2) if obtained differences among writing activities was related to how reading comprehension was measured? Meta-analysis was used to examine these questions across studies involving students in grades 1–12. Nineteen studies were located that met inclusion criteria, resulting in 4 writing activities comparisons with 4 or more studies per comparison: summary writing versus answering questions (k = 5), summary writing versus note taking (k = 7), answering questions versus note taking (k = 4), and answering questions versus extended writing activities (k = 6). Effect sizes calculated for each writing activities comparison indicated there were no statistically significant differences for any of these comparisons when effects were averaged over all reading comprehension measures, excluding treatment-inherent measures. However, statistically significant differences were found for two of the comparisons on specific measures. Extended writing enhanced reading comprehension better than question answering on measures where comprehension was assessed via an extended writing activity, whereas summary writing enhanced reading comprehension better than question answering on a free recall measure. The results provide limited support for the theoretical viewpoint that writing activities are differentially effective in improving reading comprehension based on how closely the writing activities are aligned with a particular measure.     

Investigating the impact of automated feedback on students’ scientific argumentation

ABSTRACT

This study investigates the role of automated scoring and feedback in supporting students’ construction of written scientific arguments while learning about factors that affect climate change in the classroom. The automated scoring and feedback technology was integrated into an online module. Students’ written scientific argumentation occurred when they responded to structured argumentation prompts. After submitting the open-ended responses, students received scores generated by a scoring engine and written feedback associated with the scores in real-time. Using the log data that recorded argumentation scores as well as argument submission and revisions activities, we answer three research questions. First, how students behaved after receiving the feedback; second, whether and how students’ revisions improved their argumentation scores; and third, did item difficulties shift with the availability of the automated feedback. Results showed that the majority of students (77%) made revisions after receiving the feedback, and students with higher initial scores were more likely to revise their responses. Students who revised had significantly higher final scores than those who did not, and each revision was associated with an average increase of 0.55 on the final scores. Analysis on item difficulty shifts showed that written scientific argumentation became easier after students used the automated feedback.     

Educational justice for all: the capability approach and inclusive education leadership

Leaders within education must weigh a number of fundamentals as they engage the needs of the stakeholders they represent within the political, social and economic context they operate within. Leaders must consider the unique needs and capabilities of individuals who might not possess similar abilities or talents to those of the majority. In this paper, we explore the capability approach as a frame to consider such contextual fundamentals in inclusive education leadership. In order to examine how educational leaders address the diverse needs of the groups they serve, we interviewed five central office educational administrators in the USA with the responsibility for leading programming for students with special needs. Using a human capabilities frame, we developed targeted questions to determine whether school leaders uphold the tenets of this theoretical foundation within their district's practices in an era of tight fiscal management and accountability. Findings indicate that those leaders whose districts had the highest level of inclusion for most students, including students with significant disabilities, most closely align with the capability approach. Recommendations for continued research and implications for both practice and policy are included.     

The Influence of Informal Science Education Experiences on the Development of Two Beginning Teachers’ Science Classroom Teaching Identity

In case studies of two first-year elementary classroom teachers, we explored the influence of informal science education (ISE) they experienced in their teacher education program. Our theoretical lens was identity development, delimited to classroom science teaching. We used complementary data collection methods and analysis, including interviews, electronic communications, and drawing prompts. We found that our two participants referenced as important the ISE experiences in their development of classroom science identities that included resilience, excitement and engagement in science teaching and learning–qualities that are emphasized in ISE contexts. The data support our conclusion that the ISE experiences proved especially memorable to teacher education interns during the implementation of the No Child Left Behind policy which concentrated on school-tested subjects other than science.     

Subject Matter Knowledge in a Master of Arts in Teaching Program: Ensuring candidates’ content knowledge through an arts and sciences/education collaboration

All teacher education programs face the challenge of ensuring that their graduates know their subject matter and can teach it effectively. For graduate level teacher education programs, whose candidates may complete their education in the content area at various institutions, this challenge is even greater. The following article describes one institution’s system of assessing candidates’ content knowledge and efforts to incorporate additional content coursework into a Master of Arts in Teaching program. Also provided is information on a set of courses that the arts and sciences and education faculty team teach, with the course for prospective English teachers described in detail.     

“Ain't nobody gonna get me down”: An Examination of the Educational Experiences of Four African American Women Labeled With Disabilities

This study used qualitative research methods to explore the educational experiences of four African American women with disabilities, revealing how each participant developed a critical consciousness in response to the dominant ideology surrounding the discourses of African American, woman, and disabled. The development of a critical consciousness led participants to counter the oppressive nature of their educational experiences by engaging in acts of resistance. Recommendations are made with regard to how K-18 educators might support and capitalize upon students' critical consciousness to facilitate equitable classroom communities.     

Multiple Choices,Multiple Chances: Fostering Re-entry Pathways for First Generation College Students

Student or program deficiencies are the classic explanation for student departure from community college partners of career technical education programs. However, a complex set of factors impact college attendance, persistence, and departure decision-making. Participants' experiences both within and outside program structures can reshape long-term goals. Based on a 7-year study of a population of 256 students, this article examines detailed case studies of 2 students' experiences leaving and reentering community college. It reveals how one program supported multiple exit and entry points. Interconnected educational and career pathways were made visible with the creation of a visual model that situated program experiences in a broader educational and career pathway. Additionally, both the use of creative solutions and adherence to rigid program parameters impacted students' career and educational trajectories. For more effective results, programs should support parallel career planning and encourage participants to question program structures in relation to their needs.