Use of Generalizability Theory Within K–12 School-Based Assessment: A Critical Review and Analysis of the Empirical Literature

ABSTRACT

Generalizability Theory (GT) offers several advantages relevant to educational contexts, including the fact that it can be used to estimate multiple sources of variance (e.g., raters, forms) simultaneously and to derive coefficients for the purposes of both relative and absolute decision making. Although GT has been increasingly applied in recent years to assessment data in K–12 school settings, analysis and critique of its application has not yet taken place. The goals of the current article were therefore to (1) undertake a systematic review of the school-based assessment literature in order to identify relevant applications of this statistical framework and (2) assess the degree of consistency between methodological recommendations for use of GT and reporting practices within this literature. In addition to describing the current state of this literature, suggestions for strengthening future applications are provided.     

Language Use in a Multilingual Class: a Study of the Relation Between Bilingual Students’ Languages and Their Meaning-Making in Science

In this study, we examine how bilingual students in elementary school use their languages and what this means for their meaning-making in science. The class was multilingual with students bilingual in different minority languages and the teacher monolingual in Swedish. The analysis is based on a pragmatic approach and the theory of translanguaging. The science content was electricity, and the teaching involved class instruction and hands-on activities in small groups. The findings of the study are divided into two categories, students’ conversations with the teacher and student’s conversations with each other. Since the class was multilingual, the class instruction was carried out in Swedish. Generally, when the conversations were characterised by an initiation, response and evaluation pattern, the students made meaning of the activities without any language limitations. However, when the students, during whole class instruction, were engaged in conversations where they had to argue, discuss and explain their ideas, their language repertoire in Swedish limited their possibilities to express themselves. During hands-on activities, students with the same minority language worked together and used both of their languages as resources. In some situations, the activities proceeded without any visible language limitations. In other situations, students’ language repertoire limited their possibilities to make meaning of the activities despite being able to use both their languages. What the results mean for designing and conducting science lessons in a multilingual class is discussed.     

Correction to: A Critical Review of Students’ and Teachers’ Understandings of Nature of Science

Science & Education - The original article unfortunately contains incorrect presentation of Tables 1, 3, 4, 5 and 6.     

A Critical Review of Students’ and Teachers’ Understandings of Nature of Science

Science & Education - There is widespread agreement that an adequate understanding of the nature of science (NOS) is a critical component of scientific literacy and a major goal in science...     

Improving Risk Education in Grades 6–12: A Needs Assessment of Michigan,Ohio, and Wisconsin Science Teachers

Abstract

If formal educators are to play a role in preparing future citizens for making decisions about health and environmental risks, then they must be willing and able to incorporate risk education. This article presents 10 goals for risk education. It also describes current risk education practices, future interests, and perceived barriers to risk education of Michigan, Ohio, and Wisconsin Grades 6–12 science teachers. Data were collected through a mail questionnaire (1,336 surveys were returned, adjusted return rate = 80%) and in-depth telephone interviews (completed with 45 teachers, return rate = 90%). Many teachers had covered some aspect of risk education and were interested in enhancing their efforts. Results also suggest that teachers need to be made aware of risk education resources and provided with training, particularly regarding psychological influences on risk decisions.     

A Study on Sixth Grade Students’ Misconceptions and Errors in Spatial Measurement: Length,Area, and Volume

The purpose of the present study was to portray students’ misconceptions and errors while solving conceptually and procedurally oriented tasks involving length, area, and volume measurement. The data were collected from 445 sixth grade students attending public primary schools in Ankara, Türkiye via a test composed of 16 constructed-response format tasks. The findings revealed a wide range of misconceptions and errors such as “believing that all rulers are 30 cm long,” “confusing area formula with perimeter formula,” “believing a box has more than one surface area,” “using the volume formula for surface area,” “believing that ruler must be longer than the object measured,” etc. These misconceptions and errors could be considered as the evidences indicating the sixth graders’ lack of comprehension of the fundamental concepts of spatial measurement and their relationships and the procedures and formulas used for measuring length, area, and volume. The possible causes of such misconceptions and overcoming ways were also discussed.     

Healing, Mental Energy in the Physics Classroom: Energy Conceptions and Trust in Complementary and Alternative Medicine in Grade 10–12 Students

Lay conceptions of energy often conflict with scientific knowledge, hinder science learning and scientific literacy, and provide a basis for ungrounded beliefs. In a sample of Finnish upper secondary school students, energy was attributed with features of living and animate beings and thought of as a mental property. These ontologically confused conceptions (OCC) were associated with trust in complementary and alternative medicine (CAM), and independent of scientifically valid conceptions. Substance-based energy conceptions followed the correlational pattern of OCC, rather than scientific conceptions. OCC and CAM decreased both during the regular school physics curriculum and after a lesson targeted at the ontological confusions. OCC and CAM were slightly less common among students with high actively open-minded thinking, low trust in intuition and high need for cognition. The findings are discussed in relation to the goals of scientific education.     

Use of iPads and iPods for Academic Performance and Engagement of PreK–12 Students with Disabilities: A Research Synthesis

New technology devices such as iPads and iPods have become very popular and widely used in special education settings to teach students with disabilities. The 20 selected single-case design studies published by November 2015 were comprehensively reviewed to examine the effects of using iPads and iPods on academic performance and engagement of students with disabilities. This study also examined the approaches of using the devices during instruction. Results of the review indicated that using the devices was effective for enhancing both academic performance and engagement of students with disabilities. The results also indicated that the devices could be used for instruction in various ways, including use of a variety of types of applications and delivery of videos. Directions for future research and implications for practice are discussed.     

The Big-fish–little-pond-effect Stands Up to Critical Scrutiny: Implications for Theory, Methodology, and Future Research

The big-fish–little-pond effect (BFLPE) predicts that equally able students have lower academic self-concepts (ASCs) when attending schools where the average ability levels of classmates is high, and higher ASCs when attending schools where the school-average ability is low. BFLPE findings are remarkably robust, generalizing over a wide variety of different individual student and contextual level characteristics, settings, countries, long-term follow-ups, and research designs. Because of the importance of ASC in predicting future achievement, coursework selection, and educational attainment, the results have important implications for the way in which schools are organized (e.g., tracking, ability grouping, academically selective schools, and gifted education programs). In response to Dai and Rinn (Educ. Psychol. Rev., 2008), we summarize the theoretical model underlying the BFLPE, minimal conditions for testing the BFLPE, support for its robust generalizability, its relation to social comparison theory, and recent research extending previous implications, demonstrating that the BFLPE stands up to scrutiny. Quotations (associated page numbers) to the Dai and Rinn (2008) article are based on a prepublication version of the article available to the authors of this article that may have changed during the final preparation for publication. The authors would also like to express thanks to David Dai and Anne Rinn for their encouragement and assistance to us in preparation of our article, whilst still acknowledging that they might not agree will all the views expressed here.     

A US-China Interview Study: Biology Students’ Argumentation and Explanation About Energy Consumption Issues

As China and the United States become the top two carbon emitters in the world, it is crucial for citizens in both countries to construct a sophisticated understanding of energy consumption issues. This interview study examines how U.S. and Chinese students compare in explaining and arguing about two critical energy consumption issues: burning fossil fuels and using electricity. In particular, we focused on using scientific knowledge to explain and argue about these issues. Based on relevant literature and our previous research, we developed a model to guide separate assessment and evaluation of students’ argumentation and explanation. We conducted clinical interviews with 40 biology majors, including 20 U.S. students and 20 Chinese students. This study generated several important findings. First, Chinese students tended to be less consistent across explanations and argumentation, and their levels of argumentation were lower than their levels of explanation. Second, in comparison to their Chinese counterparts, U.S. students provided more scientific arguments but many fewer scientific explanations. Finally, although all participants were college students and had completed at least one introductory level science course before the interviews, some of their explanations and arguments were based on informal ideas rather than matter and energy. We discuss the possible interpretations of these findings and their implications for teaching and learning of scientific explanation and argumentation in both countries.     

Assessment of Greek University Students’ Counselling Needs and Attitudes: An Exploratory Study

The present study is concerned with an exploration of counselling needs of students at two Greek universities as well as their attitudes to utilizing a university counselling centre. The sample consisted of 312 students who completed a Greek version of the Rutgers Needs Assessment Questionnaire as well as a subscale on Attitudes towards the Counselling Centre. The analysis of the data indicated that issues about relationships, goal setting and problem-solving were the main concerns of the participants. It was also found that students, especially those who faced some kind of psychological difficulties, held a negative attitude towards visiting a university counselling centre.     

Self-regulation of assessment beliefs and attitudes: a review of the Students’ Conceptions of Assessment inventory

How students understand, feel about and respond to assessment might contribute significantly to learning behaviour and academic achievement. This paper reviews studies that have used a relatively new self-reported survey questionnaire (Students’ Conceptions of Assessment – SCoA) about student perceptions and understandings of assessment. Confirmatory factor analyses and structural equation modelling results have shown, consistent with self-regulation theory, that the SCoA inventory has meaningful relations with academic performance among New Zealand high school students. Further, German, Hong Kong, American, and New Zealand studies have shown that the SCoA has relations to motivational constructs (e.g. effort, learning strategies, interest, self-efficacy and anxiety) that are also consistent with self-regulation. The SCoA inventory extends our understanding of how student conceptions of assessment are an integral part of self-regulation and provide a warrant for use in research studies investigating test-taker responses to assessment practices and innovations at both university and high school levels.     

Urban Students’ Perceptions of Teachers: Views of Students in General, Special, and Honors Education

Students’ satisfaction with school experiences has been linked to their sense of belongingness, connection to school, and achievement. Though the extant research addresses students’ perceptions of school climate and sense of belonging, there is a paucity of research about students’ views of teacher–student interactions. Five hundred and seventy-seven students from one ethnically and academically diverse urban high school were surveyed and interviewed about the nature of teacher talk with students. Findings from this mixed-methods investigation indicate students from general, special, and honors programs experience a wide range of interactions based on academic services received, gender, and ethnicity. More frequent perceived punitive feedback was reported by all students in special education as well as males in general and honors education programs, while Hispanic students indicated a greater frequency of perceived supportive feedback. Findings also reflect a wide range of attitudes and feelings about teachers, the educational system, and learning.     

A Critical Interrogation of Privilege, Race, Class, and Power in a University Faculty–Urban Community Relationship

I am the Coordinator of the Urban Teacher Education Center, a teacher preparation program located at a very low income, culturally diverse elementary school that serves children from two neighborhood public housing projects. As a White, middle-class, Ph.D. educated, female, I must consistently consider how people in the neighborhoods may take a racially, economically, and educationally marked view of me, marking me as an “other” while still assigning me with privilege. This paper consists of the presentation of my diary entries during my time spent in the school and its neighborhood communities. The diary entries are then critiqued with a critical interrogation of my reflections on race, class, and based on theory and research. The paper is framed by the analysis of the impact of race, class, power, and privilege, especially White privilege, and it addresses issues of power relations and school-community dynamics in low income, urban communities and schools. The paper provides an example of how a university faculty member can begin to enter an urban community, of the critical interrogations that must take place when entering such a relationship, and the challenges and rewards when such an effort is undertaken.