Effects of Ventilation on Segmental Bioimpedance Spectroscopy Measures Using Generalizability Theory

The purpose of this study was to examine the effect of three ventilation conditions (i.e., normal, regimented, and no-ventilation) on the reproducibility of bioimpedance scores in humans for the forearm and trunk segments. One hundred able-bodied North American men and women, from 18 to 71 years of age, volunteered as participants. The experimenters used a Xitron Bio-Impedance Analyzer System model 4200 instrument with Hydra software (Xitron Technologies, San Diego, California, USA) to collect bioimpedance data on extracellular fluid and intracellular fluid scores. The experimenters analyzed the data using the generalizability theory,with persons as the facet of differentiation and time as the facet of generalization. The major findings were (a) ventilation conditions did not have a significant impact on the reproducibility of the test scores, (b) the forearm segment produced consistently higher intracellular fluid generalizability coefficients across three ventilation conditions for both gender groups when compared to the trunk segment, (c) the trunk segment produced intracellular fluid generalizability coefficients that were higher for the male group, and (d) the measurement error affected extracellular fluid scores less than segmental intracellular fluid scores.     

The happy-fish-little-pond effect on enjoyment: Generalizability across multiple domains and countries

Academic enjoyment is an important educational construct given that it benefits students' engagement, persistence, wellbeing, and mental health. In this study, we examine two factors that determine this crucial emotion, namely student- and class-level achievement. Past research has been restricted to single-country or single-domain examinations of secondary school students, limiting generalizability of findings. To bridge this gap, we utilize the Trends in International Mathematics and Science Study and the Progress in International Reading Literacy Study (TIMSS-PIRLS) combined 2011 data (N = 180,084 4th-grade students, 37 countries). Our results provide robust evidence that student-level achievement positively predicts enjoyment in math, science, and reading, while the effects of class-level achievement are negative—the Happy-Fish-Little-Pond Effect. These results showed relative universality across the domains and countries examined.     

概化理论在结构化面试评分误差中的应用研究

应用概化理论对结构化面试的评分误差的控制问题进行了研究。结果表明:结构化面试评分能够较好地反映出被试的真实能力水平,评分具有较高的信度;在保证较高的面试评分信度(0.80)的情况下,建议将考官人数减少至9名,以提高结构化面试的经济性和效率性。     

Measuring instructional practice in science using classroom artifacts: lessons learned from two validation studies

With growing interest in the role of teachers as the key mediators between educational policies and outcomes, the importance of developing good measures of classroom processes has become increasingly apparent. Yet, collecting reliable and valid information about a construct as complex as instruction poses important conceptual and technical challenges. This article summarizes the results of two studies that investigated the properties of measures of instruction based on a teacher‐generated instrument (the Scoop Notebook) that combines features of portfolios and self‐report. Classroom artifacts and teacher reflections were collected from samples of middle school science classrooms and rated along 10 dimensions of science instruction derived from the National Science Education Standards; ratings based on direct classroom observations were used as comparison. The results suggest that instruments that combine artifacts and self‐reports hold promise for measuring science instruction with reliability similar to, and sizeable correlations with, measures based on classroom observation. We discuss the implications and lessons learned from this work for the conceptualization, design, and use of artifact‐based instruments for measuring instructional practice in different contexts and for different purposes. Artifact‐based instruments may illuminate features of instruction not apparent even through direct classroom observation; moreover, the process of structured collection and reflection on artifacts may have value for professional development. However, their potential value and applicability on a larger scale depends on careful consideration of the match between the instrument and the model of instruction, the intended uses of the measures, and the aspects of classroom practice most amenable to reliable scoring through artifacts. We outline a research agenda for addressing unresolved questions and advancing theoretical and practical knowledge around the measurement of instructional practice. © 2011 Wiley Periodicals, Inc. J Res Sci Teach 49: 38–67, 2012     

The Use of Generalizability (G) Theory in the Testing of Linguistic Minorities

We contend that generalizability (G) theory allows the design of psychometric approaches to testing English-language learners (ELLs) that are consistent with current thinking in linguistics. We used G theory to estimate the amount of measurement error due to code (language or dialect). Fourth- and fifth-grade ELLs, native speakers of Haitian-Creole from two speech communities, were given the same set of mathematics items in the standard English and standard Haitian-Creole dialects (Sample 1) or in the standard and local dialects of Haitian-Creole (Samples 2 and 3). The largest measurement error observed was produced by the interaction of student, item, and code. Our results indicate that the reliability and dependability of ELL achievement measures is affected by two facts that operate in combination: Each test item poses a unique set of linguistic challenges and each student has a unique set of linguistic strengths and weaknesses. This sensitivity to language appears to take place at the level of dialect. Also, students from different speech communities within the same broad linguistic group may differ considerably in the number of items needed to obtain dependable measures of their academic achievement. Whether students are tested in English or in their first language, dialect variation needs to be considered if language as a source of measurement error is to be effectively addressed.     

概化理论及其在运动成绩主观评定中的应用

概化理论是现代心理测量三大理论之一,它能针对测量结果推论范围或者使用目的的不同而提供多个不同的测量误差估计指标,确定评分者人数,较好地控制评分误差,因此广泛地用于表现性评定中.难美表现项群中,运动成绩的主观评定引发的种种问题一直未见有科学的评定手段,借助于心理测量学中的概化理论可以确定成绩误差的来源,确定裁判员评分信度以及在一定信度条件下所需的最少裁判人数,因此概化理论在体育测量领域具有较大的应用价值.     

Characteristics of School Districts That Participate in Rigorous National Educational Evaluations

Given increasing interest in evidence-based policy, there is growing attention to how well the results from rigorous program evaluations may inform policy decisions. However, little attention has been paid to documenting the characteristics of schools or districts that participate in rigorous educational evaluations, and how they compare to potential target populations for the interventions that were evaluated. Utilizing a list of the actual districts that participated in 11 large-scale rigorous educational evaluations, we compare those districts to several different target populations of districts that could potentially be affected by policy decisions regarding the interventions under study. We find that school districts that participated in the 11 rigorous educational evaluations differ from the interventions' target populations in several ways, including size, student performance on state assessments, and location (urban/rural). These findings raise questions about whether, as currently implemented, the results from rigorous impact studies in education are likely to generalize to the larger set of school districts—and thus schools and students—of potential interest to policymakers, and how we can improve our study designs to retain strong internal validity while also enhancing external validity.     

Detecting Bouts of Physical Activity in a Field Setting

Abstract

The purposes of this study were to assess the TRITRAC® and CSA® for: (a) interaccelerometer agreement; (b) agreement in detecting patterns of moderate-intensity physical activity; and (c) agreement in detecting walking patterns recorded in a diary. Thirty-one women wore both the TRITRAC® and CSA® accelerometers for three consecutive days. Interaccelerometer agreement (measured with generalizability coefficients) ranged from .88 to .99. In total, 71.3% of the accelerometers' patterns agreed in length, with CSA® patterns being on average significantly longer. Interaccelerometer agreement in detecting patterns of brisk walking, as recorded in a diary, was comparable (69.4%). Interaccelerometer discrepancies may be related in part to the threshold employed by each instrument for classifying moderate intensity patterns.     

Research Abstracts and Bibliographies

ABSTRACT

The authors address the reliability of scores obtained on the summative performance assessments during the pilot year of our research. Contrary to classical test theory, we discussed the advantages of using generalizability theory for estimating reliability of scores for summative performance assessments. Generalizability theory was used as the framework because of the flexibility this approach provides for examining sources of inconsistency within a complex assessment. Two major sources of inconsistency on scores considered in this study were raters and agencies (teachers' rating vs. researchers' rating). Overall, results showed that the inconsistency in scores attributable to raters and agencies was relatively small. Suggestions regarding improvement of consistency in the subsequent years of our research were provided.     

Challenges in Building Usable Knowledge in Education

The scientific rigor of education research has improved dramatically since the year 2000. Much of the credit for this improvement is deserved by Institute of Education Sciences (IES) policies that helped create a demand for rigorous research; increased human capital capacity to carry out such work; provided funding for the work itself; and collected, evaluated, and made available the results of that work through the What Works Clearinghouse. Major challenges still remain for education research, however. One challenge is dealing with the replication crisis that has plagued other scientific fields and is likely to be a problem in education science. A second challenge is better supporting the generalizability of education research. A third challenge is adapting our rigorous research designs to the increasing complexity of our interventions and our questions about the mechanisms by which these interventions achieve their effects. Promising approaches to meet each of these challenges are suggested.