Instructional Sensitivity as a Psychometric Property of Assessments

Standards-based reform, as codified by the No Child Left Behind Act, relies on the ability of assessments to accurately reflect the learning that takes place in U.S. classrooms. However, this property of assessments—their instructional sensitivity—is rarely, if ever, investigated by test developers, states, or researchers. In this paper, the literature on the psychometric property of instructional sensitivity is reviewed. Three categories of instructional sensitivity measures are identified—those relying on item or test scores only, those relying on item or test scores and teacher reports of instruction, and strictly judgmental methods. Each method identified in the literature is discussed alongside the evidence for its utility. Finally, recommendations are made as to the proper role of instructional sensitivity in the evaluation of assessments used under standards-based reform.     

Looking Beyond the Overall Scores of NAEP Assessments: Applications of Generalized Linear Mixed Modeling for Exploring Value-Added Item Difficulty Effects

The central idea of differential item functioning (DIF) is to examine differences between two groups at the item level while controlling for overall proficiency. This approach is useful for examining hypotheses at a finer-grain level than are permitted by a total test score. The methodology proposed in this paper is also aimed at estimating differences at the item rather than the overall score level, yet with the innovation where item-level differences for many groups simultaneously are the focus. This is a straightforward generalization of DIF as variance rather than one or several group differences; conceptually, this can be referred to as item difficulty variation (IDV). When instruction is of interest, and "groups" is a unit at which instruction is determined or delivered, then IDV signals value-added effects that can be influenced by either demographic or instructional variables.     

Evaluation of the effectiveness of remedial reading courses at community colleges

Limited research is available on the effectiveness of remedial college courses. The present study evaluated the effectiveness of two instructional approaches for developmental reading courses at a community college in the southwestern United States. The instructional approaches were traditional textbook-based instruction and strategic-reading instruction. The sample consisted of 64 participants. Half (n = 32) of the participants were in the control group (n = 32) and received traditional textbook-based instruction; the other half (n = 32) were in the experimental group and received strategic-reading instruction. All participants completed the Nelson-Denny Reading Form G at the beginning of the semester and then again 12 weeks later, at the end of the semester. The data were then statistically analyzed to identify any relationships between the type of instruction and the differences between the students’ pretest and posttest scores. The null hypotheses for H1 and H2 were rejected because the results of the paired t tests indicated that both traditional textbook-based instruction and strategic-reading instruction have a statistically significant positive effect on students’ performance on the test. A one-way ANOVA was conducted to determine whether the pretest/posttest difference scores varied based on the type of instruction. The findings showed that both methods of instruction were equally effective in improving the reading comprehension skills of community college students in a developmental reading course. Based on the findings, community college leaders are encouraged to assess the effectiveness of the instructional methods used in developmental courses to ensure at-risk community college students are receiving the most beneficial instruction.     

Sensitivity of test items to teaching quality

Instructional sensitivity is the psychometric capacity of tests or single items of capturing effects of classroom instruction. Yet, current item sensitivity measures’ relationship to (a) actual instruction and (b) overall test sensitivity is rather unclear. The present study aims at closing these gaps by investigating test and item sensitivity to teaching quality, reanalyzing data from a quasi-experimental intervention study in primary school science education (1026 students, 53 classes, M_age = 8.79 years, SD_age = 0.49, 50% female). We examine (a) the correlation of item sensitivity measures and the potential for cognitive activation in class and (b) consequences for test score interpretation when assembling tests from items varying in their degree of sensitivity to cognitive activation. Our study (a) provides validity evidence that item sensitivity measures may be related to actual classroom instruction and (b) points out that inferences on teaching drawn from test scores may vary due to test composition.     

Developing and evaluating instructionally sensitive assessments in science

The purpose of this article is to address a major gap in the instructional sensitivity literature on how to develop instructionally sensitive assessments. We propose an approach to developing and evaluating instructionally sensitive assessments in science and test this approach with one elementary life‐science module. The assessment we developed was administered to 125 students in seven classrooms. The development approach considered three dimensions of instructional sensitivity; that is, assessment items should: represent the curriculum content, reflect the quality of instruction, and have formative value for teaching. Focusing solely on the first dimension, representation of the curriculum content, this study was guided by the following research questions: (1) What science module characteristics can be systematically manipulated to develop items that prove to be instructionally sensitive? and (2) Are the instructionally sensitive assessments developed sufficiently valid to make inferences about the impact of instruction on students' performance? In this article, we describe our item development approach and provide empirical evidence to support validity arguments about the developed instructionally sensitive items. Results indicated that: (1) manipulations of the items at different proximities to vary their sensitivity were aligned with the rules for item development and also corresponded with pre‐to‐post gains; and (2) the items developed at different distances from the science module showed a pattern of pre‐to‐post gain consistent with their instructional sensitivity, that is, the closer the items were to the science module, the larger the observed gains and effect sizes. © 2012 Wiley Periodicals, Inc. J Res Sci Teach 49: 691–712, 2012     

Teachers' Planning Processes: Does ISD Make a Difference?

Do teachers who have a formal educational background in instructional systems design (ISD) differ from those who do not in how they plan and implement instruction? This question guided this pilot study of ten volunteer teachers, five who had formal education in ISD and five who did not. An open and closed item questionnaire was sent to each subject and follow-up interviews were conducted. Teachers were asked about their general planning practices, written planning procedures, mental planning, and how they implement instruction based on their plans. Results indicated that nine of the ten teachers use general ISD skills and processes in planning. Teachers reported that they (a) analyzed learners' needs and abilities, and (b) used objectives to guide the instructional process, specifically for selecting learning activities and evaluation. While there were no major differences between the two groups in their submitted daily plans, the teachers with instructional design backgrounds were more specific in their open-ended responses about the use of instructional design for some aspects of planning (e.g., using hierarchies and taxonomies to sequence instruction), and the importance of consistency among objectives, learning activities, and evaluation. In addition, four of the five ISD teachers believed that knowledge of ISD has improved both their planning processes and their teaching. Because major differences between the two groups were not observed, much of the article focuses on teacher planning processes in general.     

THE EFFECTS OF INSTRUCTION FOR THREE MATHEMATICS ITEM FORMATS1

Different instructional programs were developed for three mathematics aptitude item formats to determine the relative susceptibility of each to special instruction. Subjects were male and female high school junior volunteers in 12 schools. In the seven weeks between a pre- and posttest, experimental Ss received 21 hours of instruction for one of the three formats; control Ss received no special instrucion. Each of the three formats was found susceptible to instruction directed toward it. The complex formats were most susceptible. Female Ss were slightly less able mathematically at the outset and benefited less from instruction than males. Mean gains of nearly a full standard deviation for groups instructed for the complex formats were considered to be of practical consequence.     

Satisfied teachers are good teachers: The association between teacher job satisfaction and instructional quality

Instructional quality is associated with better academic outcomes for students. This study aimed to investigate how teachers' job satisfaction was associated with clarity of instruction and cognitive activation as measures of instructional quality. In addition, we investigated whether this association between teachers' job satisfaction and instructional quality was mediated by teacher–student relationships. Drawing on the 2018 Teaching and Learning International Survey (TALIS), we compared participants from both Eastern (N = 27,106; Japan, Taipei, Korea, Shanghai) and Western sociocultural contexts (N = 20,209; Canada, Australia, New Zealand, United States, United Kingdom). Multilevel structural equation modelling results indicated that teachers' job satisfaction was positively associated with instructional quality across Eastern and Western settings. The relationship between teachers' job satisfaction and instructional quality was partially mediated by better student–teacher relationships. There were some differences between the cultural settings in how job satisfaction correlated with clarity of instruction and cognitive activation. We suggest that these differences may be accounted for by cultural characteristics leading to different approaches to teaching. Our results suggest that teachers' job satisfaction and the quality of classroom-level relationships may be important indicators of positive instructional outcomes. While schools focus on student outcomes, they should also address teachers' job satisfaction and prioritise the importance of relationships between teachers and students in classrooms.     

Modeling Instructional Sensitivity Using a Longitudinal Multilevel Differential Item Functioning Approach

Students’ performance in assessments is commonly attributed to more or less effective teaching. This implies that students’ responses are significantly affected by instruction. However, the assumption that outcome measures indeed are instructionally sensitive is scarcely investigated empirically. In the present study, we propose a longitudinal multilevel‐differential item functioning (DIF) model to combine two existing yet independent approaches to evaluate items’ instructional sensitivity. The model permits for a more informative judgment of instructional sensitivity, allowing the distinction of global and differential sensitivity. Exemplarily, the model is applied to two empirical data sets, with classical indices (Pretest–Posttest Difference Index and posttest multilevel‐DIF) computed for comparison. Results suggest that the approach works well in the application to empirical data, and may provide important information to test developers.     

Opportunities for Professional Learning in Mathematics Teacher Workgroup Conversations: Relationships to Instructional Expertise

Increasingly, instructional improvement efforts include teacher communities as part of their overall strategy, yet the relationship between teachers’ talk and professional learning remains underspecified. Using a discourse perspective on learning, this article compares opportunities to learn (OTLs) in the collaborative conversations of 3 mathematics teacher workgroups. We examined the differences in OTLs in 17 hr of videotaped meetings from 3 groups at different levels of instructional accomplishment in secondary mathematics. Using mixed methods, we uncovered differences in the groups’ interactions and found that OTLs were not equally distributed. Instead, teacher groups whose active participants demonstrated the greatest facility with ambitious instruction also had the richest conversational OTLs. We interpret this as an accumulated advantage developmental story: Because collaborative work in teaching involves problem posing and the articulation of practice, teachers’ conceptions get built into the framing and discussion of pedagogical problems. Accomplished teachers are thus positioned to learn more from talking with colleagues. This analysis contributes to understanding of how OTLs are constituted in teacher workgroups, with implications for making better use of teacher collaboration for professional learning.     

Which instructional models influence more on perceived exertion,affective valence,physical activity level,and class time in physical education?

Abstract

The aim of this study was to examine the effects of two different instructional models (i.e. direct instructional model (DIM) and tactical games model (TGM)) with primary education students. Perceived exertion, affective valence, physical activity (PA) level, instruction time, active learning time, and relation of instruction/active learning time were measured with 256 primary education students during DIM and TGM 8 weeks’ intervention programs. Significant differences were found between the DIM and TGM in perceived exertion, PA level, instruction time and in the relation between instruction/active learning time (p?<?.01, ES=?0.4 to 1.1). By age group, the DIM had significantly higher PA level in students of 3^rd grade of primary education (G3), 4^th grade (G4) and 6^th grade (G6) (p?<?.05 or p?<?.01, ES=?0.8 to ?0.4). The instructional model used during physical education classes can affect perceived exertion, PA level, instruction time, and the relation instruction/active learning time.     

Instructional Sensitivity of a Complex Language Arts Performance Assessment

Validation of assessments intended to improve instruction and learning should include evidence of instructional sensitivity. This study investigated the instructional sensitivity of a standards-based ninth-grade performance assessment that required students to write an essay about conflict in a literary work. Before administering the assessment, teachers of 886 ninth-grade students were randomly assigned to one of three instructional groups: literary analysis, organization of writing, and teacher-selected instruction. Despite the short duration of instruction (8 class periods), results support the instructional sensitivity of the assessment in two ways: Instruction on literary analysis significantly improved students' ability to analyze and describe conflicts in literature, and instruction on the organization of writing led to significantly higher scores on measures of coherence and organization.     

DIFFERENCES IN INSTRUCTIONAL SENSITIVITY BETWEEN ITEM FORMATS AND BETWEEN ACHIEVEMENT TEST ITEMS

Building achievement tests which are sensitive to the instructional effects of school programs concerns both practitioners and researchers in education. To produce such tests, empirical procedures to guide item selection are needed. In this paper, an operational framework and a set of empirical procedures for this task are presented. Within this framework, item sensitivity is linked to instructional implementation. A simple components of variance model has been used to provide actual estimates of instructional sensitivity. These procedures are illustrated using data from a comparative study of alternative item formats for a criterion-referenced test. Even when items were closely matched to instructional content specifications, important differences in instructional sensitivity emerged. These differences were found between the same items presented in different formats as well as between different items presented within the same format. Implications of these results for developing criterion-referenced achievement tests are discussed.     

The impact of the instructional planning self-reflective tool on preservice teacher performance, disposition, and self-efficacy beliefs regarding systematic instructional planning

This study investigated the impact of a self-regulatory tool, the Instructional Planning Self-Reflective Tool (IPSRT), on preservice teachers' performance, disposition, and self-efficacy beliefs regarding systematic instructional planning. Participants enrolled in an introductory educational technology course were taught how to develop an instructional plan as part of the course. An experimental group was provided with instruction on how to use the IPSRT while engaging in instructional planning. Results indicated that the experimental group demonstrated greater skill acquisition, showed more positive disposition, and reported higher perceived instrumentality of instructional planning. In terms of self-efficacy, no significant differences were found between the two groups. However, further analyses revealed that participants who were initially high in self-efficacy reported significantly lower self-efficacy following the tool intervention; in contrast, participants initially low in self-efficacy showed significantly higher self-efficacy following the tool intervention. Findings are discussed from a social-cognitive perspective.     

An Examination of the Instructional Sensitivity of the TIMSS Math Items: A Hierarchical Differential Item Functioning Approach

In recent years, students’ test scores have been used to evaluate teachers’ performance. The assumption underlying this practice is that students’ test performance reflects teachers’ instruction. However, this assumption is generally not empirically tested. In this study, we examine the effect of teachers’ instruction on test performance at the item level using a hierarchical differential item functioning approach. The items are from the U.S. TIMSS 2011 4th-grade math test. Specifically, we tested whether students who had received instruction on a given item performed significantly better on that item compared with students who had not received such instruction when their overall math ability was controlled for, whether with or without controlling for student-level and class-level covariates. This study provides preliminary findings regarding why some items show instructional sensitivity and sheds light on how to develop instructionally sensitive items. Implications and directions for further research are also discussed.     

Effects of cooperative learning on second graders’ learning from text

Two studies were conducted to investigate the effects of cooperative learning on second‐graders’ motivation and learning from text. In Study 1, students (n = 160) in cooperative learning groups were compared with their counterparts (n = 107) in traditional instruction groups. The results revealed a statistically significant difference between the two groups, with more favourable perceptions of teachers’ instructional practices and better reading comprehension in the instructional intervention groups than in the traditional instruction groups. In Study 2, 51 second‐graders participated in the instructional intervention programme. The results showed that students’ positive cooperative behaviour and attitudes were related to their motivation and reading comprehension. When students perceived that their peers were willing to help each other and were committed to the group, they tended to be more motivated and performed better in reading comprehension.     

The Effect of Training in Test Item Writing on Test Performance of Junior High Students

High stakes testing, a phenomena born out of intense accountability across the United States, produces instructional settings that marginalize both curriculum and instruction. Teachers and other school personnel have minimized instruction to drill and practice in an effort to raise standardized and criterion referenced test scores. This study presents an alternative to current practice that engages students in learning and increases their awareness of the internal aspects of standardized tests. The Test Item Construction Model (TICM) guides students through the process of studying test item stems and subsequently creating items using a 12 week process of incrementing from understanding to creating test items. Students grew in their understanding of the test item stems and the generation of these. An ANOVA did not yield significant differences between random groups of trained and untrained test writers. However, students in the experimental group demonstrated gains in understanding of test items.     

Associations between Fine Motor and Mathematics Instruction and Kindergarten Mathematics Achievement

Research Findings: This study investigates the role of fine motor and mathematics instruction in mathematics achievement in an international sample of kindergarteners from the United States and China. Multilevel modeling was used to assess the interaction between students’ entering skills and classroom time spent on basic math, higher-order math and fine-motor instruction. For American children, the effect of basic math and higher-order math instruction on student achievement depended on entering skills; however, fine motor instruction had negative average effects on student achievement and did not depend on students’ entering skills. Instruction time was not a significant predictor of achievement for Chinese students. Practice or Policy: Though fine motor skills have a robust correlation with mathematics achievement, a causal link has not been established. Our study indicates that time spent in fine motor instruction does not advance mathematics achievement in kindergarten and in fact may weaken mathematics achievement, given the limited time in the instructional day. American teachers in our sample who spent more time in fine-motor instruction tended to spend less time on basic math and higher-order mathematics instruction. Educators should weigh instructional trade-offs carefully and work to tailor instruction to students’ skill levels.     

Longitudinal analysis of time, engagement, and achievement in at-risk versus non-risk students

This longitudinal study investigated the effects of time spent in academic instruction and time engaged on elementary students' academic achievement gains. Three groups were compared over grades as follows: (a) an at-risk experimental group of low-socioeconomic status (SES) students for whom teachers implemented classwide peer tutoring (CWPT) beginning with the second semester of first grade continuing through Grade 3; (b) an equivalent at-risk control group; and (c) a non-risk comparison group of students of average- to high-SES. In both the control and comparison groups, teachers employed conventional instructional practices over Grades 1 through 3. Results indicated significant group differences in the time spent in academic instruction, engagement, and gains on the subtests of the Metropolitan Achievement Test that favored the experimental and comparison groups over the control group. Implications include the effectiveness of CWPT for at-risk students and the continuing vulnerability of at-risk students whose daily instructional programs provide less instructional time and foster lower levels of active academic engagement.