Multifaceted Assessment of Inquiry-Based Science Learning

To improve student science achievement in the United States we need inquiry-based instruction that promotes coherent understanding and assessments that are aligned with the instruction. Instead, current textbooks often offer fragmented ideas and most assessments only tap recall of details. In this study we implemented 10 inquiry-based science units that promote knowledge integration and developed assessments that measure student knowledge integration abilities. To measure student learning outcomes, we designed a science assessment consisting of both proximal items that are related to the units and distal items that are published from standardized tests (e.g., Trends in International Mathematics and Science Study). We compared the psychometric properties and instructional sensitivity of the proximal and distal items. To unveil the context of learning, we examined how student, class, and teacher characteristics affect student inquiry science learning. Several teacher-level characteristics including professional development showed a positive impact on science performance.     

Test-language effects in bilingual education: Evidence from CLIL classes in Germany

In the last decades, many bilingual programs such as content and language integrated learning (CLIL) have emerged. A crucial question in CLIL is whether to assess students' content-subject knowledge in the common language or the dominant language of instruction. In two experimental field studies with N = 361 6th-graders and N = 703 8th-graders, we investigated test-language effects and their moderation by CLIL-experience, language background, instructional language use and item format. In both studies, students' achievement was higher when items were presented in the common language. This effect was less pronounced in experienced CLIL-learners and for multiple-choice than cloze-test items. Moderation effects of language background were only partially and for instructional language use not statistically significant. The results show that using the dominant language of instruction to assess content-subject knowledge in bilingual education might underestimate students’ learning and that test-language and item-format effects need to be considered.     

面向时代需求 实现范式转变——赖格卢特论信息时代视角下的教学理论演进

教学是帮助学习者构建内化的知识的各种行为。信息时代所出现的变革呼唤着教学新范式。教学新范式既保留了已经被实践证明行之有效的基本教学方法(讲解、示证和练习),同时也鼓励采用变通教学方法(问题教学、项目学习、模拟学习、辅导学习和认知学徒等)。信息时代的教学理论遵循着四项基本原理,这就是个性化与多样性,主动性与自导性,协作性与情感性、整体性与一体化,而技术在记录学习进步、规划学习蓝图、提供学习指导和评估学习效果等方面也起到了不可或缺的保障作用。     

Knowledge Integration and Displaced Volume

This study contrasted spontaneous and reflective knowledge integration instruction delivered using a computer learning environment to enhance understanding of displaced volume. Both forms of instruction provided animated experiments and required students to predict outcomes, observe results, and explain their ideas. In addition, the reflective instruction diagnosed specific inconsistencies in student reasoning and encouraged students to reflect on these dilemmas as well as to construct general principles. We distinguished the impact of instruction on students who believed scientific phenomena are governed by principles (cohesive beliefs) versus students who believed that science is a collection of unrelated facts (dissociated beliefs). Students typically held multiple models of displacement, using different explanations depending on the form of assessment. For example, we found that 17% of these middle school students made accurate predictions about displacement experiments prior to instruction and 25% could construct an accurate general principle. However, only 12% consistently used the same explanation across assessments. After instruction, students were more accurate and more consistent: over 50% accurately predicted experimental outcomes, 79% gave an accurate general principle, and about 40% gave consistent responses. We found no advantages for enhanced animations over straightforward animated experiments. The reflective integration instruction led to more substantial long-term changes in student understanding than did spontaneous integration instruction. Furthermore, on a delayed posttest we found that students with cohesive beliefs not only sustained their understanding of displaced volume, but, when exposed to reflective integration instruction, actually continued to construct more predictive views following instruction. In contrast, students with dissociated beliefs made no long-term progress independent of the form of instruction.     

An Investigation of Explanation Multiple-Choice Items in Science Assessment

Both multiple-choice and constructed-response items have known advantages and disadvantages in measuring scientific inquiry. In this article we explore the function of explanation multiple-choice (EMC) items and examine how EMC items differ from traditional multiple-choice and constructed-response items in measuring scientific reasoning. A group of 794 middle school students was randomly assigned to answer either constructed-response or EMC items following regular multiple-choice items. By applying a Rasch partial-credit analysis, we found that there is a consistent alignment between the EMC and multiple-choice items. Also, the EMC items are easier than the constructed-response items but are harder than most of the multiple-choice items. We discuss the potential value of the EMC items as a learning and diagnostic tool.     

Scaffolding Middle School Students’ Construction of Scientific Explanations: Comparing a cognitive versus a metacognitive evaluation approach

This study investigated the effects of scaffolds as cognitive prompts and as metacognitive evaluation on seventh-grade students' growth of content knowledge and construction of scientific explanations in five inquiry-based biology activities. Students' scores on multiple-choice pretest and posttest and worksheets for five inquiry-based activities were analyzed. The results show that the students' content knowledge in all conditions significantly increased from the pretest to posttest. Incorporating cognitive prompts with the explanation scaffolds better facilitated knowledge integration and resulted in greater learning gains of content knowledge and better quality evidence and reasoning. The metacognitive evaluation instruction improved all explanation components, especially claims and reasoning. This metacognitive approach also significantly reduced students' over- or underestimation during peer-evaluation by refining their internal standards for the quality of scientific explanations. The ability to accurately evaluate the quality of explanations was strongly associated with better performance on explanation construction. The cognitive prompts and metacognitive evaluation instruction address different aspects of the challenges faced by the students, and show different effects on the enhancement of content knowledge and the quality of scientific explanations. Future directions and suggestions are provided for improving the design of the scaffolds to facilitate the construction of scientific explanations.     

Measuring knowledge integration: Validation of four‐year assessments

Science education needs valid, authentic, and efficient assessments. Many typical science assessments primarily measure recall of isolated information. This paper reports on the validation of assessments that measure knowledge integration ability among middle school and high school students. The assessments were administered to 18,729 students in five states. Rasch analyses of the assessments demonstrated satisfactory item fit, item difficulty, test reliability, and person reliability. The study showed that, when appropriately designed, knowledge integration assessments can be balanced between validity and reliability, authenticity and generalizability, and instructional sensitivity and technical quality. Results also showed that, when paired with multiple‐choice items and scored with an effective scoring rubric, constructed‐response items can achieve high reliabilities. Analyses showed that English language learner status and computer use significantly impacted students' science knowledge integration abilities. Students who took the assessment online, which matched the format of content delivery, performed significantly better than students who took the paper‐and‐pencil version. Implications and future directions of research are noted, including refining curriculum materials to meet the needs of diverse students and expanding the range of topics measured by knowledge integration assessments. © 2011 Wiley Periodicals, Inc. J Res Sci Teach 48: 1079–1107, 2011     

Assessing Partial Knowledge in Vocabulary

This study reports an attempt to assess partial knowledge in vocabulary. Fifty multiple-choice vocabulary items were constructed so that the incorrect choices followed the stages of vocabulary acquisition defined by O'Connor (1940). Ability estimates based on Rasch dichotomous and polychotomous models were compared to determine if there were any gains in validity or reliability as a result of using the polychotomous scoring model rather than the dichotomous scoring model. An attempt was also made to determine the appropriateness of O'Connor's stage theory of vocabulary acquisition for predicting the type of errors that examinees of differing ability would make on the test items. The results indicate that the reliability and concurrent validity of the polychotomous scoring of a subset of items that fit the polychotomous scoring model were significantly higher than those for dichotomous scoring of the same subset of items. The results also indicate moderate support for O'Connor's theory of vocabulary acquisition.     

Measuring and Validating Cognitive Modifiability as an Ability: A Study in the Spatial Domain

Measuring cognitive modifiability from the responsiveness of an individual's performance to intervention has long been viewed (e.g., Dearborne, 1921) as an alternative to traditional (static) ability measurement. Currently, dynamic testing, in which cues or instruction are presented with ability test items, is a popular method for assessing cognitive modifiability. Despite the long-standing interest, however, little data exists to support the validity of cognitive modifiability measures in any ability domain. Several special methodological difficulties have limited validity studies, including psychometric problems in measuring modifiability (i.e., as change), lack of appropriate validation criteria, and difficulty in linking modifiability to cognitive theory. In this article, relatively new developments for solving the validation problems are applied to measuring and validating spatial modifiability. Criterion-related validity for predicting learning in an applied knowledge domain, as well as construct validity, is supported.     

Promoting Integration of Multiple Texts: a Review of Instructional Approaches and Practices

The ability to meaningfully and critically integrate multiple texts is vital for twenty-first-century literacy. The aim of this systematic literature review is to synthesize empirical studies in order to examine the current state of knowledge on how intertextual integration can be promoted in educational settings. We examined the disciplines in which integration instruction has been studied, the types of texts and tasks employed, the foci of integration instruction, the instructional practices used, integration measures, and instructional outcomes. The studies we found involved students from 5th grade to university, encompassed varied disciplines, and employed a wide range of task and text types. We identified a variety of instructional practices, such as collaborative discussions with multiple texts, explicit instruction of integration, modeling of integration, uses of graphic organizers, and summarization and annotation of single texts. Our review indicates that integration can be successfully taught, with medium to large effect sizes. Some research gaps include insufficient research with young students; inadequate consideration of new text types; limited attention to students’ understandings of the value of integration, integration criteria, and text structures; and lack of research regarding how to promote students’ motivation to engage in intertextual integration.     

建构主义教学模型在大学生素质教育中的应用

建构主义的教学观认为学生应是认知的主体、是知识意义的主动建构者，教师对学生的意义建构起着帮助和促进作用，在此基础上形成的建构主义教学模型，对大学生的素质教育有着很大的启示，特别是抛锚式教学模式对增强大学生的信息意识、提高信息能力个有高效、实用的作用。     

Problem-Solving Correlates of new Assessment Forms in Architecture

In the figural response item format, proficiency is expressed by manipulating elements of a picture or diagram. Figural response items in architecture were contrasted with multiple-choice counterparts in their ability to predict architectural problem-solving proficiency. Problem-solving proficiency was measured by performance on two architecture design problems, one of which involved a drawing component, whereas the other required only a written verbal response. Both figural response and multiple-choice scores predicted verbal design problem solving, but only the figural response scores predicted graphical problem solving. The presumed mechanism for this finding is that figural response items more closely resemble actual architectural tasks than do multiple-choice items. Some evidence for this explanation is furnished by architects' self-reports, in which architects rated figural response items as "more like what an architect does" than multiple-choice items.     

Development of evidence-based,student-learning-oriented rubrics for pre-service science teachers’ pedagogical content knowledge

ABSTRACT

This paper offers pedagogical content knowledge (PCK) rubrics, that is, guides providing criteria for grading that are potentially applicable to a range of science topics and levels of teacher experience. Grading criteria applied in the rubrics are based on qualitative analyses of planned topic-specific professional knowledge (TSPK) and content knowledge (CK). Data were collected via three topic-specific vignettes from 239 pre-service science teachers (PSTs) starting a university-based, full-time, one year teacher education programme in England. The statements were analysed for TSPK and CK. PSTs’ statements proposed instructional strategies comprising demonstrations, explanations, illustrations and analogies, classified as Relevant to the science topics, others Irrelevant. Some Relevant strategies missed an aspect that may, if enacted, help students’ learning, so were judged Incomplete. Statements were also analysed for evidence of relevant and correct CK. CK and TSPK statements are aligned into grids, creating PCK rubrics. These demonstrate the precise nature of knowledge likely to lead to instruction that impacts positively on student learning. The rubrics present the possibility of PCK repertoires that contribute clarity and precision to teaching instruction. Although findings cannot be generalised, the methodology offers a strategy for supporting out-of-field teachers, and those seeking instructional strategies to add to existing repertoires.     

Instructional metacognitive knowledge: A qualitative study on conceptions of freshmen about instruction

Many studies have pointed to the limited direct effectiveness of instructional interventions and one explanation of this has drawn attention to the importance of 'instructional metacognitive knowledge'. In this exploratory study, instructional metacognitive knowledge of university freshmen is addressed by means of a survey covering: (a) instruction in general, (b) instruction in two different environments, and (c) specific delivery systems and concrete instructional interventions. While some differences appeared in the answers on different sets of questions, also large similarities were found. The results show students' knowledge about instruction to be mainly affected by their instructional experiences. The students' view is 'reactive' and places instructional agents at the core of the instructional process. The findings provide additional support for the importance of correspondence between learners' and teachers' views on instructional interventions if forms of support for students' learning are to be effective.     

Measurement Efficiency of Innovative Item Formats in Computer-Based Testing

The psychometric literature provides little empirical evaluation of examinee test data to assess essential psychometric properties of innovative items. In this study, examinee responses to conventional (e.g., multiple choice) and innovative item formats in a computer-based testing program were analyzed for IRT information with the three-parameter and graded response models. The innovative item types considered in this study provided more information across all levels of ability than multiple-choice items. In addition, accurate timing data captured via computer administration were analyzed to consider the relative efficiency of the multiple choice and innovative item types. As with previous research, multiple-choice items provide more information per unit time. Implications for balancing policy, psychometric, and pragmatic factors in selecting item formats are also discussed.     

An Empirical Investigation Demonstrating the Multidimensional DIF Paradigm: A Cognitive Explanation for DIF

Differential Item Functioning (DIF) is traditionally used to identify different item performance patterns between intact groups, most commonly involving race or sex comparisons. This study advocates expanding the utility of DIF as a step in construct validation. Rather than grouping examinees based on cultural differences, the reference and focal groups are chosen from two extremes along a distinct cognitive dimension that is hypothesized to supplement the dominant latent trait being measured. Specifically, this study investigates DIF between proficient and non-proficient fourth- and seventh-grade writers on open-ended mathematics test items that require students to communicate about mathematics. It is suggested that the occurrence of DIF in this situation actually enhances, rather than detracts from, the construct validity of the test because, according to the National Council of Teachers of Mathematics (NCTM), mathematical communication is an important component of mathematical ability, the dominant construct being assessed. However, the presence of DIF influences the validity of inferences that can be made from test scores and suggests that two scores should be reported, one for general mathematical ability and one for mathematical communication. The fact that currently only one test score is reported, a simple composite of scores on multiple-choice and open-ended items, may lead to incorrect decisions being made about examinees.     

The development of an instrument for assessing students' perceptions of teachers' knowledge

The purpose of this study was to describe the development and validation of an instrument on Student Perceptions of Teachers' Knowledge (SPOTK) in relation to their pedagogy. Features of teachers' knowledge from the research literature related to instruction, representation, subject matter knowledge, and knowledge of how to assess students' understanding were used to generate categories in the SPOTK. The result of a pilot study with 634 Taiwanese junior high school students showed high reliability of the scales, a good factor structure, and provided suggestions to delete weak items. In the main study, for which nine to ten items under each category were generated making a total of 37 items in the SPOTK, the instrument was administered to 1879 Taiwanese and 1081 Australian junior high school students varying in grades, sex and ability levels. Reliability and validity measures of the instrument were established based on Cronbach alpha and factor analysis. After the validating process, 28 items remained in the final instrument and reliabilities of the scales ranged from 0.97 to 0.82. Comment is made about the differences between Australian and Taiwanese students' responses and suggestions for using the instrument in future research.     

Developing and evaluating instructionally sensitive assessments in science

The purpose of this article is to address a major gap in the instructional sensitivity literature on how to develop instructionally sensitive assessments. We propose an approach to developing and evaluating instructionally sensitive assessments in science and test this approach with one elementary life‐science module. The assessment we developed was administered to 125 students in seven classrooms. The development approach considered three dimensions of instructional sensitivity; that is, assessment items should: represent the curriculum content, reflect the quality of instruction, and have formative value for teaching. Focusing solely on the first dimension, representation of the curriculum content, this study was guided by the following research questions: (1) What science module characteristics can be systematically manipulated to develop items that prove to be instructionally sensitive? and (2) Are the instructionally sensitive assessments developed sufficiently valid to make inferences about the impact of instruction on students' performance? In this article, we describe our item development approach and provide empirical evidence to support validity arguments about the developed instructionally sensitive items. Results indicated that: (1) manipulations of the items at different proximities to vary their sensitivity were aligned with the rules for item development and also corresponded with pre‐to‐post gains; and (2) the items developed at different distances from the science module showed a pattern of pre‐to‐post gain consistent with their instructional sensitivity, that is, the closer the items were to the science module, the larger the observed gains and effect sizes. © 2012 Wiley Periodicals, Inc. J Res Sci Teach 49: 691–712, 2012