Alignment of Content and Effectiveness of Mathematics Assessment Items

Alignment has taken on increased importance given the current high-stakes nature of assessment. To make well-informed decisions about student learning on the basis of test results, assessment items need to be well aligned with standards. Project 2061 of the American Association for the Advancement of Science (AAAS) has developed a procedure for analyzing the content and quality of assessment items. The authors of this study used this alignment procedure to closely examine 2 mathematics assessment items. Student work on these 2 items was analyzed to determine whether the conclusions reached through the use of the alignment procedure could be validated. It was found that the Project 2061 alignment procedure was effective in providing a tool for in-depth analysis of the mathematical content of the item and a set of standards and in identifying 1 particular content standard that was most closely aligned with the standard. Through analyzing student work samples and student interviews, it was also found that students' thinking may not correspond to the standard identified as best aligned with the learning goals of the item. This finding highlights the potential usefulness of analyzing student work to clarify any additional deficiencies of an assessment item not revealed by an alignment procedure.     

Using Focus Groups,Expert Advice,and Cognitive Interviews to Establish the Validity of a College Student Survey

This study focused on how the design of a national student survey instrument was informed and improved through the combined use of student focus groups, cognitive interviews, and expert survey design advice. We were specifically interested in determining (a) how students interpret the items and response options, (b) the frequency of behaviors or activities associated with the response options, (c) if the items are clearly worded and specific enough to produce reliable and valid results, and (d) if the items and response categories accurately represent students' behaviors and perceptions. We collected focus group data from 8 colleges and universities as part of a nationally funded research project on student engagement. The findings provide additional insight into the importance of using focus groups and cognitive interviews to learn how students interpret various items and what different responses really mean.     

Building the BIKE: Development and Testing of the Biotechnology Instrument for Knowledge Elicitation (BIKE)

Identifying students’ conceptual scientific understanding is difficult if the appropriate tools are not available for educators. Concept inventories have become a popular tool to assess student understanding; however, traditionally, they are multiple choice tests. International science education standard documents advocate that assessments should be reform based, contain diverse question types, and should align with instructional approaches. To date, no instrument of this type targeting student conceptions in biotechnology has been developed. We report here the development, testing, and validation of a 35-item Biotechnology Instrument for Knowledge Elicitation (BIKE) that includes a mix of question types. The BIKE was designed to elicit student thinking and a variety of conceptual understandings, as opposed to testing closed-ended responses. The design phase contained nine steps including a literature search for content, student interviews, a pilot test, as well as expert review. Data from 175 students over two semesters, including 16 student interviews and six expert reviewers (professors from six different institutions), were used to validate the instrument. Cronbach’s alpha on the pre/posttest was 0.664 and 0.668, respectively, indicating the BIKE has internal consistency. Cohen’s kappa for inter-rater reliability among the 6,525 total items was 0.684 indicating substantial agreement among scorers. Item analysis demonstrated that the items were challenging, there was discrimination among the individual items, and there was alignment with research-based design principles for construct validity. This study provides a reliable and valid conceptual understanding instrument in the understudied area of biotechnology.     

The implications of content versus item validity on science tests

The use of content validity as the primary assurance of the measurement accuracy for science assessment examinations is questioned. An alternative accuracy measure, item validity, is proposed. Item validity is based on research using qualitative comparisons between (a) student answers to objective items on the examination, (b) clinical interviews with examinees designed to ascertain their knowledge and understanding of the objective examination items, and (c) student answers to essay examination items prepared as an equivalent to the objective examination items. Calculations of item validity are used to show that selected objective items from the science assessment examination overestimated the actual student understanding of science content. Overestimation occurs when a student correctly answers an examination item, but for a reason other than that needed for an understanding of the content in question. There was little evidence that students incorrectly answered the items studied for the wrong reason, resulting in underestimation of the students' knowledge. The equivalent essay items were found to limit the amount of mismeasurement of the students' knowledge. Specific examples are cited and general suggestions are made on how to improve the measurement accuracy of objective examinations.     

Development and application of a diagnostic instrument to evaluate grade-11 and -12 students' concepts of covalent bonding and structure following a course of instruction

This article initially outlines a procedure used to develop a written diagnostic instrument to identify grade-11 and -12 students' misconceptions and misunderstandings of the chemistry topic covalent bonding and structure. The content to be taught was carefully defined through a concept map and propositional statements. Following instruction, student understanding of the topic was identified from interviews, student-drawn concept maps, and free-response questions. These data were used to produce 15 two-tier multiple-choice items where the first tier examined content knowledge and the second examined understanding of that knowledge in six conceptual areas, namely, bond polarity, molecular shape, polarity of molecules, lattices, intermolecular forces, and the octet rule. The diagnostic instrument was administered to a total of 243 grade-11 and -12 chemistry students and has a Cronbach alpha reliability of 0.73. Item difficulties ranged from 0.13 to 0.60; discrimination values ranged from 0.32 to 0.65. Each item was analyzed to ascertain student understanding of and identify misconceptions related to the concepts and propositional statements underlying covalent bonding and structure.     

Virtual praxis: Constraints,approaches, and innovations of online creative arts teacher educators

Higher education, and in particular, initial teacher education, has been significantly transformed through the introduction of e-learning. However, online teacher education presents particular challenges in the creative arts, which has traditionally developed student understanding through embodied and collaborative learning experiences. In this qualitative study, in-depth interviews were conducted with eight online arts educators in teacher education programs to understand their perspectives and pedagogy in online arts coursework. Using Engeström’s Activity Theory as an analytical lens, the findings highlight how these academics navigated challenges and opportunities to facilitate authentic, praxis-focused arts experiences to prepare pre-service teachers for the classroom.     

Why Students Answer TIMSS Science Test Items the Way They Do

The purpose of this study was to explore how Year 8 students answered Third International Mathematics and Science Study (TIMSS) questions and whether the test questions represented the scientific understanding of these students. One hundred and seventy-seven students were tested using written test questions taken from the science test used in the Third International Mathematics and Science Study. The degree to which a sample of 38 children represented their understanding of the topics in a written test compared to the level of understanding that could be elicited by an interview is presented in this paper. In exploring student responses in the interview situation this study hoped to gain some insight into the science knowledge that students held and whether or not the test items had been able to elicit this knowledge successfully. We question the usefulness and quality of data from large-scale summative assessments on their own to represent student scientific understanding and conclude that large scale written test items, such as TIMSS, on their own are not a valid way of exploring students' understanding of scientific concepts. Considerable caution is therefore needed in exploiting the outcomes of international achievement testing when considering educational policy changes or using TIMSS data on their own to represent student understanding.     

The Virtual Solar System Project: Developing Conceptual Understanding of Astronomical Concepts Through Building Three-Dimensional Computational Models

The Virtual Solar System (VSS) course described in this paper is one of the first attempts to integrate three-dimensional (3D) computer modeling as a central component of an introductory undergraduate astronomy course. Specifically, this study assessed the changes in undergraduate university students' understanding of astronomy concepts as a result of participating in an experimental introductory astronomy course in which the students constructed 3D models of different astronomical phenomena. In this study, we examined students' conceptual understanding concerning three foundational astronomical phenomena: the causes of lunar and solar eclipses, the causes of the Moon's phases, and the reasons for the Earth's seasons. Student interviews conducted prior to the course identified a range of student alternative conceptions previously identified in the literature regarding the dynamics and mechanics of the Solar System. A previously undocumented alternative conception to explain lunar eclipses is identified in this paper. The interviews were repeated at the end of the course in order to quantitatively and qualitatively assess any changes in student conceptual understanding. Generally, the results of this study revealed that 3D computer modeling can be a powerful tool in supporting student conceptualization of abstract scientific phenomena. Specifically, 3D computer modeling afforded students the ability to visualize abstract 3D concepts such as the line of nodes and transform them into conceptual tools, which in turn, supported the development of scientifically sophisticated conceptual understandings of many basic astronomical topics. However, there were instances where students' conceptual understanding was incomplete and frequently hybridized with their existing conceptions. These findings have significant bearing on when and in what domains 3D computer modeling can be used to support student conceptual understanding of astronomy concepts.     

edTPA,videotape, and occupational identity: a study of pre-service teachers

In conducting this study, I sought to contribute to the scholarly discourse of understanding how North American elementary pre-service teachers experienced evaluation via teacher performance assessments. Through extensive interviews and thematic data analysis, this study generally supported the contention that the process of completing edTPA deepened student teachers’ understanding of their educational experience in a number of domains, which in turn suggested a broader awareness and appreciation of the complexities of learning to teach. Data from a larger study indicated that the identified ‘realms of understanding’ fell into four areas of insight related to education: Understanding Children, Understanding Instructional Strategies, Understanding via Collective Learning, and Understanding Self as Teacher. For the purposes of this paper, the identified area of insight ‘Understanding Self as Teacher’ is explored. These findings lead to several practical ways in which teacher education might be improved, particularly in the area of utilization of video in teacher education programs.     

师范生实践教学同侪模式探究

师范院校英语专业学生学习教学的过程在形式上很大程度是一种同侪训练,即水平相同的同伴之间的相互教学和练习。每个人承担着既做教师又当学生的双重身份:当同伴讲课时,自己要充当学生的角色,积极配合教师的教学;当自己讲课时,同伴又是自己的学生。这样,每个人都经历着从执教者到学习者再转为评议者的体验,在这个探索问题、学习教学的过程中,从换位思考的角度出发,探究教学发生的过程,建构起自己对教学的理解。     

Student engagement scale: development,reliability and validity

In this study, the purpose was to develop a student engagement scale for higher education. The participants were 805 students. In the process of developing the item pool regarding the scale, related literature was examined in detail and interviews were held. Six factors – valuing, sense of belonging, cognitive engagement, peer relationships (emotional engagement-I), relationships with faculty members (emotional engagement-II) and behavioural engagement within the scope of the components of campus engagement and class engagement – made up of 41 items were identified. The total variance explained was 59%. A higher student engagement score was regarded as an indicator that the student had a high level of engagement with the university, campus and class, while a lower score demonstrated that the student’s engagement with the campus and class was weak or that disengagement could occur.     

First-year students working with exemplars: promoting self-efficacy,self-monitoring and self-regulation

In this paper, we explore first-year undergraduate students’ experiences and perceptions regarding the use of exemplars, with reference to the development of student self-regulation. The study was carried out at a tertiary education institution in New Zealand where the second author was a lecturer. This author used exemplars with two classes of first-year students enrolled in a core introductory paper taught as part of the students' undergraduate business studies degree. Data were gathered through three complementary approaches: students’ responses to a short anonymous survey at the end of each of the four sessions where exemplars were used, written notes in the teacher’s Teaching journal and semi-structured interviews with student volunteers. When strategically employed as part of the teaching design, it was found that exemplars have the capacity to motivate, enhance students’ self-efficacy, promote understanding of task requirements, expand knowledge, understanding and skills in relation to the structure of academic tasks, support and advance subject knowledge, and develop evaluative and productive knowledge and expertise. We concluded the use of exemplars with these students demonstrated their potential as tools for the promotion of student self-efficacy, self-monitoring and to a lesser extent, self-regulation.     

Are there any questions? one teacher’s view of students and their questions in a fourth-grade classroom

While many teachers agree that the questions students ask are valuable, researchers have found that student questions are notably absent from most classrooms. We know almost nothing about the exceptions to this—classrooms where teachers manage to elicit and use student questions effectively in instruction. One fourth grade teacher, known for her use of student questions, was selected for this study. Classroom observations and interviews were used to gather data; qualitative methods were utilized to analyze the teacher interviews and the interactions surrounding 260 questions students asked during observations. This paper highlights the teacher’s stance toward her students and their questions. 1) The teacher viewed her students as learners who asked questions to increase their understanding. 2) The teacher viewed the assumptions revealed by student questions, as ‘steps’ to the curriculum. 3) The teacher viewed her students’ questions through the lens of potential; that is, she responded to questions based on how they might promote the understanding of her students.     

A comparative study of the effects of a concept mapping enhanced laboratory experience on Turkish high school students’ understanding of acid-base chemistry

The research reported here consists of the introduction of an intervention based on a series of laboratory activities combined with concept mapping. The purpose of this intervention was to enhance student understanding of acid-base chemistry for tenth grade students’ from two classes in a Turkish high school. An additional aim was to enhance student attitude toward chemistry. In the research design, two cohorts of students were compared; those from the intervention group (N = 31) and a second group (N = 28) who were taught in a more traditional manner. Student understanding of acid-base chemistry was evaluated with a pretest/posttest research design using a purpose-designed instrument, the Concept Achievement Test (CAT) consisting of 25 items, 15 multiple choice and ten multiple choice with explanation. Alternative conceptions identified in the pretest were incorporated into the intervention, which thereby sought to move students toward views more in accord with scientific views for the concepts. Statistical tests indicate the instrument is reliable (with an alpha reliability of 0.81) and the analysis of the findings revealed statistically significant differences between the intervention and traditional groups with respect to conceptual understanding. Examination of student explanations and analyses of semi-structured interviews conducted with selected students suggest that the main influence was the laboratory activities. Analysis of the findings in the context of relevant literature that concept mapping in conjunction with laboratory activities is more enjoyable, helps student link concepts, and reduces their alternative conceptions.     

Faculty ratings of course evaluation items

Instructors whose teaching was evaluated by students were given the opportunity to rate how applicable the evaluation items were to their classes. This study examined the kinds of items which instructors felt to be applicable or inapplicable, the relationships between the student ratings and the instructor applicability ratings, and the effect on an overall evaluation score of using the instructor applicability judgments as weights.Results generally support the consensus procedure of establishing rating forms; they suggest that the common criticism that faculty judgments of item applicability are influenced by anticipation of student ratings may be true for specific items and that while weighting composite evaluation scores by means of faculty applicability judgments does not affect those overall scores, the distributions of certain items may be altered.