Evaluating student assessments: the use of optimal foraging theory

The concepts of optimal foraging theory and the marginal value theorem are used to investigate possible student behaviour in accruing marks in various forms of assessment. The ideas of predator energy consumption, handling and search times can be evaluated in terms of student behaviour and gaining marks or ‘attainment’. These ideas can be used to examine student responses to dealing with assessments by examining a marks awarded/time-on-task curve. The non-linear, cumulative mark gain, as a Gompertz function, has implications for how students tackle continuously assessed projects as well as examination questions. The attainment of a student can be viewed in these general terms, as well as in specific aspects such as question ‘difficulty’ and mark gain in an examination answer. Prospect theory, from econometrics and psychology, can also be used to suggest ways in which students might tackle problems in examinations. The implications of this analysis are considered with respect to setting questions, criterion referencing of assignments and dealing with ‘troublesome knowledge’. The ideas can also be used to assist problems regarding mark fidelity and integrity as well as mark comparability.     

Utilizing online learning data to design face-to-face activities in a flipped classroom: a case study of heterogeneous group formation

This study investigates the possibility of utilizing online learning data to design face-to-face activities in a flipped classroom. We focus on heterogeneous group formation for effective collaborative learning. Fifty-three undergraduate students (18 males, 35 females) participated in this study, and 8 students (3 males, 5 females) among them joined post-study interviews. For this study, a total of 6 student characteristics were used: three demographic characteristics obtained from a simple survey and three academic characteristics captured from online learning data. We define three demographic group heterogeneity variables and three academic group heterogeneity variables, where each variable is calculated using the corresponding student characteristic. In this way, each heterogeneity variables represents a degree of diversity within the group. Then, a two-stage hierarchical regression analysis was conducted to identify the significant group heterogeneity variables that influence face-to-face group achievement. The results show that the academic group heterogeneity variables, which were derived from the online learning data, accounted for a significant proportion of the variance in the group achievement when the demographic group heterogeneity variables were controlled. The interviews also reveal that the academic group heterogeneity indeed affected group interaction and learning outcome. These findings highlight that online learning data can be utilized to obtain relevant information for effective face-to-face activity design in a flipped classroom. Based on the results, we discuss the advantages of this data utilization approach and other implications for face-to-face activity design.

     

Teaching and Learning Science Through Song: Exploring the experiences of students and teachers

This qualitative, multi-case study explored the use of science-content music for teaching and learning in six middle school science classrooms. The researcher sought to understand how teachers made use of content-rich songs for teaching science, how they impacted student engagement and learning, and what the experiences of these teachers and students suggested about using songs for middle school classroom science instruction. Data gathered included three teacher interviews, one classroom observation and a student focus-group discussion from each of six cases. The data from each unit of analysis were examined independently and then synthesized in a multi-case analysis, resulting in a number of merged findings, or assertions, about the experience. The results of this study indicated that teachers used content-rich music to enhance student understanding of concepts in science by developing content-based vocabulary, providing students with alternative examples and explanations of concepts, and as a sense-making experience to help build conceptual understanding. The use of science-content songs engaged students by providing both situational and personal interest, and provided a mnemonic device for remembering key concepts in science. The use of songs has relevance from a constructivist approach as they were used to help students build meaning; from a socio-cultural perspective in terms of student engagement; and from a cognitive viewpoint in that in these cases they helped students make connections in learning. The results of this research have implications for science teachers and the science education community in developing new instructional strategies for the middle school science classroom.     

Heat energy and temperature concepts of adolescents,adults, and experts: Implications for curricular improvements

We conducted two studies of beliefs about laboratory and everyday thermal phenomena. The first study identified concepts of heat energy and temperature held by adolescents, adults, and scientists. We found a classic separation of “school” and “everyday” knowledge in each population. We conducted clinical interviews with 37 middle school students, 9 adults, and 8 chemists and physicists to obtain their predictions and explanations of real-world phenomena. Many students believed that metals “conduct,” “absorb,” “trap,” or “hold” cold better than other materials and that aluminum foil would be better than wool or cotton as a wrapping material to keep cold objects cold. Respondents in each group held many intuitive ideas that were well established. Although scientists made more accurate predictions than students and gave theoretical definitions of terms, they too had difficulty explaining everyday phenomena. The second study investigated the impact of a middle school science curriculum designed to help students understand everyday thermal events. We found marked improvements in posttest scores and clinical interview responses as a result of instruction that built on students' intuitions.     

Exploring Biology Teachers’ Pedagogical Content Knowledge in the Teaching of Genetics in Swaziland Science Classrooms

This study explored the pedagogical content knowledge (PCK) and its development of four experienced biology teachers in the context of teaching school genetics. PCK was defined in terms of teacher content knowledge, pedagogical knowledge and knowledge of students’ preconceptions and learning difficulties. Data sources of teacher knowledge base included teacher-constructed concept maps, pre- and post-lesson teacher interviews, video-recorded genetics lessons, post-lesson teacher questionnaire and document analysis of teacher's reflective journals and students’ work samples. The results showed that the teachers’ individual PCK profiles consisted predominantly of declarative and procedural content knowledge in teaching basic genetics concepts. Conditional knowledge, which is a type of meta-knowledge for blending together declarative and procedural knowledge, was also demonstrated by some teachers. Furthermore, the teachers used topic-specific instructional strategies such as context-based teaching, illustrations, peer teaching, and analogies in diverse forms but failed to use physical models and individual or group student experimental activities to assist students’ internalization of the concepts. The finding that all four teachers lacked knowledge of students’ genetics-related preconceptions was equally significant. Formal university education, school context, journal reflection and professional development programmes were considered as contributing to the teachers’ continuing PCK development. Implications of the findings for biology teacher education are briefly discussed.     

Students' mathematical modeling of motion

We present results of an investigation of university students' development of mathematical models of motion in a physical science course for preservice teachers and graduate students in science and mathematics education. Although some students were familiar with the standard concepts of position, velocity, and acceleration from physics classes, most students had difficulty using these concepts to characterize actual or hypothetical motions. Furthermore, some students developed their own nonstandard method of describing accelerated motion in terms of changes in the average velocity, from the start of the motion up to a given time. This is in contrast to the physics community's use of the acceleration construct, defined in terms of changes in the instantaneous velocity, to describe such motion. Although the change in average velocity is not typically identified as an important construct in traditional physics texts, some students found it intuitively appealing, and were able to use it successfully to describe and predict motion. We conclude that by focusing on standard constructs, and ignoring possible intuitive ways that students might view motion, standard kinematics instruction may miss an opportunity to maximize student understanding. © 2007 Wiley Periodicals, Inc. J. Res. Sci. Teach 45: 153–173, 2008.     

A theory-driven approach to assessing the cognitive effects of PBL

Medical schools are increasingly looking to case-based formats such as problem-based learning (PBL) for their medical students. However, the effects of PBL have not been adequately assessed for an informed decision. An approach to assessment should consider not just the knowledge outcomes expected of all students, but should be tailored to the theoretical goals of PBL: clinical reasoning, integration of scientific and clinical knowledge, and lifelong learning skills. This means that problem-solving processes as well as products need to be measured. In addition, cognitive measures associated with expert performance can be used to assess the extent to which PBL affects the development of expertise. In this study, students taking an elective in PBL were compared with students taking other electives on a realistic pathophysiological explanation task. The problem-solving protocols were examined for coherence, use of science concepts, strategy use, and self-directed learning. The results indicate that cognitive measures can be used to distinguish students who have participated in PBL from their counterparts in terms of knowledge, reasoning, and learning strategies. This suggests that such measures may play a meaningful role in assessment of student learning.     

The EvoDevoCI: A Concept Inventory for Gauging Students’ Understanding of Evolutionary Developmental Biology

The American Association for the Advancement of Science 2011 report Vision and Change in Undergraduate Biology Education encourages the teaching of developmental biology as an important part of teaching evolution. Recently, however, we found that biology majors often lack the developmental knowledge needed to understand evolutionary developmental biology, or “evo-devo.” To assist in efforts to improve evo-devo instruction among undergraduate biology majors, we designed a concept inventory (CI) for evolutionary developmental biology, the EvoDevoCI. The CI measures student understanding of six core evo-devo concepts using four scenarios and 11 multiple-choice items, all inspired by authentic scientific examples. Distracters were designed to represent the common conceptual difficulties students have with each evo-devo concept. The tool was validated by experts and administered at four institutions to 1191 students during preliminary (n = 652) and final (n = 539) field trials. We used student responses to evaluate the readability, difficulty, discriminability, validity, and reliability of the EvoDevoCI, which included items ranging in difficulty from 0.22–0.55 and in discriminability from 0.19–0.38. Such measures suggest the EvoDevoCI is an effective tool for assessing student understanding of evo-devo concepts and the prevalence of associated common conceptual difficulties among both novice and advanced undergraduate biology majors.     

Exploring the use of multiple analogical models when teaching and learning chemical equilibrium

This study describes the multiple analogical models used to introduce and teach Grade 12 chemical equilibrium. We examine the teacher's reasons for using models, explain each model's development during the lessons, and analyze the understandings students derived from the models. A case study approach was used and the data were drawn from the observation of three consecutive Grade 12 lessons on chemical equilibrium, pre‐ and post‐lesson interviews, and delayed student interviews. The key analogical models used in teaching were: the “school dance”; the “sugar in a teacup”; the “pot of curry”; and the “busy highway.” The lesson and interview data were subject to multiple, independent analyses and yielded the following outcomes: The teacher planned to use the students' prior knowledge wherever possible and he responded to student questions with stories and extended and enriched analogies. He planned to discuss where each analogy broke down but did not. The students enjoyed the teaching but built variable mental models of equilibrium and some of their analogical mappings were unreliable. A female student disliked masculine analogies, other students tended to see elements of the multiple models in isolation, and some did not recognize all the analogical mappings embedded in the teaching plan. Most students learned that equilibrium reactions are dynamic, occur in closed systems, and the forward and reverse reactions are balanced. We recommend the use of multiple analogies like these and insist that teachers always show where the analogy breaks down and carefully negotiate the conceptual outcomes. © 2005 Wiley Periodicals, Inc. J Res Sci Teach 42: 1135–1159, 2005     

Variables affecting physics achievement

Much research has focused on student views about physics concepts, with an emphasis on the identification of alternative conceptions, and how curricula and professional development may ameliorate the situation. However, there has been little work on determining the extent of, and in separating, the student and teacher/classroom level variables that may impact student physics achievement. This study examined the effect of different student and teacher/classroom level variables on student understanding of physics concepts using hierarchical linear modeling (HLM), a regression based technique. The data were collected from 68 different teachers and 3,119 students who were using a reform curriculum, Active Physics. Teachers and students completed surveys asking about their beliefs, their classes and their personal characteristics. Students also completed a physics achievement test. The data show that students of teachers who used Active Physics for a greater portion of the year scored higher on the achievement test than did students of teachers who did not use the curriculum as much. Furthermore, the data show that the achievement gap was narrowed between boys and girls and between students with different attitudes toward physics. Additionally teachers who received inservice instruction on how to implement Active Physics narrowed the gap between students with different views of their classroom involvement. © 2009 Wiley Periodicals, Inc. J Res Sci Teach 46: 961–976, 2009     

Is physicality an important aspect of learning through science experimentation among kindergarten students?

The purpose of this study was to investigate whether physicality (actual and active touch of concrete material), as such, is a necessity for science experimentation learning at the kindergarten level. We compared the effects of student experimentation with Physical Manipulatives (PM) and Virtual Manipulatives (VM) on kindergarten students’ understanding of concepts related to the use of a beam balance as a means to compare and differentiate materials according to their mass. A pre-post comparison study design was used that involved four conditions, with 20 kindergarten students in each condition. The first and second conditions included students who had correct prior knowledge of what a balance beam does and who were given PM and VM, respectively, to use for the study's purposes. The third and fourth conditions included students who had an incorrect prior knowledge of what a balance beam does and who were also given PM and VM, respectively, to use. All conditions followed the same learning tasks, which was a series of experiments. The learning process occurred in the context of a clinical interview, which was used as a means to collect data before, during, and after the learning process. Findings revealed that kindergarteners learned more from experimentation in both PM conditions and the VM condition whose participants had correct prior knowledge of what a balance beam does than in the VM condition whose participants had an incorrect knowledge of what a balance beam does. Physicality appears to be a prerequisite for students’ understanding of concepts concerning the use of a beam balance, as a means to compare and differentiate materials according to their mass, only when the students have incorrect prior knowledge of what a beam balance does.     

Research in Science Education at the University of Glasgow

The purpose of this study was to examine how prior knowledge of cellular transport influenced how high school students in the USA viewed and interpreted graphic representations of this topic. The participants were Advanced Placement Biology students (n = 65); each participant had previously taken a biology course in high school. After assessing prior knowledge using the Diffusion and Osmosis Diagnostic Test, two graphical representations of cellular transport processes were selected for analysis. Three different methods of data collection—eye tracking, interviews, and questionnaires—were used to investigate differences in perceived salient features of the graphics, interpretations of the graphics, and processing difficulty experienced while attending to and interpreting the graphics. The results from the eye tracking data, interviews, and instructional representation questionnaires were triangulated and revealed differences in how high and low prior knowledge students attended to and interpreted particle differences, concentration gradient, the role of adenosine triphosphate , endocytosis and exocytosis, and text labels and captions. Without adequate domain knowledge, low prior knowledge students focused on the surface features of the graphics (ex. differences in particle colour) to build an understanding of the concepts represented. On the other hand, with more abundant and better‐organised domain knowledge, high prior knowledge students were more likely to attend to the thematically relevant content in the graphics, which enhanced their understanding. The findings of this study offer a more complete understanding of how differentially prepared learners view and interpret graphics and have the potential to inform instructional design.     

Evaluating the impact of an environmental education programme: an empirical study in Mexico

This study draws on information from 11 in‐depth interviews, two focus groups and 72 written questionnaires to evaluate an extra‐curricular environmental education programme on forestry designed for preparatory school students from a small rural community in Mexico. Specifically, the study assessed the impact of the programme on the ecological knowledge of 72 students. Qualitative feedback suggests that students learnt about forestry, acquired greater awareness of the importance of conservation for the local environment and enjoyed the participatory teaching methods used in the programme. Quantitative results show a positive and significant association between the number of times a student participated in the programme and the student’s ecological knowledge. Students who participated in the programme once had a 16.3% higher knowledge on ecological concepts and knew, on average, 1.5 more local forest plants than students who never attended it (p<.001). Findings suggest that the inclusion of participatory environmental education programmes in preparatory schools would improve the acquisition of ecological knowledge. Further research could consider the consistency of the findings by replicating participatory methods presented here and by using an experimental research design.     

The comparison study of Chinese and American secondary school students’ knowledge structure —an experimental research based on concept map assessment technique

A concept map is a schematic device for representing a set of concept meanings embedded in a framework of propositions. It can be used to evaluate students’ knowledge structure. This article introduces the comparative study of Chinese and American secondary school students’ knowledge structure. They are compared quantitatively and qualitatively in terms of mean score, individual proposition scores, proposition choice and map structure. The results indicate that students’ knowledge structures in the two countries are remarkably different. Compared with American students, Chinese students’ ability to take an exam is stronger and their mean score is higher. However, Chinese students need to improve their general knowledge and creativity although their basic knowledge is solid and they are better in mastering discipline knowledge and knowledge application.     

Concept maps for assessing change in learning: a study of undergraduate business students in first-year marketing in China

This paper explains the application of concept mapping to help foster a learning-centred approach. It investigates how concept maps are used to measure the change in learning following a two-week intensive undergraduate Marketing Principles course delivered to 162 Chinese students undertaking a Bachelor of Business Administration programme in China. Using four scoring procedures (breadth of knowledge, relational quality, structural quality and holistic quality), student learning is assessed in terms of prior and new knowledge, as well as improvement in knowledge structure. Concept map scores are correlated with conventional measures of overall student performance. The results provide strong evidence for improvement in students’ ability to externalise new learned concepts resulting from intensive instruction. Incorporating concept maps as a contemporary method of learning in the curriculum potentially enriches student learning, provides a measure of the impact of teaching on students’ learning and adds value to students’ higher education experience.     

Measurement and evaluation issues in education: The value of multivariate techniques in evaluating an innovative high school reform program

The purpose of this chapter is twofold. The first is to explore uses of new methods for the analysis of multilevel data in school effectiveness research with limited observed data, where the word “limited” is used because more often than not interesting information about changes in teaching styles and the influence of teachers on students is limited, and sometimes incomplete. The statistical and mathematical models we use to analyze our data should allow us to deal with these facts. Secondly a contribution is made to school effectiveness research by establishing that multilevel causes determine student achievement. The combination of student ability, a new curriculum and teacher satisfaction predict student achievement in a multilevel multivariate analysis. The rich data set contains 342 students, 20 teachers and more than 50 variables for each level, the student level and the teacher level. Half of the teachers were in an experimental condition, teaching a new type of class, called Humanitas, the other half taught history in a traditional way, and served as the control group in twelve schools in Los Angeles county.The statistical models used in this chapter can be described as (1) a data reduction model, where several observed variables are combined into a single scale, (2) a regression model for multilevel data, with a single dependent variable, and (3) a model for multilevel multivariate analysis. One of the conclusions of this chapter is that researchers should have a certain amount of knowledge of how the data are generated regarding the important relationships between the variables, before choosing a complicated model. It is advisable to explore the data, guided by a theoretical model, before fitting complicated hierarchical bilevel path models with latent variables.Above statements are in support of Rubin's statement (1991, p. 306): “that complicated analyses are often necessary in complicated problems, but unless their conclusions can be represented in clear summaries that can be supported by the real data and the observations in the field, they should not be trusted. Too often, the results of complicated analyses are summarized without making clear how the conclusions are consistent with the raw data”. The multilevel analyses in this chapter are such complicated analysis tools. The data are experimental data, collected under close observation by the researchers. The knowledge gained from the complicated data analysis is supported by observations, interviews and simple exploratory data analysis. The positive effect, on students and teachers, of an experimental curriculum in ten secondary high schools is revealed by multilevel path analysis with latent variables.     

What Motivates Today’s Criminal Justice Student to Become an Engaged Learner?

The use of student-centered High Impact Practices (HIPs) has become increasingly popular across university and college faculty. HIPs increase student development of critical thinking, problem solving, and communication skills. HIPs also provide students the opportunity to engage in real-world application of course knowledge. While HIPs increase student engagement and intrinsic motivation, little research has explored student perception of these concepts. This article describes a pilot course offered to upper level criminal justice students that incorporated three HIPs: undergraduate research, collaborative assignments, and service/community-based learning. Semi-structured interviews were used to facilitate individual student discussion regarding their perceptions and experiences of the course related to their level of motivation and engagement as compared to traditional classes. Results indicated student expectations for the course were exceeded; students believed the course would positively impact their future career/education plans; and the HIPs motivated the students to engage more than their traditional lecture-oriented courses.     

High school students’ difficulties in understanding osmosis

Preconceptions and misconceptions of 500 secondary students in years 9‐12 regarding osmosis were studied through the use of five different kinds of instruments: prior learning inventory, self‐report knowledge inventory, true/false test, definitions and clinical interviews. Some of the major findings were: (1) The most frequent explanation offered to osmosis is ‘a desire or drive towards equalizing concentrations’. (2) Hardly any student uses the concept ‘water concentration’. (3) Most students fail to realize that in dynamic equilibrium water molecules keep moving. (4) Students have special difficulty in understanding osmotic relations in plants. (5) Many students have difficulty in grasping solute‐solvent and concentration‐quantity relations. (6) Students often perform laboratory experiments without really understanding the underlying principles. Practical and theoretical implications are discussed.