Problems and issues in the use of concept maps in science assessment

The search for new, authentic science assessments of what students know and can do is well under way. This has unearthed measures of students' hands-on performance in carrying out science investigations, and has been expanded to discover more or less direct measures of students' knowledge structures. One potential finding is concept mapping, the focus of this review. A concept map is a graph consisting of nodes representing concepts and labeled lines denoting the relation between a pair of nodes. A student's concept map is interpreted as representing important aspects of the organization of concepts in his or her memory (cognitive structure). In this article we characterize a concept map used as an assessment tool as: (a) a task that elicits evidence bearing on a student's knowledge structure in a domain, (b) a format for the student's response, and (c) a scoring system by which the student's concept map can be evaluated accurately and consistently. Based on this definition, multiple concept-mapping techniques were found from the myriad of task, response format, and scoring system variations identified in the literature. Moreover, little attention has been paid to the reliability and validity of these variations. The review led us to arrive at the following conclusions: (a) an integrative working cognitive theory is needed to begin to limit this variation in concept-mapping techniques for assessment purposes; (b) before concept maps are used for assessment and before map scores are reported to teachers, students, the public, and policy makers, research needs to provide reliability and validity information on the effect of different mapping techniques; and (c) research on students' facility in using concept maps, on training techniques, and on the effect on teaching is needed if concept map assessments are to be used in classrooms and in large-scale accountability systems. © 1996 John Wiley & Sons, Inc.     

Comparison of two concept‐mapping techniques: Implications for scoring,interpretation, and use

We examine the equivalence of two construct‐a‐concept‐map techniques: construct‐a‐map with created linking phrases (C), and construct‐a‐map with selected linking phrases (S). The former places few constraints on the respondent and has been considered the gold standard; the latter is cost and time efficient. Both their products and processes are compared quantitatively and qualitatively as to total accuracy score, individual proposition scores, proposition choice, map structure complexity, proposition generation rate, and proposition generation procedures. We conclude that the two mapping techniques are not equivalent: The C technique is better than the S technique in capturing students' partial knowledge, even though the S can be scored more efficiently than C. Based on their characteristics, if used as an assessment tool, the C technique is more suitable for formative assessment while the S technique is a better fit for large‐scale assessments. © 2005 Wiley Periodicals, Inc. J Res Sci Teach 42: 166–184, 2005     

Using concept maps to assess student learning in the science classroom: Must different methods compete?

This yearlong study was implemented in seventh-grade life science classes with the students' regular teacher serving as teacher/researcher. In the study, a method of scoring concept maps was developed to assess knowledge and comprehension levels of science achievement. By linking scoring of concept maps to instructional objectives, scores were based upon the correctness of propositions. High correlations between the concept map scores and unit multiple choice tests provided strong evidence of the content validity of the map scores. Similarly, correlations between map scores and state criterion-referenced and national norm-referenced standardized tests were indicators of high concurrent validity. The approach to concept map scoring in the study represents a distinct departure from traditional methods that focus on characteristics such as hierarchy and branching. A large body of research has demonstrated the utility of such methods in the assessment of higher-level learning outcomes. The results of the study suggest that a concept map might be used in assessing declarative and procedural knowledge, both of which have a place in the science classroom. One important implication of these results is that science curriculum and its corresponding assessment need not be dichotomized into knowledge/comprehension versus higher-order outcomes. © 1998 John Wiley & Sons, Inc. J Res Sci Teach 35: 1103–1127, 1998.     

Identifying critical junctures in learning in a college course on evolution

The purpose of this research study was to (a) describe how concept mapping can be used as an integral instructional strategy for teaching a college course on evolution, (b) evaluate the utility of incorporating concept mapping in a college course on evolution, (c) determine whether students' concept maps reveal “critical junctures” in learning as the course unfolds, and (d) assess the impact of concept mapping on students' study practices and on students' understanding of course content. Key findings include: (a) Critical junctures in learning evolution can be identified by monitoring the degree of concordance of superordinate concepts appearing on the class set of concept maps submitted after each of the course lectures; (b) students who made concept maps reported spending an average of 37% more study time on this college biology course than on their previous biology courses; and (c) the use of “seed concepts,” “micromapping,” a standard concept map format, and a standard concept map checklist made the strategy feasible for the instructor to implement and for the student to adopt. A concept map performance index formulas was also developed for this research study in order to assess students' overall mapping performance.     

概念图的布局算法研究

概念图是一种有效的知识可视化工具,而概述图的布局是影响其信息传递能力的重要因素,因此布局算法是实现概念图工具的重要技术基础.本文在Sugiyama层次布局算法的基础上,针对概念图的特点和布局要求,提出了一种概念图的布局算法：首先对概念图的概念节点和关系节点进行简化处理,然后应用Sugiyama层次布局算法的第一步（分层）,并对图中的关系节点进行特殊处理,调整分层,最后应用Sugiyama层次布局算法的“交叉减少”和“坐标分配”模块,得出概念图各个节点（包括概念节点、关系节点和虚拟节点）的坐标,最终得出概念图的布局.该算法已通过编程实现,并成功应用于我们自主研发的概念图工具软件中.     

Analysing Concept Maps as an Assessment Tool in Teaching Physics and Comparison with the Achievement Tests

Concept mapping is a technique that paves the way to represent knowledge schematically. In this research, concept mapping was used as an assessment method on the impulse–momentum topic. The purpose of this study was to determine teacher candidates’ knowledge about understanding of the concepts of impulse and momentum by comparing and contrasting two different methods; namely, students’ concept maps and an achievement test. The mean of teacher candidates’ concept map scores are extremely low when compared with the scores of the achievement test. In addition, it was seen that although a great number of concepts were written down, not many relationships were established between these concepts. There is a weak correlation between the achievement test and the concept map scores since concept maps assess the students’ knowledge from a conceptual perspective while the achievement tests measure the level of students’ knowledge on the topic and his/her ability to apply this knowledge on different occasions.     

Concept map assessment of classroom learning: Reliability,validity, and logistical practicality

The psychometric characteristics and practicality of concept mapping as a technique for classroom assessment were evaluated. Subjects received 90 min of training in concept mapping techniques and were given a list of terms and asked to produce a concept map. The list of terms was from a course in which they were enrolled. The maps were scored by pairs of graduate students, each pair using one of six different scoring methods. The score reliability of the six scoring methods ranged from r = .23 to r = .76. The highest score reliability was found for the method based on the evaluation of separate propositions represented. Correlations of map scores with a measure of the concept maps' similarity to a master map provided evidence supporting the validity of five of the six scoring methods. The times required to provide training in concept mapping, produce concepts, and score concept maps were compatible with the adoption of concept mapping as classroom assessment technique. © 1999 John Wiley & Sons, Inc. J Res Sci Teach 36: 475–492, 1999     

An analysis of prospective teachers’ knowledge for constructing concept maps

Background: Literature contends that a teacher’s knowledge of concept map-based tasks influence how their students perceive the task and execute the creation of acceptable concept maps. Teachers who are skilled concept mappers are able to (1) understand and apply the operational terms to construct a hierarchical/non-hierarchical concept map; (2) identify the legitimacy of the constructed concept map by verifying its graphical structure and its educational utility; and (3) determine the inherent ‘good’ and ‘poor’ qualities of the resulting graphical structure to reiterate the ‘good’ qualities and to coach and provide feedback to alleviate ‘poor’ qualities.

Purpose: The purpose of this study was to investigate the nature of prospective teachers’ knowledge underpinning the technique used to construct concept maps and thus, explicate their facility to construct concept maps.

Sample, Design and Methods: Data consisted of 200 concept maps constructed by prospective teachers in an elementary science methods course.

Results: Analysis revealed that the prospective teachers had predominantly constructed either hierarchical and/or non-hierarchical concept maps. It is likely that their maps reflect the teaching that they themselves would have experienced in their science classrooms during their own education. Additionally, most of these concepts maps only contained the root concept and subordinate concepts and lacked directional linking lines, linking phrases, labelled lines and propositions.

Conclusions: We argue that teacher educators need to assess their prospective teachers’ understanding of concept mapping in relation to the legitimacy (the nature and quality) of the end-products (graphical structures) of such practices. Prospective teachers also need to understand the educational utility of concept mapping in terms of how these end-products impact and/or effectuate learning.     

Concept map structure,gender and teaching methods: an investigation of students' science learning

Background: This study deals with the application of concept mapping to the teaching and learning of a science topic with secondary school students in Germany.

Purpose: The main research questions were: (1) Do different teaching approaches affect concept map structure or students' learning success? (2) Is the structure of concept maps influenced by gender? (3) Is the concept map structure a reliable indicator of students' learning success?

Sample: One hundred and forty-nine high-achieving 5th-grade students from four German secondary schools participated in the study. The average age of participants was 10½ years. Gender distribution was balanced. Students produced concept maps working in small, single-sex groups.

Design and methods: There were two teaching approaches used: one based upon teacher-centred instruction and one consisting of student-centred learning. Both were followed by a concept-mapping phase. Student groups experienced either one or the other teaching approach. Concept map structures were analysed using of the method of Kinchin, Hay and Adams. We defined three different possible types of concept map structure: spokes, chains and nets. Furthermore, for assessing a student's short- and longer-term learning success, we constructed a multiple-choice knowledge test applied in a pre-, post-, retention-test design. Parametric tests, such as MANOVA, one-way ANOVA and t-tests were used to identify any differences in gender, teaching approach, number of nets per concept map and their interactions.

Results: Type of teaching approach had an effect on concept map structure but not on students' longer-term learning success. Students of the teacher-centred approach produced more net structures than those students who participated in the hands-on instruction. Subsequent analyses showed in total more net structures for female groups. The interaction of gender and number of nets per concept map showed a significant effect on students' longer-term learning success.

Conclusion: The study suggests that Kinchin's classification scheme for assessing concept map quality may be a good indicator of students' learning success when applied in combination with a knowledge test.     

Measuring the quality of e‐learning

This paper shows how concept mapping can be used to measure the quality of e‐learning. Six volunteers (all of them 3rd‐year medical students) took part in a programme of e‐learning designed to teach the principles of magnetic resonance imaging (MRI). Their understanding of MRI was measured before and after the course by the use of concept mapping. The quality of change in individuals' maps was assessed using criteria developed to distinguish between meaningful and rote‐learning outcomes. Student maps were also scored for evidence of conceptual richness and understanding. Finally, each map was compared directly with the content of the electronic teaching material. The results show that many of the student misconceptions were put right in the course of their learning but that many of the key concepts introduced in the teaching were ignored (or sometimes learnt by rote) by the students. This was because the teaching material locked these new ideas in structures and terminology that precluded meaning‐making among non‐experts. Our data suggest that students' prior knowledge is a key determinant of meaningful learning. We suggest that this must be acknowledged if the design and use of electronic teaching material is also to be meaningful. Ultimately, measures of student learning are the only authentic indicators of the quality of teaching through technology.     

An innovative concept map approach for improving students' learning performance with an instant feedback mechanism

Concept maps have been widely employed for helping students organise their knowledge as well as evaluating their knowledge structures in a wide range of subject matters. Although researchers have recognised concept maps as being an important educational tool, past experiences have also revealed the difficulty of evaluating the correctness of a concept map. It usually takes days or weeks for teachers to manually evaluate the concept maps developed by students; consequently, the students cannot receive timely feedback from the teachers, which not only affects their learning schedules, but also significantly influences the students' learning achievements. In this paper, a computer‐based concept map‐oriented learning strategy with real‐time assessment and feedback is proposed in order to cope with the problems mentioned above. Our approach provides immediate evaluation of concept maps and gives also real‐time feedback to the students. An experiment has been conducted to evaluate the effectiveness of this new strategy in comparison with the conventional computer‐based concept map approach. It is found that our innovative approach can be significantly beneficial to promote learning achievements as well as the learning attitudes of students.     

Using a computerized concept mapping program to assess preservice teachers' thinking about effective teaching

Concept mapping with Learning Tool, a computerized mapping program, was used to assess changes in the content and organization of 17 preservice teachers' concept maps for the topic of effective teaching. Preservice teachers in their senior year of an early childhood teacher education program constructed and revised concept maps with a partner. They entered reflections on each mapping experience into their reflective journals, developing implications for teaching. Analysis of the concept maps revealed that these students had a primary concern with classroom management throughout the year, linking diverse information to that concept. They evidenced detailed and diverse understandings under the labels of knowledge and organization. Professionalism was a common item, though it was less well developed. Their reflective journals indicated that these students moved from describing emotional reactions to using the computer program as a basis for reflection on the teaching/learning process. This study illustrates how concept mapping can be useful in describing students' evolving constructions of knowledge in a particular subject area, and in promoting reflection.     

Students' achievement in relation to reasoning ability,prior knowledge and gender

This study investigated students' achievement regarding photosynthesis and respiration in plants in relation to reasoning ability, prior knowledge and gender. A total of 117 eighth‐grade students participated in the study. Test of logical thinking and the two‐tier multiple choice tests were administered to determine students' reasoning ability and achievement, respectively. An analysis of covariance (ANCOVA) was conducted to assess the effect of reasoning ability on students' achievement. The independent variable was the reasoning ability (low, medium, high), the dependent variable was the scores on the two‐tier test. Students' grades in science in previous year were used as a covariate. Analysis revealed a statistically significant mean difference between students at high and low formal levels with respect to achievement. Stepwise multiple regression analysis revealed that reasoning ability, prior knowledge and gender were significant predictors of students' achievement in photosynthesis and respiration in plants, explaining 42% of the variance.     

Development of software to support argumentative reading and writing by means of creating a graphic organizer from an electronic text

This paper describes the development of a software program that supports argumentative reading and writing, especially for novice students. The software helps readers create a graphic organizer from the text as a knowledge map while they are reading and use their prior knowledge to build their own opinion as new information while they think about writing their essays. Readers using this software can read a text, underline important words or sentences, pick up and dynamically cite the underlined portions of the text onto a knowledge map as quotation nodes, illustrate a knowledge map by linking the nodes, and later write their opinion as an essay while viewing the knowledge map; thus, the software bridges argumentative reading and writing. Sixty-three freshman and sophomore students with no prior argumentative reading and writing education participated in a design case study to evaluate the software in classrooms. Thirty-four students were assigned to a class in which each student developed a knowledge map after underlining and/or highlighting a text with the software, while twenty-nine students were assigned to a class in which they simply wrote their essays after underlining and/or highlighting the text without creating knowledge maps. After receiving an instruction regarding a simplified Toulmin’s model followed by instructions for the software usage in argumentative reading and writing along with reading one training text, the students read the target text and developed their essays. The results revealed that students who drew a knowledge map based on the underlining and/or highlighting of the target text developed more argumentative essays than those who did not draw maps. Further analyses revealed that developing knowledge maps fostered an ability to capture the target text’s argument, and linking students’ ideas to the text’s argument directly on the knowledge map helped students develop more constructive essays. Accordingly, we discussed additional necessary scaffolds, such as automatic argument detection and collaborative learning functions, for improving the students’ use of appropriate reading and writing strategies.

     

Cognitive foundations for science assessment design: Knowing what students know about evolution

To improve assessments of academic achievement, test developers have been urged to use an “assessment triangle” that starts with research‐based models of cognition and learning [NRC (2001) Knowing what students know: The science and design of educational assessment. Washington, DC: National Academy Press]. This approach has been successful in designing high‐quality reading and math assessments, but less progress has been made for assessments in content‐rich sciences such as biology. To rectify this situation, we applied the “assessment triangle” to design and evaluate new items for an instrument (ACORNS, Assessing Contextual Reasoning about Natural Selection) that had been proposed to assess students' use of natural selection to explain evolutionary change. Design and scoring of items was explicitly guided by a cognitive model that reflected four psychological principles: with development of expertise, (1) core concepts facilitate long‐term recall, (2) causally‐central features become weighted more strongly in explaining phenomena, (3) normative ideas co‐exist but increasingly outcompete naive ideas in reasoning, and (4) knowledge becomes more abstract and less specific to the learning situation. We conducted an evaluation study with 320 students to examine whether scores from our new ACORNS items could detect gradations of expertise, provide insight into thinking about evolutionary change, and predict teachers' assessments of student achievement. Findings were consistent with our cognitive model, and ACORNS was revealing about undergraduates' thinking about evolutionary change. Results indicated that (1) causally‐central concepts of evolution by natural selection typically co‐existed and competed with the presence of naïve ideas in all students' explanations, with naïve ideas being especially prevalent in low‐performers' explanations; (2) causally‐central concepts were elicited most frequently when students were asked to explain evolution of animals and familiar plants, with influence of superficial features being strongest for low‐performers; and (3) ACORNS scores accurately predicted students' later achievement in a college‐level evolution course. Together, findings illustrate usefulness of cognitive models in designing instruments intended to capture students' developing expertise. © 2012 Wiley Periodicals, Inc. J Res Sci Teach 49: 744–777, 2012     

Influence of levels of information as presented by different technologies on students' understanding of acid,base, and ph concepts

We investigated how different levels of information presented by various technologies affected secondary students' understanding of acid, base, and pH concepts. Secondary students who were selected for the study had just completed their study of acid–base chemistry. No attempt was made to provide further instruction. We analyzed changes in the understanding of individual students by constructing concept maps from the propositions that the students used in interviews conducted before and after a series of acid–base titrations. After the initial interview, students were divided into three groups. Within each group, students individually performed the same set of titrations using different technologies: chemical indicators, pH meters, and microcomputer-based laboratories (MBL). After the titrations were completed, all students were interviewed again. We found that students using MBL exhibited a larger positive shift in their concept map scores, which indicates a greater differentiation and integration of their knowledge of acids and bases. The chemical indicator students exhibited a more moderate positive shift in their concept map scores, and the pH meter students exhibited a smaller positive shift. We also found that the MBL students constructed more inappropriate links in their concept maps than the chemical indicator or pH meter students. However, we speculate that this increased number of inappropriate links indicates a high level of involvement with the technology. We therefore argue that the level of information offered by the technology affected students' understanding of the chemical concepts.     

Exploring high school students' use of theory and evidence in an everyday context: the role of scientific thinking in environmental science decision‐making

This study examined 10th‐grade students' use of theory and evidence in evaluating a socio‐scientific issue: the use of underground water, after students had received a Science, Technology and Society‐oriented instruction. Forty‐five male and 45 female students from two intact, single‐sex, classes participated in this study. A flow‐map method was used to assess the participants' conceptual knowledge. The reasoning mode was assessed using a questionnaire with open‐ended questions. Results showed that, although some weak to moderate associations were found between conceptual organization in memory and reasoning modes, the students' ability to incorporate theory and evidence was in general inadequate. It was also found that students' reasoning modes were consistent with their epistemological perspectives. Moreover, male and female students appear to have different reasoning approaches.