Comparison of the reliability and validity of scores from two concept‐mapping techniques*

This paper reports the results of a study that compared two concept‐mapping techniques, one high‐directed, “fill‐in‐the‐map,” and one low‐directed, “construct‐a‐map‐from‐scratch.” We examined whether: (1) skeleton map scores were sensitive to the sample of nodes or linking lines to be filled in; (2) the two types of skeleton maps were equivalent; and (3) the two mapping techniques provided similar information about students' connected understanding. Results indicated that fill‐in‐the‐map scores were not sensitive to the sample of concepts or linking lines to be filled in. Nevertheless, the fill‐in‐the‐nodes and fill‐in‐the‐lines techniques were not equivalent forms of fill‐in‐the‐map. Finally, high‐directed and low‐directed maps led to different interpretations about students' knowledge structure. Whereas scores obtained under the high‐directed technique indicated that students' performance was close to the maximum possible, the scores obtained with the low‐directed technique revealed that students' knowledge was incomplete compared to a criterion map. We concluded that the construct‐a‐map technique better reflected differences among students' knowledge structure. © 2001 John Wiley & Sons, Inc. J Res Sci Teach 38: 260–278, 2001     

The Assessment Agent System: design, development, and evaluation

This article reports the design, development, and evaluation of an online software application for assessing students’ understanding of curricular content based on concept maps. This computer-based assessment program, called the Assessment Agent System, was designed by following an agent-oriented software design method. The Assessment Agent System is composed of five types of software agents: instructor agent, student agent, management agent, assessment agent, and reporting agent. Software agents in the system, through communication and cooperation, collectively provide the functionalities of user-system interaction, user management, task authoring and management, assessment delivery, task presentation, response collection, automatic assessment, and reporting. One-to-one evaluations and group evaluations were conducted to reveal students’ perceptions of the Assessment Agent System. Measures of visual clarity, system functionality, consistency, as well as error prevention and correction indicate that the Assessment Agent System is a useful tool for large-scale assessment based on concept maps. Through the process of design, development, and evaluation of the Assessment Agent System, this study demonstrates the agent-oriented approach for producing educational software applications. Furthermore, this research explored the concept map assessment method for the Assessment Agent System. When node terms and linking phrases are provided, the assessment of student concept maps can be conducted automatically by comparing student concept maps with the criterion concept map, proposition by proposition. However, the validity of the proposition-comparing method depends on the accuracy and thoroughness of the criterion propositions. Therefore, assessment criteria need to be continually refined and improved through the examination of student-created propositions.     

An analysis of prospective teachers’ knowledge for constructing concept maps

Background: Literature contends that a teacher’s knowledge of concept map-based tasks influence how their students perceive the task and execute the creation of acceptable concept maps. Teachers who are skilled concept mappers are able to (1) understand and apply the operational terms to construct a hierarchical/non-hierarchical concept map; (2) identify the legitimacy of the constructed concept map by verifying its graphical structure and its educational utility; and (3) determine the inherent ‘good’ and ‘poor’ qualities of the resulting graphical structure to reiterate the ‘good’ qualities and to coach and provide feedback to alleviate ‘poor’ qualities.

Purpose: The purpose of this study was to investigate the nature of prospective teachers’ knowledge underpinning the technique used to construct concept maps and thus, explicate their facility to construct concept maps.

Sample, Design and Methods: Data consisted of 200 concept maps constructed by prospective teachers in an elementary science methods course.

Results: Analysis revealed that the prospective teachers had predominantly constructed either hierarchical and/or non-hierarchical concept maps. It is likely that their maps reflect the teaching that they themselves would have experienced in their science classrooms during their own education. Additionally, most of these concepts maps only contained the root concept and subordinate concepts and lacked directional linking lines, linking phrases, labelled lines and propositions.

Conclusions: We argue that teacher educators need to assess their prospective teachers’ understanding of concept mapping in relation to the legitimacy (the nature and quality) of the end-products (graphical structures) of such practices. Prospective teachers also need to understand the educational utility of concept mapping in terms of how these end-products impact and/or effectuate learning.     

The impact of a rubric and friendship on peer assessment: Effects on construct validity,performance, and perceptions of fairness and comfort

Construct validity of peer assessment (PA) is important for PA application, yet difficult to achieve. The present study investigated the impact of an assessment rubric and friendship between the assessor and assessee on construct validity of PA. Two-hundred nine bachelor students participated: half of them assessed a peer's concept map with a rubric whereas the other half did not use a rubric. The results revealed a substantial reliability and construct validity for PA. All students over-score their peers’ performance, but students using a rubric were more valid. Moreover, when using a rubric a high level of friendship between assessor and assessee resulted in more over-scoring. Use of a rubric resulted in higher quality concept maps for peer and expert ratings.     

Using rasch measurement to score,evaluate, and improve examinations in an anatomy course

Any examination that involves moderate to high stakes implications for examinees should be psychometrically sound and legally defensible. Currently, there are two broad and competing families of test theories that are used to score examination data. The majority of instructors outside the high‐stakes testing arena rely on classical test theory (CTT) methods. However, advances in item response theory software have made the application of these techniques much more accessible to classroom instructors. The purpose of this research is to analyze a common medical school anatomy examination using both the traditional CTT scoring method and a Rasch measurement scoring method to determine which technique provides more robust findings, and which set of psychometric indicators will be more meaningful and useful for anatomists looking to improve the psychometric quality and functioning of their examinations. Results produced by the more robust and meaningful methodology will undergo a rigorous psychometric validation process to evaluate construct validity. Implications of these techniques and additional possibilities for advanced applications are also discussed. Anat Sci Educ 7: 450–460. © 2014 American Association of Anatomists.     

Concept map assessment of classroom learning: Reliability,validity, and logistical practicality

The psychometric characteristics and practicality of concept mapping as a technique for classroom assessment were evaluated. Subjects received 90 min of training in concept mapping techniques and were given a list of terms and asked to produce a concept map. The list of terms was from a course in which they were enrolled. The maps were scored by pairs of graduate students, each pair using one of six different scoring methods. The score reliability of the six scoring methods ranged from r = .23 to r = .76. The highest score reliability was found for the method based on the evaluation of separate propositions represented. Correlations of map scores with a measure of the concept maps' similarity to a master map provided evidence supporting the validity of five of the six scoring methods. The times required to provide training in concept mapping, produce concepts, and score concept maps were compatible with the adoption of concept mapping as classroom assessment technique. © 1999 John Wiley & Sons, Inc. J Res Sci Teach 36: 475–492, 1999     

中美中学生学科知识结构的比较研究——基于概念图评价技术的实验研究

A concept map is a schematic device for representing a set of concept meanings embedded in a framework of propositions. It can be used to evaluate students’ knowledge structure. This article introduces the comparative study of Chinese and American secondary school students’ knowledge structure. They are compared quantitatively and qualitatively in terms of mean score, individual proposition scores, proposition choice and map structure. The results indicate that students’ knowledge structures in the two countries are remarkably different. Compared with American students, Chinese students’ ability to take an exam is stronger and their mean score is higher. However, Chinese students need to improve their general knowledge and creativity although their basic knowledge is solid and they are better in mastering discipline knowledge and knowledge application. __________ Translated from Bijiao Jiaoyu Yanjiu 比较教育研究 (Comparative Education Review), 2008, (2): 12–16     

Developing a construct‐based assessment to examine students' analogical reasoning around physical models in Earth Science

In recent years, science education has placed increasing importance on learners' mastery of scientific reasoning. This growing emphasis presents a challenge for both developers and users of assessments. We report on our effort around the conceptualization, development, and testing the validity of an assessment of students' ability to reason around physical dynamic models in Earth Science. Building from the research literature on analogical mapping and informed by the current perspectives on learning progressions, we present a three‐tiered construct describing the increasing sophistication of students' analogical reasoning around the correspondences and non‐correspondences between models and the Earth System: at the level of entities (Level 1), configurations in space or relative motion of entities (Level 2), and the mechanism or cause for observed phenomena (Level 3). Grounded in a construct‐centered design approach, we describe our process for developing assessments in order to examine and validate this construct, including how we selected topics and models, designed items, and developed outcome spaces. We present the specific example of one assessment centered on moon phases, which was administered to 164 8th and 9th grade Earth Science students as a pre/postmeasure. Two hundred ninety‐four responses were analyzed using a Rasch modeling approach. Item difficulties and student proficiency scores were calculated and analyzed regarding their relative performance with respect to the three levels of the construct. The analysis results provided initial evidence in support of the construct as conceived, with students displaying a range of analogical reasoning spanning all three construct levels. It also identified problematic items that merit further examination. Overall, the assessment has provided us the opportunity to better describe and frame the cognitive uses of models by students during learning situations in Earth Science. Implications for instruction and future directions for research in this area are discussed. © 2012 Wiley Periodicals, Inc. J Res Sci Teach 49: 713–743, 2012     

概念图的布局算法研究

概念图是一种有效的知识可视化工具,而概述图的布局是影响其信息传递能力的重要因素,因此布局算法是实现概念图工具的重要技术基础.本文在Sugiyama层次布局算法的基础上,针对概念图的特点和布局要求,提出了一种概念图的布局算法：首先对概念图的概念节点和关系节点进行简化处理,然后应用Sugiyama层次布局算法的第一步（分层）,并对图中的关系节点进行特殊处理,调整分层,最后应用Sugiyama层次布局算法的“交叉减少”和“坐标分配”模块,得出概念图各个节点（包括概念节点、关系节点和虚拟节点）的坐标,最终得出概念图的布局.该算法已通过编程实现,并成功应用于我们自主研发的概念图工具软件中.     

美国公立学校优秀教师激励体系及其评价

20世纪80年代以来，美国政府和教育机构为吸引、培养和挽留公立学校的优秀教师，在评估和激励机制方面进行了许多有益的探索。美国现行优秀教师激励体系大致可归纳为四个部分：优秀教师的国家级评价和标准、州级优秀教师评估标准和相应激励措施、把教师的工作表现与工资报酬紧密联系的教师报酬制度以及联邦、州政府和学区对优秀教师所授予的各种荣誉。该体系无疑也为我国优秀教师的培养和激励提供了制度上的借鉴。     

Assessing Readiness for E‐Learning

ABSTRACT Today, e‐learning is a common delivery media for education and training within many organizations. Yet, while both the supply and demand for e‐learning opportunities has risen in recent years, many professionals are beginning to question whether e‐learn‐ers are prepared to be successful in an online learning environment (e.g., Gug‐lielmino & Guglielmino, 2003; Watkins & Corry, 2005). After all, a learner's demonstrated success in a conventional education and training classroom may not be an adequate predictor of success in an e‐learning classroom. One way of gauging a potential online learner's readiness is through self‐assessment. As a first step in defining an instrument that measures an e‐learner's readiness, with the cooperation of volunteer participants from the U.S. Coast Guard, this study looked into the validity and internal consistency of items within a self‐assessment of e‐learning readiness that is under development, and provided data for the continuing development and revision of the instrument. Having demonstrated evidence of internal consistency and construct validity, the self‐assessment now provides a tool for continuing research into the prediction of e‐learning performance. Funding for this study was provided by the International Society for Performance Improvement.     

The comparison study of Chinese and American secondary school students’ knowledge structure —an experimental research based on concept map assessment technique

A concept map is a schematic device for representing a set of concept meanings embedded in a framework of propositions. It can be used to evaluate students’ knowledge structure. This article introduces the comparative study of Chinese and American secondary school students’ knowledge structure. They are compared quantitatively and qualitatively in terms of mean score, individual proposition scores, proposition choice and map structure. The results indicate that students’ knowledge structures in the two countries are remarkably different. Compared with American students, Chinese students’ ability to take an exam is stronger and their mean score is higher. However, Chinese students need to improve their general knowledge and creativity although their basic knowledge is solid and they are better in mastering discipline knowledge and knowledge application.     

中美中学生学科知识结构的比较研究——基于概念图评价技术的实验研究

概念图是一种以图表的形式反映概念和概念之间关系的空间网络知识结构图,它能全面地评价学生的知识结构。文章用平均分、茎叶图、命题构建和构图结构四个变量对中美中学生的学科知识结构进行了定量和定性的比较实验研究。实验表明,中美中学生的学科知识结构有显著差异。相对于美国中学生,中国中学生的应试能力强,平均分高,但分数分布不均匀;虽然他们的基础知识扎实,学科知识的掌握及运用较好,但在常识性知识和知识创新上有待提高。     

概念图策略在化学教学中的实验研究

概念图是一种能形象表达命题网络中一系列概念含义及其关系的图解。概念图作为一种教学策略,强调学生自身对知识的建构过程,注重师生之间的互动关系。实验研究表明,概念图教学策略应用于中学化学教学中,有利于学生更好地掌握化学概念体系,且可显著提高场依存性学生的学习成绩。     

Moving house for education in the pre‐school years

This paper uses data from the Millennium Cohort Study (MCS) to examine house moves that take place in the pre‐school years, focusing on families who move for the education of their children. We present results showing that education‐ related house moves do indeed occur in the pre‐school years with particular types of parents making these education‐related moves to ensure their children attend a good primary school. We then examine whether this demand for high quality schooling is associated with a house price premia by linking the MCS data to Land Registry data and show that parents are prepared to pay significantly more to buy a house located near to better performing primary schools, even before their children reach school starting age. We interpret this as evidence of demand for school quality in the early years as parents (especially more educated and advantaged parents) ‘gear up’ their quest for what they perceive to be better schooling for their children before they start school.     

Automatic testing and assessment of neuroanatomy using a digital brain atlas: Method and development of computer‐ and mobile‐based applications

Preparation of tests and student's assessment by the instructor are time consuming. We address these two tasks in neuroanatomy education by employing a digital media application with a three‐dimensional (3D), interactive, fully segmented, and labeled brain atlas. The anatomical and vascular models in the atlas are linked to Terminologia Anatomica. Because the cerebral models are fully segmented and labeled, our approach enables automatic and random atlas‐derived generation of questions to test location and naming of cerebral structures. This is done in four steps: test individualization by the instructor, test taking by the students at their convenience, automatic student assessment by the application, and communication of the individual assessment to the instructor. A computer‐based application with an interactive 3D atlas and a preliminary mobile‐based application were developed to realize this approach. The application works in two test modes: instructor and student. In the instructor mode, the instructor customizes the test by setting the scope of testing and student performance criteria, which takes a few seconds. In the student mode, the student is tested and automatically assessed. Self‐testing is also feasible at any time and pace. Our approach is automatic both with respect to test generation and student assessment. It is also objective, rapid, and customizable. We believe that this approach is novel from computer‐based, mobile‐based, and atlas‐assisted standpoints. Anat Sci Educ 2:244–252, 2009. © 2009 American Association of Anatomists.     

The Skilled Counselor Training Model: Skills Acquisition,Self‐Assessment,and Cognitive Complexity

The authors evaluated the effectiveness of the Skilled Counselor Training Model (SCTM; M. H. Smaby, C. D. Maddux, E. Torres‐Rivera, & R. Zimmick, 1999) in teaching counseling skills and in fostering counselor cognitive complexity. Counselor trainees who completed the SCTM had better counseling skills and higher levels of cognitive complexity than did counselor trainees who did not receive the training. Before training, both experimental and control group participants overestimated their skills performance. The control group persisted in this overestimation after training, whereas students who completed the SCTM had a much more accurate self‐assessment. Results suggest that skills‐based training may improve counseling skills and cognitive complexity in counselor trainees.     

Bayesian mapping of neural tube defects prevalence in Heshun County, Shanxi Province, China during 1998-2001

Objective： To estimate the prevalence rates of neural tube defects （NTDs） in Heshun County, Shanxi Province, China by Bayesian smoothing technique. Methods： A total of 80 infants in the study area who were diagnosed with NTDs were analyzed. Two mapping techniques were then used. Firstly, the GIS software ArcGIS was used to map the crude prevalence rates. Secondly, the data were smoothed by the method of empirical Bayes estimation. Results： The classical statistical approach produced an extremely dishomogeneous map, while the Bayesian map was much smoother and more interpretable. The maps produced by the Bayesian technique indicate the tendency of villages in the southeastern region to produce higher prevalence or risk values. Conclusions： The Bayesian smoothing technique addresses the issue of heterogeneity in the population at risk and it is therefore recommended for use in explorative mapping of birth defects. This approach provides procedures to identify spatial health risk levels and assists in generating hypothesis that will be investigated in further detail.     

Developing students' ability to ask more and better questions resulting from inquiry‐type chemistry laboratories

This study focuses on the ability of high‐school chemistry students, who learn chemistry through the inquiry approach, to ask meaningful and scientifically sound questions. We investigated (a) the ability of students to ask questions related to their observations and findings in an inquiry‐type experiment (a practical test) and (b) the ability of students to ask questions after critically reading a scientific article. The student population consisted of two groups: an inquiry‐laboratory group (experimental group) and a traditional laboratory‐type group (control group). The three common features investigated were (a) the number of questions that were asked by each of the students, (b) the cognitive level of the questions, and (c) the nature of the questions that were chosen by the students, for the purpose of further investigation. Importantly, it was found that students in the inquiry group who had experience in asking questions in the chemistry laboratory outperformed the control grouping in their ability to ask more and better questions. © 2005 Wiley Periodicals, Inc. J Res Sci Teach 42: 791–806, 2005