Measuring twenty-first century skills: development and validation of a scale for in-service and pre-service teachers

A self-report scale that measures teachers’ confidence in teaching students about twenty-first century skills was developed and validated with pre-service and in-service teachers. First, 16 items were created to measure teaching confidence in six areas: information literacy, collaboration, communication, innovation and creativity, problem solving, and responsible citizenship. Then, the items were tested in two groups of pre-service teachers and two groups of in-service teachers respectively. Exploratory factor analyses and confirmatory factor analyses were conducted to examine the scale’s validity and reliability. The analyses identified a three-construct scale including innovation and problem solving, collaboration, and utility of technology for pre-service teachers and a one-dimension scale, cross-functional skills, for in-service teachers. This study suggests that different scales should be used to measure in-service teachers’ and pre-service teachers’ confidence in teaching twenty-first century skills.     

Development and validation of an instrument for assessing high-school students’ perceptions of socio-scientific issues-based learning in biology

In this research, we developed and validate an instrument for assessing high-school students’ perceptions of socio-scientific issues (SSI)-based learning in biology. The development of the instrument was carried out using the following four steps: item formulation, content validation, construct validation and reliability calculation. According to an extensive literature review, 28 initial items were formulated for four SSI-based instruction aspects: (1) contextualisation of SSI, (2) student involvement, (3) student attitudes towards SSI-learning and (4) SSI-based learning objectives. Based on the pilot study conducted for construct validation, 24 items were successfully constructed for the four scales and subsequently administered in biology classrooms to 151 students. Factor analysis showed that each of the 24 items had a factor loading at least 0.40 on its own scale and no other scale. Moreover, the four scales accounted for 58.32% of the total variance and the alpha reliability coefficients for the scales ranged from 0.75 to 0.85. According to these findings, the SSI-based Learning Environment Inventory in Biology is valid, reliable and useful.

     

Assessing test‐taking strategies of university students: developing a scale and estimating its psychometric indices

Test‐taking strategies are important cognitive skills that strongly affect students’ performance in tests. Using appropriate test‐taking strategies improves students’ achievement and grades, improves students’ attitudes toward tests and reduces test anxiety. This results in improving test accuracy and validity. This study aimed at developing a scale to assess students’ test‐taking strategies at university level. The scale developed was passed through several validation procedures that included content, construct and criterion‐related validity. Similarly, scale reliability (internal reliability and stability over time) was assessed through several procedures. Four samples of students (50, 828, 553 and 235) participated by responding to different versions of the scale. The scale developed consists of 31 items distributed into four sub‐scales: Before‐test, Time management, During‐test and After‐test. To the researcher’s knowledge, this is the first comprehensive scale developed to assess test‐taking strategies used by university students.     

Measuring ocean literacy of high school students: psychometric properties of a Chinese version of the ocean literacy scale

This study established a Chinese scale for measuring high school students’ ocean literacy. This included testing its reliability, validity, and differential item functioning (DIF) with the aim of compensating for the lack of DIF tests focusing on current scales. The construct validity and reliability were verified and tested by analyzing the established scale’s items using the Rasch model, and a gender DIF test was conducted to ensure the test results’ fairness when distinct groups were compared simultaneously. The results indicated that the scale established in this study is unidimensional and possesses favorable internal consistency and construct validity. The gender DIF test results indicated that several items were difficult for either female or male students to correctly answer; however, the experts and scholars discussed these items individually and suggested retaining them. The final Chinese version of the ocean literacy scale developed here comprises 48 items that can reflect high school students’ understanding of ocean literacy—which helps students understand the topics of marine science encountered in real life.     

Exploring Secondary Students' Knowledge and Misconceptions about Influenza: Development,validation, and implementation of a multiple-choice influenza knowledge scale

Understanding infectious diseases such as influenza is an important element of health literacy. We present a fully validated knowledge instrument called the Assessment of Knowledge of Influenza (AKI) and use it to evaluate knowledge of influenza, with a focus on misconceptions, in Midwestern United States high-school students. A two-phase validation process was used. In phase 1, an initial factor structure was calculated based on 205 students of grades 9–12 at a rural school. In phase 2, one- and two-dimensional factor structures were analyzed from the perspectives of classical test theory and the Rasch model using structural equation modeling and principal components analysis (PCA) on Rasch residuals, respectively. Rasch knowledge measures were calculated for 410 students from 6 school districts in the Midwest, and misconceptions were verified through the χ ² test. Eight items measured knowledge of flu transmission, and seven measured knowledge of flu management. While alpha reliability measures for the subscales were acceptable, Rasch person reliability measures and PCA on residuals advocated for a single-factor scale. Four misconceptions were found, which have not been previously documented in high-school students. The AKI is the first validated influenza knowledge assessment, and can be used by schools and health agencies to provide a quantitative measure of impact of interventions aimed at increasing understanding of influenza. This study also adds significantly to the literature on misconceptions about influenza in high-school students, a necessary step toward strategic development of educational interventions for these students.     

Reliability and validity of the Learning Styles Questionnaire

The reliability and predictive validity of a short form of Honey and Mumford's Learning Styles Questionnaire were examined. Students' learning preferences were measured by self-report ratings on their degree of likings for 19 teaching and learning activities commonly used in higher and professional education. Subjects were 381 second-year undergraduates enrolled in accountancy, engineering and communication programmes at the Hong Kong Polytechnic. The alpha coefficients for the four learning styles scales were quite low, ranging from 0.311 for the Pragmatist scale to 0.421 for the Reflector scale. Factor analysis of the items did not reveal any coherent factor structure congruent with the underlying constructs. However, significant though weak correlations were found between the learning styles scores and the learning preferences of the students as hypothesised.     

Assessing 10- to 11-year-old children’s performance and misconceptions in number sense using a four-tier diagnostic test

Background: Number sense is a key topic in mathematics education, and the identification of children’s misconceptions about number is, therefore, important. Information about students’ serious misconceptions can be quite significant for teachers, allowing them to change their teaching plans to help children overcome these misconceptions. In science education, interest in children’s alternative conceptions has led to the development of three- and four-tier tests that not only assess children’s understandings and misconceptions, but also examine children’s confidence in their responses. However, there are few such tests related to mathematical content, especially in studies of number sense.

Purpose: The purpose of this study was to investigate children’s performance and misconceptions with respect to number sense via a four-tier diagnostic test (Answer Tier → Confidence rating for Answer Tier → Reason Tier → Confidence rating for Reason Tier).

Design and method: A total of 195 fifth graders (10–11 years old) from Taiwan participated in this study. The four-tier test was web-based and contained 40 items across five components of number sense.

Findings: The results show that (1) students’ mean confidence rating for the answer tier was significantly higher than for the reason tier; (2) an average of 68% of students tended to have equal confidence ratings in both answer and reason tiers; (3) students who chose correct answers or reasons had higher mean confidence ratings in most items (36 out of 40) than those who did not; and (4) 16 misconceptions were identified and most of them were at a strong level.

Conclusion: The four-tier test was able to identify several misconceptions in both the answer and reason tier and provide information about the confidence levels. By using such information, teachers may be better positioned to understand the nature of learners’ misconceptions about number sense and therefore support their pupils’ progress in mathematics.     

Student perceptions of personalised learning: development and validation of a questionnaire with regional secondary students

This project sought to evaluate regional students’ perceptions of their readiness to learn, assessment processes, engagement, extent to which their learning is personalised and to relate these to academic efficacy, academic achievement, and student well-being. It also examined teachers’ perceptions of students’ readiness to learn, the assessment process, engagement, and the extent to which students’ learning is personalised. The sample involved students in years 7–10 from six Victorian secondary schools. An instrument Personalised Learning Environment Questionnaire (PLQ) was developed to measure students’ perceptions of the factors effecting the implementation of Personalised Learning Plans (PLPs). It employed the latest scales to assess a range of PLP indicator variables, with all scales modified for use in an Australian context, and the total number of items kept to a minimum. Only scales more sensitive to PLPs were used to minimise the length of the instrument. There were three outcome variables: academic efficacy, academic achievement, and student well-being. The PLPs were assessed through scales that assess several contributing, distinct dimensions: selfdirected learning readiness, personal achievement, goal orientation, learning environment, personalised teaching and learning initiatives, curriculum entitlement and choice, and perceptions of assessment for learning. The trail PLQ was administered to 220 students, resulting in a 19 scale questionnaire with three or four items per scale. This paper reveals good data to model fit for the majority of items and each scale had good reliability. The paper describes the analytic techniques and results, how the instrument was refined and identifies common and uncommon student perceptions based on a post hoc analysis. The main study consisted of 2,407 students from four schools in the Bendigo Education Plan. They responded to this refined 19 scale version of the PLQ that was developed from the trial PLQ. All scales had satisfactory internal consistency reliability.     

Development and validation of an instrument for assessing attitudes of high school students about recycling

Recycling and its applications are growing significantly due to the great potential for solving a range of environmental problems in society. Nevertheless, there are currently very few instruments that can provide valid and reliable data on students’ attitudes toward recycling. In this regard, this article focuses on the development and validation of Recycling Attitude Scale (RAS). The items in the RAS were developed initially from the responses to three open-ended items by 53 tenth and eleventh grade students and literature review on recycling attitude. This initial form was pilot tested with 356 tenth and eleventh grade students and then subjected to exploratory factor analysis. Subsequently, the revised version of the scale was administrated to 694 tenth grade students, and the results were subjected to confirmatory factor analysis and reliability analysis. The RAS consists of 21 items in three subscales, with responses recorded on a four-point Likert scale, options ranging from strongly agree to strongly disagree. Cronbach’s alpha reliability coefficient (α) of the scale was found to be .87. The results indicate that the RAS a potentially valuable tool for both instructors and researchers in Turkey for the assessment of the attitudes toward recycling held by students in secondary education.     

Development and validation of Children’s Responsible Environmental Behavior Scale

Though environmentally responsible behavior (ERB) has been a focus of many studies in the field of environmental education, very few scales have been developed to assess children’s ERB. In this regard, this article focuses on the development and validation of Children’s Responsible Environmental Behavior Scale (CREBS) and also reports the psychometric properties of this scale. The items in the CREBS were developed initially from the responses to four open-ended items by 229 fourth and fifth grade students. This initial form was pilot tested with 673 fourth and fifth graders and then subjected to exploratory factor analysis. Later, the revised version of the scale was administrated to 2412 fifth graders, and those results were subjected to confirmatory factor analysis and reliability analysis. CREBS consists of 23 items measured using a seven-point Likert-type scale, which have been organized into four sub-scales: political action (six items, α?=?.92); eco-management (six items, α?=?.70); consumer and economic action (five items, α?=?.70); and Individual and Public Persuasion (six items, α?=?.80). Study results indicate that CREBS can be used for exploring the extent to which elementary school students in Turkey demonstrate four types of behavior to help prevent and resolve environmental problems and issues.     

Developing and validating test items for first-year computer science courses

We report on the development, validation, and implementation of a collection of test items designed to detect misconceptions related to first-year computer science courses. To this end, we reworked the development scheme proposed by Almstrum et al. (SIGCSE Bulletin 38(4):132–145, 2006) to include students’ artifacts and to simultaneously incorporate think-aloud interviews and flash tests. We also investigated to what extent the practical efficiency of detecting certain misconceptions could be increased without significantly affecting the sensitivity of the instrument, and present positive and negative results regarding this goal. The results of a first transfer and implementation study suggest that it is indeed possible to use the test items in a large-scale practical setting – both as diagnostic instruments and as interventions.     

Development and validation of an instrument to assess secondary school students’ perceptions of assessment tasks

Research aimed at developing and validating an instrument to assess secondary school students’ perceptions of assessment tasks was conducted. Following a review of literature, a five‐scale instrument of 40 items was trialled with a sample of 658 science students in 11 English secondary schools. Based on internal consistency reliability data and exploratory factor analysis, refinement decisions resulted in a five‐scale instrument called the Perceptions of assessment tasks inventory (PATI). The scales of the PATI are Congruence with planned learning, Authenticity, Student consultation, Transparency and Diversity.     

Cognitive validity of students’ self-reports of classroom mastery goal structure: What students are thinking and why it matters

Cognitive interviews were employed to systematically examine the cognitive validity of self-report survey items extensively used to assess classroom mastery goal structure. In a sample of elementary and middle school students, items were identified that functioned according to their intended meaning and those eliciting less accurate interpretations as conceptually defined by mastery goal structure cognitive validity criteria. Evidence suggested that items framed to focus on students’ teachers (i.e., teacher goals) were more cognitively valid than were items that focused students on their classroom context. Items with abstract terms yielded less accurate interpretations. We discuss implications of determining the cognitive validity of scales used to assess achievement goal structure and related self-report instruments.     

Development and validation of an educational robot attitude scale (ERAS) for secondary school students

Humanoid robots equipped with social skills have come to be used increasingly in the field of education across various subfields such as science education, special education, and foreign language education. In order to enhance the use of humanoid robots in educational settings, and to comprehensively evaluate its impact on the transformation of the class, understanding students’ attitudes towards the use of robots for educational purposes plays a critical role. This paper outlines the implementation and validation procedures of an educational robot attitude scale (ERAS) developed to measure the attitudes of secondary school students towards the use of humanoid robots in educational settings. The sample of the study comprised of 232 secondary school students. The development and validation process consisted of exploratory factor analysis and convergent validity. The developed scale consists of 17 items and represents four factors of students’ attitude: engagement, enjoyment, anxiety and intention. These four factors accounted for 66% of the total variance of the scale. Internal consistency coefficient for the whole scale was found .90 according to the reliability analysis. The results of the study suggest that the scale is a valid, reliable, and efficient tool for measuring the dimensions of students’ attitudes towards humanoid robots in educational settings.     

Rasch Validation of a Measure of Reform-Oriented Science Teaching Practices

Growing evidence from recent curriculum documents and previous research suggests that reform-oriented science teaching practices promote students’ conceptual understanding, levels of achievement, and motivation to learn, especially when students are actively engaged in constructing their ideas through scientific inquiries. However, it is difficult to identify to what extent science teachers engage students in reform-oriented teaching practices (RTPs) in their science classrooms. In order to exactly diagnose the current status of science teachers’ implementation of the RTPs, a valid and reliable instrument tool is needed. The principles of validity and reliability are fundamental cornerstones in developing a robust measurement tool. As such, this study was motivated by the desire to point out the limitations of the existing statistical and psychometric analyses and to further examine the validation of the RTP survey instrument. This paper thus aims at calibrating the items of the RTPs for science teachers using the Rasch model. The survey instrument scale was adapted from the 2012 National Survey of Science and Mathematics Education (NSSME) data. A total of 3701 science teachers from 1403 schools from across the USA participated in the NSSME survey. After calibrating the RTP items and persons on the same scale, the RTP instrument well represented the population of US science teachers. Model-data fit determined by Infit and Outfit statistics was within an appropriate range (0.5–1.5), supporting the unidimensional structure of the RTPs. The ordered category thresholds and the probability of the thresholds showed that the five-point rating scale functioned well. The results of this study support the use of the RTP measure from the 2012 NSSME in assessing usage of RTPs.     

Students’ confidence in their performance judgements: a comparison of different response scales

We report results of two studies on metacognitive accuracy with undergraduate education students. Participating students were asked to judge their personal performance in a multiple-choice exam as well as to state their confidence in their performance judgement (second-order judgement [SOJ]). In each study, we compared four conditions that differed in the type of the presented 5-point confidence scale for SOJs. In Study 1, four bipolar scales with different labels were applied; in Study 2, unipolar and bipolar scales were implemented. The results of Study 1 with N?=?420 students show that undergraduates on average provided accurate performance estimations. However, students were not aware of their judgement accuracy, shown by a low fit of SOJ and judgement accuracy. In addition, the type of provided scale significantly influenced the SOJs. Study 2 with N?=?348 students replicated the findings of the first study and gave further insight into the effects of uni- vs. bipolar response scales.     

Click-On-Diagram Questions: a New Tool to Study Conceptions Using Classroom Response Systems

Geoscience instructors depend upon photos, diagrams, and other visualizations to depict geologic structures and processes that occur over a wide range of temporal and spatial scales. This proof-of-concept study tests click-on-diagram (COD) questions, administered using a classroom response system (CRS), as a research tool for identifying spatial misconceptions. First, we propose a categorization of spatial conceptions associated with geoscience concepts. Second, we implemented the COD questions in an undergraduate introductory geology course. Each question was implemented three times: pre-instruction, post-instruction, and at the end of the course to evaluate the stability of students’ conceptual understanding. We classified each instance as (1) a false belief that was easily remediated, (2) a flawed mental model that was not fully transformed, or (3) a robust misconception that persisted despite targeted instruction. Geographic Information System (GIS) software facilitated spatial analysis of students’ answers. The COD data confirmed known misconceptions about Earth’s structure, geologic time, and base level and revealed a novel robust misconception about hot spot formation. Questions with complex spatial attributes were less likely to change following instruction and more likely to be classified as a robust misconception. COD questions provided efficient access to students’ conceptual understanding. CRS-administered COD questions present an opportunity to gather spatial conceptions with large groups of students, immediately, building the knowledge base about students’ misconceptions and providing feedback to guide instruction.     

大学生公共安全素质量表的编制

研究者通过访谈调查,编制出80个题目,在737名大学生中进行初测,通过项目鉴别度分析和探索性因素分析,保留了24个题目。24个题目包括安全体质、安全心理、安全技能和安全意识等四个因子,内部一致性系数均大于0.73。在571名大学生中进行了复测,该量表的内部一致性系数为0.88;从复测的大学生中随机抽取51人进行重测,重测信度在0.66~0.85之间。验证性因素分析验证了量表与构想模型拟合较好,具有很好的结构效度。最后,研究者进行了大学生公共安全素质量表的结构分析、信度和效度分析,并讨论了存在的问题。     

Development of the Transferable Learning Orientations tool: providing metacognitive opportunities and meaningful feedback for students and instructors

This study encapsulates the development and testing of the Transferable Learning Orientations (TLO) tool. It is a triangulated measure built on select scales from the Motivated Strategies for Learning Questionnaire (MSLQ), together with multiple-choice items adapted from the lifelong learning VALUE rubric, and an open-ended response for each dimension. Select scales from the MSLQ were tested in a range of undergraduate courses, and the TLO (version one) was developed and piloted in a first-year engineering course. Minor refinements were made, and the TLO (version two) was retested with second-year undergraduates. The TLO is designed to engage students in meta-cognitive processes and provide meaningful feedback to students. The dimensions are outcome motivation, learning belief, self-efficacy, transfer and organisation. Results from the second-year group were more consistent and reliable than the first-year group, suggesting that context is an important factor. The scales demonstrate acceptable reliability, and the moderate correlations between scale scores and rubric ratings provide support for concurrent validity. We recommend the TLO be tested with broader populations to confirm psychometric properties and that it be implemented longitudinally to investigate the development of learning skills and changes in orientations over time.