(Mis)appropriations of criteria and standards‐referenced assessment in a performance‐based subject

This paper draws on semi‐structured interview data and participant observations of senior secondary Physical Education (PE) teachers and students at two school sites across 20 weeks of the school year. The data indicated that the teachers in this study made progressive judgements about students’ level of achievement across each unit of work without explicit or overt reference to the criteria and standards represented in the schools’ work programmes and in the Senior PE syllabus. The teachers’ justification for such an approach was that the criteria and standards had become for them sufficiently ‘internalised’. Determining students’ levels of achievement was for the teachers somewhat ‘intuitive’, being reliant on their memory of students’ performances, and influenced by the construct‐irrelevant affective characteristics of the students. It is argued in this paper that such construct‐irrelevance compromised the construct validity and possible inter‐rater reliability of the decisions made and advantaged some students and marginalised others on the basis of characteristics that were not specifically related to the learning expected from following the syllabus. The potential inequities of such an approach are discussed and suggestions are made for the consolidation of the validity and reliability of teachers’ judgements.     

Teachers’ Roles,Students’ Personalities,Inquiry Learning Outcomes,and Practices of Science and Engineering:The Development and Validation of the McGill Attainment Value for Inquiry Engagement Survey in STEM Disciplines

Inquiry engagement is a newly defined construct that represents the participation in carrying out practices of science and engineering to achieve learning outcomes and is influenced by learners’ personalities and teachers’ roles. Expectancy value theory posits that attainment values are important components of task values that, in turn, directly influence students’ achievement-related choices and performance. The current paper developed and validated the McGill Attainment Value for Inquiry Engagement Survey (MAVIES) with undergraduate students in STEM disciplines. The MAVIES is a 67-item, learner-focused survey that addresses four components that are theoretically important for engaging in scientific inquiry: (a) teachers’ roles, (b) students’ personalities, (c) inquiry learning outcomes, and (d) practices of science and engineering. Evidence for internal consistency and construct, content, and criterion validity was obtained from 85 undergraduates who had experience with scientific inquiry in diverse STEM fields. Confirmatory factor analyses confirmed factors that were consistent with role theory, Big Five personality traits, revised Bloom’s learning outcomes, and the Next Generation Science Standards. The MAVIES instrument is a reliable and valid instrument for measuring undergraduate students’ attainment values for scientific inquiry engagement in STEM disciplines.     

The assessment of intrinsic and extrinsic motivation and amotivation: Validity and reliability of the Greek version of the Academic Motivation Scale

Self‐determination theory provides an integrated conception of school‐ and academic motivation. The theory proposes a continuum comprising three types of motivation: intrinsic motivation (IM), extrinsic motivation (EM), and amotivation (AM), characterised by seven dimensions (IM = to know, to accomplish and to experience stimulation, EM = external regulation, introjection and identification, and Amotivation). The purpose of the present study was to examine the psychometric properties of the Academic Motivation Scale (AMS) with Greek high school students. Two studies were conducted to examine the factorial, construct, concurrent and predictive validity of the scale along with its reliability properties. Confirmatory factor analyses supported the proposed seven‐factor structure. The scale showed satisfactory levels of internal consistency and temporal stability. Additionally, indices of the scale’s construct, concurrent, and predictive validity were in the desired direction. These findings support the use of the Greek version of the AMS for the assessment of intrinsic motivation, extrinsic motivation, and amotivation.     

高职院校教师课堂教学质量学生评价的问卷编制及信效度检验

为编制一个可用于高职院校的教师课堂教学质量学生评价测量工具,根据高职院校教师课堂教学质量学生评价存在的问题,在《大学教师教学效果评价问卷(学生用)》基础上创新性地设计了高职院校教师课堂教学质量学生评价问卷(简称VSEEQ),开发了符合教育测量学标准的、现代教学与学习理论支持的VSEEQ评价问卷,施测并搜集了信效度资料。结果表明,VSEEQ评价问卷具有合理的维度结构,较好的内部一致性信度、重测信度、内容效度和结构效度。     

Measuring ocean literacy of high school students: psychometric properties of a Chinese version of the ocean literacy scale

This study established a Chinese scale for measuring high school students’ ocean literacy. This included testing its reliability, validity, and differential item functioning (DIF) with the aim of compensating for the lack of DIF tests focusing on current scales. The construct validity and reliability were verified and tested by analyzing the established scale’s items using the Rasch model, and a gender DIF test was conducted to ensure the test results’ fairness when distinct groups were compared simultaneously. The results indicated that the scale established in this study is unidimensional and possesses favorable internal consistency and construct validity. The gender DIF test results indicated that several items were difficult for either female or male students to correctly answer; however, the experts and scholars discussed these items individually and suggested retaining them. The final Chinese version of the ocean literacy scale developed here comprises 48 items that can reflect high school students’ understanding of ocean literacy—which helps students understand the topics of marine science encountered in real life.     

Achievement-oriented beliefs and their relation to academic expectations and school achievement among Qatari students

This study investigated the relation between motivational goals and university intentions, school valuing and school achievement. The premise of this study is that motivational goals play a key role in academic values and achievement. Confirmatory factor analysis was conducted to establish the construct validity of the motivational measures drawn from the Inventory of School Motivation (ISM): mastery (effort and task), performance (competition and social power), social solidarity (social concern and affiliation), extrinsic (praise and token). A set of eight regression analyses was conducted to examine the relations among these eight motivational measures, future goals to attend college, the value of school and six school subjects. Eight regression equations revealed that effort and task were the most significant predictors of the dependent variables of university intention, valuing school and school achievement scores. Students in Qatar exhibited ‘effort and task’ in goal motivation in relation to students’ beliefs regarding future academic goals and achievement scores in school subjects. As Qatar is a highly gender-segregated society, analyses were also conducted by gender.     

Assessing students’ perceptions of fit between secondary and higher education: a validation study of the SPFQ

ABSTRACT

Supporting first-year students to adjust to their new academic environment is a crucial task in higher education. Investigating students’ perceptions of fit between secondary and higher education could give higher education institutions valuable information for student feedback and support, when captured in a reliable and valid way. This study examines the construct validity, reliability, and criterion-related validity of the Students’ Perceived Fit Questionnaire (SPFQ) by using a longitudinal dataset (N?=?930, first-year students). The results of exploratory and confirmatory factor analysis supported a three-factor structure measuring students’ perceptions of: 1) the extent to which secondary education prepared them for higher education in terms of content knowledge; 2) resemblance in the teaching approach between secondary and higher education, and 3) the need to adapt to their higher education study programme. Concerning the criterion-related validity, it was found that first-year students who experienced similarities in teaching approach are more likely to persist in their studies. Furthermore, our results indicated that first-year students who experienced less need to adapt to their higher education study programme felt more self-efficacious regarding their own learning and, consequently, were more likely to persist in their studies.     

Using Student Video Cases to Assess Pre-service Elementary Teachers’ Engineering Teaching Responsiveness

Our study addresses the need for new approaches to prepare novice elementary teachers to teach both science and engineering, and for new tools to measure how well those approaches are working. This in particular would inform the teacher educators of the extent to which novice teachers are developing expertise in facilitating their students’ engineering design work. One important dimension to measure is novice teachers’ abilities to notice the substance of student thinking and to respond in productive ways. This teacher noticing is particularly important in science and engineering education, where students’ initial, idiosyncratic ideas and practices influence the likelihood that particular instructional strategies will help them learn. This paper describes evidence of validity and reliability for the Video Case Diagnosis (VCD) task, a new instrument for measuring pre-service elementary teachers’ engineering teaching responsiveness. To complete the VCD, participants view a 6-min video episode of children solving an engineering design problem, describe in writing what they notice about the students’ science ideas and engineering practices, and propose how a teacher could productively respond to the students. The rubric for scoring VCD responses allowed two independent scorers to achieve inter-rater reliability. Content analysis of the video episode, systematic review of literature on science and engineering practices, and solicitation of external expert educator responses establish content validity for VCD. Field test results with three different participant groups who have different levels of engineering education experience offer evidence of construct validity.     

My Attitudes Toward Science (MATS): the development of a multidimensional instrument measuring students’ science attitudes

The number of students in the United States choosing science, technology, engineering or mathematics careers is declining at a time when demand for these occupations is rapidly increasing. Numerous efforts have been undertaken to reverse this trend, yet results are uncertain. One’s attitude is key to many choices one makes, and this includes, for many, what career is pursued. Hence, teachers, informal science educators and researchers often wish to measure children’s attitudes towards science using a pretest and a posttest to determine the effects of a curriculum, an activity or an intervention. However, measuring children’s attitudes toward science has been problematic because of both the limited use of basic psychometrics in checking reliability and validity of instruments and the lack of a single construct of students’ attitudes towards science being surveyed. This article reports the development and testing of an instrument for measuring students’ science attitudes across several dimensions. Thirty-two scientists and teachers from the northeastern and south central United States participated in content validity trials. The instrument was field tested with 549 children (92 elementary-school students, 327 middle-school students and 130 high-school students) from 6 rural and suburban school systems located in the northeastern United States to determine inter-item reliability for each dimension. The resulting instrument, entitled My Attitudes Toward Science (MATS), has 40 items that measure four dimensions: (1) Attitude towards the subject of science; (2) Desire to become a scientist; (3) Value of science to society; and (4) Perception of scientists. The MATS, as a multidimensional instrument, can measure several facets of students’ attitude toward science and is designed to be used across grades levels and to be scored easily.     

Identifying exemplary science teachers through students’ perceptions of their learning environment

The purpose of this study was to examine students’ psychosocial perceptions of their science classroom learning environment in order to identify exemplary teachers. This mixed-method study used the valid and reliable What Is Happening In this Class? (WIHIC) questionnaire with over 3,000 middle school students in 150 classrooms in Australia. Exemplary teachers were defined as those whose students’ perceptions were more than one standard deviation above the mean for three of the five scales of Student Cohesiveness, Teacher Support, Involvement, Task Orientation and Equity. The reliability of the WIHIC scales ranged from 0.75 to 0.87. Eta² showed that the WIHIC was able to distinguish between the perceptions of the students in the classroom. The construct validity of the WIHIC for identifying these exemplary teachers was confirmed through interviews with students and these views are reported in the article.     

The reliability and validity of the cultural congruity and university environment scales with Chicana/o community college students

Following the calls for increased research on the educational experiences of Chicana/o community college students, and the development of culturally applicable measures for communities of color, this study examined the utility and the applicability of the Cultural Congruity Scale (CCS) and University Environment Scale (UES) for use with Chicana/o community college students. Applying a psychosociocultural framework, the reliability, construct, and criterion-related validity of the scales for use with a sample of 110 Chicana/o community college students was examined. Results demonstrated adequate reliability and construct validity, with indication of applicability of these scales for the study’s sample. Overall, the study challenges normative practices in educational research that students—despite their race/ethnicity, backgrounds, and histories—face similar educational experiences. Implications are discussed.     

Why increasing the number of raters only helps sometimes: Reliability and validity of peer assessment across tasks of different complexity

Number of raters is theoretically central to peer assessment reliability and validity, yet rarely studied. Further, requiring each student to assess more peers’ documents both increases the number of evaluations per document but also assessor workload, which can decline performance. Moreover, task complexity is likely a moderating factor, influencing both workload and validity. This study examined whether changing the number of required peer assessments per student / number of raters per document affected peer assessment reliability and validity for tasks at different levels of task complexity. 181 students completed and provided peer assessments for tasks at three levels of task complexity: low complexity (dictation), medium complexity (oral imitation), and high complexity (writing). Adequate validity of peer assessments was observed for all three task complexities at low reviewing loads. However, the impacts of increasing reviewing load varied by reliability vs. validity outcomes and by task complexity.     

Construction and validation of a classroom climate scale: a mixed methods approach

Students’ perceptions of their classroom climate have been found to relate significantly to students’ learning outcomes. The purpose of the present study was to construct an instrument for assessing elementary-school students’ perceptions of classroom climate, based on a previous instrument that was being used in Chile by a public national school mental health program as a tool for aiding teachers in improving classroom management, but which showed poor psychometric properties. We used a six-staged mixed-methods approach to construct relevant items and dimensions based on this measure and by adapting previously-existing scales. Item development included participatory construction of items involving program officials, focus groups with students, and a pilot study. The final version was administered to a sample of 6813 elementary-school students. Results showed adequate reliability and construct validity, convergent validity with school climate, and divergent validity with peer victimisation. When consequential validity was explored through semi-structured interviews with program officials and school administrators, we found that the instrument was being used as a tool for helping teachers to improve their school climate and management skills. We discuss the importance of constructing instruments using a mixed-methods approach.     

中学生社会适应状况评估的理论模型建构与验证

在整合已有理论和实证成果的基础上,从关键性任务分析和积极心理学视角出发,将中学生社会适应状况评估指标划分为内容领域和功能状态。通过开放问卷调查进一步确定了评估社会适应状态的四个具体领域,并据此编制自我报告问卷进行验证。在对全国2314名中学生所填写问卷进行分析后,形成青少年社会适应状况评估问卷。问卷最终共50个项目,包括自我肯定、自我烦扰、亲社会倾向、社会疏离、行事效率、违规行为、积极应对和消极退缩等8个一阶因子;分为自我适应、人际适应、行为适应和环境适应4个领域,以及积极适应和消极适应两种功能状态。经多质多法模型的交叉验证分析,问卷的二阶结构与理论模型拟合较好,可以从四个适应领域和两种功能状态考察中学生社会适应状况。初步验证了社会适应状况评估的领域-功能理论模型。     

Establishing the Initial Validity of the REDFLAGS Model: Implications for College Counselors

The aim of this study was to initially validate the REDFLAGS model, 8 cautionary warning signs of mental distress in college students. A test of internal consistency reliability and factor analysis supported the model's reliability and construct validity. Hierarchical logistic regression models endorsed the model's predictive validity; students’ recognition of the REDFLAGS model was significantly associated with increases in the odds of a peer‐to‐peer referral to the counseling center. Implications for college counselors are discussed.     

Using generalizability theory to examine the accuracy and validity of large-scale ESL writing assessment

Using generalizability (G-) theory, this study examined the accuracy and validity of the writing scores assigned to secondary school ESL students in the provincial English examinations in Canada. The major research question that guided this study was: Are there any differences between the accuracy and construct validity of the analytic scores assigned to ESL students and to NE students for the provincial English writing examination across three years? A series of G-studies and decision (D-) studies for three years were conducted to examine accuracy and validity issues. Results showed that differences in score accuracy did exist between ESL and NE students when initial (pre-adjudication) scores were used. The observed G-coefficients for ESL students were significantly lower than those for NE students in all three years, indicating that there were less accuracy and increased errors associated with the writing scores assigned to ESL students. Further, there were significantly less convergent validity in one year and less discriminant validity in all three years of the writing scores assigned to ESL students than to NE students. These findings raise a potential question about the presence of bias in the assessment of ESL students’ writing if initial scores were used.     

大学生心理亚健康自评量表的编制

介绍了高校大学生心理亚健康自评量表编制及对量表进行信效度分析。研究结果显示,量表含心理调适力、躯体状况、学习适应、人际关系、社会适应及睡眠障碍6个维度,6个维度与总量表间的相关系数在0.695-0.824间,有较好信度和结构效度,量表总体Cronbach’α系数为0.916,分量表α系数分别为0.810、0.809、0.785、0.807、0.796及0.661。量表可用于大学生心理亚健康症状筛查。     

Examining students’ affective commitment toward country: a case study of a Singapore primary school

The purpose of this study was to examine students’ affective commitment toward Singapore. Affective commitment refers to the sense of attachment to the nation state. The sample was taken from 286 students in a primary school. In the first section of the paper, we described the design of a Likert-type Affective Commitment to Country questionnaire. Factor analyses (principal component analysis and confirmatory factor analysis) showed evidence of construct validity for the 10-item scale, and an overall Cronbach alpha reliability coefficient of 0.91. In the second section, we reported the statistics related to the students’ affective commitment scores. Overall, a positive affective commitment toward the country was found. Results of our t-test analyses revealed that no statistically significant difference was found between boys and girls for each of the questionnaire items. However, students who had higher academic achievement reported significantly higher scores than their lower ability counterparts with regard to six items of the questionnaire. Suggestions for future research are discussed.