The role of treatment acceptability,effectiveness, and understanding in treatment fidelity: Predicting implementation variation in a middle school science program

Evaluating a school-based program is a particular challenge when teachers implementing the program. Variations in implementation can be difficult to measure and predict. We developed measures to explore variation in treatment implementation and serve as predictors of variation in a school-based science education program. Based on previous work, we focused on assessing treatment acceptability, effectiveness, and understanding among teachers as critical determinants of variations in program implementation using self-reported and objective measures of implementation. All three constructs were associated with implementation of the program. Our measures of these constructs show promise for use in formative and summative evaluations. Our stratification of program elements with implementation predictors can provide a template for future exploration of treatment fidelity.     

The way students’ internalize assessment criteria on inquiry reports

This paper, based on two case studies, presents an interpretative research on the processes used by high school students to internalize the assessment criteria of Physics and Chemistry inquiry reports. Findings support that understanding the assessment criteria is complex, mainly because of its terminology. However the discussion of exemplars can have a crucial role in this process. In order to operationalize the assessment criteria, students used strategies stemming from the social context (teacher, peers and other didactic sources) as well as from their individual experience (errors made, engagement in critical thinking and the creation of a favorable environment). However it also showed that some differences in operationalizing the assessment criteria were related to different students’ profile. The results point to a multi-strategy pedagogical approach to enhance students’ internalization of assessment criteria. Nevertheless reducing the tension between teachers’ expectations and students’ own standards of quality showed to be a complex process.     

中小学校本课程领导力的内涵及评价指标

提高校本课程领导力是推进中小学校校本课程建设、深化课程改革、提高教育教学质量的重要举措。当前,在中小学缺乏对课程领导力的评价,尤其是缺乏对校本课程领导力的课程规划、开发、实施和资源建设的评价。 规划,可以宏观地预测并调控学校校本课程发展的整体方向;开发,可以把学校课程规划的内容落实到实践;实施,可以把开发的内容落实到真正的课堂;资源建设,可 以使学校课程资源源源不断地得到补充。 基于此,通过厘清校本课程领导力的内涵,提出从校本课程的规划能力、开发能力、实施能力以及资源建设能力四个评价指标对校本课程领导力开展评价,以期构建评价中小学校本课程领导力的一个框架,提高校本课程领导力,促进学校的内涵发展。     

Rubrics for developing students’ professional judgement: A study of sustainable assessment in arts education

The participation of students in assessment is known to generate higher-order learning outcomes. This study aims to determine the usefulness of rubrics in aiding the incorporation of undergraduate students into assessor teams for developing their professional judgement. A quasi-experimental study examined the effects of a brief training programme on the use of rubrics, and of the participation of students in rubric creation and moderation discussions. We calculated Cronbach’s alphas, and intraclass correlation coefficients in order to examine the intra- and inter-rater reliability between all the members of the assessor teams. The results demonstrate that only participation in the rubric design and in the moderation discussions regulating their use helped undergraduate students to develop sound assessment skills. We infer that rubrics can help to promote professional judgement if they are conceived as instructional resources for defining and supporting the processes of negotiation and agreement that characterize an assessment culture.     

The content aspect of validity in a rubric-based assessment system for course syllabuses

The growing trend among universities to promote systems of programme and course evaluation entails more responsibility for faculties and departments. These systems require resources to ensure that they are not only valid and reliable but also effective and sustainable. The design of rubric-based assessment systems may provide a solution, but there is a gap in the research on curriculum evaluation concerning their use and validation. We examine the content aspect of validity in a rubric-based assessment system for course syllabuses using a mixed method that combines an analysis of the agreement among 23 experts with a phenomenographic study. With data gathered through a questionnaire linked to the Delphi technique, content validity indexes were calculated and the experts' different perspectives were identified. The content validity indexes (greater than .80) met the standards set out in literature, and the qualitative study of the experts' feedback showed three different perspectives on the system's use. Beyond providing evidence of the system's content validity, the study highlights the extent to which it is important to give appropriate consideration to experts' – and by extension final users' – experience in order to ensure the successful implementation of rubric-based assessment systems.     

We agree on what we see: Teacher and student perceptions of formative assessment practice

This study examines teacher and student perceptions of formative assessment (FA) activities used by teachers in the classroom. These activities are divided into five phases that together comprise the FA cycle: (1) clarifying expectations, (2) eliciting responses, (3) analysing and interpreting responses, (4) communicating about responses and (5) adjusting teaching and learning. Reliable questionnaires were used to measure the perceptions of 96 teachers and 1,095 students with regard to FA practice. Paired t-tests indicated no differences between the perceptions of teachers and students, except for with regard to clarifying expectations. Teachers used FA activities primarily to clarify expectations and elicit student responses, and they were least likely to apply them to adjust teaching and learning. The results suggest that the framework of the FA cycle could be used as an analytical lens for the reliable evaluation of the FA activities of teachers.     

E-assessment of online academic courses via students' activities and perceptions

The rapid growth of online courses in higher education has led to developments in the field of e-assessment. This paper presents a study, which examined the quality of online academic courses using a multidimensional assessment of students' activities and perceptions, using educational data mining and an online questionnaire. The assessment focused on four aspects: instructional, communication, course workload and overall learning experience. The course instructional model was found well-structured. The video lectures, assignments and materials designed for the online course were the most used and contributing learning resources. However, the number of students who entered the video lectures decreased as the course progressed. Low activity was found in the discussion forums. Students perceived the course workload as low. Overall, the learning experience was high and the students were highly satisfied. These findings provide insights that may assist in improving the quality of future online courses.     

Assessing conceptual knowledge using similarity judgments

This study evaluates the validity of a similarity judgments test (SJT) for assessing conceptual knowledge. 89 high school seniors judged the semantic similarity of a set of concepts. The resulting scores are calculated using a correlational approach as well as the MDS and Pathfinder approaches, respectively. The validity of the SJT is estimated by comparing (a) the scores with the test scores resulting from a multiple-choice test, (b) the scores resulting from pre- and post-instruction, and (c) the scores of students with high vs. low expertise. Our results support the validity of the SJT and show that both the correlational and the Pathfinder approach outclass the MDS approach. Recommendations to evaluate the SJT are provided.     

The construction and validation of a usability evaluation survey for mobile learning environments

The advent of mobile technologies in learning context, has been increased the requirements for developing appropriate usability model to align with mobile learning applications. Even though mobile learning has been studied from different aspects of pedagogy environment and technology acceptance, there is little scientific and published research on usability of mobile learning applications. To fill up the gap, in this study, a usability evaluation model with the inclusion of timeliness is developed to assess the usability of mobile learning applications. Timeliness or response time as an important feature in mobile learning, which influences learning satisfaction, can be considered to evaluate the peers and instructors’ timely response. The main objective of this study is to construct and validate a usability evaluation survey for mobile learning environments. This study employed a two round Delphi method to empirically verify the usability questionnaire by obtaining a consensus from fourteen experts regarding the questionnaire items. Results indicate that over 88% of experts have consented on all usability items represented in the usability questionnaire. The usability evaluation survey for mobile learning applications can help to improve user satisfaction and reductions in training costs. The decrease in costs attracts many researchers, interface designers and project managers to employ the usability evaluation when designing the interfaces for mobile learning applications.     

方法与模型：引领教育评估的变革

大数据时代,依赖经验,依赖粗放方式、单一手段、定性思维进行学习行为判断和教学决策,这种沿袭自远古的教育评价范式必须改变。基于数据的教育评估,是教育评估最前沿的课题。重庆市教育评估院自2009年成立以来,一直致力于教育数据分析研究,研发教育质量监测工具,建构了分析系统,形成一套较完整的教育数据采集、处理模型或方法,挖掘了多元多层数据,促进了教育评估的改革。     

基于学员综合素质评估的模型与算法研究

为实现学员综合素质的评估,提供一种针对人才培养质量考核的有效手段,在对目前通用的评估模型与算法比较分析的基础上,构建了基于学员综合素质评估的两类模型一线性评估模型和平面评估模型,设计并实现了针对两类模型的评估算法;结合指标体系,对两类评估模型及算法在实际系统中加以应用,结果符合专家预测,验证了模型和算法的可用性和科学性。     

以学生为中心构建高校专业评估模式

目前我国专业评估中存在以学生为中心意识不够强、过于强调政府的主导、评估工具有待创新、评价结果使用不足等问题。新时期面对新的需求和挑战,专业评估应立足国内实践,借鉴国际专业评估的有益经验,深化以学生为中心的评估理念,营造多方参与的评估格局,创新评估工具,完善评估结果运用,进而不断提升专业评估的质量水准及社会公信力。     

Mission,messages, and measures: Engaging zoo educators in environmental education program evaluation

Here we describe the development and implementation of a large-scale monitoring system to systematically evaluate various Chicago Zoological Society (CZS) education programs. Our primary goal was to engage program staff in developing a consistent measurement and evaluation strategy across and within education programs. We did this by using the CZS mission as a framework and incorporating participatory, theory-based, and utilization-focused evaluation approaches into our education programs. As we carried out the process, we learned several lessons that helped us to succeed. This process has allowed us to begin building the perspective among our staff and leaders that evaluation is an ongoing process that occurs alongside program delivery to inform cycles of reflection and improvement and measure program performance over time.     

我国教育评价政策的注意力配置分析

教育评价政策注意力配置是影响政策的重要变量。运用文本分析法从“目标维度、任务维度、路径维度”三个维度对1987-2022年的《教育部工作要点》进行分析的注意力配置情况。研究发现,依据注意力配置情况和政策重点,我国教育评价政策注意力配置的发展可划分为以体系建立为重点的恢复阶段、强调“公平”与“多元评价”的改进阶段、关注机制改革和系统优化的完善阶段;目标维度从规范管理、保障质量走向关注公平,任务维度较为关注学校、政府和学生评价,路径维度依赖制度建设。结合分析结果揭示教育评价政策注意力配置存在的问题并提出改进建议。     

本科教学工作水平评估与学校整体办学实力的提升

本科教学工作水平评估是教育部依据《中华人民共和国高等教育法》,在赋予高校办学自主权的同时,进行宏观质量监控的一项重要措施。评估工作坚持“以评促建、以评促改、以评促管、评建结合、重在建设”的方针,其着眼点在建设,最终目的是推动和促进本科院校各项事业的发展,提升本科院校的整体办学实力。这对本科院校的改革、发展无疑具有非常重要的意义。     

The impact of the scoring system of a large-scale standardized EFL writing assessment on its score variability and reliability: Implications for assessment policy makers

Using generalizability (G-) theory and rater interviews as research methods, this study examined the impact of the current scoring system of the CET-4 (College English Test Band 4, a high-stakes national standardized EFL assessment in China) writing on its score variability and reliability. One hundred and twenty CET-4 essays written by 60 non-English major undergraduate students at one Chinese university were scored holistically by 35 experienced CET-4 raters using the authentic CET-4 scoring rubric. Ten purposively selected raters were further interviewed for their views on how the current scoring system could impact its score variability and reliability. The G-theory results indicated that the current single-task and single-rater holistic scoring system would not be able to yield acceptable generalizability and dependability coefficients. The rater interview results supported the quantitative findings. Important implications for the CET-4 writing assessment policy in China are discussed.     

构建促进高职教师发展的督导与评价体系

文章叙述了高职院校教师评价管理中的问题。     

“以学评教”的数学课堂教学评价指标体系建构

文章从以学评教的内涵和课堂教学评价指标体系建构出发, 综合了有效教学的相关理论依据,紧密结合数学学科特点,围绕学生有效学习行为的针对性、能动性、多样性、选择性这四个维度,建立了“以学评教”的课堂教学评价指标体系,并对以学评教的意义做了分析。力图通过改变教学评价方式促进课堂教学观念的转变,尊重学生个体差异,努力实现学生全体发展和全面发展。     

高校学生教育和管理评价体系的研究

本论采用了系统论的方法、比较研究的方法及数学加权的方法，通过对国内外高校学生教育和管理的应用研究，得出符合我国高校的评价体系，对我国目前高校众多的教育和管理方法评价体系，提出了很好的规范的和科学的可借鉴的方法。