A framework for designing and developing multimedia-based performance assessment in vocational education

The development of any assessment should be an iterative and careful process. Ideally, this process is guided by a well-defined framework (see for example Downing in: Downing and Haladyna (eds) Handbook of test development, Lawrence Erlbaum Associates, Mahwah, 2006; Mislevy et al. in On the roles of task model variables in assessment design (CSE Technical Report 500), Educational Testing Service, Princeton, 1999; AERA et al. in Standards for educational and psychological testing, AERA, Washington, DC, 2004), but such a framework is not always available when the instrument to be developed is new or innovative. Frameworks for the development of traditional computer-based tests have been published and experimented with since the late 1990s, by which time CBT had already existed for more than a decade. In an earlier empirical pilot study, we described a new type of assessment for Dutch vocational education, called multimedia-based performance assessment (MBPA) (self-revealing reference 2014). This CBT uses multiple media formats and interactive tasks to measure skills that are currently measured by performance-based assessment. In conducting that pilot study, deficits in the existing literature made it difficult to ground all developmental steps in sound scientific theory. To remedy those deficits, this article presents and validates a framework for the design and development of MBPA, combining a search of the relevant literature from several subfields of educational assessment and consultation with assessment experts. The framework unites assessment development and multimedia development theory, focus solely on vocational education, and answers the call for a framework from the scientific community. The first step in validating the prototype framework involved five semi-structured interviews with Dutch assessment and multimedia experts to produce a final version of the framework. Second, the pilot MBPA was reconstructed in accordance with this finalized framework, resulting in an improved MBPA and demonstrating that the proposed framework is a useful and applicable tool for the design and development of MBPA in vocational education.     

Comparing the influence of various measurement error presentations in test score reports on educational decision-making

This study investigated (1) the extent to which presentations of measurement error in score reports influence teachers’ decisions and (2) teachers’ preferences in relation to these presentations. Three presentation formats of measurement error (blur, colour value and error bar) were compared to a presentation format that omitted measurement error. The results from a factorial survey analysis showed that the position of a score in relation to a cut-off score impacted most significantly on decisions. Moreover, the teachers (N = 337) indicated the need for additional information significantly more often when the score reports included an error bar compared to when they omitted measurement error. The error bar was also the most preferred presentation format. The results were supported in think-aloud protocols and focus groups, although several interpretation problems and misconceptions of measurement error were identified.     

Mental health problems of Dutch youth with hearing loss as shown on the Youth Self Report

The prevalence OF mental health problems among youth with hearing loss was assessed with an adjusted version of the Dutch rendition of the Youth Self Report, or YSR (Achenbach, 1991). The sample totaled 202 youth, aged 11-18 years, with auditory disabilities. The prevalence rates of externalizing problems, internalizing problems, and moderate to severe overall mental health problems were found to be 2-3 times higher than in a normative sample. Deaf participants scored significantly higher than hard of hearing participants in these areas. Main-streamed participants scored significantly lower than peers in schools for the deaf or for the hard of hearing. Participants with low IQ scores showed significantly more internalizing and social problems than those with moderate to high scores. The adjusted YSR is recommended for screening in schools and in mental health services for youth with hearing loss for prevention and early intervention.     

Investigating Feedback on Practice Among Teachers: Coherence of Observed and Perceived Feedback

Despite that benefits of feedback in student learning are reported in much research, little has been reported regarding the use of feedback from teachers to other teachers—a key tool in professional development. In this study, we triangulated data from videotaped peer coaching sessions, questionnaires, and interviews regarding 12 primary school teachers in four peer groups in the Netherlands. We focused our research on two issues: the interplay of observed feedback dimensions and elements and perceptions of that feedback. Feedback dimensions were generally effective and the influence of the elements on the dimensions mostly aligned with the expectations. Teachers generally perceived feedback as effective. Moreover, effective observed feedback was perceived as effective. Findings indicate that peer coaches should stimulate coached teachers to become goal directed, specific, detailed, and neutral (neither positive nor negative) by using feedback elements so as to optimize feedback processes.     

Do honors students have more potential for excellence in their professional lives?

Universities in many countries increasingly value talent, and do so by developing special honors programs for their top students. The selection process for these programs often relies on the students’ prior achievements in school. Research has shown, however, that school grades do not sufficiently predict academic success. According to Renzulli’s (1986) three-ring model, student characteristics relating to intelligence, motivation and creativity are the most important predictors of excellent achievements in professional life. In this paper, we will investigate whether honors students differ from non-honors students in terms of these characteristics. By means of a questionnaire, more than 1,100 honors and non-honors students at Utrecht University were asked to assess themselves on six characteristics: intelligence, creative thinking, openness to experience, the desire to learn, persistence, and the drive to excel. The results showed that the honors students differed significantly from the non-honors students in terms of the combined variables as well as for the separate variables, with the exception of ‘persistence’. The strongest distinguishing factors between honors and non-honors students appeared to be the desire to learn, the drive to excel and creativity, whilst there was little difference in terms of intelligence and persistence. However, the profiles of these differences varied according to the study program. While Law and Humanities honors students differed from their non-honors peers in terms of their drive to excel, Physics honors students were primarily more eager to learn than their non-honors peers, while the LA&S honors students scored higher on creative thinking than non-honors students.     

Interpretation of the score reports from the Computer Program LOVS by teachers,internal support teachers and principals

Data-driven decision making, such as the decision making that is conducted through the use of pupil monitoring systems, has become increasingly popular in the Netherlands, as it is considered to have promise as a means of increasing pupils’ learning outcomes. The reports generated by the pupil-monitoring Computer Program LOVS (Cito) provide educators with reliable and objective data feedback; however, research has suggested that many users struggle with interpreting these reports. This study aims to investigate the extent to which the reports are correctly interpreted by educators, and to identify various potential stumbling blocks with regards to the interpretation of the reports. The results suggest that users encounter many stumbling blocks in these reports and often cannot interpret them entirely correctly.     

Teacher research in Dutch professional development schools: perceptions of the actual and preferred situation in terms of the context,process and outcomes of research

This study aims to provide deeper insight into participants’ (e.g. school principals, teachers and student teachers) perceptions of the actual and preferred situation in terms of the context, process and outcomes of practice-based research through teachers-as-researchers at Dutch professional development schools. We interviewed eight school principals, 10 teachers, and six student teachers from four professional development schools in the Netherlands. A trend displayed across all types of participants was their focus on the context dimension of research. Moreover, respondents often focussed in rather general terms on the process of conducting teacher research within the school and the content of this research. A major difference exists between perceptions of the actual and preferred situation related to the effects of teacher research on pupils’ outcomes: despite the central focus of research on pupil learning and learning results, according to participants effects on pupil level were not (yet) reached at this moment. These results suggest that in Dutch professional development schools increased attention is needed both by researchers and practitioners on the process and outcome dimensions of doing teacher research.     

真实评价设计的五维框架

真实性(authenticity)是新型评价模式的核心要素之一.然而,对于什么是真实评价,学术界还未予以充分具体的探索,更远未达成共识.本文在评述有关真实评价的研究文献基础上,讨论了教学、学习与评价的关系,论述了基于真实能力评价的意义,探讨了真实性和真实评价的定义,建构了面向专业实践能力发展的真实教学和真实评价设计的五维框架.结合真实教学的五维框架,重点论述了真实评价的五个维度:真实任务、物理情境、社会情境、评价结果/形式和评价标准,并且提出了设计或开发真实评价应当考虑的三大问题.     

Developmental Links between Teacher-Child Closeness and Disobedience for Boys Placed in Special Education

The aim of this study was to examine developmental links between disobedience and teacher-child closeness in a sample of boys with psychiatric disorders (i.e., emotional and behavioral disorders (EBD) and autism spectrum disorders (ASD)) and special educational needs who are placed in special education. More specifically, this study examined whether developmental links were different between boys with EBD (n = 150) versus boys with ASD (n = 122). Developmental links between disobedience and teacher-child closeness were investigated by incorporating a multi-informant perspective using teacher, child, and peer ratings and analyzed using autoregressive cross-lagged models across three waves within one school year. Results showed that in general, developmental links between teacher-child closeness and disobedience were stronger for boys with EBD than for boys with ASD. Specifically, boys with EBD experiencing less teacher-child closeness showed more disobedience, which in turn negatively affected their relationship development according to teacher ratings. Surprisingly, for boys with ASD, higher levels of disobedience predicted higher levels of teacher-rated teacher-child closeness during the school year. Our results offer more insight into the differential impact of teacher-child closeness on the behavioral problems for children with EBD and ASD.