Consistency of Angoff-Based Predictions of Item Performance: Evidence of Technical Quality of Results From the Angoff Standard Setting Method

Judgmental standard-setting methods, such as the Angoff(1971) method, use item performance estimates as the basis for determining the minimum passing score (MPS). Therefore, the accuracy, of these item peformance estimates is crucial to the validity of the resulting MPS. Recent researchers (Shepard, 1995; Impara & Plake, 1998; National Research Council. 1999) have called into question the ability of judges to make accurate item performance estimates for target subgroups of candidates, such as minimally competent candidates. The propose of this study was to examine the intra- and inter-rater consistency of item performance estimates from an Angoff standard setting. Results provide evidence that item pelformance estimates were consistent within and across panels within and across years. Factors that might have influenced this high degree of reliability, in the item performance estimates in a standard setting study are discussed.     

Development of a distance education assessment instrument

The study described in this article takes the view that distance education programs are dynamic systems with multiple constituencies. Literature on distance education programs as complex systems and on the critical constituency approach to program evaluation was used to guide the development of a questionnaire designed to assess the components of a distance education program. Instrument development was conducted in four phases using both quantitative and qualitative methods. The data from this study suggest that there are four major components of effectiveness in distance education programs: instruction, management, telecommuting, and support. Within each of these components there are from two to five subcomponents, 14 components in all, which can be used to assess distance education programs. The assessment scale which was developed measures these four components and their subcomponents.     

Aligning assessment with the needs of work-integrated learning: the challenges of authentic assessment in a complex context

Abstract

Work-integrated learning (WIL) is a feature of university courses, both in professional areas, where it is commonplace, but also across many different disciplines. Assessment of WIL can be complex as it involves parties and settings external to the university, and it can be problematic because of difficulties in aligning learning activities during placements with what is or can be assessed by the university. This paper explores the relationship between students’ placement experiences and accompanying assessments in contexts where activities are tightly coupled with the curriculum, and in those where it is not. It draws on a qualitative analysis of student interviews and drawings by the interviewees of their WIL experiences, supplemented with analysis of unit guides. Our findings highlight that students’ perceptions of authenticity of assessment were undermined by misalignments between the student, university and industry. Assessment authenticity was perceived by students as based on alignment between their current and future selves in the assessment process, involvement of industry supervisors and relevance of placement activities to assessment activities. The paper discusses the complexity of coordination of educational activities with external partners, especially when one party drives assessment. It then suggests a reframing of WIL assessment to promote alignment and authenticity.     

Implications of technological change for higher education reform

Technological change and social change are viewed as interdependent in this paper. At the same time technology's push is seen as outpacing the capability of social institutions to respond. Consequently, America risks losing its competitive edge economically and, further, is rapidly evolving toward a two-tiered occupational and social structure. This paper examines the relationship between technological and social change, the nature and significance of technological innovation and dissemination, and impacts on organizational life. Finally, the paper explores the role of higher education as well as strategies the university must adopt if it is to participate effectively in addressing the changes that are emerging as society moves from being production-and-material based to an information-based system.     

Situational influences upon children's beliefs about global warming and energy

This paper explores children's beliefs about global warming and energy sources from a psychological perspective, focusing upon situational influences upon subjective beliefs, including perceived self‐efficacy. The context of the research is one of growing concern at the potential impacts of global warming, yet demonstrably low levels of self‐efficacy amongst both adults and children to effectively respond to this large‐scale environmental problem. Empirical research was conducted on a sample of 198 UK children and adults to explore the influence of a cooperative learning environment upon children's beliefs about global warming and energy. A comparative design was adopted, contrasting 9–12 year old members of the Woodcraft Folk educational organisation with non‐members of similar age and with adult members of the same organisation. Results indicate that cooperative learning environments can have a significant and positive effect upon children's beliefs about large‐scale environmental problems. In particular, Woodcraft folk children reported significantly higher levels of personal awareness and perceived self‐efficacy in relation to global warming in comparison to their peers. Secondly, unexpected differences were identified between levels of perceived self‐efficacy in children and adult Woodcraft folk. The implications of these differences for the design of educational programmes seeking to empower children to respond to global warming are discussed.     

Embodied engagements with online pornography

Even though sexual arousal is a central feature in browsing online pornography, embodiment has largely gone unexamined in much of the research on pornography in information studies and human–computer interaction. Through existential phenomenology, which emphasizes a synthesis of cognitive reflection and embodied experience, this article examines how our embodied engagements with the technological apparatus of the computer help reconstitute the way we feel time, objects of desire, and pleasure in the process of browsing online pornography.     

Innovation in regions: What does really matter?

The literature on systems of innovation conceptualises innovation as an evolutionary and social process of collective learning. But three main questions remain open: is this learning process rather internal or external? What are the boundaries of this process? Is this social process voluntary or rather unintended? This paper strives to tackle those questions by analysing the determinants of regional innovative levels in French manufacturing industries. By estimating a knowledge production function we find that building external interactions is of greater importance than developing internal innovative competences. Second, unintended knowledge flows have less influence on regional innovative performance than deliberate ones. Lastly, the French innovation system looks regional rather than sectoral.