How do instructional designers evaluate? A qualitative study of evaluation in practice

This study employed a qualitative research design to investigate how instructional designers use evaluation in everyday design practice. While past research has examined how designers spend their time, how they generally make decisions, and expert-novice differences, little attention has been paid to use of context, input, process, or product evaluation, from the perspective of practicing designers. Based on interviews of practitioners, our findings included ten themes regarding how designers use evaluation to improve their products. While these results substantiate to some degree the claim that practitioners believe clients will not pay for formal evaluations, they also suggest that practitioners use evaluation in important but less formal ways. Other conclusions regarding the role of evaluation in design are provided and future directions for training and research are discussed.     

Observer ratings of instructional quality: Do they fulfill what they promise?

Despite considerable interest in the topic of instructional quality in research as well as practice, little is known about the quality of its assessment. Using generalizability analysis as well as content analysis, the present study investigates how reliably and validly instructional quality is measured by observer ratings. Twelve trained raters judged 57 videotaped lesson sequences with regard to aspects of domain-independent instructional quality. Additionally, 3 of these sequences were judged by 390 untrained raters (i.e., student teachers and teachers). Depending on scale level and dimension, 16–44% of the variance in ratings could be attributed to instructional quality, whereas rater bias accounted for 12–40% of the variance. Although the trained raters referred more often to aspects considered essential for instructional quality, this was not reflected in the reliability of their ratings. The results indicate that observer ratings should be treated in a more differentiated manner in the future.     

We train,but what do they think? Preservice teachers’ perceptions of the adequacy of their teacher education in Turkey

The study aimed to assess the adequacy of pre-service teacher education in Turkey as perceived by preservice teachers. Specifically, the researchers investigated preservice teachers’ perceptions of the adequateness of their education in terms of developing teaching skills and the major components of teacher education. The factors affecting their perceptions were also examined. In total, 1856 preservice teachers participated in the study. A questionnaire was prepared to collect data from the participants. According to the findings, teacher education should be enhanced in terms of preparing candidates for skills such as teaching in different contexts and cooperating with parents; and practicum hours needed to be improved for more effective preparation of preservice teachers. Moreover, it was also found that the preservice teachers’ subject, college, orientation towards teaching and the perceived adequacy of program components played significant roles in affecting their perceptions of teacher education.     

How do instructional designers manage learners’ cognitive load? An examination of awareness and application of strategies

Educational technology research and development - This study examined how practicing instructional designers manage cognitive load in a standardized scenario as they select and implement...     

Explicit promotion of experimental skills. And what about the content-related skills?

Experimental skills should be acquired by learners at school and university alike. To promote experimental skills, various approaches exist within a spectrum between implicit and explicit instruction. Regarding these instructional approaches, numerous findings are available which predominantly relate to pupils. It is an open question whether it is better to instruct experimental skills explicitly or implicitly in university courses. And if experimental skills are fostered explicitly, what about the content-related skills? Especially in teaching programmes with both experiment-related and content-related objectives, it is questionable whether content-related objectives are achieved to a lesser degree via explicit instruction of experimental skills. Both questions are addressed in this paper. We present an intervention study involving 144 students. All students carry out the same experiments. The intervention groups differ merely in the degree of explicit instruction. Learning gains in experimental and content-related skills are assessed in a pre-/post-test design. The results show that the type of instruction does not significantly influence the acquisition of experimental and content-related skills in this target group. But by trend, the expected differences in learning gains can be observed. This leads to new research perspectives and implications for teaching which are discussed.     

Fractions: Could they really be the gatekeeper’s doorman?

The National Mathematics Advisory Panel (NMAP, 2008) asserts that a foundational knowledge of fractions is crucial for students’ success in algebra; however, empirical evidence for this claim is relatively nonexistent. In the present study, we examine the impact of middle school students’ fraction and whole number magnitude knowledge on various components of their algebra readiness. Results suggest that fraction knowledge is related to algebra readiness, more so than number magnitude knowledge in general; students’ magnitude knowledge of unit fractions (i.e., those with a numerator of 1) appears particularly important. Findings confirm the intuition of the NMAP (2008) and support the recommendation of the Common Core Standards (National Governors Association Center for Best Practices (NGA Center), 2010) that students’ fraction knowledge should be cultivated using number lines.     

RESEARCH SECTION: Why do students think they fail? The relationship between attributions and academic self‐perceptions

Attributions can have a significant effect on academic achievement and students with learning difficulties are more likely to display negative attributions than their peers. In this article, the attributions of students identified as having learning difficulties are compared with those of other non-labelled low achievers, and non-labelled average achievers. Margaret Banks, MSc research methods student, and Dr Lisa Woolfson, Reader in Educational Psychology, both at the University of Strathclyde, involved 26 low-achieving students (15 identified as having learning difficulties and 11 without any such label) and 27 averagely achieving students in their exploratory study. The students attempted to solve two sets of puzzles that, unknown to the participants, were actually impossible. They then gave a reason for their failure and rated the stability and controllability of their failure. Participants were also asked to give a self-rating of academic achievement. It was found that teacher and student perceptions of academic achievement were not synonymous. In terms of attributions, Margaret Banks and Lisa Woolfson argue that teachers' perceptions of student achievement may not be as important as students' self concept. They suggest that low self-perception of achievement and negative attributions are both associates of low self-esteem.     

What matters to supervisors and is this reflected in what they do? Analysing the work of university supervisors of the practicum

This study analyses the contemporary work and potential contribution of university supervisors supporting preservice teachers´ during their practicum experiences in initial English language teacher preparation. The study draws on a complex data derived from surveys, interviews, and observations of university supervisors working in a diverse array of English language teacher education programmes across Chile. A cultural historical activity theory perspective is employed to explore and analyse the motives, activities, and actions that underpinned the work of these supervisors. Findings suggest that supervisors have to respond to often contradictory demands and an overemphasis towards quality assurance actions that are potentially detrimental to preservice teachers´ development. The work of supervisors is analysed to understand how potentially to re-envision their role and functions, toward proposing how to productive expand supervisory practices as an integral part of teacher education.     

Pedagogical content knowledge of experts and novices—what knowledge do they activate when analyzing science lessons?

In the current debate on pedagogical content knowledge (PCK), the term is used to refer to the context‐specific knowledge that teachers activate when reflecting on practice. Against the background of this debate, we conducted an empirical study and sought to answer the question of which knowledge experts and novices activated in assessing a videotaped lesson in relation to its effectiveness for learning. Our assumption was that the participants activate their PCK as a blending of content knowledge (CK) and pedagogical knowledge (PK) as suggested by Shulman's amalgam thesis. The participants (9 experts and 9 novices) were shown a lesson on optics, in which the law of refraction (Snell's law) was being studied. In a subsequent interview, the participants were asked to analyze the observed lesson. A qualitative and quantitative evaluation of the interviews showed that experts activated both CK and PK intensively and in this respect they differed significantly from novices. Further analysis of the expert statements also proved that they do not activate their CK and PK in isolation, but instead combine both kinds of knowledge together, in line with Shulman's amalgam thesis. © 2017 Wiley Periodicals, Inc. J Res Sci Teach 55: 44–67, 2018     

Gender differences in adult numeracy skills: what is the role of education?

Previous research suggests that as societies empower women educationally, gender differences in numeracy skills will decline. Using direct measures of 56,142 adults’ numeracy skills from the Programme for International Assessment for Adult Competencies (PIAAC), this article studies whether this claim is evidenced across 20 Organisation for Economic Co-operation and Development (OECD) countries. Results show that in many countries, as education has equalised over generations, gender differences in adult numeracy skills have indeed declined. However, women’s advances in education have not always been matched by a reduction in the gender difference in adult numeracy. Contrary to expectations, gender differences in fields of study in further and higher education cannot systematically explain gender differences in adult numeracy. This suggests that to achieve gender equality in numeracy skills, societies must do more than empower women educationally. More research is needed on the educational policies, contextual, and life course factors contributing to gender differences in adult numeracy in post-industrial societies.     

Young working-class men do not read: or do they? Challenging the dominant discourse of reading

ABSTRACT

In the majority of the research on boys’ and young men’s relation to reading, it is argued that boys and young men read too little, read poorly and in all the wrong ways. However, few studies focus on how boys and young men read the texts they do encounter. In particular, there is a lack of research on young men of working-class background, whose relationship to reading is stressed as particularly problematic in many studies. In this article, we approach the reader histories of three young working-class men from a life story perspective, on the basis of a broad definition of text and reading, to capture how reading and texts are used in their identity construction. Our analysis shows that the young men engage in reading in different ways, from listening to audio books and reading aloud to watching films. We argue that our approach makes it possible to re-envision working-class men as readers who, among other things, use reading to construct softer masculinities, thus challenging the dominant narrative of working-class masculinity and working-class men’s relationship to reading and texts.     

Perceptions of the Bologna process: what do students’ choices reveal?

A major element in the creation of the European area of higher education is the adoption of a common structure of degrees, implying in several countries the reduction of the duration of the first degree to 3 years, which is a controversial change. Cardoso et al. (CESifo Econ Stud 54(2): 229–247, 2008) have analyzed student confidence in the curricula change, quantifying its impact on students’ first choices of academic programs. This paper goes two steps further. First, it concentrates on a variable that better translates total demand for an academic program, namely the total number of students who place the program among their six revealed preferences, instead of just the first option; and, second, an econometric model that better fits the data is estimated. Results confirm a positive impact of the Bologna process on the demand for programs, which varies with program size and across fields of study. Our results complement those in Cardoso et al. (2008), as they uncover that being a program leader, i.e. the only institution in the country that restructured a given program, was associated with higher demand by prospective students, which nevertheless stemmed from their “second best" options and not from their first choices.

Teacher and student responses to interdisciplinary aspects of sustainability education: what do we really know?

That the value of interdisciplinarity in sustainability education is largely taken for granted reflects widely held views about the inherent complexity and uncertainty of sustainability issues and the need for holistic responses to them. Although there is an extensive literature on interdisciplinarity, there has been relatively little investigation of the ways in which teachers and learners deal with and respond to this central aspect of sustainability education. This paper presents data from two case studies of masters course in sustainability education, one in an English university and one in a Chinese university, and examines some of the contextual factors that support or limit their attempts to go beyond the bounded ways of thinking associated with more traditional academic disciplines. While some learners respond enthusiastically to interdisciplinarity, others experience forms of ‘cognitive dissonance’. Interdisciplinarity requires individual teachers to operate as members of teams and to find ways of promoting dialogue between different perspectives, discourses and methods of approaching problems. The key to interdisciplinarity is an open attitude, a willingness to learn and an ability to engage with different ways of thinking about issues we face in common.     

Children's perceptions of wider reading: to what extent do 7- and 8-year-old children read beyond the scheme books?

A case study was conducted with a class of twenty 7- to 8-year-old children in a primary school in the North West of England, to explore possible links between the use of a school reading scheme and children's involvement in wider reading. A questionnaire was administered to all children in the class, followed by semi-structured interviews with three children. The findings revealed that the vast majority of the children held extremely positive views of their reading scheme books but were also very enthusiastic wider readers, be it either independent reading or reading with a parent or carer. Further findings emerged suggest that environmental factors, such as access to books at home and library usage, also play an important part in children's involvement in wider reading.     

Measures of reading comprehension: do they measure different skills for children learning English as a second language?

The validity of two measures of English reading comprehension was examined across three different groups of English language learners (ELLs; 64 Portuguese, 66 Spanish and 65 Cantonese). All three groups were achieving within the average range in second grade. An exploratory principal components analysis of reading skills was carried out to determine which skills were related to two commonly used tests of reading comprehension, the Woodcock Language Proficiency Battery??s test of Passage Comprehension (WLPB-PC; Woodcock, 1991) and the Gray Oral Reading Test-4 (GORT-4; Wiederholt & Bryant, 2001). The factor solutions were different for the three language groups but showed many similarities in that the GORT-4 and WLPB-R tests of reading comprehension fell on the same factor within each group. Hierarchical regression analyses examining relationships among vocabulary, decoding and reading comprehension showed that language group membership did not significantly predict performance on either measure of reading comprehension. Differences that arose are likely due to issues with task validity and not ELL status. Limitations and future research are discussed.     

Beginning teachers’ perceptions of their levels of pedagogical knowledge and skills: did they change since their graduation from initial teacher preparation?

This study investigated changes in beginning teachers’ self-perceptions of their pedagogical knowledge and skills after their first year of teaching. Surveys were administered to 322 graduating student teachers at the end of the initial teacher preparation programme and at the end of their first year of teaching to compare if there were any differences in their self-perceptions. The results of the study showed significant increases in beginning teachers’ perceptions of their pedagogical knowledge and skills in three factors: Instructional Support, Accommodating Diversity and Classroom Management. Results established the development of teachers to be an ongoing process that is initiated but not completed in initial teacher preparation. Implications of findings for induction and mentoring of beginning teachers are discussed.