Approaches to Classroom Assessment Inventory: A New Instrument to Support Teacher Assessment Literacy

Teacher assessment literacy has become a central priority across many educational systems in North America and elsewhere in response to growing accountability demands. Although many scholars have aimed to measure teacher assessment literacy, recent research has identified that current assessment literacy instruments do not fully reflect current transformations in the assessment landscape and remain predicated on dated standards for teacher classroom assessment practice. Given significant shifts in classroom assessment over the past 20 years, the purpose of this article is to construct a reliable instrument reflective of contemporary assessment practices and contexts. Specifically, this article describes our instrument development process including construct validation and reliability testing with more than 400 teachers. The result of this research is the Approaches to Classroom Assessment Inventory, which can be used by researchers and practitioners to support teacher assessment literacy in relation to the current accountability framework evident across educational systems.     

A rubric to track the development of secondary pre-service and novice teachers’ summative assessment literacy

Abstract

Teachers require specialised assessment knowledge and skills in order to effectively assess student learning. These knowledge and skills develop over time through ongoing teacher learning and experiences. The first part of this paper presents a Summative Assessment Literacy Rubric (SALRubric) constructed to track the development of secondary science teachers’ summative assessment literacy. The analytic rubric consists of 10 dimensions spread across three categories drawn from the literature and context-specific empirical evidence: knowledge of assessment, understanding the context for assessment, and recognising the impact of assessment. The second part of this paper applies the SALRubric in a case study to explore the development of summative assessment literacy of New Zealand secondary science pre-service and novice teachers. An increasing sophistication in these teachers’ summative assessment literacy was evident over 20 months albeit in a nuanced manner for individual teachers. The rubric was a very useful tool for evaluating and documenting shifts in teachers’ summative assessment literacy over time. Implications of the use of SALRubric are discussed in terms of summative assessment literacy practice and development.     

Implementation Quality: Lessons Learned in the Context of the Head Start REDI Trial

This study uses data collected in the intervention classrooms (N = 22) of Head Start REDI (Research-based, Developmentally Informed), a randomized clinical trial testing the efficacy of a comprehensive preschool curriculum targeting children's social-emotional competence, language, and emergent literacy skills delivered by teachers who received weekly coaching support. Multiple dimensions of implementation (Dosage, Fidelity, Generalization, and Child Engagement) were assessed across curriculum components. Results indicated that REDI Trainers perceived significant growth in teacher implementation quality over time but that patterns differed by implementation dimension. Dosage and Fidelity of all intervention components were high at the beginning of the year and remained so over time while Generalization was low at baseline but increased significantly across the year. Variation in implementation was associated with variation on several child outcome measures in the social-emotional domain but not in the language and literacy domains.     

Measures of preschool children's interest and engagement in literacy activities: Examining gender differences and construct dimensions

Multiple approaches to measuring preschool children's literacy interest and engagement (i.e., parent-, teacher-, child-reported child literacy interest and observer-reported child literacy engagement) were examined in a sample of 167 four- and five-year-old children (M = 56.62 months, SD = 6.01) enrolled in Head Start. Associations among measures as well as gender differences and dimensions of preschooler's literacy interest and engagement were examined across measures. Measures were not strongly associated. There were small, but significant correlations between parents’ and teachers’ reports of children's literacy interest, and teachers’ reports of children's literacy interest and observers’ reports of children's literacy engagement. Gender differences were found for parent- and teacher-report measures, with teachers and parents rating girls higher on interest. Patterns of factor loadings differed between adult and child measures. Implications of findings are discussed.     

Learning to spell: raising standards in spelling and independent writing

This paper considers evidence from an intervention designed to raise standards in spelling and independent writing. The Complete Spelling Programme (McMurray and Fleming, 1998) is designed to ensure that all processes involved in learning to spell are activated and that the interaction between them is facilitated. In addition, a number of innovative features within the programme's design allow sources of spelling knowledge to be presented in a developmental sequence, allowing all children to learn together, regardless of their ability. The programme provides day‐to‐day spellings and an extensive range of follow‐up activities, clearly differentiated, linking spelling to wider literacy teaching, e.g. grammar, language understanding, punctuation. The impact of the programme on progress, not only in spelling tests but also in spelling accuracy and quality in independent writing, was assessed by means of a longitudinal study with a 2 × 2 quasi‐experimental design. The two independent variables were (i) experimental school (intervention) or control school (no intervention) and (ii) high or low social disadvantage. A sample of 81 children aged 5–6 years across the four schools was followed for three years from Year 2 to Year 4 (Northern Ireland). A range of quantitative measures was used for baseline assessment and to establish the quantifiable outcomes for children in both the experimental (N=43) and the control schools (N=38). Qualitative measures were used to illuminate the processes involved in the programme in the experimental schools (McMurray, 2004). At the end of the study pupils who had been taught using the programme had made significant improvement in spelling and independent writing. On the basis of the findings it is argued that spelling needs to be seen as an integral tool in raising standards in literacy and that it should not be taught in isolation from other literacy skills.     

Co-constructors of data,co-constructors of meaning: Teacher professional development in an age of accountability

This research examines how engaging teachers in collaboratively constructing situated assessments enhances teacher professional development, fosters meaningful shifts to practice, and supports productive conceptions of accountability. We conducted case studies of six teachers engaged as partners in investigating new approaches to assessing and fostering literacy. Findings suggested that supporting teachers to generate, interpret, and act upon assessment data within instructional change cycles assisted them in monitoring student performance, grounding instructional decisions in data, and enacting changes to practice. We also document how collaboratively constructing assessments with teachers supported conceptions of accountability considered meaningful by individuals adopting differing socio-political perspectives.     

Making learning last: teachers’ long-term retention of improved nature of science conceptions and instructional rationales

ABSTRACT

Despite successful attempts to improve learners’ nature of science (NOS) conceptions through explicit, reflective approaches, retention of improved conceptions is rarely addressed in research. The issue of context for NOS instruction has implications for this retention. Whether to contextualise has been the question occupying science educators’ attention. We think this question is misplaced. Instead, we build upon recent research addressing a context continuum – drawing on the strengths of both contextualised and noncontextualised NOS instruction – to improve retention. Although there are many different potential contexts for NOS instruction, this investigation focuses on science content as context. The present investigation focused on long-term retention of improved NOS conceptions and rationales for NOS instruction. Participants were all 25 teachers who completed a professional development programme (PDP) utilising a mixed contextualisation approach to NOS instruction. We classified teachers’ NOS conceptions into three levels of understanding using the Views of the Nature of Science Form-C responses and interviews three times over the year: pre-, post-, and 10-month delayed post-PDP. Results indicated that initially participants held many alternative NOS conceptions. Post-instruction, responses were substantially improved across all NOS concepts. Furthermore, nearly all of the participants’ conceptions were retained across the academic year following the PDP. Participants offered varied rationales for NOS instruction including its potential to improve students’ scientific literacy, perceptions of the relevance of science, improve positive risk-taking, and increase tolerance for differences. These results contrast favourably with previous reports of the retention of improvements in NOS conceptions over time.     

Assessment literacy and student learning: the case for explicitly developing students ‘assessment literacy’

In this paper, we report on a study to quantify the impact on student learning and on student assessment literacy of a brief assessment literacy intervention. We first define ‘assessment literacy’ then report on the development and validation of an assessment literacy measurement instrument. Using a pseudo-experimental design, we quantified the impact of an assessment literacy-building intervention on students’ assessment literacy levels and on their subsequent performance on an assessment task. The intervention involved students in the experimental condition analysing, discussing and applying an assessment rubric to actual examples of student work that exemplified extremes of standards of performance on the task (e.g. poor, excellent). Results showed that such a procedure could be expected to impact positively on assessment literacy levels and on student performance (on a similar or related task). Regression analyses indicated that the greatest predictor of enhanced student marks (on the assessment task that was the subject of the experiment), was the development of their ability to judge standards of performance on student work created in response to a similar task. The intervention took just 50 minutes indicating a good educational return on the pedagogical investment.     

Adult Literacy: Monitoring and Evaluation for Practice and Policy

With the advent of the UN Literacy Decade launched in 2003, there is increased emphasis on monitoring and evaluation of adult literacy around the globe. The present paper presents an overview of the main approaches that have been taken to measure adult literacy within and across countries. A particular focus of the present review is to compare and contrast three models of literacy assessment: (1) the “traditional” model based largely on census enumeration, which has been used over recent decades to collect national and regional rates of illiteracy; (2) the large-scale survey techniques employed with the International Adult Literacy Survey and similar models; and (3) an intermediate type of assessment that borrows from the previous two models, and attempts to “tailor” the size and complexity of the survey methodology to the policy questions needing answers (called the SQC model). The present paper suggests that there is no single model or set of methods that are universally appropriate to monitoring and measuring in adult literacy around the world, but that blending both programmatic and comparative models through the SQC approach may bring greater involvement in, and insight into, adult literacy evaluations.     

Diagnosing misconceptions: Revealing changing decimal fraction knowledge

Conceptual change is a gradual process that occurs as students integrate new information into their existing conceptions. Throughout this process, assessing learning requires measures to diagnose misconceptions and understand how knowledge is changing. We developed three measures of misconceptions to assess students' knowledge early in instruction on decimals that measured the: 1) prevalence of misconception errors based on response patterns, 2) existence of misconceptions in a more abstract context, and 3) strength of misconceptions using confidence ratings. Students ages 9–11 (N = 297) completed the assessment at three time points. These measures revealed that whole number and role of zero misconceptions decreased and fraction misconceptions increased over time. The current measures also differentiated between weaker misconceptions that were changed after brief instruction and strongly held misconceptions. The current measures can create a more complete picture of knowledge than only measuring students' accuracy, providing a window into the conceptual change process.     

Who Is at Risk and Why? Teachers' Reasons for Concern and Their Understanding and Assessment of Early Literacy

This study focuses on an issue of recent policy significance—the need to aid teachers in successfully identifying why children struggle to acquire literacy. This study (a) asked K–1 teachers to nominate students that they believed to be at risk for literacy difficulties and to provide reasons for their concern, (b) examined how these reasons relate to teachers' broader conceptions of literacy, and (c) investigated whether teachers' initial reasons and checklist-guided ratings align with concurrently administered standardized assessments. Results revealed that teachers have a wide array of initial concerns for students. There was some discordance between teachers's specific reasons for concern and their broader conceptions of early literacy. Comparison of student performance on standardized measures with teacher rationale also revealed discordance. Specific guidelines to teachers on use of a literacy checklist increased concordance between subsequent teacher ratings and standardized measures in some reading-related skills but not others. Implications for the use of multiple sources of evidence for student performance, as well as professional development, are discussed.     

A Systematic Review of Assessment Literacy Measures

This work systematically reviews teacher assessment literacy measures within the context of contemporary teacher evaluation policy. In this study, the researchers collected objective tests of assessment knowledge, teacher self‐reports, and rubrics to evaluate teachers’ work in assessment literacy studies from 1991 to 2012. Then they evaluated the psychometric work from these measures against a set of claims related to score interpretation and use. Across the 36 measures reviewed, they found support for these claims was weak. This outcome highlights the need for increased work on assessment literacy measures in the educational measurement field. The authors conclude with recommendations and a resource to inform a research agenda focused on assessment literacy measurement to inform policy and practice.     

Experienced Middle School Science Teachers’ Assessment Literacy: Investigating Knowledge of Students’ Conceptions in Genetics and Ways to Shape Instruction

Using a framework of assessment literacy that included teachers?? view of learning, knowledge of assessment tools, and knowledge of assessment interpretation and action taking, this study explored the assessment literacy of five experienced middle school teachers. Multiple sources of data were: teachers?? predictions about students?? ideas, students?? written and verbal responses to assessment tasks, teacher background questionnaire, and a videotaped teacher focus group. We investigated middle school teachers?? predictions, interpretations, and recommended actions for formative assessment in genetics. Results documented a variety of ways that teachers would elicit students?? ideas in genetics, focusing on discussion strategies. Findings showed how well teachers predicted student conceptions compared to actual student conceptions. We also found that teachers mostly described general topics they would use to address students?? alternative conceptions. Less often, they explained specific content they would use to challenge ideas or pedagogical strategies for conceptual change. Teachers also discussed barriers to addressing ideas. Teacher professional development should provide more support in helping teachers close the formative assessment cycle by addressing conceptions that are elicited with assessments.     

Establishing Performance Standards Using Common Assignments and Rubrics: Measurable and Achievable Student Learning Outcomes

The struggle for many community colleges is how to model and/or remodel what happens in the classroom into a plan to measure and achieve student learning outcomes (Syed & Mojock, 2008). Encountering the same struggle, the psychology department of El Centro College (ECC) of the Dallas County Community College District implemented a methodology for measuring and achieving student learning outcomes. The methodology included the following: (a) using common assignments across all classes; (b) an informational literacy component integrated into the curriculum; (c) establishing performance standards via a common rubric; (d) model submissions depicting high quality work; and (e) a rating system of exemplars. By using this methodology, student learning outcomes within a discipline were both measurable and achievable.     

Teachers’ academic training for literacy instruction

Literacy instruction is a powerful determinant of children’s academic and school outcomes. Teachers’ training for literacy instruction is therefore critical for children’s reading learning. The present study examined the contents of 130 courses related to literacy instruction from a representative national sample of primary teachers’ undergraduate programs (N = 81), to address the following issues: which courses, related to literacy instruction, are included in the curriculum of undergraduate training programs? Which is the weight of the courses in undergraduate programs? What are the contents of the courses? The results show that most critical features of literacy instruction are included in the course contents of most programs (e.g. phonics, theory of literature). Still, some other critical features are underrepresented (reading/writing comprehension skills) or apparently missing (assessment and intervention in reading/writing problems). Moreover, the time allocated to literacy instruction seems to be scarce. Still, wide differences across university programs and courses were found.     

The Phenomenon of Information Literacy

Information literacy is being increasingly recognised as an important educational outcome for university graduates. How it is experienced, however, has only recently become the subject of scrutiny. The study reported here examines varying conceptions of information literacy amongst a group of lecturers, librarians, staff developers and learning counsellors. A phenomenographic approach was used to discover their conceptions. Data were gathered from participants, both male and female, through interviews, e‐mail discussions and workshops. As an outcome of the analysis, seven categories, or “faces” of information literacy were discovered. These categories depict information literacy as it is conceived or experienced. They provide target conceptions for the educational process which differ from the more conventional competencies or skill‐based objectives.     

‘I’m an assessment illiterate’: towards a shared discourse of assessment literacy for external examiners

The quality assurance ‘regime’ that spans the higher education sector internationally is underpinned by a number of unchallenged assumptions. Nowhere is this more apparent than within the external examiner system in the UK, an instrument for the professional self-regulation of higher education. One such assumption is that external examiners are assessment literate. Within higher education assessment literacy is a concept in its infancy, but one that has the capacity to reverse the deterioration of confidence in academic standards. Whilst the concept is becoming integrated into the sectorial vernacular, its fluid and negotiated nature is under-conceptualised. The primary aim of this article lies in investigating the extent of external examiners’ assessment literacy and to use this as a base to initiate a discussion surrounding the development of a shared discourse of assessment literacy, thereby providing a tool for greater conceptualisation.