Alternative Interpretations of Alternative Assessments: Some Validity Issues in Educational Performance Assessments

The use of alternative assessments has led many researchers to reexamine traditional views of test qualities, especially validity. Because alternative assessments generally aim at measuring complex constructs and employ rich assessment tasks, it becomes more difficult to demonstrate (a) the validity of the inferences we make and (b) that these inferences extrapolate to target domains beyond the assessment itself. An approach to addressing these issues from the perspective of language testing is described. It is then argued that in both language testing and educational assessment we must consider the roles of both language and content knowledge, and that our approach to the design and development of performance assessments must be both construct-based and task-based.¹     

Peer assessment in the digital age: a meta-analysis comparing peer and teacher ratings

Given the wide use of peer assessment, especially in higher education, the relative accuracy of peer ratings compared to teacher ratings is a major concern for both educators and researchers. This concern has grown with the increase of peer assessment in digital platforms. In this meta-analysis, using a variance-known hierarchical linear modelling approach, we synthesise findings from studies on peer assessment since 1999 when computer-assisted peer assessment started to proliferate. The estimated average Pearson correlation between peer and teacher ratings is found to be .63, which is moderately strong. This correlation is significantly higher when: (a) the peer assessment is paper-based rather than computer-assisted; (b) the subject area is not medical/clinical; (c) the course is graduate level rather than undergraduate or K-12; (d) individual work instead of group work is assessed; (e) the assessors and assessees are matched at random; (f) the peer assessment is voluntary instead of compulsory; (g) the peer assessment is non-anonymous; (h) peer raters provide both scores and qualitative comments instead of only scores; and (i) peer raters are involved in developing the rating criteria. The findings are expected to inform practitioners regarding peer assessment practices that are more likely to exhibit better agreement with teacher assessment.     

Exploring the use of complexity theory and action research as frameworks for curriculum change

This paper considers the impact of a small-scale action research project which focused on the development of an emergent approach to curriculum making in a general certificate in secondary education course in geography. In this context, we argue that complexity thinking offers a useful theoretical foundation from which to understand the nature of dynamic pedagogic change resulting from the application of action research methods. Results show that process-focused curriculum change can bring about shifts in both learning and assessment. This is seen as being the result of an emergence orientated approach to action research as a counter to more reductionist approaches which are often used and advocated in educational settings by teachers. We conclude that a combination of complexity thinking and action research can offer a valuable medium through which the educational needs of learners and teachers can be addressed in different, localized contexts.     

Personal understanding of assessment and the link to assessment practice: the perspectives of higher education staff

The study investigates how higher education staff understand assessment, and the relationship between these understandings and their assessment practices. Nine individuals attended a workshop that guided them through the creation of a concept map about assessment, which was subsequently discussed in one-to-one semi-structured interviews. We found considerable variation in understanding of assessment, both between and within participants, and this appeared to be a consequence of the varied contexts within which assessment operates. Some assessment practices were highly complex, and at times closely entwined with teaching. In addition, individuals’ practices helped to illuminate variation in how underlying concepts (e.g. assessment for learning) were understood. The approach supported the construction of the participants’ understanding of assessment, and enabled the exploration of the interplay between thinking and reported practice, which were closely aligned. It also drew attention to the need to further develop methodologies which capture both the complexity of thinking about assessment and real-world assessment practices.     

Using technology to facilitate effective assessment for learning and feedback in higher education

The aims of this paper are to examine and critically evaluate a selection of different technological methods that were specifically chosen for their alignment with, and potential to enhance, extant assessment for learning practice. The underpinning perspectives are that: (a) both formative and summative assessment are valuable opportunities for learning, and (b) using technology may enhance learning in assessment and feedback processes. Drawing on the literature and empirical evidence from a research study in a Scottish university, the advantages and drawbacks of using technology are examined. It is asserted that, by adopting a flexible approach and taking small incremental steps, the use of different types of technology can be beneficial in facilitating effective assessment for learning and feedback in higher education.     

Exploring classroom assessment practices: the case of teachers of English as a foreign language

The research investigated the factors which underlie the perceptions and usage of alternative assessment procedures among EFL teachers in Israel. The research was conducted within the framework of an earlier model by Hargreaves and colleagues comprising four perspectives – technological, cultural, political and postmodern – to account for teachers’ assessment practices and beliefs. The sample included 113 EFL teachers who responded to a self‐report questionnaire. The model’s four perspectives were validated using a two‐stage factor analysis. Results show that the predominant factor related to the usage of alternative assessment is the technological one, followed by the cultural and postmodern perspectives. The political perspective yielded mixed results. The findings highlight the complexity of teachers’ assessment practices reflecting not merely a testing approach but a social and educational paradigm encompassing micro constraints (technological), macro influences (political), ideologies and commonly‐held beliefs (cultural) as well as evidence of critical pedagogy (postmodern).     

Constructivist learning environments and the (im)possibility to change students’ perceptions of assessment demands and approaches to learning

Recent research shows that, as students interpret the demands of the assessment tasks, they vary their approaches to learning in order to cope with the assessment tasks. Three research questions are central in the present paper: (1) Do students who participate in a constructivist learning environment change their perception of assessment demands towards more deep level demands? (2) Do students in a constructivist learning environment change their approaches to learning towards a more deep approach to learning? (3) Is there a relation between change in approaches to learning and change in the perceptions of the assessment demands? Students following the course ‘Education and psychology’ of the teacher training program at the University of Antwerp completed questionnaires during the first, the second and the final lesson of the course. One questionnaire measured their approaches to learning and the other their general perceptions of the assessment demands. The course ‘Education and psychology’ can be labelled as a ‘constructivist learning environment’ with congruent assessment methods. Results of the paired sampled t-tests indicated that students indeed do change their perceptions of assessment demands towards more deep level demands. However, the results also indicated that students did not change their approach to learning towards a more deep approach. On the contrary, students seem to develop more surface approaches to learning during the course. Correlation analyses indicated that only changes of perceptions of assessment demands towards less surface levels are significantly related to changes in approaches to learning, towards a more surface approach. Results of the stepwise multiple regression analyses indicated that students’ approach to learning at the beginning of the course seems to have a higher impact on the extent to which they change their approach to learning than how students perceive the demands of the assessment within the course. These results point us to the complexity of the relationship between the learning environment, the students’ perceptions of assessment demands, and students’ approaches to learning.     

Why increasing the number of raters only helps sometimes: Reliability and validity of peer assessment across tasks of different complexity

Number of raters is theoretically central to peer assessment reliability and validity, yet rarely studied. Further, requiring each student to assess more peers’ documents both increases the number of evaluations per document but also assessor workload, which can decline performance. Moreover, task complexity is likely a moderating factor, influencing both workload and validity. This study examined whether changing the number of required peer assessments per student / number of raters per document affected peer assessment reliability and validity for tasks at different levels of task complexity. 181 students completed and provided peer assessments for tasks at three levels of task complexity: low complexity (dictation), medium complexity (oral imitation), and high complexity (writing). Adequate validity of peer assessments was observed for all three task complexities at low reviewing loads. However, the impacts of increasing reviewing load varied by reliability vs. validity outcomes and by task complexity.     

The Validity of National Curriculum Assessment

This paper reviews the validity of National Curriculum assessment in England. It works with the concept of 'consequential validity' (Messick, 1989) which incorporates both conventional 'reliability'issues and the use to which any assessment is put. The review uses the eight stage 'threats to validity'model developed by Crooks, Kane and Cohen (1996). The complexity of National Curriculum assessment makes evaluation difficult. These assessments are used for a variety of purposes so that the 'consequential'aspects are compounded. National Curriculum assessment also involves both Teacher Assessment and tests – each of which has strengths and limitations in relation to validity. The main finding is that the validity of National Curriculum assessment hinges on the balance between Teacher Assessment and testing. Between them they can meet Crooks et al.' s requirements of a valid assessment system. The current emphasis on the use of test results for school accountability and as a measure of national standards has undermined Teacher Assessment to a point at which the validity of the system is in question.     

Navigating the complexities at an LGBTQQI-identified charter school: An ethnography of c/overt narratives

The authors describe ethnographic research exploring the experiences of school stakeholders at a lesbian, gay, bisexual, transgender, queer, questioning, and intersex (LGBTQQI)–identified charter school. Participants evidenced use of an overt and covert narrative that appeared to reflect how they navigated the complexities at the LGBTQQI-identified charter school. Participants’ narrative included 5 broad themes of complexity: (a) a negotiation of autonomy with support and belonging, (b) ambiguities in professional roles and boundaries, (c) inconsistency across educational standards and assessment, (d) interaction between individual and collective identity, and (e) the gap between needs and resources. Implications for future practice and research are explored.     

The practice and products of communication inquiry and education

Communication is described as a discipline with an obligation to pursue any inquiry that will shed light on the process of life forces attempting a common union. A holistic perspective is cited as necessary to deal with the complexity and ambiguity this approach embodies: an approach that requires a recognition of both the art and the science of human communication. The ability to communicate effectively is termed fundamental for communication education. Consequently, for internal validity, both knowledge‐building and skills‐training are called for in the teaching of communication. For external validity, the discipline must strive to establish common understandings of its work and to send clear messages about the findings to those outside the field.     

Classification of Double Deficit Groups Across Time: An Analysis of Group Stability From Kindergarten to Second Grade

The Double Deficit Hypothesis of dyslexia is one approach to classifying students with reading disabilities. The theory offers four distinct groups of readers: (a) average readers, (b) students with phonological deficits, (c) students with naming speed deficits, and (d) students with double deficits: those having both (b) and (c). This study examines the stability of these groups from kindergarten to second grade. An initial sample of 214 students were tested at four time points on measures of rapid automatized naming, phonological awareness, and reading. Latent transition analyses were used to examine the stability of these groups over time. These analyses indicated moderate stability from kindergarten to second grade with the probability of movement between groups being higher in kindergarten and early first grade. The groups differed in reading achievement at each testing time, with the double deficit group obtaining the lowest scores. Implications for early assessment and intervention are discussed.     

Solving arithmetic word problems. An analysis of Spanish textbooks / Resolución de problemas aritméticos verbales. Un análisis de los libros de texto españoles

This study analyses whether the primary school mathematics textbooks from two Spanish publishers show a varied instructional diet of addition and multiplication problems at different levels of complexity. To do so, it analyses the problems in all the primary grades by the publishers Santillana and SM according to two levels of complexity: (a) procedural (number of steps needed to solve the problem); and (b) semantic/mathematical (addition or multiplication structures, with their different subtypes). The results show that: (a) these problems are so simple that the books themselves cannot be regarded as a sufficient tool to teach students to solve the more complex problems; and (b) if we compare them with previous studies, the design of the problems has hardly changed in 10 years. These results show that the variety of problems in books should be expanded both procedurally and semantically/mathematically, and teachers should be given assistance to compensate for these shortcomings when using these textbooks in class.     

Responsible research and innovation indicators for science education assessment: how to measure the impact?

ABSTRACT

The emerging paradigm of responsible research and innovation (RRI) in the European Commission policy discourse identifies science education as a key agenda for better equipping students with skills and knowledge to tackle complex societal challenges and foster active citizenship in democratic societies. The operationalisation of this broad approach in science education demands, however, the identification of assessment frameworks able to grasp the complexity of RRI process requirements and learning outcomes within science education practice. This article aims to shed light over the application of the RRI approach in science education by proposing a RRI-based analytical framework for science education assessment. We use such framework to review a sample of empirical studies of science education assessments and critically analyse it under the lenses of RRI criteria. As a result, we identify a set of 86 key RRI assessment indicators in science education related to RRI values, transversal competences and experiential and cognitive aspects of learning. We argue that looking at science education through the lenses of RRI can potentially contribute to the integration of metacognitive skills, emotional aspects and procedural dimensions within impact assessments so as to address the complexity of learning.     

A comparative study of effectiveness of peer assessment of individuals’ contributions to group projects in undergraduate construction management core units

In recent years, various forms of group work have been introduced in university courses across various subject domains, including construction management courses. Although the use of group work in higher education has sound pedagogical reasons and advantages, group work has its own drawbacks. Therefore, the acceptance by students and the success of group work critically depend on a fair and credible assessment of the group process. In this paper, the implementation of different approaches to peer assessment (PA) of individuals’ contributions to group projects in two core units in an undergraduate construction management course in an Australian university is reported. The effectiveness of the adopted PA approaches have been evaluated and validated by students. It has been found that contrary to doubts of the sufficiency of a simplistic approach to PAs, the fairness of a PA approach does not necessarily depend on its complexity. Besides, voluntary group discussions, learning and collaboration are found to aid in improving each of the group’s camaraderie. Hence, it is recommended that academics should develop both a structured methodology to progressively encourage group members to work cohesively in teams and effective PA approaches that measure individual member’s contribution.     

EDUCATIONAL DIAGNOSTIC ASSESSMENT

A strong demand currently exists for testing instruments that are capable of providing more informative and diagnostic results than typical tests offer. This paper reviews approaches that have been proposed for educational diagnostic assessment. Two major approaches are identified: (a) deficit assessment, which focuses on weaknesses of the student, and (b) error analysis, which focuses on the kinds of errors the student commits. This paper also reviews recent work related to diagnostic assessment that is based on the integration of methods from cognitive psychology and artificial intelligence. It is concluded that the development of powerful diagnostic instruments may require a reexamination of existing psychometric models and possibly the development of alternative ones. It is also pointed out that the traditional approach to the specification of content in terms of static taxonomies may not be appropriate given the dynamic and sequential nature of diagnostic assessment. Finally, it is noted that the psychometric and content demands of diagnostic assessment all but require test admininstration by computer.     

拆分特征选择及其在企业信用评估中应用

评估指标体系的选取是企业信用评估的首要问题，它是一个特征选择问题。文章提出了一种针时SVM组合技术的拆分特征选择方法，其主要思想是时SVM组合中的各个分类器分别进行特征选择，再采用不同的特征子集作为各子分类器的输入，进行组合建模与预测。文章从filter和wrapper相结合的思想出发，进行了子分类器的特征选择；之后，针对企业信用评估问题的特点，采用了二叉树结构作为SVM的组合策略。实验表明，拆分特征选择方法能选出规模较小、具有一定差异的关键指标集，提高了模型的分类性能，并且具有计算简单，运行快速的优点。     

Criteria Teachers Use to Score Performance Items

This article addresses the issue of language-related construct-irrelevant variance on content area tests from the perspective of systemic functional linguistics. We propose that the construct relevance of language used in content area assessments, and consequent claims of construct-irrelevant variance and bias, should be determined according to the degree of correspondence between language use in the assessment and language use in the educational contexts in which the content is learned and used. This can be accomplished by matching the linguistic features of an assessment and the linguistic features of the domain in which the assessment is measuring achievement. This represents a departure from previous work on the assessment of English language learners’ content knowledge that has assumed complex linguistic features are a source of construct irrelevant variance by virtue of their complexity.