What is computer-based testing washback,how can it be evaluated and how can this support practitioner research?

ABSTRACT

With the introduction of a new initiative in a teaching and learning environment there is an ethical responsibility to consider whether the impact of the introduction has met its intended goals, and whether it has harmed those who are influenced by it. Technology and infrastructure developments have encouraged a continued growth in the development and introduction of computer-based tests (CBTs) in educational environments. In the educational assessment literature, enquiry into the impact of testing (of all types) is known as ‘washback’. This is a reference to the way in which a test might have a range of influences on learners and teachers prior to the test-taking event. This article reviews the literature on CBT washback and outlines a framework for studying its effects as it is introduced into educational contexts. We then outline a research framework that we have developed (based on the literature) that can be used to evaluate CBT washback. We go on to argue that, to fulfil its potential in supporting the development of change, the research framework needs to act as a mediating device that brings together teaching-practitioner and researcher perspectives. The framework that we propose conceptualises the nature of washback in CBT contexts, as well as the research process and the methods required to understand it. This framework provides an element of common ground between practitioners (i.e. teachers who are involved in a CBT development process) and external researchers, and supports collaboration at three distinct levels.     

When Are Multidimensional Data Unidimensional Enough for Structural Equation Modeling? An Evaluation of the DETECT Multidimensionality Index

In structural equation modeling (SEM), researchers need to evaluate whether item response data, which are often multidimensional, can be modeled with a unidimensional measurement model without seriously biasing the parameter estimates. This issue is commonly addressed through testing the fit of a unidimensional model specification, a strategy previously determined to be problematic. As an alternative to the use of fit indexes, we considered the utility of a statistical tool that was expressly designed to assess the degree of departure from unidimensionality in a data set. Specifically, we evaluated the ability of the DETECT “essential unidimensionality” index to predict the bias in parameter estimates that results from misspecifying a unidimensional model when the data are multidimensional. We generated multidimensional data from bifactor structures that varied in general factor strength, number of group factors, and items per group factor; a unidimensional measurement model was then fit and parameter bias recorded. Although DETECT index values were generally predictive of parameter bias, in many cases, the degree of bias was small even though DETECT indicated significant multidimensionality. Thus we do not recommend the stand-alone use of DETECT benchmark values to either accept or reject a unidimensional measurement model. However, when DETECT was used in combination with additional indexes of general factor strength and group factor structure, parameter bias was highly predictable. Recommendations for judging the severity of potential model misspecifications in practice are provided.     

Cross-Validation of a Model of Intrinsic Motivation With Students Enrolled in High School Elective Courses

The purpose of this study was to cross-validate a model of relationships among social-contextual factors, individual differences, and intrinsic motivation in adolescent students enrolled in required courses (E. Ferrer-Caja & M. R. Weiss, 2000) with an independent sample of students taking elective courses. Female and male high school students (N = 219) completed measures of motivational climate, teaching style, perceived competence, self-determination, goal orientation, and intrinsic motivation. Motivated behavior was assessed by teachers who rated the students on effort and persistence in class activities. First, the authors used structural equation modeling to examine model invariance between the original and the new samples, which yielded a lack of equivalence. Next, the authors examined several alternative theory-based models using the elective sample. The results indicated that the data were best represented by a model that separated social-contextual factors, individual factors, intrinsic motivation, and motivated behaviors. The strongest predictors of intrinsic motivation were task-goal orientation and perceived competence. These results are discussed from both theoretical and methodological perspectives.     

Taking the Time to Improve the Validity of Low-Stakes Tests: The Effort-Monitoring CBT

The attractiveness of computer-based tests (CBTs) is due largely to their capability to expand the ways we conduct testing. A relatively unexplored application, however, is actively using the computer to reduce construct-irrelevant variance while a test is being administered. This investigation introduces the effort-monitoring CBT, in which the computer monitors examinee effort (based on item response time) in a low-stakes test and displays warning messages to those exhibiting rapid-guessing behavior. The results of an experimental study are presented, which showed that an effort-monitoring CBT increased examinee effort and yielded more valid test scores than a conventional CBT. Thus, unlike previous research that has focused on identifying rapid-guessing behavior after it has occurred, the effort-monitoring CBT proactively attempts to suppress rapid-guessing behavior. This innovative testing procedure extends the capabilities of measurement practitioners to manage the psychometric challenges posed by unmotivated examinees.     

A Validity Framework for Evaluating the Technical Quality of Alternate Assessments

This article presents findings from two projects designed to improve evaluations of technical quality of alternate assessments for students with the most significant cognitive disabilities. We argue that assessment technical documents should allow for the evaluation of the construct validity of the alternate assessments following the traditions of Cronbach (1971) , Messick (1989, 1995) , Linn, Baker, and Dunbar (1991) , and Shepard (1993) . The projects used the work of Knowing What Students Know ( Pellegrino, Chudowsky, & Glaser, 2001 ) to structure and focus the collection and evaluation of assessment information. The heuristic of the assessment triangle ( Pellegrino et al., 2001 ) was particularly useful in emphasizing that the validity evaluation needs to consider the logical connections among the characteristics of the students tested and how they develop domain proficiency (the cognition vertex), the nature of the assessment (the observation vertex), and the ways in which the assessment results are interpreted (the interpretation vertex). This project has shown that in addition to designing more valid assessments, the growing body of knowledge about the psychology of achievement testing can be useful for structuring evaluations of technical quality.     

The Model Size Effect in SEM: Inflated Goodness-of-Fit Statistics Are Due to the Size of the Covariance Matrix

The size of a model has been shown to critically affect the goodness of approximation of the model fit statistic T to the asymptotic chi-square distribution in finite samples. It is not clear, however, whether this “model size effect” is a function of the number of manifest variables, the number of free parameters, or both. It is demonstrated by means of 2 Monte Carlo computer simulation studies that neither the number of free parameters to be estimated nor the model degrees of freedom systematically affect the T statistic when the number of manifest variables is held constant. Increasing the number of manifest variables, however, is associated with a severe bias. These results imply that model fit drastically depends on the size of the covariance matrix and that future studies involving goodness-of-fit statistics should always consider the number of manifest variables, but can safely neglect the influence of particular model specifications.     

Alternative Methods for Assessing Mediation in Multilevel Data: The Advantages of Multilevel SEM

Multilevel modeling (MLM) is a popular way of assessing mediation effects with clustered data. Two important limitations of this approach have been identified in prior research and a theoretical rationale has been provided for why multilevel structural equation modeling (MSEM) should be preferred. However, to date, no empirical evidence of MSEM's advantages relative to MLM approaches for multilevel mediation analysis has been provided. Nor has it been demonstrated that MSEM performs adequately for mediation analysis in an absolute sense. This study addresses these gaps and finds that the MSEM method outperforms 2 MLM-based techniques in 2-level models in terms of bias and confidence interval coverage while displaying adequate efficiency, convergence rates, and power under a variety of conditions. Simulation results support prior theoretical work regarding the advantages of MSEM over MLM for mediation in clustered data.     

Computer-aided authoring of assessment instruments: An activity-theoretical approach

Abstract

Various researches have been conducted on the role and importance of assessment in education as well as its impact on the learner and the overall learning process. In fact, the way assessment is formulated in a particular subject shapes the way students learn. They focus their learning to comply with assessment requirements that they anticipate. In this article, the study is focused on the written examination papers (teacher-made tests) that are normally prepared at the end of a semester or an academic year to assess students of secondary and tertiary levels. The study also investigates how well papers are set and balanced according to the cognitive levels defined by Bloom (1956) and the learning outcomes/objectives as defined for the subjects. A collaborative process model as a framework for the design of such tests that can enhance the evaluation process is proposed. A brief argument is made for a case for a computer-supported collaborative environment to implement such a framework and which is based on activity theory. Such a framework is implemented in the form of MYSTIC; a collaborative authoring software for assessment instruments. The software allows stand-alone as well as collaborative authoring of examination papers and also helps academics' decision-making concerning the examination paper balancing and moderating process by graphically displaying and comparing marks allocated per question paper against the learning objectives     

A Comparison of CFA,ESEM, and BSEM in Test Structure Analysis

Minor cross-loadings on non-targeted factors are often found in psychological or other instruments. Forcing them to zero in confirmatory factor analyses (CFA) leads to biased estimates and distorted structures. Alternatively, exploratory structural equation modeling (ESEM) and Bayesian structural equation modeling (BSEM) have been proposed. In this research, we compared the performance of the traditional independent-clusters-confirmatory-factor-analysis (ICM-CFA), the nonstandard CFA, ESEM with the Geomin- or Target-rotations, and BSEMs with different cross-loading priors (correct; small- or large-variance priors with zero mean) using simulated data with cross-loadings. Four factors were considered: the number of factors, the size of factor correlations, the cross-loading mean, and the loading variance. Results indicated that ICM-CFA performed the worst. ESEMs were generally superior to CFAs but inferior to BSEM with correct priors that provided the precise estimation. BSEM with large- or small-variance priors performed similarly while the prior mean for cross-loadings was more important than the prior variance.     

The Reliability of Observations of Talkativeness and Social Contact among Nursery School Children by the “Short Time Sample” Technique

In 1958, Page conducted a large multiple experiment: 74 teachers gave one class its normal quiz, scored and graded it in the usual way, assigned three comment treatments to students in stratified-random blocks, and then reported scores from the next objective quiz. There was a highly significant effect of comments. Others have borrowed some study features, with results that have appeared mixed. Here, a critical overall analysis shows much agreement with the ordered hypothesis of comments and with specified comments over no comments (p < .01). Despite great variety of designs and subtlety of effect, results broadly support teachers who comment. A typical effect size is demonstrated for ranks, and lessons are taken about the proper strategies for designs and the future of such research.