A systematic review on the deployment and effectiveness of data analytics in higher education to improve student outcomes

Abstract

This is a systematic review conducted of primary research literature published between 2007 and 2018 on the deployment and effectiveness of data analytics in higher education to improve student outcomes. We took a methodological approach to searching databases; appraising and synthesising results against predefined criteria. We reviewed research on the effectiveness of three differentiated forms of data analytics: learning, academic and learner analytics. Student outcomes are defined as retention, academic performance and engagement. Our results find that three quarters of studies report the use of educational data analytics to be effective in improving student outcomes but their relationship with student outcomes requires further and more robust investigation and assessment. We argue that research must interpret and communicate effectiveness qualitatively, as well as quantitatively, by including the student voice in assessments of impact.     

Assessment criteria for competency-based education: a study in nursing education

This study examined the effects of type of assessment criteria (performance-based vs. competency-based), the relevance of assessment criteria (relevant criteria vs. all criteria), and their interaction on secondary vocational education students’ performance and assessment skills. Students on three programmes in the domain of nursing and care (N = 93) participated in the study. Results show that students who were given the relevant criteria made more accurate assessments of an expert model, performed better on a test and achieved higher instructional efficiency (defined as the relationship between performance and mental effort) compared to students who were given all criteria. Students who were given performance-based assessment criteria made more accurate assessments of an expert model and scored higher on task performance during practice compared to students who were given competency-based assessment criteria. They invested less mental effort in the assessments, resulting in higher instructional efficiency. An interaction effect for the concreteness of answers shows that the combination of performance-based and relevant criteria leads to superior analysis of worked examples compared to the other combinations of criteria.     

Integrating Scenario‐Based and Component Reading Skill Measures to Understand the Reading Behavior of Struggling Readers

In recent years, researchers, educators, and policy makers have called for a new generation of reading comprehension assessments (e.g., Partnership for 21st Century Skills, 2008). Advocates of this movement argue for a deeper type of reading assessment, one that captures students’ ability to not only understand single texts in isolation but also to engage in purposeful, multisource integration of sources. While this shift in how we define and measure reading comprehension is laudable, assessments must also measure the fundamental reading skills that may impede higher‐level comprehension processes. This article presents data from two assessments that were designed to work in tandem to provide a more complete picture of reading comprehension. Middle school students were given a component skills battery which measured core reading skills such as word recognition, decoding, vocabulary, and morphology, as well a second assessment designed to measure reading comprehension. Reading comprehension was measured using a scenario‐based assessment approach, which required students to read a range of sources to fulfill a particular reading goal. The results indicate that students, including struggling readers, were able to read, understand, and problem solve in complex learning environments, but students’ ability to do so was often tempered by their basic reading skills. We argue that including a measure of component skills alongside a measure of higher‐level comprehension is beneficial in interpreting student performance. Accordingly, we present the results on the scenario‐based measure as a function of reading component skills and argue for the value of using this approach for struggling readers.     

Exploring the Impact of Rater Effects on Person Fit in Rater-Mediated Assessments

Researchers have documented the impact of rater effects, or raters’ tendencies to give different ratings than would be expected given examinee achievement levels, in performance assessments. However, the degree to which rater effects influence person fit, or the reasonableness of test-takers’ achievement estimates given their response patterns, has not been investigated. In rater-mediated assessments, person fit reflects the reasonableness of rater judgments of individual test-takers’ achievement over components of the assessment. This study illustrates an approach to visualizing and evaluating person fit in assessments that involve rater judgment using rater-mediated person response functions (rm-PRFs). The rm-PRF approach allows analysts to consider the impact of rater effects on person fit in order to identify individual test-takers for whom the assessment results may not have a straightforward interpretation. A simulation study is used to evaluate the impact of rater effects on person fit. Results indicate that rater effects can compromise the interpretation and use of performance assessment results for individual test-takers. Recommendations are presented that call researchers and practitioners to supplement routine psychometric analyses for performance assessments (e.g., rater reliability checks) with rm-PRFs to identify students whose ratings may have compromised interpretations as a result of rater effects, person misfit, or both.     

Rhetoric and reality in science performance assessments: An update

This article addresses the rhetoric of performance assessment with research on important claims about science performance assessments. We found the following: (a) Concepts and terminology used to refer to performance assessments often were not consistent within and across researchers, educators, and policy-makers. (b) Performance assessments are highly sensitive not only to the tasks and the occasions sampled, but also to the method (e.g., hands-on, computer simulation) used to measure performance. (c) Performance assessments do not necessarily tap higher-order thinking, especially when they are poorly designed. (d) Performance assessments are expensive to develop and use: technology is needed for developing these assessments in an efficient way. (e) Performance assessments do not necessarily have the expected positive impact on teachers' teaching and students' understanding. (f) If teachers are to use performance assessments in their classrooms, they need professional development to help them construct the necessary knowledge and skills. This article attempts to address some of these realities by presenting a conceptual framework that might guide the development and the evaluation of performance assessments, as well as steps that might be taken to create a performance assessment technology and develop teacher inservice programs. © 1996 John Wiley & Sons, Inc.     

The Effects of Motivational Instruction on College Students' Performance on Low-Stakes Assessment

Assessments of student learning outcomes (SLO) have been widely used in higher education for accreditation, accountability, and strategic planning purposes. Although important to institutions, the assessment results typically bear no consequence for individual students. It is important to clarify the relationship between motivation and test performance and identify practical strategies to boost students' motivation in test taking. This study designed an experiment to examine the effectiveness of a motivational instruction. The instruction increased examinees' self-reported test-taking motivation by .89 standard deviations (SDs) and test scores by .63 SDs. Students receiving the instruction spent an average of 14 more seconds on an item than students in the control group. Score difference between experimental and control groups narrowed to .23 SDs after unmotivated students identified by low response time were removed from the analyses. The findings provide important implications for higher education institutions which administer SLO assessments in a low-stakes setting.     

Students’ views of oral performance assessment in mathematics: straddling the ‘assessment of’ and ‘assessment for’ learning divide

This paper explores the views of a group of students who took an oral performance assessment in a first-year mathematics module. Such assessments are unusual for most subjects in the UK, but particularly within the generally homogenous assessment diet of undergraduate mathematics. The evidence presented here resonates with some, but not all, of the existing literature on oral assessment and suggests that, despite concerns about anxiety and fairness, students see oral assessments as encouraging a focus on understanding, being relatively authentic and reactive to their needs. We argue that, suitably implemented, oral assessment may be a viable assessment method for straddling the ‘assessment for’ and ‘assessment of’ learning divide in higher education.     

The Design of Competency-Based Performance Assessment in E-Learning

This article focuses on the design of competency-based performance assessment in e-learning. Though effort has been invested in designing powerful e-learning environments, relatively little attention has been paid to the design of valid and reliable assessments in such environments, leaving many questions to educational developers and teachers. As a solution to this problem, a systematic approach to designing performance assessments in e-learning contexts is presented, partly based on the 4C/ID model. This model enables the construction of realistic whole tasks instead of advocating education that is restricted to more isolated skills. A new assessment procedure also implies an alternative view of instructional design, learning and assessment. The requirements for the learning environment are addressed. Examples from a virtual seminar are presented to illustrate the design approach. The article concludes with the identification of possible pitfalls related to the approach and gives directions for future research.     

Estimating the quality of performance assessments: The case of an ‘interactive examination’ for teacher competencies

Professional schools prepare students to become competent professionals. Consequently, there is a need for assessments that can determine the acquisition of the relevant professional competencies. Although using performance assessment to replace traditional paper-and-pencil tests might provide one way to move forward, the use of performance assessments for summative purposes has been shown to be problematic (e.g. marker consistency and construct representation). With the aid of a comprehensive framework of quality criteria for competence assessments, this article considers if one particular existing competence assessment methodology is suitable for summative as well as formative use. It is argued that the comprehensive quality estimation of the examination procedure aids in identifying strengths and weaknesses in the assessment methodology, and that this information can be used to facilitate the inclusion of performance assessment in higher education, both for summative and formative use.     

Supporting practitioners in school-based evaluations of autism spectrum disorders

Students with autism spectrum disorder (ASD) have vastly different educational needs. Although some students with ASD may perform well across subjects within the general education classroom, other students with ASD may need more individualized support outside of the classroom. Historically, ASD assessments in schools have primarily focused on the measurement of cornerstone behaviors of ASD such as deficits in social communication and restricted or repetitive behaviors while underestimating the focus of academic skills even though both factors are required in the evaluation process and for an eligibility classification. The purpose of this article is to provide an overview of academic assessment practices in the schools by highlighting best practices in ASD evaluation, outlining ASD academic profiles and how to identify academic skill deficits, and reviewing the available literature that comes from different formal- and function-based assessment practices that are commonly used in schools today. The authors also discuss how formal- and function-based assessments both serve a purpose and are recommended to be used in conjunction with one another to best demonstrate a student's academic profile. Practitioners are ultimately encouraged to use a multi-informant, multi-modal approach when it comes to psychoeducational evaluations for individuals with ASD as it is evident that an individualized approach is essential due to the discrepancy of academic profiles related to this disability category as well as understanding that no measure is a perfect representation of any one skill.     

The application of pairwise comparisons to form scaled exemplars as a basis for setting and exemplifying standards in teacher education

ABSTRACT

This paper details a two-stage process for standard setting using an extended pairwise comparison method. It describes the application of a new process of establishing a set of scaled and ordered performances as a basis for exemplifying standards. In stage one, the method of pairwise comparison was used to generate data from pairwise judgements made at the criterion level and overall grade. In stage two, judges identified where the pass/fail boundary lay in relation to a selected subset of ordered performances generated from stage one. The combined evidence from the two-stage process demonstrates the method’s application to establish exemplified standards using on-balance judgements across multiple criteria. The approach combines expert judgement and statistical data to provide a basis for comparable cross-institution assessments against standards in the case of a summative assessment of professional competence in teacher education. It may be relevant to any discipline with professional standards and complex performance assessments.     

Outcomes assessment and its role in self-reviews of undergraduate education: in the context of Japanese higher education reforms since the 1990s

Under the recent state of higher education, “Outcomes Assessment” has become a familiar term in Japan as in other nations all over the world. However, actual conditions of outcomes assessment and its contribution toward educational improvement are not always obvious. Thus, this article attempts to clarify: (1) Japanese higher education reforms since the 1990s focusing on assessment and evaluation; (2) the influence of reforms on outcomes assessment implemented by higher education institutions; (3) how much the results of outcomes assessment are used in self-reviews of undergraduate education; and (4) whether application of outcomes assessment contribute to educational improvement. The results of national survey conducted in Japan reveal that assessments are clearly being carried out, but may not have been supporting the improvement of education.     

Assessing Writing Portfolios: issues in the Validity and Meaning of Scores

This article discusses an approach to analyzing performance assessments that identifies potential reasons for misfitting items and uses this information to improve on items and rubrics for these assessments. Specifically, the approach involves identifying psychometric features and qualitative features of items and rubrics that may possibly influence misfit; examining relations between these features and the fit statistic; conducting an analysis of student responses to a sample of misfitting items; and finally, based on the results of the previous analyses, modifying characteristics of the items or rubrics and reexamining fit. A mathematics performance assessment containing 53 constructed-response items scored on a holistic scale from 0 to 4 is used to illustrate the approach. The 2-parameter graded response model (Samejima, 1969) is used to calibrate the data. Implications of this method of data analysis for improving performance assessment items and rubrics are discussed as well as issues and limitations related to the use of the approach.     

Criterion-referenced and norm-referenced assessments: compatibility and complementarity

The tension between criterion-referenced and norm-referenced assessment is examined in the context of curriculum planning and assessment in outcomes-based approaches to higher education. This paper argues the importance of a criterion-referenced assessment approach once an outcomes-based approach has been adopted. It further discusses the implementation of criterion-referenced assessment, considering to what extent the criteria and standards adopted are implicitly norm referenced. It introduces a compatible interpretation of criterion-referenced and norm-referenced assessments in higher education, and illustrates how their combined use can avoid grade inflation and also provide useful information to educators, employers and learners. Instead of seeing criterion referencing and norm referencing as a dichotomy, assessment in higher education benefits from their synthesis through a feedback loop that emphasises alignment between learning and assessment; such feedback and alignment are essential features of quality assurance and enhancement.     

Psychometric quality of a student evaluation of teaching survey in higher education

Student evaluations of teaching (SET) are used globally by higher education institutions for performance assessment of academic staff and evaluation of course quality. Higher education institutions commonly develop their own SETs to measure variables deemed relevant to them. However, ‘home-grown’ SETs are rarely assessed psychometrically. One potential consequence of this limitation is that an invalidated instrument may not provide accurate information for the intended purposes. Moreover, in the absence of psychometric assessment, the students’ voices collected by the SETs often fail to provide insight relative to their intended purpose. The present study evaluates a ‘home-grown’ SET using a Rasch model and confirmatory factor analysis. Our results identified weaknesses in two areas: the rating categories and the number of items used to measure the intended constructs. Suggestions are provided to address these weaknesses. This work provides an additional tool set for critical analysis of SET that is generally applicable for a variety of institutions, including those in Asia.     

增值评价:一种绿色升学率理念下的学校评价模式

增值评价以学校教育活动对学生预期成绩的增值为教育评价标准,用来判定学校对学生发展的影响,是一种绿色升学率理念下的发展性学校评价模式。与传统的学校评价模式相比,这种评价模式具有综合考察学生发展影响、强调发展性评价标准、重视起点关注过程、促进学校改进等特点,对推进我国教育的均衡发展、促进学校特色发展、实现教育教学公平会起到积极的作用。实证研究得出的一系列结论进一步证实了增值评价应该作为学校评价的重要组成部分。因此,各级教育行政部门、学校、家长和社会要重视运用增值评价的结果来客观分析和评价学校的作用和效能,使增值评价在促进学生和学校的全面发展,促进社会和谐方面发挥其应有的作用。     

A Polytomous Scoring Approach to Handle Not-Reached Items in Low-Stakes Assessments

In low-stakes assessments, some students may not reach the end of the test and leave some items unanswered due to various reasons (e.g., lack of test-taking motivation, poor time management, and test speededness). Not-reached items are often treated as incorrect or not-administered in the scoring process. However, when the proportion of not-reached items is high, these traditional approaches may yield biased scores and thereby threatening the validity of test results. In this study, we propose a polytomous scoring approach for handling not-reached items and compare its performance with those of the traditional scoring approaches. Real data from a low-stakes math assessment administered to second and third graders were used. The assessment consisted of 40 short-answer items focusing on addition and subtraction. The students were instructed to answer as many items as possible within 5 minutes. Using the traditional scoring approaches, students’ responses for not-reached items were treated as either not-administered or incorrect in the scoring process. With the proposed scoring approach, students’ nonmissing responses were scored polytomously based on how accurately and rapidly they responded to the items to reduce the impact of not-reached items on ability estimation. The traditional and polytomous scoring approaches were compared based on several evaluation criteria, such as model fit indices, test information function, and bias. The results indicated that the polytomous scoring approaches outperformed the traditional approaches. The complete case simulation corroborated our empirical findings that the scoring approach in which nonmissing items were scored polytomously and not-reached items were considered not-administered performed the best. Implications of the polytomous scoring approach for low-stakes assessments were discussed.     

Bayesian Model Averaging Over Directed Acyclic Graphs With Implications for the Predictive Performance of Structural Equation Models

This article examines Bayesian model averaging as a means of addressing predictive performance in Bayesian structural equation models. The current approach to addressing the problem of model uncertainty lies in the method of Bayesian model averaging. We expand the work of Madigan and his colleagues by considering a structural equation model as a special case of a directed acyclic graph. We then provide an algorithm that searches the model space for submodels and obtains a weighted average of the submodels using posterior model probabilities as weights. Our simulation study provides a frequentist evaluation of our Bayesian model averaging approach and indicates that when the true model is known, Bayesian model averaging does not yield necessarily better predictive performance compared to nonaveraged models. However, our case study using data from an international large-scale assessment reveals that the model-averaged submodels provide better posterior predictive performance compared to the initially specified model.     

Feedback,feedforward: evaluating the effectiveness of an oral peer review exercise amongst postgraduate students

Assessment for learning approaches, such as peer review exercises may improve student performance in summative assessments and increase their satisfaction with assessment practices. We conducted a mixed methods study to evaluate the effectiveness of an oral peer review exercise among post-graduate students. We examined: (1) final assessment grades among students who did and did not take part in the peer review exercise; (2) student perceptions of the impact of the peer review exercise; and (3) student understanding of, and satisfaction with, this new assessment practice. We found that students who took part in the exercise had a significantly higher mean grade in a subsequent summative oral presentation assessment than students who did not take part in the exercise. Students gained a better understanding of assessment and marking criteria and expressed increased confidence and decreased anxiety about completing the subsequent summative assessment. Assessment for learning improves academic attainment and the learning experience in postgraduate students.