Using multidimensional scaling to improve functionality of the Revised Learning Process Questionnaire

The Revised Learning Process Questionnaire has been part of the development of a conceptual understanding of how students learn and what motivates them to engage in particular tasks. We obtained responses from 329 student volunteers at a mid‐sized public university in the southeast USA. We first investigated whether the psychometric properties and latent factor structure of this questionnaire are replicable in a different educational context and with students from a different country than that originally used to create and validate the questionnaire. We found this to be true. Second, we used Profile Analysis via Multidimensional Scaling (PAMS) to improve the diagnostic functionality of the instrument as well as further explore the latent structure of the questionnaire. The factor structure was evident in this solution, but we found that interpreting the latent structure in terms of the dimensions of Strategy and Motive as opposed to the factors of Deep and Surface approaches to be more appropriate for diagnostic use. We also found that PAMS has the inherent ability to assess an individual’s fit within the model, thereby acting as a measure of self‐report credibility. The Strategy dimension was found to have ecological validity through analysing its relationship to academic performance.     

Allowing course compensation in higher education: a latent class regression analysis to evaluate performance on a follow-up course

Abstract

In this study, the consequences of allowing course compensation in a higher education academic dismissal policy are evaluated by examining performance on a second-year follow-up (i.e. sequel) course that builds on material from a first-year precursor course. Up to now, differences in the consequences of compensation on student performance across groups of students who portray different unobserved study processes were not considered. In this study we used a latent class regression model to distinguish latent groups of students. Data from two undergraduate curricula were used and latent classes were formed based on similar patterns in averages, variability in grades, the number of compensated courses, and the number of retakes in the first year. Results show that students can be distinguished by three latent classes. Although the first-year precursor course is compensated in each of these latent classes, low performance on the precursor course results in low performance on the second-year sequel course for psychology students who belong to a class in which the average across first-year courses is low and the average number of compensated courses and retakes are high. For these students, compensation on a precursor course seems more likely to relate to insufficient performance on a sequel course.     

Early prediction of student knowledge in game-based learning with distributed representations of assessment questions

Game-based learning environments hold significant promise for facilitating learning experiences that are both effective and engaging. To support individualised learning and support proactive scaffolding when students are struggling, game-based learning environments should be able to accurately predict student knowledge at early points in students' gameplay. Student knowledge is traditionally assessed prior to and after each student interacts with the learning environment with conventional methods, such as multiple choice content knowledge assessments. While previous student modelling approaches have leveraged machine learning to automatically infer students' knowledge, there is limited work that incorporates the fine-grained content from each question in these types of tests into student models that predict student performance at early junctures in gameplay episodes. This work investigates a predictive student modelling approach that leverages the natural language text of the post-gameplay content knowledge questions and the text of the possible answer choices for early prediction of fine-grained individual student performance in game-based learning environments. With data from a study involving 66 undergraduate students from a large public university interacting with a game-based learning environment for microbiology, Crystal Island , we investigate the accuracy and early prediction capacity of student models that use a combination of gameplay features extracted from student log files as well as distributed representations of post-test content assessment questions. The results demonstrate that by incorporating knowledge about assessment questions, early prediction models are able to outperform competing baselines that only use student game trace data with no question-related information. Furthermore, this approach achieves high generalisation, including predicting the performance of students on unseen questions.

Practitioner notes

What is already known about this topic

• A distinctive characteristic of game-based learning environments is their capacity to enable fine-grained student assessment.
• Adaptive game-based learning environments offer individualisation based on specific student needs and should be able to assess student competencies using early prediction models of those competencies.
• Word embedding approaches from the field of natural language processing show great promise in the ability to encode semantic information that can be leveraged by predictive student models.

What this paper adds

• Investigates word embeddings of assessment question content for reliable early prediction of student performance.
• Demonstrates the efficacy of distributed word embeddings of assessment questions when used by early prediction models compared to models that use either no assessment information or discrete representations of the questions.
• Demonstrates the efficacy and generalisability of word embeddings of assessment questions for predicting the performance of both new students on existing questions and existing students on new questions.

Implications for practice and/or policy

• Word embeddings of assessment questions can enhance early prediction models of student knowledge, which can drive adaptive feedback to students who interact with game-based learning environments.
• Practitioners should determine if new assessment questions will be developed for their game-based learning environment, and if so, consider using our student modelling framework that incorporates early prediction models pretrained with existing student responses to previous assessment questions and is generalisable to the new assessment questions by leveraging distributed word embedding techniques.
• Researchers should consider the most appropriate way to encode the assessment questions in ways that early prediction models are able to infer relationships between the questions and gameplay behaviour to make accurate predictions of student competencies.

     

Modeling Predictors of Latent Classes in Regression Mixture Models

The purpose of this study is to provide guidance on a process for including latent class predictors in regression mixture models. We first examine the performance of current practice for using the 1-step and 3-step approaches where the direct covariate effect on the outcome is omitted. None of the approaches show adequate estimates of model parameters. Given that Step 1 of the 3-step approach shows adequate results in class enumeration, we suggest using an alternative approach: (a) decide the number of latent classes without predictors of latent classes, and (b) bring the latent class predictors into the model with the inclusion of hypothesized direct covariate effects. Our simulations show that this approach leads to good estimates for all model parameters. The proposed approach is demonstrated by using empirical data to examine the differential effects of family resources on students’ academic achievement outcome. Implications of the study are discussed.     

Building a whole school approach and teacher efficacy with English language learners

Current professional development initiatives show a preference for whole school approaches. However, pedagogy related to English language learners (ELLs) is often not widely dispersed in New Zealand primary schools, impacting on teacher efficacy with these learners. This paper discusses findings from a qualitative study which aimed to identify influences shaping the dispersal of ELL-related pedagogy in two schools in contrasting socio-economic areas. Data from semi-structured individual interviews with teachers at different class levels revealed unique patterns for the dispersal of ELL-related pedagogy in each school. Teacher data indicated that these patterns were influenced by the performance of ELLs in national standards testing, as well as the special characteristics of ELLs in each setting and the distinctive nature of each context. The dispersal of ELL pedagogy in each school impacted differentially on opportunities to build a sense of collective teaching efficacy with ELLs in that setting, but also raised questions about whether one size should be expected to fit all.     

Robustness of Stepwise Latent Class Modeling With Continuous Distal Outcomes

Recently, several bias-adjusted stepwise approaches to latent class modeling with continuous distal outcomes have been proposed in the literature and implemented in generally available software for latent class analysis. In this article, we investigate the robustness of these methods to violations of underlying model assumptions by means of a simulation study. Although each of the 4 investigated methods yields unbiased estimates of the class-specific means of distal outcomes when the underlying assumptions hold, 3 of the methods could fail to different degrees when assumptions are violated. Based on our study, we provide recommendations on which method to use under what circumstances. The differences between the various stepwise latent class approaches are illustrated by means of a real data application on outcomes related to recidivism for clusters of juvenile offenders.     

The missing link: an econometric analysis on the impact of curriculum standards on student achievement

Nationwide, state policy-makers have increasingly adopted curriculum standards as a means of improving education. However, relatively little empirical research has been done to investigate if a link actually exists between curriculum standards and student performance. Using data collected by the New York State Department of Education, the impact of standards on high school student achievement is examined through estimation of a one-way fixed-effects model of the education production function. Curriculum standards are operationalized as the award of a “50–64” variance, which is considered a means of increasing the number of students taking demanding courses. The findings suggest that curriculum standards can improve student performance, but they do little to improve equity. While larger portions of pupils pass curriculum-based assessment exams, continued associations between select student characteristics and student performance remain.     

Latent Class Detection and Class Assignment: A Comparison of the MAXEIG Taxometric Procedure and Factor Mixture Modeling Approaches

Taxometric procedures such as MAXEIG and factor mixture modeling (FMM) are used in latent class clustering, but they have very different sets of strengths and weaknesses. Taxometric procedures, popular in psychiatric and psychopathology applications, do not rely on distributional assumptions. Their sole purpose is to detect the presence of latent classes. The procedures capitalize on the assumption that, due to mean differences between two classes, item covariances within class are smaller than item covariances between the classes. FMM goes beyond class detection and permits the specification of hypothesis-based within-class covariance structures ranging from local independence to multidimensional within-class factor models. In principle, FMM permits the comparison of alternative models using likelihood-based indexes. These advantages come at the price of distributional assumptions. In addition, models are often highly parameterized and susceptible to misspecifications of the within-class covariance structure.

Following an illustration with an empirical data set of binary depression items, the MAXEIG procedure and FMM are compared in a simulation study focusing on class detection and the assignment of subjects to the latent classes. FMM generally outperformed MAXEIG in terms of class detection and class assignment. Substantially different class sizes negatively impacted the performance of both approaches, whereas low class separation was much more problematic for MAXEIG than for the FMM.     

Illuminating growth and struggles using mixed methods: Practice-based professional development and coaching for differentiating SRSD instruction in writing

In this mixed methods study, qualitative, quantitative, and single-case methods were combined to provide a comprehensive investigation of teacher and student outcomes following practice-based professional development (PBPD) for self-regulated strategy development (SRSD) in writing. Qualitative observations were used to determine outcomes among the three-fourth grade teachers involved, a nested quantitative model was used to analyze classwide student writing outcomes across 53 students in the three classes, and single case design was used to determine differential outcomes among randomly selected struggling and average writers in each class. PBPD was followed by coaching for differentiation based on student performance and fidelity as teachers taught their students strategies for writing timed imaginary stories (with self as main character), as required by the state writing test. Qualitative results indicated two teachers did not differentiate writing instruction without coaching; one teacher was unresponsive to coaching and did not differentiate instruction. Classwide analysis demonstrated significant growth in writing at the class level for students from baseline to posttesting. Single case design results indicated mixed outcomes among struggling and average writers and instances where instruction was not effective. Teachers indicated high social validity for PBPD and for SRSD; students indicated high social validity for SRSD. Results across the multiple methods illuminate growth and struggles often disguised in group data and are addressed to aid in understanding and enhancing PBPD as well as instruction in SRSD and other evidence-based practices. Limitations and future research directions are discussed.     

A Note on Cluster Effects in Latent Class Analysis

This article examines the effects of clustering in latent class analysis. A comprehensive simulation study is conducted, which begins by specifying a true multilevel latent class model with varying within- and between-cluster sample sizes, varying latent class proportions, and varying intraclass correlations. These models are then estimated under the assumption of a single-level latent class model. The outcomes of interest are measures of bias in the Bayesian Information Criterion (BIC) and the entropy R ² statistic relative to accounting for the multilevel structure of the data. The results indicate that the size of the intraclass correlation as well as between- and within-cluster sizes are the most prominent factors in determining the amount of bias in these outcome measures, with increasing intraclass correlations combined with small between-cluster sizes resulting in increased bias. Bias is particularly noticeable in the BIC. In addition, there is evidence that class separation interacts with the size of the intraclass correlations and cluster sizes in producing bias in these measures.     

Defining and Measuring College and Career Readiness: A Validation Framework

This article reviews the intended uses of these college‐ and career‐readiness assessments with the goal of articulating an appropriate validity argument to support such uses. These assessments differ fundamentally from today's state assessments employed for state accountability. Current assessments are used to determine if students have mastered the knowledge and skills articulated in state standards; content standards, performance levels, and student impact often differ across states. College‐ and career‐readiness assessments will be used to determine if students are prepared to succeed in postsecondary education. Do students have a high probability of academic success in college or career‐training programs? As with admissions, placement, and selection tests, the primary interpretations that will be made from test scores concern future performance. Statistical evidence between test scores and performance in postsecondary education will become an important form of evidence. A validation argument should first define the construct (college and career readiness) and then define appropriate criterion measures. This article reviews alternative definitions and measures of college and career readiness and contrasts traditional standard‐setting methods with empirically based approaches to support a validation argument.     

Improving the Background Knowledge of Non-traditional Students

Three intervention methods to assist non-traditional college students in evaluating and improving their chemical and mathematical backgrounds were developed and implemented over three successive years. The goal was to improve students' self-confidence and success in the course. The first method was a one-day optional symposium featuring a variety of tutorial sessions. In the second, a pre-test helped students determine potential deficits in requisite knowledge, and topical workshops were made available outside of class time. The third method used the same assessment pre-test combined with instructional videos on selected topics drawn from the pre-test. The three approaches were compared in a formative sense with respect to 1) student identification of areas of potential weakness, 2) accessibility of each method to the non-traditional student population. The success of the third method at improving student course performance was also examined.     

Latent Class Modeling with A Time-To-Event Distal Outcome: A Comparison of One,Two and Three-Step Approaches

Latent class methods can be used to identify unobserved subgroups which differ in their observed data. Researchers are often interested in outcomes for the identified subgroups and in some disciplines time-to-event outcome measures are common, e.g., overall survival in oncology. In this study Monte Carlo simulation is used to evaluate the empirical properties of latent class effect estimates on a time-to-event distal outcome using one, two and three-step approaches. Both standard and inclusive bias-corrected three-step approaches are considered. One-step latent class effect estimates are shown to be superior to the evaluated alternatives. Both the two-step approach and a standard three-step approach, where subjects are partially assigned to latent classes, produced unbiased estimates with nominal confidence interval coverage when latent classes were well separated, but not otherwise.

     

Using interactive content and online activities to accommodate diversity in a large first year class

As in many universities, class sizes have increased more quickly than teaching and learning resources. A related challenge is the increasing diversity of the student body in terms of socio-economic background, learning styles, English language ability and preparedness. This paper explores ways in which traditional face-to-face teaching methods (such as lectures and tutorials) can be combined with online teaching and learning activities in a “blended” learning approach to improve teaching and learning and to accommodate student diversity. Using a large first year Economics class as a case study, data were collected on student perceptions of the use of blended learning. A statistical model was used on a sample of 50 students to determine which online activities were most beneficial in improving student performance. The majority of students agreed that replacing one lecture a week with online activities and resources improved their learning, although about 20 % of the class would have preferred more lecturing and fewer online resources. Statistical finding showed that more active online resources, such as multiple choice and graphing questions, were most beneficial in improving student performance, but that more passive lecture capture was also useful in modelling the discourse of the discipline.     

Marking the Difference: An investigation of the criteria used for assessing undergraduate dissertations in a business school

Although the outcomes of a first degree are complex, UK universities continue to use a single measure for grading the student's performance—namely the degree classification. This classification has significant implications for the student, prospective employers, and for the awarding institution. There is some evidence of changing perceptions of degree classes, for example that third class (and even, in some cases, lower second class) awards are unacceptable. At the same time there is concern about maintenance of quality standards in HE, and of possible grade inflation. The final degree class, although a single measure, results from an accumulation of marks achieved throughout the degree course; it is the combined weighting of individual tutor marks which ultimately determines the student's degree class. This study focuses on the marking process, and the application of criteria within one final year module—the dissertation. The dissertation formed a significant part of the final degree weighting, and there was evidence of the use of different criteria and approaches to marking by staff within the faculty. The study employed a qualitative research methodology—including: a structured group discussion where the issue of criteria was raised and brainstormed; and a questionnaire to staff which asked open questions about the criteria which each used when marking dissertations. The responses were analysed, and common ground and differences identified. Questions are raised concerning (a) the tension which surfaces within the marking process, between individual tutor freedom and institutional demands, and (b) staff development opportunities within the marking process.     

A Comparison of Approaches for Estimating Covariate Effects in Nonparametric Multilevel Latent Class Models

The inclusion of covariates improves the prediction of class memberships in latent class analysis (LCA). Several methods for examining covariate effects have been developed over the past decade; however, researchers have limited to the comparisons of the performance among these methods in cases of the single-level LCA. The present study investigated the performance of three different methods for examining covariate effects in a multilevel setting. We conducted a simulation to compare the performance of the three methods when level-1 and level-2 covariates were simultaneously incorporated into the nonparametric multilevel latent class model to predict latent class membership at each level. The simulation results revealed that the bias-adjusted three-step maximum likelihood method performed equally well as the one-step method when the sample sizes were sufficiently large and the latent classes were distinct from each other. However, the unadjusted three-step method significantly underestimated the level-1 covariate effect in most conditions.

     

Tailoring National Standards to Early Science Teacher Identities: Building on Personal Histories to Support Beginning Practice

Individual recommendation plans (IRP) for student teaching practice were co-constructed with two methods students based on the select application of National Science Teachers Association’s National Standards for Science Teacher Preparation. Methods students completed a resume, an interview on pedagogical preferences, and a learning styles survey to determine the reform-based standards and pedagogical approaches that better fit their personal histories and identity formation as science teachers. Each case was unique with one student better meeting the Standards of “Issues” and “Science in the Community” and the other student better meeting the standards of “Inquiry” and the “Nature of Science”. Student teachers planned and taught lessons based on their IRP and were mostly successful in meeting their prescribed standards and utilizing their preferred pedagogies. However, their success in use of specific strategies supporting their approach was highly dependent upon classroom context. The use of the IRP process as a reflective tool strengthening identity formation and early practice is discussed.     

Class Size and Student Performance at a Public Research University: A Cross-Classified Model

This study addresses several methodological problems that have confronted prior research on the effect of class size on student achievement. Unlike previous studies, this analysis accounts for the hierarchical data structure of student achievement, where grades are nested within classes and students, and considers a wide range of class sizes across various disciplines. Based on data from undergraduate class sections at a single institution, the study provides consistent evidence of a negative effect of class size on grade performance, most substantially affecting the achievement of “A” grades with lesser effect on grades of “C” or higher. Because models with logarithmic specification show superior fit, this study also demonstrates that the effect of class size on students’ final grades diminishes as the class size increases. These findings suggest that a larger impact on student performance could be attained by further trimming enrollment in small classes than by reducing class size overall.     

Exploring latent class based on growth rates in number sense ability

The purpose of this study was to explore latent class based on growth rates in number sense ability by using latent growth class modeling (LGCM). LGCM is one of the noteworthy methods for identifying growth patterns of the progress monitoring within the response to intervention framework in that it enables us to analyze latent sub-groups based not on an arbitrary cut-point but on each group’s growth pattern. Progress monitoring data for number sense were administered in four times for age of 4(n = 58), 5(n = 95), and 6(n = 58) children, by the measure named basic academic skill assessment: number sense developed to assess students’ number sense and includes Number identification, Missing number, Quantity discrimination and estimation. To perform LGCM analysis, M plus 5.0 was used, and Bayesian information criteria and entropy values were used as criteria to determine the number of sub-groups. Results showed that there were 2, 4, and 4 sub-groups according to each age group based on the growth patterns. Each group’s growth patterns were classified differently based on their initial performance and growth level. Advantages and limitations of using LGCM method to analyze latent groups’ growth patterns for screening and identifying children at risk were discussed.     

Flipping the classroom and instructional technology integration in a college-level information systems spreadsheet course

The purpose of this research was to explore how technology can be used to teach technological skills and to determine what benefit flipping the classroom might have for students taking an introductory-level college course on spreadsheets in terms of student achievement and satisfaction with the class. A pretest posttest quasi-experimental mixed methods design was utilized to determine any differences in student achievement that might be associated with the instructional approach being used. In addition, the scalability of each approach was evaluated along with students’ perceptions of these approaches to determine the affect each intervention might have on a student’s motivation to learn. The simulation-based instruction tested in this study was found to be an extremely scalable solution but less effective than the regular classroom and flipped classroom approaches in terms of student learning. While students did demonstrate learning gains, the process focus of the simulation’s instruction and assessments frustrated students and decreased their motivation to learn. Students’ attitudes towards the topic, their willingness to refer the course to others, and the likelihood that they would take another course like this were considerably lower than those of students in the flipped or regular classroom situations. The results of this study support the conclusion that a technology enhanced flipped classroom was both effective and scalable; it better facilitated learning than the simulation-based training and students found this approach to be more motivating in that it allowed for greater differentiation of instruction.