Developing formative assessments for postgraduate students in engineering

This paper outlines an approach taken to produce computer-based formative assessments for two modules in a one-year taught MSc programme in Road Management and Engineering. It presents the aims of the assessments, the taxonomy adopted to ensure that the formulation of the questions addressed learning outcomes related to the development of higher order skills and the choice of software used. Further, the students’ qualitative perception of the assessments is presented together with a discussion on key elements that affected the implementation procedure. This included an understanding of the higher order skills assessed, knowledge of the facilities offered by the software to be used, and the commitment needed to designing, delivering and improving flexible learning materials.     

Tensions between knowledge transmission and student-focused teaching approaches to assessment purposes: helping students improve through transmission

This study surveyed 1064 Chinese school teachers’ approaches to teaching and conceptions of assessment, and examined their inter-relationship using confirmatory factor analysis and structural equation modeling. Three approaches to teaching (i.e. Knowledge Transmission, Student-Focused, and Examination Preparation) and six conceptions of assessment (i.e. Student Development, Teaching Improvement, Examination, Control, School Accountability, and Irrelevance) were identified. Teachers indicated they used Student-Focused most frequently and this positively predicted the assessment purposes of Student Development and Teaching Improvement, while loading negatively on Control, School Accountability, and Irrelevance. The Knowledge Transmission teaching approach, in contrast, positively predicted the assessment purposes of Examination, School Accountability, Control, Student Development, and Teaching Improvement. Thus, despite a predominantly student-focused approach to teaching, knowledge transmission was seen as a teaching approach that contributed positively to student learning. Possible explanations for this anomalous result are discussed.     

A Second-Order Longitudinal Model for Binary Outcomes: Item Response Theory Versus Structural Equation Modeling

Measuring academic growth, or change in aptitude, relies on longitudinal data collected across multiple measurements. The National Educational Longitudinal Study (NELS:88) is among the earliest, large-scale, educational surveys tracking students’ performance on cognitive batteries over 3 years. Notable features of the NELS:88 data set, and of almost all repeated measures educational assessments, are (a) the outcome variables are binary or at least categorical in nature; and (b) a set of different items is given at each measurement occasion with a few anchor items to fix the measurement scale. This study focuses on the challenges related to specifying and fitting a second-order longitudinal model for binary outcomes, within both the item response theory and structural equation modeling frameworks. The distinctions between and commonalities shared between these two frameworks are discussed. A real data analysis using the NELS:88 data set is presented for illustration purposes.     

Reading assessments for students with ASD: a survey of summative reading assessments used in special educational schools in the UK

Schools have an obligation to assess the literacy skills of their students, and the provision of reading instruction to students includes the ability to measure progress in this area. However, the design of reading tests includes the ability not only to read words, but also the ability to verbalise them. This presents a particular challenge for practitioners working with students with autism spectrum disorders (ASD) who can be nonverbal in many cases. How this issue is generally overcome is currently unknown. A survey was developed, in the form of an online multiple‐choice questionnaire, in order to determine which tests are currently being used in the UK to assess the reading abilities of nonverbal students, and to examine the opinions of the education practitioners who use them, in relation to their suitability. Using the schools web directory, e‐mail invitations were sent to 1,050 special educational needs schools across the UK, and 70 schools responded to the invitation. Respondents suggested that the majority of practitioners hold little faith in the ability of current reading assessments to provide an accurate picture of reading ability for students with ASD, and this holds particularly true for nonverbal pupils with ASD. One purpose of education assessment is to establish a baseline of students’ ability in order to plan for lifelong learning and achievement. If there is an inability on the part of schools accurately to assess the reading abilities of nonverbal students with ASD, then it would be fair to assume that this could have a negative impact on the provision of learning opportunities for this population.     

The effects of computer-based attribution retraining on the attributions, persistence, and mathematics computation of students with learning disabilities.

The purpose of the present study was to examine the impact of attribution retraining, embedded within a mathematics computer-assisted instructional (CAI) program, on students' attributions, persistence, and mathematics computation. Twenty-nine school-identified students with learning disabilities from five urban schools participated in the study. The sample's mean age was 13.3 years. After blocking on initial attributional patterns, students were randomly assigned to a mathematics CAI program that provided either attribution retraining or neutral feedback. Students used their assigned program for eight 30-minute sessions. Results did not support the contention that attribution retraining would have a significant impact on students' attributions. However, students who participated in the attribution retraining condition completed significantly more levels of the program than their counterparts who received neutral feedback. Attribution retraining students also obtained significantly higher scores on a test of problems practiced during the CAI program. These results suggest that attribution retraining may be a desirable addition to the type of feedback typically provided by CAI programs. However, they also highlight the need for further research that examines the conditions under which specific attributions are most advantageous.     

Inference and Interval Estimation Methods for Indirect Effects With Latent Variable Models

Although much is known about the performance of recent methods for inference and interval estimation for indirect or mediated effects with observed variables, little is known about their performance in latent variable models. This article presents an extensive Monte Carlo study of 11 different leading or popular methods adapted to structural equation models with latent variables. Manipulated variables included sample size, number of indicators per latent variable, internal consistency per set of indicators, and 16 different path combinations between latent variables. Results indicate that some popular or previously recommended methods, such as the bias-corrected bootstrap and asymptotic standard errors had poorly calibrated Type I error and coverage rates in some conditions. Likelihood-based confidence intervals, the distribution of the product method, and the percentile bootstrap emerged as leading methods for both interval estimation and inference, whereas joint significance tests and the partial posterior method performed well for inference.     

Perceived—and not manipulated—self-control depletion predicts students’ achievement outcomes in foreign language assessments

Abstract

In the present study we investigated which role manipulated (i.e., experimentally induced) and perceived (i.e., self-reported) self-control depletion plays in students’ (N?=?176 seventh graders) achievement-related experiences and behaviour during a test of English as a foreign language, while controlling for trait self-control. Our successful experimental manipulation of self-control depletion revealed that there were no effects on any of the students’ outcome variables. However, students who reported high self-control depletion immediately after the experimental manipulation were less motivated to work on the subsequent test, reported more distracting thoughts, showed lower performance, and felt more depleted at the end of the test session. Trait self-control turned out to be a protective and supportive factor for most of our outcome variables. Our results provide evidence that the perceived and not the manipulated level of self-control depletion is a predictor of achievement-related behaviour in tests on English as a foreign language.     

Gauging Item Alignment Through Online Systems While Controlling for Rater Effects

The alignment of test items to content standards is critical to the validity of decisions made from standards‐based tests. Generally, alignment is determined based on judgments made by a panel of content experts with either ratings averaged or via a consensus reached through discussion. When the pool of items to be reviewed is large, or the content‐matter experts are broadly distributed geographically, panel methods present significant challenges. This article illustrates the use of an online methodology for gauging item alignment that does not require that raters convene in person, reduces the overall cost of the study, increases time flexibility, and offers an efficient means for reviewing large item banks. Latent trait methods are applied to the data to control for between‐rater severity, evaluate intrarater consistency, and provide item‐level diagnostic statistics. Use of this methodology is illustrated with a large pool (1,345) of interim‐formative mathematics test items. Implications for the field and limitations of this approach are discussed.     

The role of attachment in Facebook usage: a study of Canadian college students

Considering the increasingly ubiquitous and frequent use of Facebook among college students, this study sought to explicate and unravel the salient determinants of Facebook use. Specifically, the main goal was to ascertain the factors influencing Collège d'enseignement général et professionnel (CEGEP) students’ Facebook use, for which a structural equation model was proposed to examine the relationships between constructs affecting this process. Using a recently proposed extended technology acceptance model, Dhammic Technology Acceptance Model (DTAM) for Facebook use, proposed by Teo and Jarupunphol [2015. Dhammic technology acceptance model (DTAM): Extending the TAM using a condition of attachment in Buddhism. Journal of Educational Computing Research, 52(1), 136–151. doi:10.1177/0735633114568859], we present results of the study using 233 completed survey data from a sample of CEGEP students in Montreal, Quebec. The DTAM was originally tested using a sample of Thai university students; this leads to a natural question as to whether this extended Technology Acceptance Model (TAM) model holds in a Western sample. The findings from the present study support the validity of the DTAM for explicating Facebook use, and add empirical evidence to the DTAM, according to which the condition of attachment exerts influence on Facebook use. The paper concludes with a discussion of the implications, limitations, and future extensions of the study.     

Identifying and developing crosscutting environmental education outcomes for adolescents in the twenty-first century (EE21)

Abstract

While multiple valid measures exist for assessing outcomes of environmental education (EE) programs, the field lacks a comprehensive and logistically feasible common instrument that can apply across diverse programs. We describe a participatory effort for identifying and developing crosscutting outcomes for Environmental Education in the twenty-first Century (EE21). Following extensive input and debate from a wide range of EE providers and researchers, we developed, tested and statistically validated crosscutting scales for measuring consensus-based outcomes for individual participants in youth EE programs using confirmatory factor analysis across six unique sites, including two single-day field trip locations, four multiday residential programs and one science museum in the United States. The results suggest that the scales are valid and reliable for measuring outcomes that many EE programs in the United States can aspire to influence in adolescent participants, ages 10–14.     

The influence of a response format test accommodation for college students with and without disabilities

Standardised and other multiple-choice examinations often require the use of an answer sheet with fill-in bubbles (i.e. ‘bubble’ or Scantron sheet). Students with disabilities causing impairments in attention, learning and/or visual-motor skill may have difficulties with multiple-choice examinations that employ such a response style. Such students may request and receive testing accommodations that intend to mitigate these impairments, such as circling responses in a test booklet, which contains both the questions and corresponding multiple-choice answers. The current study evaluated this test accommodation as compared to using a bubble sheet or Scantron on a multiple-choice vocabulary test. College students with (n = 25) and without (n = 76) disabilities completed a vocabulary test under both booklet (accommodated) and bubble sheet (standard) conditions. Results demonstrated that answering in a test booklet, a much preferred response mode, allowed students to attempt significantly more items than using a bubble sheet, improving their overall test scores. Booklet responding tends to improve overall performance, even for students without disabilities, calling into question the specificity and validity of this accommodation.     

Facilitating participation in assessments and the general curriculum: level of symbolic communication classification for students with significant cognitive disabilities

This study empirically evaluated a classification schema based on symbolic communication level use with students with significant cognitive disabilities. Ninety‐five teachers of students with significant disabilities rated students’ level of performance on 10 academic tasks. Cluster analysis suggested a range of two to four clusters solutions. Support was found for three clusters: symbolic (abstract), early symbolic (concrete), and pre‐symbolic/awareness. The potential application of the classification system to planning general curriculum access and setting achievement expectations are discussed.     

Cultural,social, and economic capital constructs in international assessments: an evaluation using exploratory structural equation modeling

The article employs exploratory structural equation modeling (ESEM) to evaluate constructs of economic, cultural, and social capital in international large-scale assessment (LSA) data from the Progress in International Reading Literacy Study (PIRLS) 2006 and the Programme for International Student Assessment (PISA) 2009. ESEM integrates the theory-generating approach of exploratory factor analysis (EFA) and theory-testing approach of confirmatory factor analysis (CFA). It relaxes the zero-loading restriction in CFA, allowing items to load on different factors simultaneously, and it provides measurement invariance tests across countries not available in EFA. A main criticism of international LSA studies is the extended use of indicators poorly grounded in theory, like socioeconomic status, that prevent the study of mechanisms underlying associations with student outcomes. This article contributes to addressing this criticism by providing statistical criteria to evaluate the fit of well-defined sociological constructs with the empirical data.     

Relationship of creativity and critical thinking to pattern recognition among Singapore private school students

Abstract

Cognitive pattern recognition is known to be an important skill for academic subjects such as mathematics, science, languages, or even humanities. In this study, we investigate the relationships between creativity, critical thinking, and pattern recognition among 203 private school students in Singapore. The instruments used include a creativity test (modified Creativity Selected Elements Questionnaire), a Critical Thinking Test (modified Cornell Critical Thinking), and a pattern recognition test. The main data analysis is done using the SMART-PLS structural equation modeling software. The results of the study reveal that creativity is a weak predictor of pattern recognition (β?=?0.131, p?>?0.05, f² = 0.024) but critical thinking is a good predictor (β?=?0.517, p?<?0.05, f² = 0.374). An implication of the research outcome is that more training on critical thinking should be given to the students to improve their pattern recognition ability.     

A conceptual framework for trust in standardised assessments: Commercial,quasi-market and national systems

This article presents a conceptual framework for trust in standardised assessments. Standardised assessments play an important role in many education systems as they inform decisions about students' future schooling career or entry to the labour market. Also, standardised assessments are often used for teacher performance reviews and school accountability, or to monitor learning outcomes on the national level. Various stakeholders rely on the accuracy of assessment outcomes when making decisions about students' competences, or seek to improve the quality of education. Such reliance implies a need for trust in those who design and administer standardised assessments and make decisions on the basis of the outcomes. The framework presented in this article describes the type of relational and macro-level trust that is relevant for three types of assessment systems: national, quasi-market and commercial systems. Throughout the analysis presented, examples are provided to illustrate the ways in which relational and macro-level trust can vary by who is tested and by whom they are assessed; and how trust in evaluations varies by the purpose and consequences of testing, as well as the individual agency of students, their teachers and school leaders.     

National survey of accommodations and alternate assessments for students who are deaf or hard of hearing in the United States

This paper reports the results of the National Survey of Accommodations and Alternate Assessments for Students who are Deaf or Hard of Hearing in the United States (National Survey). This study focused on the use of accommodations and alternate assessments in statewide assessments used with students who are deaf or hard of hearing. A total of 258 participants responded to the survey, including 32 representing schools for the deaf, 168 from districtwide/school programs, and 58 from mainstreamed settings. These schools and programs served a total of nearly 12,000 students who are deaf or hard of hearing nationwide. The most prevalent accommodations used in 2003-2004 statewide standardized assessments in mathematics and reading were extended time, an interpreter for directions, and a separate room for test administration. Read aloud and signed question-response accommodations were often prevalent, used more often for mathematics than in reading assessments. Participants from mainstreamed settings reported a more frequent use of accommodations than those in schools for the deaf or districtwide/school programs. In contrast, schools for the deaf were most likely to have students participate in alternate assessments. The top three alternate assessment formats used across all settings were out-of-level testing, work samples, and portfolios. Using the National Survey results as a starting point, future research will need to investigate the validity of accommodations used with students who are deaf or hard of hearing. In the context of the No Child Left Behind Act of 2001 accountability policies, the accommodations and alternate assessment formats used with students who are deaf or hard of hearing may result in restrictions in how scores are integrated into state accountability frameworks.     

Cross-Validation of a Model of Intrinsic Motivation With Students Enrolled in High School Elective Courses

The purpose of this study was to cross-validate a model of relationships among social-contextual factors, individual differences, and intrinsic motivation in adolescent students enrolled in required courses (E. Ferrer-Caja & M. R. Weiss, 2000) with an independent sample of students taking elective courses. Female and male high school students (N = 219) completed measures of motivational climate, teaching style, perceived competence, self-determination, goal orientation, and intrinsic motivation. Motivated behavior was assessed by teachers who rated the students on effort and persistence in class activities. First, the authors used structural equation modeling to examine model invariance between the original and the new samples, which yielded a lack of equivalence. Next, the authors examined several alternative theory-based models using the elective sample. The results indicated that the data were best represented by a model that separated social-contextual factors, individual factors, intrinsic motivation, and motivated behaviors. The strongest predictors of intrinsic motivation were task-goal orientation and perceived competence. These results are discussed from both theoretical and methodological perspectives.     

Model Specification Searches in Structural Equation Modeling with R

This article introduces and demonstrates the application of an R statistical programming environment code for conducting structural equation modeling (SEM) specification searches. The implementation and flexibility of the provided code is demonstrated using the Tabu search procedure, although the underlying code can also be directly modified to implement other search procedures like Ant Colony Optimization, Genetic Algorithms, Ruin-and-Recreate, or Simulated Annealing. The application is illustrated using data with a known common factor structure. The results demonstrate the capabilities of the program for conducting specification searches in SEM. The programming codes are provided as open-source R functions.     

The relative effects of university success courses and individualized interventions for students with learning disabilities

Little is known about the relative effects of post‐secondary learning services for students with learning disabilities. We compared outcomes for students with learning disabilities who selected to: (1) take an academic learning success course (course‐intervention), (2) have regular individual interventions (high‐intervention) or (3) use services only as needed (low‐intervention). Pre‐ and post‐test comparisons revealed improvements in academic self‐efficacy and academic resourcefulness for students in the course‐ and high‐intervention groups. The course‐intervention group also showed decreases in their failure attributions to bad luck and increases in their general repertoire of learned resourcefulness skills in comparison to the high‐intervention group and had significantly higher year‐end GPAs in comparison to the low‐intervention group. Here we find positive outcomes for students with learning disabilities taking a course that teaches post‐secondary learning and academic skills.     

Exploring coping and optimism as predictors of academic burnout and performance among university students

This study examined the relationship between coping strategies, dispositional optimism, academic burnout and academic performance using structural equation modelling. Data were collected from a sample of 532 Spanish undergraduate students. Participants completed a battery of questionnaires including the LOT-R to assess optimism, CSI for the measurement of coping (adaptive and maladaptive coping strategies), and MBI-SS to evaluate academic burnout (exhaustion, cynicism, and efficacy). Academic performance was evaluated by the grade point average (GPA). The results showed that academic burnout was directly and positively associated with maladaptive coping but directly and negatively explained by adaptive coping. In addition, emotional exhaustion was significantly and negatively predicted by optimism. Finally, academic performance was significantly predicted by academic burnout. In conclusion, the findings suggest that both adaptive coping and optimism help to prevent academic burnout and, therefore, positively affect academic performance. Implications for intervention and future research are discussed.