The Undergraduate Research Student Self-Assessment (URSSA): Validation for Use in Program Evaluation

This article examines the validity of the Undergraduate Research Student Self-Assessment (URSSA), a survey used to evaluate undergraduate research (UR) programs. The underlying structure of the survey was assessed with confirmatory factor analysis; also examined were correlations between different average scores, score reliability, and matches between numerical and textual item responses. The study found that four components of the survey represent separate but related constructs for cognitive skills and affective learning gains derived from the UR experience. Average scores from item blocks formed reliable but moderate to highly correlated composite measures. Additionally, some questions about student learning gains (meant to assess individual learning) correlated to ratings of satisfaction with external aspects of the research experience. The pattern of correlation among individual items suggests that items asking students to rate external aspects of their environment were more like satisfaction ratings than items that directly ask about student skills attainment. Finally, survey items asking about student aspirations to attend graduate school in science reflected inflated estimates of the proportions of students who had actually decided on graduate education after their UR experiences. Recommendations for revisions to the survey include clarified item wording and increasing discrimination between item blocks through reorganization.Undergraduate research (UR) experiences have long been an important component of science education at universities and colleges but have received greater attention in recent years, as they have been identified as important ways to strengthen preparation for advanced study and work in the science fields, especially among students from underrepresented minority groups (Tsui, 2007 ; Kuh, 2008 ). UR internships provide students with the opportunity to conduct authentic research in laboratories with scientist mentors, as students help design projects, gather and analyze data, and write up and present findings (Laursen et al., 2010 ). The promised benefits of UR experiences include both increased skills and greater familiarity with how science is practiced (Russell et al., 2007 ). While students learn the basics of scientific methods and laboratory skills, they are also exposed to the culture and norms of science (Carlone and Johnson, 2007 ; Hunter et al., 2007 ; Lopatto, 2010 ). Students learn about the day-to-day world of practicing science and are introduced to how scientists design studies, collect and analyze data, and communicate their research. After participating in UR, students may make more informed decisions about their future, and some may be more likely to decide to pursue graduate education in science, technology, engineering, and mathematics (STEM) disciplines (Bauer and Bennett, 2003 ; Russell et al., 2007 ; Eagan et al. 2013 ).While UR experiences potentially have many benefits for undergraduate students, assessing these benefits is challenging (Laursen, 2015 ). Large-scale research-based evaluation of the effects of UR is limited by a range of methodological problems (Eagan et al., 2013 ). True experimental studies are almost impossible to implement, since random assignment of students into UR programs is both logistically and ethically impractical, while many simple comparisons between UR and non-UR groups of students suffer from noncomparable groups and limited generalizability (Maton and Hrabowski, 2004 ). Survey studies often rely on poorly developed measures and use nonrepresentative samples, and large-scale survey research usually requires complex statistical models to control for student self-selection into UR programs (Eagan et al., 2013 ). For smaller-scale program evaluation, evaluators also encounter a number of measurement problems. Because of the wide range of disciplines, research topics, and methods, common standardized tests assessing laboratory skills and understandings across these disciplines are difficult to find. While faculty at individual sites may directly assess products, presentations, and behavior using authentic assessments such as portfolios, rubrics, and performance assessments, these assessments can be time-consuming and not easily comparable with similar efforts at other laboratories (Stokking et al., 2004 ; Kuh et al., 2014 ). Additionally, the affective outcomes of UR are not readily tapped by direct academic assessment, as many of the benefits found for students in UR, such as motivation, enculturation, and self-efficacy, are not measured by tests or other assessments (Carlone and Johnson, 2007 ). Other instruments for assessing UR outcomes, such as Lopatto’s SURE (Lopatto, 2010 ), focus on these affective outcomes rather than direct assessments of skills and cognitive gains.The size of most UR programs also makes assessment difficult. Research Experiences for Undergraduates (REUs), one mechanism by which UR programs may be organized within an institution, are funded by the National Science Foundation (NSF), but unlike many other educational programs at NSF (e.g., TUES) that require fully funded evaluations with multiple sources of evidence (Frechtling, 2010 ), REUs are generally so small that they cannot typically support this type of evaluation unless multiple programs pool their resources to provide adequate assessment. Informal UR experiences, offered to students by individual faculty within their own laboratories, are often more common but are typically not coordinated across departments or institutions or accountable to a central office or agency for assessment. Partly toward this end, the Undergraduate Research Student Self-Assessment (URSSA) was developed as a common assessment instrument that can be compared across multiple UR sites within or across institutions. It is meant to be used as one source of assessment information about UR sites and their students.The current research examines the validity of the URSSA in the context of its use as a self-report survey for UR programs and laboratories. Because the survey has been taken by more than 3400 students, we can test some aspects of how the survey is structured and how it functions. Assessing the validity of the URSSA for its intended use is a process of testing hypotheses about how well the survey represents its intended content. This ongoing process (Messick, 1993 ; Kane, 2001 ) involves gathering evidence from a range of sources to learn whether validity claims are supported by evidence and whether the survey results can be used confidently in specific contexts. For the URSSA, our method of inquiry focuses on how the survey is used to assess consortia of REU sites. In this context, survey results are used for quality assurance and comparisons of average ratings over years and as general indicators of program success in encouraging students to pursue graduate science education and scientific careers. Our research questions focus on the meaning and reliability of “core indicators” used to track self-reported learning gains in four areas and the ability of numerical items to capture student aspirations for future plans to attend graduate school in the sciences.     

Binomial index of model fit: An elaboration

This article is an elaboration on the use of the binomial test of model fit value, which in this article will be referred to as the binomial index of model fit value, to gauge the degree that the data fit a path analytic or structural equation model. In addition, this article responds to the criticisms and comments made by Hsu (this issue), Drezner and Drezner (this issue), and Raykov and Penev (this issue) regarding the use of this approach to measuring the degree of model fit. We appreciate the comments provided by these authors. Their comments have assisted us in clarifying our reason for developing the binomial index of model fit procedure as well as our perception of its use.     

The impact of individual forms of childhood maltreatment on health behavior

OBJECTIVE: This study examines the unique contribution of five types of maltreatment (sexual abuse, physical abuse, emotional abuse, physical neglect, emotional neglect) to adult health behaviors as well as the additive impact of exposure to different types of childhood maltreatment. METHOD: Two hundred and twenty-one women recruited from a VA primary care clinic completed questionnaires assessing exposure to childhood trauma and adult health behaviors. Regression models were used to test the relationship between childhood maltreatment and adult health behaviors. RESULTS: Sexual and physical abuse appear to predict a number of adverse outcomes; when other types of maltreatment are controlled, however, sexual abuse and physical abuse do not predict as many poor outcomes. In addition, sexual, physical, and emotional abuse and emotional neglect in childhood were all related to different adult health behaviors. The more types of childhood maltreatment participants were exposed to the more likely they were to have problems with substance use and risky sexual behaviors in adulthood. IMPLICATIONS: The results indicate that it is important to assess a broad maltreatment history rather than trying to relate specific types of abuse to particular adverse health behaviors or health outcomes.     

Can educational engagement prevent Early School Leaving? Unpacking the school’s effect on educational success

There is growing evidence that school engagement, or more specifically disengagement, is a key indicator for predicting Early School Leaving. The aim of this article is to explore the impact of secondary schools in student (dis)engagement and subsequent opportunities to succeed in school. Drawing on data from a qualitative study in five secondary schools in Barcelona, the article discusses the role of school context in inhibiting or facilitating school engagement by exploring compositional effects, organisational and pedagogical practices, and teachers’ expectations. To do so a twofold perspective is adopted: firstly, we look at the main school features and their impact on students’ educational opportunities; and secondly, a systematic analysis of the dimensions of school engagement (behavioural, emotional and cognitive) is carried out. As a result, the article contributes to the identification of the most significant variables at school level that influence student engagement and their opportunities for school success.     

Using student norms to create positive learning environments

Ever since brash young students goaded one another to race chariots through the streets of the ancient world, elders have tried to understand and harness the energy of late adolescent normative group behavior. John Braxton and Timothy Caboni introduce a perspective and survey instrument that may help today's educators do just that.     

Comparisons of achievement, effort, and self-perceptions among students with learning disabilities and their peers from different achievement groups

This study compared the social-emotional implications of academic achievement for students with and without learning disabilities (LD) and identified predictors of effort investment. Students with LD showed lower levels of achievement, effort investment, academic self-efficacy, sense of coherence, positive mood, and hope, and higher levels of loneliness and negative mood. When compared to peers without LD (n = 447) at four different academic achievement levels, students with LD showed higher achievement than the low-average group, but their social-emotional profiles were similar to the low and low-average groups. Hierarchical multiple regression analysis showed that achievement, academic self-efficacy, negative mood, and hope predicted effort investment for students with LD. These results demonstrated the importance of hope in understanding the functioning of students with LD.     

Review and criticism

An earlier version of this paper was presented to the AEJ convention at Boulder, Colorado in 1973 by Dr. Milan Meeske for the authors. Dr. O'Keefe is an associate professor of communication at Florida Technological University where Mr. Sheinkopf is a student.     

Professional and Technical Communication in a Web 2.0 World

This article reports on results of a nationwide survey of alumni in professional and technical communication. It presents a series of snapshots from the results, including the types of texts written and valued, where those types are written, with and for whom, and with what technologies. A range of implications are explored.