How do instructional designers evaluate? A qualitative study of evaluation in practice

This study employed a qualitative research design to investigate how instructional designers use evaluation in everyday design practice. While past research has examined how designers spend their time, how they generally make decisions, and expert-novice differences, little attention has been paid to use of context, input, process, or product evaluation, from the perspective of practicing designers. Based on interviews of practitioners, our findings included ten themes regarding how designers use evaluation to improve their products. While these results substantiate to some degree the claim that practitioners believe clients will not pay for formal evaluations, they also suggest that practitioners use evaluation in important but less formal ways. Other conclusions regarding the role of evaluation in design are provided and future directions for training and research are discussed.     

The nexus between college choice and persistence

Initial student commitments have long been considered an influence on persistence, but the reasons why students choose to attend a college have seldom been considered as dimensions of initial commitments that could influence persistence processes and outcomes. This study used NPSAS-87 to examine the influence of finance-related reasons for choosing a college on persistence decisions. The findings include (1) finance-related choices have direct and indirect influences on whether students persist in college; and (2) market-based, monetary measures of financial aid, tuition costs, housing costs, and other living costs have a substantial direct effect on persistence.     

Gender Differences in College Mathematics Grades and SAT-M Scores: A Reanalysis of Wainer and Steinberg

Wainer and Steinberg (1992) showed that within broad categories of first-year college mathematics courses (e.g., calculus) men had substantially higher average scores on the mathematics section of the SAT (SAT-M) than women who earned the same letter grade. However, three aspects of their analyses may lead to unwarranted conclusions. First, they focused primarily on differences in SAT-M scores given course grades when the more important question for admissions officers is the difference in course grades given scores on the predictor. Second, they failed to account for differences among calculus courses (e.g., calculus for engineers versus calculus for liberal arts students). Most importantly, Wainer and Steinberg focused on the use of SAT-M as a single predictor. A reanalysis presented here indicated that a more appropriate composite indicator made up of both SAT-M and high school grade point average demonstrated minuscule gender differences for both calculus and precalculus courses.     

What really influences minority attendance? Sequential analyses of the high school and beyond sophomore cohort

While the factors that influence college minority student attendance have been the subject of recent study, there remain unresolved questions about how different factors influence college attendance decisions of applicants from historically disadvantaged backgrounds. In particular, there is ambiguity about whether blacks are more or less likely to attend than whites and what factors might improve their attendance rates. This study uses two sets of logistic regressions to identify the factors that can promote minority attendance: a set that examines attendance by all high school seniors in the high school class of 1982 and a set that examines attendance by college applicants in this class. Consistent with prior research on student access, three factors are identified that can potentially improve college attendance by minority students: (1) improved academic preparation in elementary and high school; (2) increased aspirations for higher levels of educational attainment; (3) increased levels of financial aid. Public interventions that would improve any of these factors for minority high school students are likely to improve minority participation rates.     

The Status of Career and Technical Education Undergraduate and Graduate Programs in the United States

The purpose of this study was to examine undergraduate and graduate student enrollments, course delivery modes, and curricular trends and issues of CTE programs. Based on findings from 139 program/department coordinators, results emphasized that although CTE programs within institutions of higher education have declined in number (Fletcher, Gordon, Asunda, &; Zirkle, 2015 Fletcher, E., &; Djajalaksana, Y. (2015). Instructional strategy preferences in the career and technical education classroom. Journal of Research in Business Education, 57(1), 32–56. [Google Scholar]), student enrollments within those programs have remained steady compared to prior studies (Bruening et al., 2001 Bruening, T., Scanlon, D., Hodes, C., Dhital, P., Shao, X., &; Liu, S. (2001). The status of career and technical education teacher preparation programs. (Report No. V051A990004). Minneapolis, MN: The National Research Center for Career and Technical Education. [Google Scholar]; Lynch, 1990 Lynch, R. (1990). A national database on vocational teacher education (Report No. V051A80004-89). Berkeley, CA: National Center for Research in Vocational Education. [Google Scholar]). Even so, CTE program/department administrators have articulated that declining student enrollment is a major challenge within their programs. Additionally, the development of online courses and the impact of state/national educational reform initiatives were identified as major curricular trends for CTE programs. Recommendations include the need for CTE faculty to embrace online instruction, and for programs to use broader nomenclature of CTE/Workforce Education, instead of names of more traditional subdisciplines within the field to remain more accessible and sustainable in the future. The condition of undergraduate and graduate CTE programs will likely impact the viability and sustainability of CTE teacher preparation programs, recruitment of a talented and diverse CTE teaching force, continuity of the U.S. teaching workforce, and ability of K–12 teachers to maintain high-quality CTE programs.     

The Earlier the Better? Pausing for Thought …

Is earlier intervention always superior? Using two complementary forms of meta-analysis, Gardner and colleagues find no support for the “earlier is better” hypothesis in outcomes of parenting programs for child behavior problems across the 2–11 year age range. This commentary explores possible methodological and substantive reasons for the pattern of their findings. We need additional careful analyses of this kind, assessing age variations in intervention effects across broader age ranges, and in other developmental domains, for strong tests of the “earlier is better” hypothesis. At this stage, however, Gardner et al.'s findings give us some pause for thought.     

Supporting Human Performance Across Disciplines: A Converging of Roles and Tools

Performance support is close to the center of a host of related fields and specialties, including human performance technology, electronic performance support systems, technical communications, and instructional design. Because of their common interest in performance support, and common external influences such as cognitive psychology and digital technologies, roles and tools within these fields are beginning to converge, resulting in unprecedented overlap. In times of rapid change, related fields have an opportunity to learn from one another, borrowing useful elements and incorporating them into their own practices. The purpose of this paper is to explore the similarities, differences, and emerging trends among some of these fields and to gain insights into how their evolution affects performance support. Across these fields, we find a continuing tension between designed messages and tools allowing users more flexibility and control. The best performance-support systems include both of these components as well as a strong human support component. We also observe a trend toward greater reliance on users and user communities in defining and controlling support systems.