Using the retrospective pretest to get usable,indirect evidence of student learning

Minimal evidence exists regarding the retrospective pretest’s effectiveness as a tool to obtain usable, indirect evidence of student learning in open admissions institutions. The researcher conducted this study to determine if a retrospective pretest more accurately detected a change in students’ knowledge compared to a conventional pretest–posttest. The researcher designed a 17‐item inventory covering course objectives, and students were randomly assigned to either a retrospective pretest (referred to as post‐then) or a conventional pretest–posttest group. Findings indicated that participants in the post‐then group reported a greater change in knowledge compared to the conventional group. This finding supported the use of the retrospective pretest as a tool for getting usable, indirect evidence of student learning. Future research needs to examine response effect, alternative ways of calculating treatment effect and the confounding influences of other perceptual variables.     

Using a Retrospective Pretest to Measure Learning in Professional Development Programs

Millions of dollars are spent annually on professional development, and designers of training events recognize they must demonstrate to all stakeholders the value added by these programs. The purpose of this study was to measure the learning that occurred during professional development seminars for community college academic department chairs. The research question was the following: What increases in knowledge and skills did program participants attribute to their training experience? Using Alliger and Janak's model for levels of learning as a conceptual framework, data were collected through a preliminary questionnaire on learning needs, session and overall evaluations, and a retrospective pretest addressing 35 knowledge and skill areas. Statistical analysis of retrospective pretest results for six areas identified by participants as priorities indicated that participants felt learning had occurred as a result of the seminar (significant at .0001 level). The retrospective pretest instrument was deemed an effective tool for evaluating learning in continuing education offerings, higher education professional development events, and similar settings.     

RUMORS REGARDING THE DEATH OF THE EQUIPERCENTILE ASSUMPTION MAY HAVE BEEN GREATLY EXAGGERATED

Powers, Slaughter, and Helmick (1983) recently analyzed selection, pretest, and posttest scores collected from large numbers of students in two cohorts. These analyses led them to conclude that the equipercentile assumption underlying norm-referenced evaluation methodology is "inappropriate." Re-examination of the data suggests that there is strong support for the validity of the equipercentile assumption in the selection and pretest scores they present. The observed "gains" from pre- to posttests are better attributed to stakeholder bias, posttests that match the curriculum content too closely, or a combination of these two factors than to inappropriateness of the equipercentile assumption. Annual testing where the posttest from one year also serves as the pretest for the next is suggested as a promising solution to both of the cited threats to the internal validity of norm-referenced evaluations.     

Some Effects of a Five-Year Developmental Experiment Sponsored by a Private Social Agency in a Public School

The present study investigated the relative validity of the conventional pretest-posttest design and the retrospective pretest-posttest design using self-report measures as outcome criteria. As predicted the experimental training in Seeing Problems Strategy was effective in improving subjects’ task performance. Traditional self-report pre-post comparison did not reflect this actual change of performance, whereas the retrospective self-report approach did. Comparison of pretest mean scores and retrospective pretest mean scores indicated that subjects initially over-estimated their performance. This result represents further evidence on the validity of the response-shift notion as introduced by Howard et al. (8) and supports earlier research in showing that the retrospective pretest is a valid means to control for this source of contamination.     

Evaluation of university teaching as sound performance appraisal

As research on student evaluations of teaching (SET) has dominated our understanding of teaching evaluation, debate over SET implementation has turned attention away from basic principles for appraising employee performance as established in human resources literature. Considering SET research and its practical significance in isolation from relevant human resources literature not only risks unlawful remedies for issues such as bias in SET but also risks replacing one form of bias with another. Meanwhile, the full potential of human resources tools to support consistent evaluation of teaching remains unrealized. To address this gap, this article clarifies how teaching evaluation can be conducted as sound performance appraisal by deploying SET and peer review of teaching within a larger framework of established human resources techniques. A review of recent literature articulates prominent themes in research on SET and peer review of teaching and outlines key principles for performance appraisal and performance management. Those principles are used to analyze representative faculty evaluation policies and procedures and clarify the weaknesses of both traditional and recently revised approaches to teaching evaluation. The final section elaborates performance appraisal techniques relevant to teaching evaluation. These include planful use of results and/or behavior approaches to performance appraisal, robust rating instruments for behavioral performance appraisal, targeted collection of information from multiple stakeholders, and job analysis. Efforts to de-emphasize quantitative SET data to address issues such as bias can be strengthened through the incorporation of performance appraisal tools that clearly articulate performance criteria and standards and that gather both qualitative and quantitative data on employee performance.     

Impact of narrative and badging on learning and engagement in a psychology learning game

The literature reflects a shortage of empirical research on educational game design. This paper presents the results from a randomized controlled study on the impact of narrative and digital badging on learning and engagement in an educational game that teaches basic brain structure and function to undergraduate psychology students. Participants were randomly assigned to one of four conditions, and correspondingly played a base version of the game, the base version plus fantasy-based narrative, the base version plus digital badges or the base version and both fantasy-based narrative and digital badges. Eighty students completed a subject matter pretest, played the game in its entirety and then completed a posttest. Quantitative analyses revealed that the manipulations did not significantly impact learning or engagement. While the manipulations were ineffective in this study, future research should examine how professional writing affects narrative impact and how difficulty level can be better assessed in digital badging in order to see if the results remain consistent.     

The Current State of Criminological Research in the United States: An Examination of Research Methodologies in Criminology and Criminal Justice Journals

A recurring concern within criminology and criminal justice (CCJ) is how to best investigate criminological theory and criminal justice policy. To assess the current state of research, we conducted a content analysis of articles that appeared in seven CCJ journals over a two-year period (2013–2014). We then examined types and frequencies of data sources, analytic techniques, methodological approaches, and subject matters. Findings demonstrate that articles are predominantly employing quantitative methodologies and data where there is no participant contact. From these findings, we discuss the current state of research and how this could be used to guide graduate education, by recommending a variety of subject matters that graduate schools should emphasize in training new academics.     

Using Learning Theory to Guide Reflection during School Experience

Reflection has been considered a key aspect that should be included in teacher‐education programmes. There have also been recent calls to base teacher education on constructivist principles of learning. In this study, a reflective‐school‐experience component was integrated into a curriculum subject, using constructivist learning theory as a guide. It shows how these two constructs may be brought together in practice. The learning theory used is explained, and relationships to teaching are explored. Teacher interventions derived from the learning theory provided guidelines for the placement and operation of the school experience, and for encouraging reflection about both the school experience and classwork. An evaluation of the subject was conducted using a variety of data. Selected findings are discussed, and the success of the subject in influencing cognitive and affective outcomes are explored. The role of the school experience and indications of the occurrence of reflection by students are also examined.     

Evaluating Intervention Effects in a Diagnostic Classification Model Framework

The evaluation of intervention effects is an important objective of educational research. One way to evaluate the effectiveness of an intervention is to conduct an experiment that assigns individuals to control and treatment groups. In the context of pretest/posttest designed studies, this is referred to as a control‐group pretest/posttest design. The transition diagnostic classification model (TDCM) was recently developed to assess growth, defined as change in attribute mastery status over time, in a diagnostic classification model framework. The TDCM, however, does not model multiple groups, and therefore is not able to analyze data from a control‐group pretest/posttest designed experiment. In this study, we extend the TDCM to model multiple groups, thereby enabling the examination of group‐differential growth in attribute mastery and the evaluation of intervention effects. The utility of the multigroup TDCM is demonstrated in the evaluation of an innovative instructional method in mathematics education.     

Students' Evaluation of Teachers and Instructional Quality--Analysis of Relevant Factors Based on Empirical Evaluation Research

This paper explores the factors that determine the student evaluation of teachers. Unlike many previous studies, the effect of potential biases on global ratings is examined in the context of teaching behaviour (that should in fact--in contrast to the potential biases--have an influence on global ratings). Our research takes into account that biases like the students' interest in the subject or their liking for the teacher may be a result of good teaching behaviour and may not be considered a mere bias of student ratings. Furthermore, it also concentrates on the students' attitudes towards evaluating their teachers. The empirical results are based on qualitative interviews with 40 students at Austrian commercial colleges and on a quantitative survey of 2,121 students who were asked to evaluate their Accounting teachers. Structural equation models show that the students' global ratings of teachers mainly depend on their teaching behaviour. However, global ratings are also affected by the students' attitudes towards evaluating their teachers, as well as by the students' liking for their teacher and their interest in the subject of Accounting.     

Empirical Performance of Covariates in Education Observational Studies

This article summarizes results from 12 empirical evaluations of observational methods in education contexts. We look at the performance of three common covariate-types in observational studies where the outcome is a standardized reading or math test. They are: pretest measures, local geographic matching, and rich covariate sets with a strong theory of treatment selection. Overall, the review demonstrates that although the pretest often reduces bias in observational studies, it does not always eliminate it. Its performance depends on the pretest's correlation with treatment selection and the outcome, and whether preintervention trends are present. We also find that although local comparisons are prioritized for matching, its performance depends on whether comparable no-treatment cases are available. Otherwise, local comparisons may produce badly biased results. In cases where researchers have a strong theory of selection and rich covariate sets, observational methods perform well, but additional replication studies are needed. Finally, observational methods that rely on demographic covariates without a theory of selection rarely produce unbiased treatment effects. The article concludes by offering education researchers empirically based guidance on covariate selection in observational studies.     

Controlling response shift bias: the use of the retrospective pre‐test design in the evaluation of a master's programme

Traditionally the measures used to evaluate the impact of an educational programme on student outcomes and the extent to which students change is a comparison of the student's pre‐test scores with his/her post‐test scores. However, this method of evaluating change may be problematic due to the confounding factor of response shift bias when student self‐reports of change are used. Response shift bias occurs when the student's internal frame of reference of the construct being measured, for example research ability or critical thinking, changes between the pre‐test and the post‐test due to the influence of the educational programme. To control for response shift bias the retrospective pre‐test method was used to evaluate the outcomes achieved from students completing a research module at master's level. The retrospective pre‐test method differs from the traditional pre‐test–post‐test design in that both post‐test and pre‐test perceptions of respondents are collected at the same time. The findings indicated that response shift bias was evident in that the programme had significantly greater impact on outcomes than identified using the traditional pre‐test–post‐test design leading to the conclusion that students may overestimate their ability at the commencement of an educational programme. The retrospective pre‐test design is not a replacement for the traditional pre‐test–post‐test measures but may be a useful adjunct in the evaluation of the impact of educational programmes on student outcomes.     

An Agency Theory Perspective on Student Performance Evaluation

The emphasis in recent research on the responsibility of college and university business instructors to prepare students for future employment underscores a need to refine the evaluation of student performance. In this article, an agency theory framework is used to understand the trade‐offs that may be involved in the selection of various approaches to student evaluation. Understanding these trade‐offs may be particularly important as faculty members seek to balance competing obligations, such as research and service requirements, while ensuring instructional effectiveness. This article presents propositions for examining how various institutional, instructor, and student characteristics influence the selection and use of student performance evaluation techniques (i.e., exams, papers, and group assignments). In conclusion, we suggest that agency theory may serve as a foundation for understanding current evaluation practices and guiding instructors in their selection of appropriate evaluation mechanisms.     

The Interrelation of D,V, T,and P Scores

Evaluating changes in student competencies and attitudes in studies of foreign political systems and ideologies is a problematic but important issue in research on social science instruction. This article outlines a set of instructional objectives on Policy Making in Communist Party States (PMCPS) and presents the results of an empirical evaluation based upon data gathered from nine classes taught on the subject of communist political systems. The study determined how and when students memorized and comprehended certain pieces of information, the extent to which they were able to successfully apply, analyze, and synthesize this information, and finally, how the learning process influenced beliefs and values toward selected communist states and issues involving socialism and communism.     

Acquiescence,instructor’s gender bias and validity of student evaluation of teaching

Abstract

The validity of student evaluation of teaching (SET) scores depends on minimum effect of extraneous response processes or biases. A bias may increase or decrease scores and change the relationship with other variables. In contrast, SET literature defines bias as an irrelevant variable correlated with SET scores, and among many, a relevant biasing factor in literature is the instructor’s gender. The study examines the extent to which acquiescence, the tendency to endorse the highest response option across items and bias in the first sense affects students’ responses to a SET rating scale. The study also explores how acquiescence affects the difference in teaching quality (TQ) by instructor’s gender, a bias in the latter sense. SET data collected at a faculty of education in Ontario, Canada were analysed using the Rasch rating scale model. Findings provide empirical support for acquiescence affecting students’ responses. Latent regression analyses show how acquiescence reduces the difference in TQ by instructor’s gender. Findings encourage greater attention to the response process quality as a way to better defend the utility of SET and prevent potentially misleading conclusions from the analysis of SET data.     

我国中小学信息技术课堂评价研究的可视化分析

采用文献计量分析法和词频分析法,以中国知网(CNKI)2002—2019年间检索的207篇中小学信息技术课堂评价研究文献为研究对象,运用Cite Space V软件对文献的时间、作者、机构及关键词等进行可视化分析,客观生成知识图谱,实现数据的可视化,揭示我国中小学信息技术课堂评价研究领域的年度发文量、研究主体、研究力量、关键词热度等情况,总结出我国中小学信息技术课堂评价研究的现状,为该领域的理论研究与实践提供参考。     

Cognitive biases in the interpretation of autonomic arousal: A test of the construal bias hypothesis

According to Bandura’s construal bias hypothesis, derived from social cognitive theory, persons with the same heightened state of autonomic arousal may experience either pleasant or deleterious emotions depending on the strength of perceived self-efficacy. The current study tested this hypothesis by proposing that college students’ preexisting efficacy beliefs may affect how physiological arousal during exams is construed as a valence of positive emotion. Survey data were collected from 172 students in a statistics course at a large research university in the Midwest. Students provided self-reported intensity of autonomic arousal during previous exams, statistics self-efficacy, and positive affect toward a proximal final exam. The construal bias hypothesis was supported by a significant interaction between autonomic arousal and self-efficacy on the dependent variable positive affect. Results revealed that as autonomic arousal increased, so did the moderating effect of self-efficacy. Specifically, as physiological arousal intensified, students with high or low self-efficacy reported divergent levels of positive emotion. Implications for theory, research, and practice are discussed.     

MathCAD在金属塑性成形原理教学中的应用研究

该文介绍了利用MathCAD软件在数值计算、符号运算和曲线自动绘制等方面的综合功能,以其在基础理论推导、实验数据分析、塑性成形解题模型等方面的应用为要点,通过实例介绍该软件在构造金属塑性成形原理CAI课件中的应用。     

Rating of personality of normally-developing and mentally disabled children and the mothers’ beliefs about their child’s development

The purpose of this paper focuses on mothers’ beliefs about their child’s personality and development (stability or change and causal attributions) and how these beliefs can be affected by mothers’ experience (being a mother of a six or of a twelve-year-old child and being a mother of a normally-developing or of a mentally disabled child). The scales were completed by several samples of mothers. The effect of the temporal orientation (retrospective or perspective) was also considered. The results show that the mothers rate their child with an optimistic bias while the mothers of disabled children are less positive than those of the broad-ranging sample. The results are also in accordance with the decrease of positiveness as the child grows. Finally, the optimistic bias differs in function of the temporal orientation for the mothers of normally-developing children and for the mothers of mentally disabled children. The retrospective change is also overestimated with regard to the prospective one in the broad-ranging sample.