Balancing federal and local evaluations of federally-funded programs

Some researchers have suggested that both federal and local studies of early childhood programs could be improved by delegating a portion of the responsibility for federally funded national evaluations to the local level so that local project staff have important roles in national studies. This article describes some of the ways in which the federal government has decentralized the responsibility for national evaluations of federally-funded early childhood programs, the methods used in local evaluation studies, and the strengths and weaknesses of centralized and decentralized national evaluations.     

On the importance of a cognitive processing perspective: an introduction

Children with learning problems require early intervention. If it is evidence based and implemented with integrity and intensity, it will accelerate the academic progress of many students. This is the hope and expectation of the many supporters of responsiveness-to-intervention (RTI). A minority of children, however, will not respond sufficiently to such intervention because of learning disorders like specific learning disabilities (SLD). Some RTI models do not include research-backed methods to identify these children, nor do RTI practitioners often produce the data necessary to develop individualized instruction for them. The authors suggest practitioners go beyond typical RTI assessment data documenting responsiveness/ unresponsiveness to conduct comprehensive evaluations of these most difficult-to-teach students and to include in their evaluations carefully chosen cognitive measures. This special issue presents the work of teams of researchers, which suggests that cognitive and neuropsychological assessments can provide information to further understand SLD, which in turn can guide development of promising interventions.     

The Implications of Teacher Selection and the Teacher Effect in Individually Randomized Group Treatment Trials

Abstract

In some experimental evaluations of classroom-level interventions it is not practically feasible to randomly assign teachers to experimental conditions. Given such restrictions, researchers may randomly assign students to experimental conditions and consider the teacher to be a part of the intervention. However, in an individually randomized evaluation of a classroom-level intervention, unless teachers are randomized to experimental conditions, it will not be clear whether differences in outcomes between program and control group students are a result of the core components of the intervention or the teachers (i.e., teacher effects). This article clarifies the correct interpretation of “program impacts” when this study design is used. In addition, using the magnitude of estimated teacher effects from past research, this article demonstrates that, if teachers are not randomly assigned to experimental conditions, it is difficult to establish whether the program works or whether the types of teachers selected to teach in program classrooms are simply more or less effective than their control group counterparts. The significant implications of the correct causal interpretation are discussed, and the limitations of this research design are explored.     

Evaluating child and family demonstration initiatives: lessons from the comprehensive child development program

The Comprehensive Child Development Program (CCDP) was a demonstration project designed to test a specific model of service delivery for young children and families in poverty. Following the evaluation’s failure to show strong impacts, early intervention has come under fire from opponents in the popular literature and in Congress. We conclude that shortcomings in both implementation and evaluation contributed to the failure to demonstrate effectiveness. Lessons learned from the CCDP are articulated, addressing the roles of demonstration projects and their evaluations, the problems associated with evaluating programs early in their implementation, the importance of assuring appropriate treatment quantity and quality, and the judicious use and interpretation of large-scale randomized evaluations.     

How do instructional designers evaluate? A qualitative study of evaluation in practice

This study employed a qualitative research design to investigate how instructional designers use evaluation in everyday design practice. While past research has examined how designers spend their time, how they generally make decisions, and expert-novice differences, little attention has been paid to use of context, input, process, or product evaluation, from the perspective of practicing designers. Based on interviews of practitioners, our findings included ten themes regarding how designers use evaluation to improve their products. While these results substantiate to some degree the claim that practitioners believe clients will not pay for formal evaluations, they also suggest that practitioners use evaluation in important but less formal ways. Other conclusions regarding the role of evaluation in design are provided and future directions for training and research are discussed.     

Professional Development for K–12 Math and Science Teachers: What Do We Really Know?

Abstract

As investments in K–12 math and science professional development programs expanded over the past decade, researchers and policymakers have questioned whether and how such programs work to improve student learning. This article summarizes the current knowledge-base and offers a theoretical framework researchers can use to design studies that explore mechanisms through which professional development programs influence teacher knowledge, teacher practices, and ultimately student achievement. We quantitatively pool the most rigorous evaluation research available to determine whether currently held beliefs in the field are supported by the evidence. Although this study does find some support for the guidance propounded by experts, it cautions policymakers and practitioners that the current evidence base is thin. We urge researchers to build on what we have learned through theory and practice to increase the rigor of evaluations on this important topic by incorporating valid and reliable measures of professional development components as well as valid measures of teacher knowledge and practices.     

Reading, Reasoning, and Literacy: Strategies for Early Childhood Education From the Analysis of Classroom Observations

Researchers have posited that children generally learn to read in environments that are trusting, comfortable, and offer small group or one-to-one adult support, all of which are characteristics of a high quality early childhood education program. This evaluation research study examines the Pre-Kindergarten Incentive Program, an early childhood demonstration program in Washington, DC designed to study several urban community-based sites. The authors facilitate a discourse on how all early childhood education programs can become high quality programs through standardized observations, evaluations, and constructive feedback. The article describes the demonstration program, highlights general findings from standardized classroom observations, discusses areas needing improvement, and presents strategies for addressing areas of challenge. It will reveal to early childhood educators how evaluation findings can improve teaching and learning techniques and environments in early childhood programs. The goal is to promote overall improvement in reading, reasoning, and literacy for children enrolled in early childhood programs and thereby better prepare them for kindergarten.     

Student Evaluation of Teaching Effectiveness: An assessment of student perception and motivation

Over the past century, student ratings have steadily continued to take precedence in faculty evaluation systems in North America and Australia, are increasingly reported in Asia and Europe and are attracting considerable attention in the Far East. Since student ratings are the most, if not the only, influential measure of teaching effectiveness, active participation by and meaningful input from students can be critical in the success of such teaching evaluation systems. Nevertheless, very few studies have looked into students' perception of the teaching evaluation system and their motivation to participate. This study employs expectancy theory to evaluate some key factors that motivate students to participate in the teaching evaluation process. The results show that students generally consider an improvement in teaching to be the most attractive outcome of a teaching evaluation system. The second most attractive outcome was using teaching evaluations to improve course content and format. Using teaching evaluations for a professor's tenure, promotion and salary rise decisions and making the results of evaluations available for students' decisions on course and instructor selection were less important from the students' standpoint. Students' motivation to participate in teaching evaluations is also impacted significantly by their expectation that they will be able to provide meaningful feedback. Since quality student input is an essential antecedent of meaningful student evaluations of teaching effectiveness, the results of this study should be considered thoughtfully as the evaluation system is designed, implemented and operated.     

The Dynamic Association of Interest and Confusion: The Potential for Moderation by Utility Value and Gender

Abstract

To better understand students’ experiences while learning in an online computer-science course, we adopted a dynamical systems analytic framework in which we tested linked patterns of confusion and interest over time and investigated whether and how those patterns might change as a function of students’ exposure to a utility value (UV) intervention and gender. We assessed interest and confusion via event contingent experience-sampled questionnaires that displayed immediately after students had engaged with key examples or exercises. Results indicated that students’ exposure to a UV intervention and gender affected the levels of confusion that students reported and the associations between interest and confusion levels over time. Implications for educators and researchers are discussed.     

Evaluate to improve: useful approaches to student evaluation

Many teachers in higher education use feedback from students to evaluate their teaching, but only some use these evaluations to improve their teaching. One important factor that makes the difference is the teacher’s approach to their evaluations. In this article, we identify some useful approaches for improving teaching. We conducted focus groups with award-winning university teachers who use student evaluations to improve their teaching, and we identified how they approach their evaluation data. We found that these teachers take a reflective approach, aiming for constant improvement, and see their evaluation data as formative feedback, useful for improving learning outcomes for their students. We summarise this as the improvement approach, and we offer it for other teachers to emulate. We argue that if teachers take this reflective, formative, student-centred approach, they can also use student evaluations to improve their teaching, and this approach should be fostered by institutions to encourage more teachers to use student evaluations to improve their teaching.     

Evidence,Evaluation and the ‘Tyranny of Effect Size’: a proposal to more accurately measure programme impacts in adult and family literacy

As literacy grows in importance, policymakers’ demands for programme quality grow, too. Evidence on the effectiveness of adult and family literacy programmes is limited at best: research gaps abound, and programme evaluations are more often than not based on flawed theories of programme impact. In the absence of robust evidence on the full range of short- and long-term programme impacts, it is difficult to accurately measure intervention effectiveness. Too frequently, researchers and policymakers focus only on short-term, easily measured outcomes, creating a ‘tyranny of effect size’ that may systematically underestimate impact while simultaneously distorting practice. However, the answer does not lie in turning away from quantitative research. Doing so will consign adult and family literacy to the margins of public policy, when they should be in the mainstream. Longitudinal research from Turkey and the US suggests a need for revised, more subtle theories of how adult literacy and family literacy programmes work, and the diverse ways they benefit participants. By working together more closely and intelligently, researchers, policymakers and practitioners can develop evaluation strategies that more accurately measure programme effects. The key is combining methodological rigour with fully fleshed out theories of literacy development and programme impact.     

Effective evaluation through appreciative inquiry

Evaluators in the HPI field can improve their performance program results with effective evaluation through appreciative inquiry. Appreciative inquiry and evaluation have many similarities, and when combined they add value and effectiveness to the measurement of intervention results. Appreciative inquiry is beneficial in many evaluation contexts: for example, when previous evaluations have failed or when change needs to be accelerated. Success is associated with applying appreciative inquiry to the evaluation process for performance improvement projects.     

Autism spectrum disorder assessment and evaluation research in 10 school psychology journals from 2007 to 2017

School psychologists are increasingly engaged in service provisions for students eligible for special education services under the eligibility category of autism, including conducting school‐based assessments and evaluations. Evaluations occur for a variety of reasons such as special education eligibility decision‐making, treatment and intervention planning, and progress monitoring. Publications in school psychology journals emphasizing autism spectrum disorder (ASD) assessment and evaluation are vital to quality training and practitioner utilization of quality practices. In the current study, researchers conducted a systematic review of publicaftions in 10 school psychology journals from 2007 to 2017 to assess the current state of ASD assessment and evaluation research in the field of school psychology. Implications for researchers, trainers, and practitioners are discussed.     

“学生评教”信奉什么教育哲学?

管理走向简单化、庸俗化,学生评教的积极意义受到了极度的重视。物极必反,倘若教师工作业绩如何,由学生的评判一锤定音,教师的无奈选择就可能是顺从、迎合学生的趣味,而不是也无法去教育、引导学生。评教的导向功能异化,师生关系的扭曲,必须引起教育研究者的警惕:不能把育人哲学等同于市场哲学,也不能用经济规律取代教育规律。     

A critical meta-analysis of all evaluations of state-funded preschool from 1977 to 1998: implications for policy, service delivery and program evaluation

The number of state-funded preschool programs for low-income children has increased dramatically over the past few decades, and recent research has indicated that these programs vary considerably along a variety of dimensions. By 1998 only 13 of the current 33 state preschool programs (which serve children 3 to 5, provide some form of classroom-based educational service, and are primarily funded and administered at the state level) had completed a formal evaluation of the program’s impact on child outcomes. This paper presents a critical meta-analytic review of these evaluations, providing measures of standardized effects for all significant impacts to facilitate comparisons across differing domains of outcome and evaluative methods. Although several methodological flaws in these studies are identified, the pattern of overall findings may offer modest support for positive impacts in improving children’s developmental competence in a variety of domains, improving later school attendance and performance, and reducing subsequent grade retention. Significant impacts were mostly limited to kindergarten and first grade; however, some impacts were sustained several years beyond preschool. The results of these studies were similar to evaluations of other large-scale preschool programs for low-income children, such as Head Start. Modest outcome goals are warranted for preschool programs serving low-income children, for example, the promotion of school readiness. Suggestions are presented for improved preschool and early intervention program evaluation.     

Training transfer: Not the 10% solution

Training transfer is a key concern for organizational stakeholders, training professionals, and researchers. Since Georgenson's (1982) article on transfer, his conversational gambit that only 10% of training transfers to performance on the job has been quoted often. This estimate suggests a low level of success with training programs in general, and given its pervasiveness, it has the potential to undermine confidence in using training as an intervention. Although there is considerable evidence that learning from training can and does transfer, assessing how much of it transfers in general is far from simple. Researchers conducting recent meta‐analyses of transfer studies have illustrated some of the difficulties in assigning a general effect size for training transfer. Measuring transfer is complicated by how data are collected, from whom, at what interval from the training event, and a number of other factors. Taking those factors into account, along with existing transfer evaluation methods, training managers and instructional designers should develop protocols to define how they will conduct transfer evaluation within their organizations.     

莆仙地区普通话水平测试评分差异分析

着重分析普通话水平测试中评分差异存在的原因,指出评分标准不统一,定性分析界限不清都会造成评分差异,要提高测试信度,应统一标准,并建设一支高素质的测试员队伍。     

If you build it,will they come? Lessons learned in recruiting students for randomized controlled trials in postsecondary settings

Randomized controlled trials (RCTs) are considered the most methodologically sound methods of conducting empirical research, but a successful RCT is contingent on the recruitment of subjects into the intervention and study. Few research articles and evaluations in education discuss recruitment, and even fewer discuss challenges in recruitment. This is worrisome, given that recruitment can jeopardize internal and external validity – the most valuable aspects of RCTs. This paper initiates a conversation about planning and carrying out a successful recruitment campaign, specifically in postsecondary settings where students are more autonomous, programmes are likely to be voluntary, the flow of information is disparate, and incentives are often misaligned for external evaluation. We also share our challenges and lessons learned in recruiting students for a student support programme and its evaluation. We also advocate that more researchers devote attention to, and sufficiently plan for the recruitment of subjects into RCTs.     

Evaluating the Effectiveness of Reading Recovery: a critique

This paper attempts to examine critically evaluation studies concerned with the implementation of Reading Recovery, an intensive individualised early intervention programme, by focusing on teacher, child and systemic change. Some conceptual and methodological shortcomings which have characterised many evaluations to date are discussed, together with ways in which these weaknesses could be redressed in future studies.